Recommendations — Best-in-Class Feature Set

This section translates the feature catalogs and redundancy analysis into actionable per-class recommendations. For each feature class (and its subclasses), it names the best-in-class tool(s), the key features to extract, whether the tool runs locally, and a priority tier: core (the minimal tractable set — one extractor per signal, maximal cross-subclass reuse, single consumer GPU + CPU) or extended (added only when resources or a specific accuracy/distinct-signal need justifies it).

Recommendation table

Tradeoffs

Cross-subclass reuse dominates within-subclass pruning. Several signals appear in multiple classes and must be extracted once and routed everywhere, not re-run per catalog. The clearest cases: face detection/identity (InsightFace) feeds Faces, Social, and High-level visual; face emotion (HSEmotion/OpenFace 3.0) feeds Faces, Social, and Affect; body pose (MMPose RTMPose) feeds Faces, Social, and Dynamic visual; diarization (pyannote) feeds Speech and Social; prosody (Parselmouth + openSMILE) feeds Speech, Low-level acoustic, and Affect; shot boundaries (TransNetV2/PySceneDetect) feed Dynamic visual, Situation, and Action; vocal affect (audEERING wav2vec2-dim) feeds Speech and Affect; LLM surprisal and hidden states (minicons) feed Lexical and Semantics; and lexical-VAD/GoEmotions feed Lexical and Affect. Wiring the pipeline around shared passes, rather than per-class extractors, is what makes the whole set tractable on a single consumer GPU.

The single largest compute saving is consolidating high-level reasoning into one video-LLM pass. Action description, situational dimensions (space/time/causation/intention/protagonist), social interaction-type/dominance/ToM, and depicted emotion are all realized by Qwen2.5/3-VL-class models. Running one Qwen-VL pass per shot with a unified multi-field JSON schema replaces four separate VLM deployments. InternVL3, VideoLLaMA3, LLaVA-OneVision, Emotion-LLaMA, and AffectGPT are interchangeable for this role; keep at most one primary plus, optionally, one frontier-API judge reserved for gold-label validation rather than routine annotation.

Foundation models supersede most fixed-taxonomy CNNs, but not always. SigLIP 2 zero-shot probing covers the object/scene/place/attribute role that Places365 and ImageNet backbones (and YAMNet/AST in audio) used to fill. Keep the fixed-taxonomy models only when a frozen, calibrated, reproducible probability vector is explicitly required for interpretability. Likewise, classic flow (RAFT/Farneback), depth (MiDaS/ZoeDepth), saliency (TASED-Net/UNISAL), and shot detection (AutoShot) are superseded by SEA-RAFT, Depth-Anything-V2, ViNet, and TransNetV2 respectively; the older tools survive only as no-GPU fallbacks.

Some surface-similar pairs are genuinely distinct and must both be kept. DINOv2 (label-free dense embedding for RSA) versus SigLIP 2 (text-probed interpretable scores) are different representations. pymoten (brain-aligned motion-energy regressor) versus SEA-RAFT (interpretable camera-vs-object flow), and semantic action recognition (VideoMAE) versus pre-semantic motion (SEA-RAFT/pymoten), are deliberately separate layers. BEATs (calibrated 527-class) versus CLAP (arbitrary open-vocabulary prompts), inaSpeechSegmenter (speech/music/noise partition) versus pyannote (who-spoke-when), Mask2Former (area-fraction composition) versus Grounding DINO (object counts/positions), and CREPE (robust pitch contour) versus Parselmouth (canonical voice-quality) all carry complementary information.

The one non-negotiable “keep two” is depicted versus elicited emotion. Face/voice/text emotion models predict character-expressed (depicted) affect, whereas LIRIS-ACCEDE/MuSe-trained regressors predict viewer-induced (elicited) affect. These are correlated but not interchangeable; logging them as a single stream would be a scientific error, not a parsimony gain. Similarly, interpretable per-modality affect signals (wav2vec2 + HSEmotion + GoEmotions) should remain the primary features, with any fusion MLLM as an added reasoned layer rather than a replacement.

Local-first is feasible throughout. Every core recommendation runs locally; most low-level audio and lexical tools are CPU-only, and the heaviest core components (Qwen-VL, VideoMAE, SigLIP 2, DINOv2, pyannote, Whisper) fit on a single 24-48 GB GPU. API embeddings and frontier-API VLM judges are the only API-only entries and are confined to the extended tier, used only when leaderboard-topping accuracy is required and off-site transfer of transcripts/frames is acceptable.