Affect features
Emotion & affect (multimodal)
Time-resolved emotion annotation for movies/audio splits cleanly by signal source: per-frame face models (EmoNet, HSEmotion/EmotiEffLib), per-utterance/windowed voice models (audeering wav2vec2-dim), per-window text/dialogue models (RoBERTa-GoEmotions, EmoBank-VAD regressors), per-window music models (MERT-based regressors), and clip/window multimodal fusion or MLLM reasoning systems (Emotion-LLaMA, AffectGPT, Qwen2.5-VL/GPT-4o judges). A crucial distinction: most of these models predict character-DEPICTED emotion (the expressed affect of a speaker/face), whereas viewer-ELICITED emotion requires separate models trained on induced-affect ground truth (LIRIS-ACCEDE / MediaEval continuous, MuSe/SEWA benchmarks) — the two are correlated but not interchangeable and should be logged as distinct feature streams. For a small lab, the strongest open, locally deployable defaults are audeering wav2vec2 (voice VAD), HSEmotion EfficientNet (face VA + categorical), RoBERTa-GoEmotions (dialogue), MERT regressor (music), with an MLLM (Qwen2.5-VL or GPT-4o) as a window-level fusion/captioning layer and EmoNet/Emotion-LLaMA as alternatives.
| Tool | Measures | Per-timepoint output | Modality | Granularity | Local | Rating | Rec |
|---|---|---|---|---|---|---|---|
| audeering wav2vec2-large-robust-12-ft-emotion-msp-dim | Dimensional emotion from raw speech audio (depicted/expressed affect of the speaker’s voice); trained on MSP-Podcast v1.7. | 3 continuous scalars per window: arousal, dominance, valence each in ~0..1. Also exposes pooled last-layer hidden states (1024-d) usable as a voice-affect embedding. Run on a sliding window (e.g. 0.5-2 s hop) to get per-second VAD timeseries aligned to the audio timeline. | audio | sub-second to per-second (windowed; receptive field of one inference pass) | yes-gpu | best-in-class | include |
| HSEmotion / EmotiEffLib (EfficientNet-B0/B2 + MT-MobileFaceNet, AffectNet-trained) | Per-face facial expression: categorical emotion, continuous valence/arousal, and (multi-task variants) action-unit / emotion intensity. Depicted emotion of on-screen faces. | Per detected face per frame: 7- or 8-class categorical softmax (Anger, Contempt, Disgust, Fear, Happiness, Neutral, Sadness, Surprise), continuous valence and arousal scalars (va_mtl models), plus the penultimate embedding (1280-d for B0) as a face-affect feature. Aggregate across faces per frame for a scene-level signal. | video-frame | frame (needs an upstream face detector/tracker) | yes-cpu | best-in-class | include |
| EmoNet (face-analysis/emonet, Toisoul et al. Nature Mach. Intell. 2021) | Continuous valence/arousal and categorical expression from faces in naturalistic conditions; depicted facial emotion. | Per face per frame: continuous valence and arousal (-1..1), 5- or 8-class expression probabilities, and 68 facial landmarks. Ships demo_video.py (added 2024) for face detection + emotion over video frames. | video-frame | frame | yes-gpu | strong | alternative |
| RoBERTa fine-tuned on GoEmotions (SamLowe/roberta-base-go_emotions) | Categorical emotion of dialogue/narration text (depicted emotional content of language). 28-emotion taxonomy. | Per utterance/sentence: 28 independent multi-label sigmoid scores (admiration, amusement, anger, … neutral). Map to Russell valence/arousal via a VAD lexicon (NRC-VAD / EmoBank) for a continuous proxy. Align to timeline via subtitle/ASR timestamps (per-utterance). | text | per-utterance (subtitle/ASR-segment level) | yes-cpu | strong | include |
| EmoBank / NRC-VAD dimensional text regressor (VAD lexicon + transformer) | Continuous valence-arousal-dominance of text on a graded scale (complements categorical GoEmotions). Depicted language affect. | Per utterance: continuous valence, arousal, dominance scalars (typically 1..9 or normalized). Can be produced by a transformer regressor fine-tuned on EmoBank or by lexicon lookup (NRC-VAD) averaged over tokens. | text | per-utterance | yes-cpu | usable | include-if-resources |
| MERT-based music emotion regressor (MERT-v1-95M/330M embeddings + V/A head) | Emotion conveyed by the musical score/soundtrack (depicted musical affect), distinct from speech affect. | Per audio window: continuous valence and arousal scalars (and optional categorical mood tags). MERT yields ~768/1024-d self-supervised music embeddings per ~frame; a small regression head trained on DEAM/PMEmo/EmoMusic maps to V/A timeseries. | audio | per-second (windowed) | yes-gpu | strong | include-if-resources |
| Emotion-LLaMA (ZebangCheng/Emotion-LLaMA, NeurIPS 2024) | Multimodal (audio+visual+text) emotion recognition AND free-text reasoning about the emotional state of on-screen people. Depicted emotion with explanation. | Per clip/window: categorical emotion label (+ confidence) plus a natural-language explanation of cues (facial, prosodic, lexical). Strong zero-shot transfer (e.g. DFEW). Run over fixed windows (e.g. 2-5 s shots) to get a per-shot labeled + reasoned stream. | multimodal | per-shot / per-clip (windowed) | yes-heavy-gpu | strong | include-if-resources |
| AffectGPT (zeroQiaoba/AffectGPT, ICML 2025) | Explainable multimodal emotion understanding with descriptive captions; open-vocabulary (OV-MER) emotion rather than fixed classes. Depicted emotion + description. | Per clip: open-vocabulary emotion terms and a descriptive caption of the emotional state and its multimodal evidence; can be reduced to category/intensity. Built on MER-Caption / EMER datasets (MER2024-SEMI 115k videos). | multimodal | per-clip (windowed) | yes-heavy-gpu | strong | alternative |
| General MLLM as affect annotator (GPT-4o / Qwen2.5-VL-7B/72B with structured prompting) | Window-level fused emotion annotation from frames+ASR+(audio captions): both depicted character emotion and, with appropriate prompts, an estimate of viewer-intended/elicited affect, plus rationale. | Per window (configurable, e.g. per shot or per 2-5 s): JSON with categorical emotion, valence/arousal/dominance estimates (1-5 or -1..1), intensity, confidence, and a free-text justification. Flexible schema; can separately ask for ‘character feels’ vs ‘viewer likely feels’. | multimodal | per-shot / per-window (configurable) | api-only | best-in-class | include |
| Continuous induced-affect regressors (LIRIS-ACCEDE / MediaEval Emotional Impact; AttendAffectNet) | Viewer-ELICITED valence/arousal evoked by a movie segment (not the character’s expressed emotion) — the affect a typical audience feels, trained on continuous self-report + GSR ground truth. | Per second: continuous induced valence and arousal (-1..1) along the movie timeline; some pipelines also predict a fear/emotional-impact flag. AttendAffectNet fuses audio+visual (+deep) features with self-attention to regress continuous V/A. | video-clip | per-second | yes-gpu | usable | include |
Recommended (best-in-class): audeering wav2vec2-large-robust-12-ft-emotion-msp-dim (voice VAD); HSEmotion / EmotiEffLib EfficientNet (face valence/arousal + categorical); RoBERTa-GoEmotions + NRC-VAD mapping (dialogue affect); MERT-based V/A regressor (soundtrack/music affect, if relevant); Qwen2.5-VL-7B or GPT-4o as window-level multimodal fusion/annotator; LIRIS-ACCEDE/MuSe-trained continuous regressor for viewer-ELICITED valence/arousal (kept as a separate stream from all depicted-emotion models)
Likely redundant:
- EmoNet is redundant with HSEmotion for face valence/arousal+categorical — keep HSEmotion as primary (faster, lighter, more permissive license, active ABAW results), EmoNet for cross-validation only
- AffectGPT is redundant with Emotion-LLaMA as the open MLLM fusion/reasoning layer — pick one (AffectGPT for open-vocabulary/descriptive output, Emotion-LLaMA for turnkey benchmarked labels)
- Emotion-LLaMA/AffectGPT overlap with a general MLLM (GPT-4o/Qwen2.5-VL) — the specialized MLLMs add benchmarked depicted-emotion accuracy, the general MLLM adds schema flexibility and elicited-vs-depicted prompting; running all three is wasteful
- EmoBank/NRC-VAD dimensional text regressor partially duplicates RoBERTa-GoEmotions (same text, dimensional vs categorical view) — include both only if cross-modal VAD alignment is needed
- Per-modality stack (wav2vec2 + HSEmotion + GoEmotions) overlaps the internal branches of any fusion MLLM; the interpretable per-modality signals are generally preferable as primary features, with the MLLM as an added fused/reasoned layer rather than a replacement
Open questions:
- Does the project need depicted (character-expressed) emotion, viewer-elicited (induced) emotion, or both? This determines whether the LIRIS-ACCEDE/MuSe-style regressor is mandatory (it is the only viewer-elicited source) versus the depicted-emotion stack.
- Target temporal resolution: true per-second requires sliding-window inference and decisions on hop size and label smoothing; MLLM-per-shot annotation is cheaper but coarser and costlier per second of film.
- Licensing for downstream use: audeering wav2vec2 (CC-BY-NC-SA), EmoNet, and the MLLM-specialized models are non-commercial/research only; if any commercial redistribution is planned, prefer Apache/MIT options (HSEmotion, GoEmotions, Qwen2.5-VL).
- Soundtrack handling: do you need source separation to split musical-score affect (MERT) from speech affect (wav2vec2) when they are mixed in the film audio?
- Ground-truth/validation plan: which human-rated benchmark (SEND, SEWA, MuSe, LIRIS-ACCEDE) will be used to calibrate and cross-check the automated streams, and how will face/voice/text streams be fused (late fusion vs MLLM)?
- Face pipeline dependency: HSEmotion/EmoNet need an upstream detector+tracker (RetinaFace/MediaPipe) and a policy for multi-face scenes (which/aggregate) before producing a scene-level signal.
References
- audeering wav2vec2-large-robust-12-ft-emotion-msp-dim — Wagner et al. 2023 IEEE TPAMI ‘Dawn of Transformer Era in SER’; HF audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim
- HSEmotion / EmotiEffLib (EfficientNet-B0/B2 + MT-MobileFaceNet, AffectNet-trained) — Savchenko, HSEmotion (Software Impacts 2022); ABAW 6/7/8 reports arXiv:2403.11590, 2407.13184; github.com/sb-ai-lab/EmotiEffLib
- EmoNet (face-analysis/emonet, Toisoul et al. Nature Mach. Intell. 2021) — Toisoul et al. 2021 Nat. Mach. Intell.; github.com/face-analysis/emonet
- RoBERTa fine-tuned on GoEmotions (SamLowe/roberta-base-go_emotions) — Demszky et al. GoEmotions ACL 2020; HF SamLowe/roberta-base-go_emotions
- EmoBank / NRC-VAD dimensional text regressor (VAD lexicon + transformer) — Buechel & Hahn EmoBank EACL 2017; Mohammad NRC-VAD ACL 2018
- MERT-based music emotion regressor (MERT-v1-95M/330M embeddings + V/A head) — Li et al. MERT ICLR 2024 (arXiv:2306.00107); DEAM/MediaEval; ‘Unified MER’ arXiv:2502.03979
- Emotion-LLaMA (ZebangCheng/Emotion-LLaMA, NeurIPS 2024) — Cheng et al. arXiv:2406.11161 (NeurIPS 2024); github.com/ZebangCheng/Emotion-LLaMA
- AffectGPT (zeroQiaoba/AffectGPT, ICML 2025) — Lian et al. arXiv:2501.16566 / 2407.07653 (ICML 2025); github.com/zeroQiaoba/AffectGPT
- General MLLM as affect annotator (GPT-4o / Qwen2.5-VL-7B/72B with structured prompting) — Qwen2.5-VL arXiv:2502.13923; EmoBench-M arXiv:2502.04424; MMAFFBen arXiv:2505.24423; GPT-4o (OpenAI)
- Continuous induced-affect regressors (LIRIS-ACCEDE / MediaEval Emotional Impact; AttendAffectNet) — Baveye et al. LIRIS-ACCEDE IEEE TAC 2015; MediaEval Emotional Impact of Movies 2016/2018; AttendAffectNet arXiv:2010.11188; MuSe/SEWA benchmarks