Social features
Social & interpersonal features
Depicted social content for a movie/audio timeline decomposes into (a) low-level perceptual primitives that are reliably extractable today with dedicated open models – number/identity of agents (face detection + recognition), who-speaks-when (diarization), who-is-actively-talking (audio-visual ASD), gaze targets / joint attention, and body pose for proximity/touch – and (b) high-level social semantics (interaction type, cooperation/conflict, dominance/affiliation, theory-of-mind, intentions, joint attention construal) that have no robust dedicated detector and are best realized by prompting a strong video VLM (Qwen2.5-VL, InternVL3, or a frontier API) against schemas derived from social-reasoning benchmarks (Social-IQ 2.0, Social Genome, SIV-Bench). The recommended architecture is a two-layer pipeline: run the cheap deterministic primitive extractors per-second to build a character/speaker track and proximity/gaze graph, then feed those tracks plus sampled frames into a VLM with structured prompts to label interaction type and mental-state content per shot/scene. The dedicated social-reasoning models in the literature are mostly benchmarks, not deployable annotators, so the VLM-prompting layer is unavoidable for the genuinely “social” dimensions.
| Tool | Measures | Per-timepoint output | Modality | Granularity | Local | Rating | Rec |
|---|---|---|---|---|---|---|---|
| pyannote-audio 3.1 (speaker-diarization-3.1) | Who-speaks-when from the audio track: segments the soundtrack by speaker identity, the backbone for who-talks-to-whom in dialogue. | Per-segment speaker labels (anonymous SPK_00, SPK_01, …) with start/end times; derivable per-second one-hot active-speaker-id vector, speaker count, overlap/cross-talk flag, turn-taking transitions. Optional speaker embeddings (192-d) for cross-scene identity linking. | audio | per-utterance / per-segment, resampleable to per-second | yes-gpu | best-in-class | include |
| Light-ASD / LR-ASD (CVPR 2023) | Audio-visual active speaker detection: which on-screen face is speaking at each frame, linking the diarization speaker to a visible agent (essential for who-talks-to-whom in video). | Per-face, per-frame speaking probability/binary label; combined with face tracks yields per-second ‘face X is speaking’ signals, listener vs speaker roles, and addressee inference when fused with gaze. | multimodal | per-frame | yes-gpu | strong | include |
| InsightFace (SCRFD/RetinaFace detection + ArcFace/buffalo_l recognition) | Number and identity of agents present: detects faces per frame and assigns persistent character identities via face-embedding clustering across the film. | Per-frame face count + bounding boxes; 512-d ArcFace embeddings; per-track character ID (after clustering/gallery matching); per-second ‘characters present’ set; co-presence matrix (which characters share the frame) that seeds the social-network graph; head pose (yaw/pitch/roll) and age/gender estimates. | video-frame | per-frame, aggregated per-shot/per-second | yes-gpu | best-in-class | include |
| Gaze-LLE (CVPR 2025, Highlight) | Gaze target estimation and joint attention: where each person is looking in the scene, enabling looking-at-each-other (mutual gaze), looking-at-same-object (shared/joint attention), and addressee detection. | Per-person gaze heatmap over the frame + predicted gaze target point; in-frame/out-of-frame gaze flag; derivable per-second mutual-gaze (dyad looking at each other) and joint-attention (multiple people fixating same target) binary signals and gaze-direction vectors. | video-frame | per-frame (designed for VideoAttentionTarget), aggregateable per-second | yes-gpu | best-in-class | include |
| RTMPose / ViTPose (MMPose) + ByteTrack | Multi-person 2D body pose and tracking: provides inter-personal distance (proximity), orientation/facing, touch/contact candidates, and gesture cues underlying affiliation and interaction. | Per-person 17/133-keypoint skeletons per frame with track IDs; derived per-second features: pairwise inter-personal distance, body-orientation (facing toward/away), overlap/contact flags (touch proxy when keypoints of two people coincide), approach/retreat velocity, gesture/posture descriptors. | video-frame | per-frame | yes-gpu | strong | include |
| Qwen2.5-VL (7B / 72B, Instruct) | High-level social semantics via structured video prompting: interaction type (cooperation/conflict/dominance/affiliation), who-talks-to-whom narration, intentions, joint-attention construal, scene-level social description — the main engine for the genuinely social labels. | Per-shot/per-clip structured outputs you define: interaction-type categorical labels, dominance/affiliation ratings, agent-relationship descriptions, ToM/intention free-text or scalar ratings, dialogue addressee attribution, social-network edge proposals; supports temporal grounding (timestamps) for event localization. | multimodal | per-clip / per-shot (sampled frames), with timestamp grounding | yes-heavy-gpu | best-in-class | include |
| InternVL3 (8B / 38B / 78B) | Alternative high-level social-semantics VLM for the same interaction-type / ToM / relationship labeling layer; native multimodal pretraining gives strong video social-scene reasoning. | Same configurable per-clip social labels as Qwen2.5-VL: interaction categories, dominance/affiliation, intention/mental-state inferences, agent-relationship and social-network edge proposals, grounded social descriptions. | multimodal | per-clip / per-shot | yes-heavy-gpu | strong | alternative |
| Frontier multimodal APIs (Claude Opus 4.x, GPT-4o, Gemini 2.x) as social-reasoning judges | Highest-quality high-level social inference (ToM, intention, conflict/cooperation nuance, dominance/affiliation, mental-state attribution) when local VLMs are insufficient or for generating gold annotations to validate the local pipeline. | Per-shot/per-scene structured JSON: interaction-type labels, dominance/affiliation/valence ratings, ToM/mental-state inferences with rationale, addressee and relationship graphs; can produce grounded reasoning traces analogous to Social Genome. | multimodal | per-clip / per-scene (sampled frames or transcript) | api-only | strong | include-if-resources |
| Social-IQ 2.0 / Social Genome / SIV-Bench (benchmark + schema resources) | Not annotators but the definitional backbone: socially-rich video QA and grounded-reasoning datasets that define the social-interaction label taxonomy, evidence types (visual/verbal/vocal cues + external knowledge), and evaluation protocols to design and validate the VLM-prompting layer. | Provides label schemas, multiple-choice social-reasoning items, and 5,777 annotated reasoning steps (Social Genome) referencing visual/verbal/vocal evidence — used to construct prompt templates, few-shot exemplars, and held-out evaluation sets for the social-feature extractor. | multimodal | per-clip (dataset-level) | yes-cpu | usable | include |
Recommended (best-in-class): pyannote-audio 3.1 (who-speaks-when); Light-ASD/LR-ASD (active speaker / who-talks-on-screen); InsightFace SCRFD+ArcFace (agent count + character identity + co-presence); Gaze-LLE (gaze target / joint attention / mutual gaze); RTMPose or ViTPose (pose for proximity/touch/orientation); Qwen2.5-VL (primary local VLM for interaction-type, dominance/affiliation, ToM/intention labeling)
Likely redundant:
- TalkNet-ASD — superseded by Light-ASD/LR-ASD (same task, ~23x more params for equal mAP)
- OpenPose — legacy, superseded by RTMPose/ViTPose for multi-person pose
- InternVL3 — redundant with Qwen2.5-VL as the primary local social-reasoning VLM; keep at most one primary + optional agreement check
- Frontier APIs (GPT-4o/Gemini/Claude) — redundant with local VLMs for routine annotation; reserve for gold-label validation/distillation
- DeSIQ / VEGAS / Human Behavior Atlas / SIV-Bench — overlapping social benchmarks, redundant with Social-IQ 2.0 + Social Genome as the chosen schema source
- Dedicated HOI/dyadic-interaction detectors (HOTR, PRTT, ViPLO) — image-centric, no robust video social-interaction-type SOTA; the VLM layer supersedes them for movie-timeline use
References
- pyannote-audio 3.1 (speaker-diarization-3.1) — huggingface.co/pyannote/speaker-diarization-3.1; github.com/pyannote/pyannote-audio
- Light-ASD / LR-ASD (CVPR 2023) — github.com/Junhua-Liao/Light-ASD; arxiv 2303.04439 (Light-ASD); IJCV 2025 LR-ASD
- InsightFace (SCRFD/RetinaFace detection + ArcFace/buffalo_l recognition) — insightface.ai/arcface; github.com/deepinsight/insightface
- Gaze-LLE (CVPR 2025, Highlight) — arxiv 2412.09586; github.com/fkryan/gazelle
- RTMPose / ViTPose (MMPose) + ByteTrack — arxiv 2303.07399 (RTMPose); ViTPose NeurIPS 2022; github.com/open-mmlab/mmpose
- Qwen2.5-VL (7B / 72B, Instruct) — qwenlm.github.io/blog/qwen2.5-vl; HF Qwen2.5-VL
- InternVL3 (8B / 38B / 78B) — OpenGVLab InternVL3; arxiv InternVL3
- Frontier multimodal APIs (Claude Opus 4.x, GPT-4o, Gemini 2.x) as social-reasoning judges — Social Genome (arxiv 2502.15109) uses such models as graders; provider API docs
- Social-IQ 2.0 / Social Genome / SIV-Bench (benchmark + schema resources) — cmu-multicomp-lab.github.io/social-iq-2.0; cmu-multicomp-lab.github.io/social-genome; arxiv 2502.15109; arxiv 2506.05425 (SIV-Bench)