Visual features

Low-level static visual properties

Per-frame low-level image statistics (luminance, RMS contrast, color in RGB/HSV/CIELAB, colorfulness, spatial-frequency power spectra, edge/orientation energy, entropy, clutter) are a mature, well-understood domain that is best served by classic computer-vision libraries and a handful of vision-science toolboxes rather than learned models. The recommended core is scikit-image + OpenCV + NumPy for the bulk of pixel/color/edge/frequency statistics, a steerable-pyramid library (pyrtools / plenoptic) for biologically motivated orientation-energy and spatial-frequency banks, SHINE for canonical luminance/contrast/Fourier-amplitude measures aligned with the vision-science literature, and the Rosenholtz Feature-Congestion/Subband-Entropy code for visual clutter. GIST provides a compact holistic spatial-envelope descriptor that bridges low- and mid-level. Learned descriptors add little over these analytic features for this specific subclass; the main “modern” gain is wrapping the classic operators in a per-frame pipeline (e.g. pliers) and running them at 1-25 fps over a video. All of these run on CPU and are trivially feasible for a small lab; the practical work is timeline alignment and per-frame batching, not compute.

Recommended (best-in-class): scikit-image + OpenCV + NumPy as the core per-frame engine (luminance, RMS contrast, RGB/HSV/CIELAB color, entropy, edges, FFT power spectrum); pyrtools / plenoptic steerable pyramid for orientation-energy and spatial-frequency banks; SHINE for canonical, literature-matched luminance/contrast/Fourier-amplitude-spectrum definitions; Rosenholtz Feature-Congestion + Subband-Entropy for visual clutter; Hasler-Susstrunk colorfulness formula (implemented directly) for perceptual color; pliers as the timestamping/orchestration layer feeding the movie timeline

Likely redundant:

• OpenCV Gabor-kernel bank is superseded by pyrtools/GIST for principled orientation-energy (keep OpenCV mainly for speed and video decoding)
• Global Shannon entropy and raw edge density are weaker clutter proxies superseded by Rosenholtz Feature-Congestion / Subband-Entropy
• GIST overlaps heavily with pyrtools orientation-energy; only one is needed unless a compact fixed-length holistic vector is specifically wanted
• Learned low-level descriptors / CNN early-layer activations add little over analytic luminance/contrast/color/frequency features for this subclass and are better reserved for mid/high-level visual subclasses
• pliers built-in extractors duplicate scikit-image/OpenCV computations; it is an orchestration convenience, not an independent feature source
• scikit-image and OpenCV themselves overlap substantially in color/edge/FFT ops

High-level static visual: objects, scenes, places, attributes

For per-frame movie/audio-story annotation of objects, scenes, places, and attributes, the field has consolidated around a few foundation-model families. Two complementary backbones cover most needs: contrastive vision-language encoders (OpenCLIP, SigLIP 2) that give open-vocabulary probe scores and zero-shot scene/place/object/attribute labels via text prompts, and self-supervised encoders (DINOv2/DINOv3) that give label-free dense semantic embeddings. For explicit localized labels, open-vocabulary detectors (Grounding DINO 1.5/1.6, YOLO-World) and promptable/panoptic segmenters (SAM 2, Mask2Former) emit per-object boxes/masks. For rich natural-language scene descriptions and attribute/relation reasoning, open VLMs (Qwen2.5-VL, InternVL3, LLaVA-OneVision) caption frames or answer structured VQA. A small lab can run most of these locally on a single 24-48GB GPU; the main tradeoff is fixed-vocabulary classifiers (Places365, ImageNet backbones) being largely superseded by open-vocabulary CLIP-family probing, except where a frozen, reproducible, interpretable category vector is explicitly wanted.

Recommended (best-in-class): SigLIP 2 (or OpenCLIP ViT-L) as the primary open-vocabulary text-probe encoder for object/scene/place/attribute scores; DINOv2/DINOv3 as the complementary label-free dense embedding for RSA and linear probes; Qwen2.5-VL-7B as the primary captioning/VQA VLM for structured scene descriptions; Grounding DINO (1.6 Pro via API for accuracy, or open 1.0 / YOLO-World locally) for open-vocabulary per-object detection; Mask2Former/OneFormer panoptic for scene-composition (thing+stuff area fractions)

Likely redundant:

• OpenCLIP and SigLIP 2 fill the same text-probe role - run one as primary (SigLIP 2 generally stronger), not both
• Qwen2.5-VL, InternVL3, and LLaVA-OneVision are interchangeable captioning/VQA VLMs - pick one primary after benchmarking your prompt schema
• YOLO-World is superseded on accuracy by Grounding DINO 1.6 Pro; keep only when per-frame throughput/density is the constraint
• Places365 CNN and ImageNet-trained backbones are largely superseded by zero-shot CLIP/SigLIP2 probing of the same label sets - keep only for a frozen, reproducible fixed-taxonomy probability vector
• SAM 2 alone produces no labels and overlaps Mask2Former for static panoptic needs; only adds value via video-consistent tracking or as Grounded-SAM

Open questions:

• Frame sampling rate to hit true second-by-second alignment vs compute budget (1 fps probing vs shot-level keyframes for heavy VLMs)
• Whether to derive scene/place labels via zero-shot CLIP/SigLIP2 prompting or keep a frozen Places365 head for reproducibility/citation in fMRI work
• DINOv3 licensing (research/non-commercial terms) may restrict use vs Apache-2.0 DINOv2
• Grounding DINO 1.6 Pro best accuracy is API-only - acceptable for a small lab, or fall back to open 1.0/YOLO-World?
• Need for an aggregation/parsing layer to convert free-text VLM captions into consistent categorical per-second feature vectors (schema + JSON-constrained decoding)

Faces, bodies, gaze, and facial expression

This subclass covers per-frame extraction of where faces and bodies are, who they are, where they look, and what they express. The mature open-source stack is: a detector/tracker (RetinaFace/InsightFace, MediaPipe) feeding identity (ArcFace), landmarks, head pose and gaze (OpenFace 3.0, L2CS-Net), facial action units and discrete/continuous emotion (Py-Feat, OpenFace 3.0, HSEmotion/EmotiEffLib, EmoNet), and body/hand keypoints (MMPose’s RTMPose/RTMW, ViTPose, MediaPipe Holistic). As of 2025, OpenFace 3.0 (arXiv 2506.02891) consolidates landmarks+AUs+gaze+emotion into one lightweight CPU-capable multitask model, and MMPose RTMPose/RTMW dominate real-time 2D/whole-body pose; these are the recommended defaults for a small lab. All major components run on a single consumer GPU (often CPU), making second-by-second annotation of long films feasible.

Recommended (best-in-class): OpenFace 3.0 (landmarks + AUs + gaze + emotion in one CPU-capable multitask model) as the primary face-behavior extractor; InsightFace buffalo_l (RetinaFace + ArcFace) for face detection, tracking, and identity embeddings; MMPose RTMPose / RTMW for real-time 2D and whole-body (body+face+hands+feet) keypoints; MediaPipe Tasks for cheap CPU-only body/hand/face landmarks + blendshapes as a fallback/redundancy layer; HSEmotion/EmotiEffLib for fast continuous valence-arousal and expression, and L2CS-Net when high-accuracy 360-degree gaze is a primary signal

Likely redundant:

• OpenFace 2.0 -> superseded by OpenFace 3.0 (faster, multitask, better accuracy); keep only for reproducing legacy pipelines or its validated AU intensity scale
• Standalone RetinaFace -> redundant with InsightFace, which bundles it; and with detectors inside OpenFace 3.0 / Py-Feat
• ViTPose -> redundant with MMPose RTMPose/RTMW for typical lab use; only worth it for max-accuracy keypoints on a strong GPU
• EmoNet -> valence-arousal largely superseded by HSEmotion (faster, newer, more outputs); keep as a citable cross-check
• L2CS-Net gaze and Py-Feat/HSEmotion emotion overlap OpenFace 3.0’s bundled gaze/emotion -> run separately only when those specific signals need best-in-class accuracy
• MediaPipe body/hand keypoints overlap MMPose (lower accuracy, single-best-effort) -> use MediaPipe for cheap CPU-only or redundancy, MMPose for accuracy/multi-person

Dynamic visual: motion, optical flow, motion energy

This subclass covers low-level dynamic visual signal: dense optical flow (per-pixel motion vectors), aggregate motion energy from spatiotemporal filter banks, camera-vs-object motion separation, shot/cut boundaries, and frame-to-frame visual change/novelty. These are deliberately pre-semantic features (how much/which-direction things move, when the camera cuts) distinct from semantic action recognition. The field is mature and largely open-source: RAFT and its 2024 successor SEA-RAFT dominate dense flow, MemFlow/VideoFlow add multi-frame temporal smoothing, pymoten provides the canonical neuroscience motion-energy pyramid, and TransNetV2 plus PySceneDetect cover shot detection. For a small lab, a practical default pipeline is: PySceneDetect/TransNetV2 for cuts, pymoten for a brain-aligned motion-energy regressor, and SEA-RAFT (or RAFT) for dense flow from which you derive global/mean flow magnitude (proxy for camera motion) and residual flow (object motion).

Recommended (best-in-class): SEA-RAFT (dense optical flow; derive global camera-motion vs residual object-motion and mean flow magnitude); pymoten (neuroscience-standard motion-energy pyramid feature for direct fMRI encoding); TransNetV2 (shot/cut boundary detection incl. gradual transitions); PySceneDetect (fast CPU cut detection + continuous per-frame visual-change/novelty score)

Likely redundant:

• RAFT -> superseded by SEA-RAFT (better accuracy and ~2.3x faster); keep only as easy torchvision fallback
• VideoFlow -> superseded by MemFlow (faster, fewer params, higher accuracy for multi-frame flow)
• AutoShot -> overlaps TransNetV2; only preferable for short-form/social video stimuli
• OpenCV Farneback/DIS -> superseded by SEA-RAFT/RAFT in accuracy; keep only as no-GPU fallback
• GMFlow/FlowFormer family -> covered in spirit by SEA-RAFT as the chosen single dense-flow model; no need to run multiple deep flow nets

Open questions:

• Camera-vs-object motion separation is not a turnkey tool: it must be derived from a dense-flow model (e.g. robust homography/affine global-motion fit on the flow field, then residual = object motion). Decide whether to implement this or adopt a SLAM/egomotion package.
• Choose the temporal aggregation strategy: flow and motion-energy are per-frame, so define how to bin/downsample to per-second or per-TR (mean, RMS, or HRF-convolved) for the neuroimaging timeline.
• For long narrative films, decide between per-pair flow (SEA-RAFT) vs temporally-smoothed multi-frame flow (MemFlow) depending on whether jitter in the second-by-second series is a problem.
• Confirm SEA-RAFT vs MemFlow generalization on your actual stimulus domain (animation, archival film, low light) since flow models are trained mostly on Sintel/KITTI/Spring.

Action and activity recognition (video understanding)

Computational action/activity recognition for a movie/audio-story timeline splits into three usable layers: (1) clip-level recognition backbones that emit Kinetics/Something-Something class posteriors and/or pooled spatiotemporal embeddings over short sliding windows (VideoMAE V2, InternVideo2, V-JEPA 2, TimeSformer, VideoSwin, SlowFast), best run at a fixed window stride (e.g. 16 frames / 1-2 s) to produce per-second features; (2) temporal action localization (TAL) heads (ActionFormer / OpenTAD, typically fed InternVideo2 or VideoMAE features) that turn those per-window embeddings into start-end action segments with confidence; and (3) video-LLMs (Qwen3-VL, VideoLLaMA3, InternVL3, Video-LLaVA) that produce open-vocabulary dense event descriptions with timestamps, valuable for naturalistic film where a fixed label set is too restrictive. For a small lab the most leverage comes from: VideoMAE V2 / InternVideo2 as frozen feature extractors on a sliding window, X-CLIP for zero-shot open-vocabulary action probabilities, and one video-LLM for dense captioning. The pure clip classifiers (TimeSformer/VideoSwin/SlowFast) are now largely superseded as standalone choices but remain valuable as the standard, well-documented MMAction2/PySlowFast backbones and as TAL feature sources.

Recommended (best-in-class): VideoMAE V2 (ViT-L/g) or InternVideo2-1B as the frozen sliding-window feature extractor + Kinetics classifier; X-CLIP for lightweight zero-shot open-vocabulary action probabilities over a custom film vocabulary; ActionFormer/OpenTAD over InternVideo2/VideoMAE features for timestamped action segments; Qwen3-VL (8B) as the primary video-LLM for open-vocabulary dense, timestamped event description

Likely redundant:

• TimeSformer / VideoSwin / SlowFast as standalone recognizers -> superseded by VideoMAE V2 and InternVideo2 (keep only for the convenient MMAction2/PySlowFast pipeline or as legacy TAL features)
• InternVL3 (video) and VideoLLaMA3 -> overlap with Qwen3-VL for dense captioning/temporal grounding; pick one video-LLM as primary (Qwen3-VL has the strongest native timestamp grounding)
• X-CLIP zero-shot vs InternVideo2 zero-shot -> same open-vocabulary role; InternVideo2 is more accurate but far heavier, X-CLIP is the cheap default
• V-JEPA 2 vs VideoMAE V2 -> both self-supervised backbones; V-JEPA 2 is the more motion/temporal-biased complement, not a strict duplicate

Open questions:

• Licensing: VideoMAE V2 and InternVideo2 weights are non-commercial/research only; confirm this is acceptable for the project’s use and publication plans.
• Window stride and temporal resolution: clip backbones natively output per-16-frame (~0.5-1 s) features; the target second-by-second cadence requires choosing stride/overlap and a pooling rule that needs to be standardized across features.
• Label taxonomy: Kinetics-400/700 and Something-Something classes are everyday/atomic actions and may map poorly onto narrative-film events; decide whether to rely on TAL with a custom taxonomy vs open-vocabulary X-CLIP/video-LLM outputs.
• Video-LLM timestamp reliability: Qwen3-VL/VideoLLaMA3 emit timestamps but their second-level accuracy on long naturalistic films is not well validated; may need calibration against TAL or manual spot-checks.
• Compute budget: InternVideo2-6B and Qwen3-VL-30B+ exceed a single-GPU small-lab setup; confirm whether 1B/8B-class models suffice for the required annotation quality.

Visual saliency, attention, aesthetics, depth

This subclass produces per-frame attention/complexity proxies: predicted human fixation/saliency maps (static and video), monocular depth maps, and scalar image-level scores for aesthetics, technical quality, and memorability. For a movie/audio-story timeline, the most timeline-relevant outputs are spatiotemporal video-saliency maps (one map per frame, summarizable to scalars like saliency concentration/entropy and number-of-salient-regions) and depth maps (mean/range/foreground-fraction per frame). The field is mature and overwhelmingly open-source: DeepGaze IIE/III for static fixation, ViNet/TASED-Net/UNISAL for video saliency, Depth-Anything-V2 (and Apple Depth Pro for metric) for depth, and CLIP/LMM-based scorers (Q-Align, LAION-Aesthetics, ResMem) for aesthetics/quality/memorability. All run on a single consumer GPU; most also run on CPU slowly. Recommend one video-saliency model + one depth model + one aesthetics/quality scorer + memorability as the core, with static-image saliency only as an alternative.

Recommended (best-in-class): ViNet (or ViNet-A for audio-visual) for per-frame video saliency maps -> saliency concentration/entropy, peak, salient-area, temporal shift; Depth-Anything-V2 for per-frame relative depth (add Apple Depth Pro only if metric/meters depth is needed) -> mean/range depth, foreground-fraction, depth complexity; Q-Align/OneAlign for combined per-frame aesthetic + technical-quality scalars (subsample to ~1 fps), with IQA-PyTorch (MUSIQ/NIMA/LAION-AES) as a lightweight fallback; ResMem for per-frame image-memorability scalar

Likely redundant:

• MiDaS / DPT / ZoeDepth — superseded by Depth-Anything-V2 (relative) and Apple Depth Pro (metric)
• TASED-Net and UNISAL — same video-saliency output as ViNet; pick one (ViNet best accuracy, UNISAL lightest)
• DeepGaze IIE/III (static fixation) — for video timelines superseded by motion-aware video-saliency models; keep only as a pure static-image attention proxy
• LAION-Aesthetics V2 and CLIP-IQA/QualiCLIP — subsumed by Q-Align’s IAA/IQA branches (use them only as lightweight substitutes when the 7B LMM is too heavy)
• MemNet / ResMem-Net — older or less-standardized memorability predictors superseded by the maintained, pip-installable ResMem