Scoping Review — Overview & Executive Summary

Computational Tools for Second-by-Second Annotation of Movies and Stories

Naturalistic stimuli — feature films, television, and spoken narratives — have become central to cognitive and affective neuroscience because they engage perception, language, social cognition, and emotion as they actually unfold in the world. But their analytic value depends on a prerequisite that is rarely trivial: turning a continuous audiovisual stream into a dense, time-locked set of features that can be aligned to brain data on a common clock (per second, per frame, or per fMRI repetition time, TR). This review maps the computational tools that produce those features and offers a tractable, best-in-class set a small lab can actually deploy.

We organize the entire annotation space into a six-class semantic hierarchy. Three classes are defined by signal source — Visual (pixels), Audio (waveform), and Language (the time-aligned transcript) — and within each, features run from low-level signal statistics through mid-level structure to high-level semantics. Three further classes — Social, Situation, and Affect — are defined by what is represented rather than by any single sensor; they are intrinsically cross-modal and consume outputs from the first three. This crossing of modality with representational level is the review’s backbone and lets a reader walk any modality from “what the sensor measures” to “what it means.”

Our headline finding is that the field has consolidated. For most subclasses a handful of mature, open-source, locally deployable tools dominate, and a single consumer GPU (often just a CPU) suffices. The recommended core: classic computer-vision libraries (scikit-image, OpenCV) plus vision-science toolboxes (pyrtools, SHINE) for low-level visual statistics; SEA-RAFT, pymoten, and TransNetV2 for motion and shot structure; vision-language and self-supervised encoders (SigLIP 2, DINOv2/v3) plus a video-LLM (Qwen2.5/3-VL) for visual semantics and actions; OpenFace 3.0, InsightFace, and MMPose for faces, identity, gaze, and body pose. On the audio side, librosa, openSMILE, and Parselmouth cover acoustics and voice quality; BEATs/CLAP handle sound events; faster-whisper/WhisperX and pyannote produce transcripts and speaker turns. For language, spaCy plus established norm databases (SUBTLEX, Brysbaert, Warriner, NRC) supply lexical features, while instruction-tuned LLM embeddings (Qwen3-Embedding) and hidden states anchor discourse, coherence, and the neuroimaging encoding-model standard. The cross-modal classes lean on the same primitives fused by a video-LLM, with depicted affect drawn from per-modality models (audeering wav2vec2, HSEmotion, GoEmotions) and event boundaries from GSBS or BrainIAK HMMs.

Two design cautions recur throughout. First, depicted versus elicited affect (what a character expresses versus what a viewer feels) must be logged as separate streams with different ground truth. Second, temporal unit (frame, window, utterance, event, shot) must be recorded explicitly, because every feature ultimately resamples to one shared per-second/per-TR movie clock. These two fields are the most likely to be silently conflated downstream and should be required, not optional.

Introduction

Motivation

For decades, cognitive neuroscience favored tightly controlled, minimal stimuli — isolated words, gratings, static faces — because they made the mapping from stimulus to neural response interpretable. That control came at a cost: it stripped away the temporal continuity, multimodal richness, and social-emotional context in which cognition normally operates. The shift toward naturalistic paradigms — watching films, listening to stories — restores that ecological validity and has proven remarkably productive, enabling work on event segmentation, narrative comprehension, social inference, and shared neural responses across individuals.

The methodological catch is that a movie or a story is not, by itself, an independent variable. To relate brain activity to the stimulus, the stimulus must first be annotated: decomposed into quantitative feature timelines — luminance and motion energy, speaker identity and word surprisal, depicted emotion and event boundaries — each sampled densely enough to track second-by-second variation and aligned to the same clock as the neural recording. Historically this was done by hand, which is slow, subjective, and impossible to scale to feature films. The explosion of computer-vision, audio, and language models over the past decade has made automated second-by-second annotation feasible, but it has also produced a bewildering and fast-moving toolscape. A researcher who wants, say, a per-TR regressor for “social interaction intensity” or “visual clutter” faces dozens of candidate models with unclear redundancy, maturity, and hardware demands. This review exists to make that landscape navigable.

The feature taxonomy at a glance

We classify annotation features along two axes. The first is modality, the signal source: the visual stream (1), the audio waveform (2), and the time-aligned linguistic transcript (3). The second is representational level, the climb from raw signal to meaning: low-level perceptual statistics, mid-level structure, and high-level semantics — with a static-versus-dynamic split preserved within vision and audio. Layered on top are three cross-modal classes organized by what they represent: Social (4), depicted interpersonal content; Situation (5), settings, schemas, scripts, and event structure; and Affect (6), emotion in both its depicted and elicited senses. Each leaf in this tree is a feature group with a stable dotted address (for example visual.lowlevel_static.color or affect.depicted.vocal), a modality and level tag, a temporal unit, onset/duration, values, and provenance. Crucially, cross-modal classes do not copy values; they store references to the contributing leaves, making lineage explicit. A language subclass acts as a deliberate seam — speech-to-language tools (ASR, diarization, prosody) are computed from the waveform but produce the linguistic substrate, so they sit at the head of the Language class while referencing, not duplicating, the acoustic layer.

Methodology of this review

We conducted a scoping review (rather than a systematic effectiveness review) because the goal is breadth of coverage and practical orientation, not pooled effect sizes. For each leaf of the hierarchy we surveyed the current open-source and API tool ecosystem, prioritizing tools that (a) emit time-resolved, timeline-alignable output, (b) are documented and maintained, and (c) are deployable by a small lab on modest hardware. Each tool is characterized by its concrete output (what value it produces per temporal unit), its compute profile (CPU/GPU/API), its native temporal granularity, and a recommendation tier (include / include-if-resources / alternative / drop). For every subclass we then identify a best-in-class default set and flag likely-redundant tools — overlapping options where running more than one buys little. This redundancy analysis is deliberate: it converts an overwhelming menu into a defensible minimal pipeline.

Headline recommendations

Our central recommendation is to resist the temptation to run everything. For each subclass, a small overlapping set of mature tools captures nearly all the available signal; additional models are mostly redundant variants. The tractable best-in-class pipeline named in the Executive Summary lets a single lab annotate a full feature film across all six classes on one consumer GPU. The recurring decision points are practical, not exotic: which norm databases and LLM to license, what window stride sets temporal granularity, and how to resample heterogeneous units onto a shared per-second clock. Finally, we stress two non-negotiable schema fields — temporal_unit and a depicted_vs_elicited flag — as the safeguards most likely to prevent silent misinterpretation of these feature streams downstream.

Narrative Feature Extraction