Narrative Feature Extraction

Computational annotation of movies and stories for cognitive science and naturalistic neuroimaging.

This site is the entry point for browsing every part of the project: the Phase 1 scoping review, the annotation format specification and pipeline, the how-to guides for growing the corpus, the draft review paper with its figures, and the interactive segment browser.

What the project does

A human supplies a movie or an audio/text story; the pipeline returns a hierarchical, semantically organized, second-by-second set of annotations — visual, audio, language, social, situational, and affective — produced by a curated set of best-in-class models. Features that do not apply to a stimulus (e.g. visual features for an audio-only story) are returned as explicit nulls, so every annotation shares one constant shape and the corpus stacks into rectangular matrices for analysis.

At a glance

• 83 stimuli annotated (~7.8 h): 53 audiovisual, 29 audio-only, 1 text-only.
• 95 channels per stimulus across six feature classes, on a common 1 Hz grid.
• Local-first models on Apple-Silicon GPU/CPU; HDF5 + JSON output with a MATLAB reader.

How this book is organized

Start here

• Review paper (draft) — the models/algorithms behind each annotation and the empirical structure of the annotation space across the corpus.
• Contents & user guide — the full map of tools, datasets, derivatives, and how to load / view / inspect them (with MATLAB recipes).
• Interactive segment browser — rank stimulus segments by any combination of features.

The source repository holds the pipeline (src/nfe/), the MATLAB reader (matlab/), and the annotated corpus derivatives. This documentation book is generated from the canonical Markdown in docs/ by tools/build_book.py.

Narrative Feature Extraction

📖 Browse the documentation book →

The full documentation — scoping review, annotation format, pipeline, corpus, the review paper with figures, and the interactive segment browser — is published as a book at canlab.github.io/narrative_feature_annotations. It is the recommended entry point. (Build/deploy notes: docs/DEPLOYING.md.)

🗺️ Feature map — all 95 annotation channels at a glance

A map of every feature and its hierarchical organization. The source is an editable SVG (open/recolor in PowerPoint, Illustrator, or Inkscape); regenerate with python3 tools/build_feature_map.py.

Infrastructure for producing second-by-second computational annotations of movies and audio stories across the full range of stimulus, perceptual, linguistic, social, situational, and affective features — for use in cognitive science and naturalistic-neuroimaging research.

A human supplies a movie or story; the pipeline returns a hierarchical, semantically organized, time-resolved set of annotations from a curated set of best-in-class models. Annotations that do not apply to a given stimulus (e.g. visual features for an audio-only story) are returned as explicit nulls (NaN). The output reserves space for later human annotations and ships with a lightweight MATLAB reader.

Project phases

Repository layout

docs/
  scoping_review/      # Phase 1 output: the hierarchical scoping review
  design/              # plans, schema design, deployment feasibility
schema/                # annotation output format (spec + JSON schema + example)
src/nfe/               # the pipeline package
  base.py              #   types, common-grid TimeGrid, resampler
  ingest.py            #   PyAV decode + audio extraction
  emit.py              #   HDF5 + JSON-sidecar writer
  pipeline.py, run.py  #   orchestration + CLI
  extractors/          #   one module per feature pass
matlab/                # MATLAB reader (readAnnotations / getFeature / featuresToTimetable)
data/
  movies/              # input movies (spacetop/ = lab stimuli; open/ = CC-BY test films)
  stories/             # input audio stories
annotations/
  corpus/<id>/         # corpus derivatives: <id>.h5 + <id>.manifest.json per stimulus
                       #   + corpus_index.csv (batch status)
analysis/
  figures/             # generated analysis figures
  corpus_stats.json    # corpus summary numbers the docs cite
  web/                 # interactive segment-search app
tools/                 # helper scripts (manifest, template, search index, refresh)

Quickstart (Phase 2 pipeline)

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
PYTHONPATH=src .venv/bin/python -m nfe.run data/movies/open/BigBuckBunny.mp4 --rate 1.0

See docs/design/PHASE2_STATUS.md for what runs today and what’s next.

Design principles

• Common time grid. Every feature has a native rate (frame, sub-second, per-second, per-utterance); all are resampled onto a configurable common grid (default 1 Hz) with native-rate metadata preserved.
• Explicit nulls. Inapplicable annotations are NaN, never silently omitted, so downstream analysis can distinguish “not measured” from “measured zero.”
• Hierarchical + readable. Output is organized by the semantic hierarchy in docs/scoping_review/ so a human can read it and a program can traverse it.
• Local-first. Prefer open, locally deployable models; fall back to hosted large models only where they are clearly superior and irreplaceable.
• Reproducible provenance. Each feature records the model name, version, and parameters that produced it.

Start here

• docs/CONTENTS.md — full guide to the folder, tools, datasets, the extracted derivatives, and how to load / view / inspect them.
• docs/walkthrough.m — runnable MATLAB tour of the common operations.
• docs/REVIEW_PAPER.md — draft review: the models/algorithms and the empirical structure of the annotation space across the corpus.
• docs/ADDING_MOVIES.md — how to add movies/stories and refresh the whole dataset (annotations, search index, figures, and the corpus-specific doc numbers).

Key documents

• Scoping review (Phase 1 output): docs/scoping_review/ — 146 tools across 15 feature subclasses, the semantic hierarchy, redundancy analysis, and best-in-class recommendations.
• Annotation output format: docs/design/ANNOTATION_FORMAT.md — authoritative v0.2 spec (HDF5 canonical + readable JSON sidecar; common grid; explicit nulls; reserved human-annotation slots; MATLAB reader interface). schema/annotation_schema.json is the v0.1 pure-JSON profile used for small/demo files, and matlab/*.m is the reference reader for that profile.
• Phase 2 core-set freeze sheet: docs/design/PHASE2_CORE_SET_PROPOSAL.md — fillable decision doc; sign-off freezes the Phase 2 build scope.
• Deployment feasibility: docs/design/DEPLOYMENT_FEASIBILITY.md.
• Full plan: docs/design/IMPLEMENTATION_PLAN.md.

Feature map

A visual map of all 95 annotation channels and their hierarchical organization — the six feature classes, each grouped by subclass, with one chip per channel.

Editable graphic

The map is a plain, flat SVG built for editing: analysis/figures/feature_map.svg.

• Open it in PowerPoint, Illustrator, Inkscape, or any browser. In PowerPoint, Insert → Pictures, then right-click → Convert to Shape to recolor, move, or relabel individual cards, chips, and text.
• It uses real text elements, per-card groups, and inline colors (no CSS or filters), so it survives import cleanly and recolors easily.
• Regenerate it whenever the channel set changes:

  python3 tools/build_feature_map.py     # reads schema/channel_template.json
  

Related

• The designed semantic hierarchy behind these channels (the taxonomy the pipeline is organized around): feature hierarchy.
• What each pass actually computes: pipeline status.
• The on-disk layout of these channels in each .h5: annotation format.

Project Contents & User Guide

A map of everything in this repository: the tools, datasets, and extracted derivatives, what is included so far, and how to load, view, and inspect them. For a hands-on tour run docs/walkthrough.m in MATLAB.

What this project is. Infrastructure that turns a movie or audio story into second-by-second, hierarchical, computational annotations (visual, audio, language, social, situational, affective), stored in a constant-shape format that loads into MATLAB, plus tools to analyze the annotation structure across a corpus and to design high-variance / low-redundancy stimulus sets.

Status at a glance

Folder map

README.md                  project overview + quickstart
requirements.txt           Python deps (pipeline)
data/
  manifest.csv / .json     stimulus catalog (83 stimuli) — tools/build_manifest.py
  movies/spacetop/         lab fMRI stimulus clips (49)        [internal]
  movies/open/             CC-BY Blender films (3) + SOURCES.md
  lexicons/                optional psycholinguistic norm CSVs (README.md)
  stories/narratives/      29 Narratives spoken-story audio clips (+ SOURCES.md)
  stories/samples/         pure-text sample story (.txt)
  movies/hcp/, movies/camcan/  placeholders for credentialed stimuli (see EXTERNAL_STIMULI.md)
schema/
  channel_template.json    the 95-channel constant-shape template
  annotation_schema.json   v0.1 pure-JSON profile schema
  example_annotation.json  tiny worked example
src/nfe/                   the Python annotation pipeline (see "Tools")
matlab/                    the MATLAB readers + analysis (see "Tools")
annotations/
  corpus/<id>/<id>.h5      DERIVATIVES: one annotation per stimulus (+ .manifest.json)
  corpus/corpus_index.csv  batch status/index
analysis/figures/          generated analysis figures (PNG)
analysis/corpus_stats.json corpus summary numbers the docs cite (refreshAnalysis)
analysis/web/              interactive segment search (index.html + segments.json + README)
docs/
  CONTENTS.md              ← this file
  walkthrough.m            runnable MATLAB tour
  scoping_review/          Phase 1 review (hierarchy, recommendations, ...)
  design/                  format spec, plans, per-phase status
tools/                     helper scripts (manifest, template, review assembly)

Datasets

• Corpus manifest: 83 stimuli (data/manifest.csv) — 49 spacetop audiovisual clips + 4 short films (3 CC-BY Blender open films + Kung Fury) + 29 Narratives spoken-story audio clips (data/stories/narratives/, ~5.3 h; OpenNeuro ds002345, Nastase et al.) + 1 pure-text sample story. 53 audiovisual, 29 audio-only, 1 text-only. Add media under data/movies/<source>/ (movies/audio) or data/stories/<source>/ (audio/text stories) and re-run tools/build_manifest.py. See ADDING_MOVIES.md; credentialed sets (HCP, CamCAN) in EXTERNAL_STIMULI.md. All 83 are annotated; corpus-wide analysis focuses on the audio/language channels shared across modalities, with visual/social/ affective structure on the 53-stimulus audiovisual subset (see REVIEW_PAPER.md §5–6).
• Lexicons (optional): drop data/lexicons/<field>.csv (valence, arousal, dominance, concreteness, aoa) to light up those per-word channels; absent → NaN.

Extracted derivatives

One annotation per stimulus at annotations/corpus/<id>/:

• <id>.h5 — canonical HDF5. Hierarchical groups mirror the feature taxonomy:

  /time/        common 1 Hz grid (rate_hz, t_start_sec, n_samples, time_sec)
  /stimulus/    id, title, modality, duration, source, sha256
  /features/    visual/ audio/ language/ social/ situation/ affect/
                  → each leaf = one channel dataset [n_samples (× dim)] + attrs
                    (dtype, applicable, units, model, version, native_rate_hz, resample)
  /human/       reserved, empty — slots for later human annotation
  /provenance/  per-channel model registry
  

• <id>.manifest.json — readable sidecar: same hierarchy + metadata, no bulk arrays.

Constant shape. Every file has the same 95 channels (the template). Channels not produced for a stimulus (a class that doesn’t apply to the modality, or a pass not run) are present with applicable=false and all-NaN, so the corpus stacks into rectangular matrices. Full spec: design/ANNOTATION_FORMAT.md.

The 95 channels span: visual (37 — low-level, semantic SigLIP2/DINOv2, motion, depth, action, faces, pose, saliency), audio (21 — low-level, AudioSet/CLAP, speech), language (14 — lexical, syntax, surprisal), affect (14 — text emotion/ sentiment, vocal, EmoNet image emotion, facial affect, VLM depicted), situation (5 — incl. GSBS event boundaries), social (4). What each pass computes: design/PHASE2_STATUS.md.

Tools

Python pipeline — src/nfe/ (annotate media)

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
# annotate ONE movie (CPU passes always on; add MPS/VLM passes as flags):
PYTHONPATH=src .venv/bin/python -m nfe.run data/movies/open/BigBuckBunny.mp4 \
    --vision --audio-hl --events --template schema/channel_template.json
# annotate the WHOLE corpus (resumable, crash-safe):
PYTHONPATH=src .venv/bin/python -m nfe.batch --manifest data/manifest.csv \
    --out annotations/corpus --template schema/channel_template.json \
    --vision --audio-hl --events

Pass flags: --vision (SigLIP2/DINOv2/RAFT/depth/VideoMAE/faces/pose/saliency), --audio-hl (AST/CLAP/vocal-affect/text-emotion/surprisal), --reason (Qwen2.5-VL — slow), --events (GSBS). Modules: ingest (PyAV decode), extractors/ (the 20 passes), emit (HDF5+JSON), pipeline/run/batch, schema_registry (skeleton fill). Helpers in tools/: build_manifest.py, build_channel_template.py, build_search_index.py (segment index for the web search interface).

Web — analysis/web/ (interactive segment search)

Browser tool to rank segments by any combination of features and play the matching moment. Serve from the project root, then open the page:

python3 tools/serve.py           # from the project root (Range-enabled, so video seeking works)
# open http://localhost:8000/analysis/web/index.html

Rebuild the index after annotating more stimuli: PYTHONPATH=src .venv/bin/python tools/build_search_index.py --seglen 5. Details: ../analysis/web/app_readme.md.

MATLAB — matlab/ (load, view, analyze)

How to load, view, inspect (MATLAB quick recipes)

addpath matlab

% 1) Inspect ONE stimulus
ann = readAnnotations("annotations/corpus/ses-01_run-01_order-04_content-parkour");
tt  = featuresToTimetable(ann);          % scalars as a timetable
stackedplot(tt(:, ["visual__low_level_static__luminance","audio__low_level__rms"]))
mf  = getFeature(ann, "audio/low_level/mfcc");   % a vector channel [n × 13]

% 2) WATCH a movie with its annotations scrolling underneath
m = "data/movies/spacetop/videos/ses-01/ses-01_run-01_order-04_content-parkour.mp4";
annotationMovieViewer(m, "annotations/corpus/ses-01_run-01_order-04_content-parkour")

% 3) Analyze the whole CORPUS
C   = readAnnotationCorpus("annotations/corpus");
res = analyzeCorpus(C);                   % figures → analysis/figures/
sel = selectStimulusSet(C, "K", 20);      % design a stimulus set; sel.table

% Python inspection (alternative): h5py / pandas on <id>.h5 and corpus_index.csv

# 4) SEARCH segments by feature in the browser (serve from project root)
python3 tools/serve.py           # then open http://localhost:8000/analysis/web/index.html

See walkthrough.m for the same steps, runnable section by section.

Documentation index

• Feature map (all 95 channels): FEATURE_MAP.md — a graphical map of the feature hierarchy; editable SVG at analysis/figures/feature_map.svg
• Adding movies/stories: ADDING_MOVIES.md — how to grow the corpus and refresh all derivatives + corpus-specific doc numbers
• External stimuli (HCP, CamCAN): EXTERNAL_STIMULI.md — how to obtain the credentialed/copyrighted neuroimaging movie stimuli and integrate them
• Review paper (draft): REVIEW_PAPER.md — models/algorithms + the empirical structure of the annotation space (correlation, dimensionality, networks, contingency on stimulus type, design tool)
• Scoping review (Phase 1): scoping_review/
• Format spec: design/ANNOTATION_FORMAT.md
• Plan & per-phase status: design/IMPLEMENTATION_PLAN.md, PHASE2_STATUS.md, PHASE3_CORPUS.md, PHASE4_ANALYSIS.md
• Frozen feature set: design/PHASE2_CORE_SET_PROPOSAL.md
• Deployment: design/DEPLOYMENT_FEASIBILITY.md

Scoping Review — Computational Tools for Second-by-Second Annotation of Movies and Stories

Generated by a 21-agent research workflow. Sections:

1. Overview & Executive Summary
2. Semantic Feature Hierarchy
3. Visual features
4. Audio features
5. Language features
6. Social features
7. Situation features
8. Affect features
9. Redundancy & Parsimony Analysis
10. Recommendations — Best-in-Class Feature Set

Companion design docs: annotation output format, deployment feasibility, implementation plan.

Catalog covers 146 tools across 15 feature subclasses.

Scoping Review — Overview & Executive Summary

Computational Tools for Second-by-Second Annotation of Movies and Stories

A Scoping Review

Executive Summary

Naturalistic stimuli — feature films, television, and spoken narratives — have become central to cognitive and affective neuroscience because they engage perception, language, social cognition, and emotion as they actually unfold in the world. But their analytic value depends on a prerequisite that is rarely trivial: turning a continuous audiovisual stream into a dense, time-locked set of features that can be aligned to brain data on a common clock (per second, per frame, or per fMRI repetition time, TR). This review maps the computational tools that produce those features and offers a tractable, best-in-class set a small lab can actually deploy.

We organize the entire annotation space into a six-class semantic hierarchy. Three classes are defined by signal source — Visual (pixels), Audio (waveform), and Language (the time-aligned transcript) — and within each, features run from low-level signal statistics through mid-level structure to high-level semantics. Three further classes — Social, Situation, and Affect — are defined by what is represented rather than by any single sensor; they are intrinsically cross-modal and consume outputs from the first three. This crossing of modality with representational level is the review’s backbone and lets a reader walk any modality from “what the sensor measures” to “what it means.”

Our headline finding is that the field has consolidated. For most subclasses a handful of mature, open-source, locally deployable tools dominate, and a single consumer GPU (often just a CPU) suffices. The recommended core: classic computer-vision libraries (scikit-image, OpenCV) plus vision-science toolboxes (pyrtools, SHINE) for low-level visual statistics; SEA-RAFT, pymoten, and TransNetV2 for motion and shot structure; vision-language and self-supervised encoders (SigLIP 2, DINOv2/v3) plus a video-LLM (Qwen2.5/3-VL) for visual semantics and actions; OpenFace 3.0, InsightFace, and MMPose for faces, identity, gaze, and body pose. On the audio side, librosa, openSMILE, and Parselmouth cover acoustics and voice quality; BEATs/CLAP handle sound events; faster-whisper/WhisperX and pyannote produce transcripts and speaker turns. For language, spaCy plus established norm databases (SUBTLEX, Brysbaert, Warriner, NRC) supply lexical features, while instruction-tuned LLM embeddings (Qwen3-Embedding) and hidden states anchor discourse, coherence, and the neuroimaging encoding-model standard. The cross-modal classes lean on the same primitives fused by a video-LLM, with depicted affect drawn from per-modality models (audeering wav2vec2, HSEmotion, GoEmotions) and event boundaries from GSBS or BrainIAK HMMs.

Two design cautions recur throughout. First, depicted versus elicited affect (what a character expresses versus what a viewer feels) must be logged as separate streams with different ground truth. Second, temporal unit (frame, window, utterance, event, shot) must be recorded explicitly, because every feature ultimately resamples to one shared per-second/per-TR movie clock. These two fields are the most likely to be silently conflated downstream and should be required, not optional.

Introduction

Motivation

For decades, cognitive neuroscience favored tightly controlled, minimal stimuli — isolated words, gratings, static faces — because they made the mapping from stimulus to neural response interpretable. That control came at a cost: it stripped away the temporal continuity, multimodal richness, and social-emotional context in which cognition normally operates. The shift toward naturalistic paradigms — watching films, listening to stories — restores that ecological validity and has proven remarkably productive, enabling work on event segmentation, narrative comprehension, social inference, and shared neural responses across individuals.

The methodological catch is that a movie or a story is not, by itself, an independent variable. To relate brain activity to the stimulus, the stimulus must first be annotated: decomposed into quantitative feature timelines — luminance and motion energy, speaker identity and word surprisal, depicted emotion and event boundaries — each sampled densely enough to track second-by-second variation and aligned to the same clock as the neural recording. Historically this was done by hand, which is slow, subjective, and impossible to scale to feature films. The explosion of computer-vision, audio, and language models over the past decade has made automated second-by-second annotation feasible, but it has also produced a bewildering and fast-moving toolscape. A researcher who wants, say, a per-TR regressor for “social interaction intensity” or “visual clutter” faces dozens of candidate models with unclear redundancy, maturity, and hardware demands. This review exists to make that landscape navigable.

The feature taxonomy at a glance

We classify annotation features along two axes. The first is modality, the signal source: the visual stream (1), the audio waveform (2), and the time-aligned linguistic transcript (3). The second is representational level, the climb from raw signal to meaning: low-level perceptual statistics, mid-level structure, and high-level semantics — with a static-versus-dynamic split preserved within vision and audio. Layered on top are three cross-modal classes organized by what they represent: Social (4), depicted interpersonal content; Situation (5), settings, schemas, scripts, and event structure; and Affect (6), emotion in both its depicted and elicited senses. Each leaf in this tree is a feature group with a stable dotted address (for example visual.lowlevel_static.color or affect.depicted.vocal), a modality and level tag, a temporal unit, onset/duration, values, and provenance. Crucially, cross-modal classes do not copy values; they store references to the contributing leaves, making lineage explicit. A language subclass acts as a deliberate seam — speech-to-language tools (ASR, diarization, prosody) are computed from the waveform but produce the linguistic substrate, so they sit at the head of the Language class while referencing, not duplicating, the acoustic layer.

Methodology of this review

We conducted a scoping review (rather than a systematic effectiveness review) because the goal is breadth of coverage and practical orientation, not pooled effect sizes. For each leaf of the hierarchy we surveyed the current open-source and API tool ecosystem, prioritizing tools that (a) emit time-resolved, timeline-alignable output, (b) are documented and maintained, and (c) are deployable by a small lab on modest hardware. Each tool is characterized by its concrete output (what value it produces per temporal unit), its compute profile (CPU/GPU/API), its native temporal granularity, and a recommendation tier (include / include-if-resources / alternative / drop). For every subclass we then identify a best-in-class default set and flag likely-redundant tools — overlapping options where running more than one buys little. This redundancy analysis is deliberate: it converts an overwhelming menu into a defensible minimal pipeline.

Headline recommendations

Our central recommendation is to resist the temptation to run everything. For each subclass, a small overlapping set of mature tools captures nearly all the available signal; additional models are mostly redundant variants. The tractable best-in-class pipeline named in the Executive Summary lets a single lab annotate a full feature film across all six classes on one consumer GPU. The recurring decision points are practical, not exotic: which norm databases and LLM to license, what window stride sets temporal granularity, and how to resample heterogeneous units onto a shared per-second clock. Finally, we stress two non-negotiable schema fields — temporal_unit and a depicted_vs_elicited flag — as the safeguards most likely to prevent silent misinterpretation of these feature streams downstream.

Semantic Feature Hierarchy

This semantic hierarchy organizes all annotation features into a coherent tree. The natural organizing axes are MODALITY (visual / audio / language / multimodal) crossed with LEVEL (low-level signal → mid → high-level semantic/social/situational/affective).

Semantic Hierarchy for Naturalistic-Stimulus Annotation Features

The tree is organized first by modality (the signal source: visual pixels, audio waveform, linguistic transcript) and second by representational level (low-level perceptual signal → mid-level structure → high-level semantic/social/situational meaning). Cross-modal classes (Social, Situation, Affect) sit at the top level because they are defined by what is represented, not by a single signal source, and they consume features from multiple modalities.

ANNOTATION FEATURES
│
├── 1. VISUAL  — features derived from the image/video stream
│   │
│   ├── 1.1 Low-level static (per-frame signal statistics)
│   │   ├── 1.1.1 Luminance & contrast        (mean luminance, RMS/Michelson contrast, luminance histogram)
│   │   ├── 1.1.2 Color                        (RGB/HSV/CIELAB means+SDs, colorfulness, warmth, saturation, dominant hue)
│   │   ├── 1.1.3 Spatial frequency & orientation energy  (FFT power spectrum/slope, steerable-pyramid subband energy, Gabor banks)
│   │   ├── 1.1.4 Texture & edges             (edge density, GLCM, LBP, Laplacian sharpness/focus)
│   │   ├── 1.1.5 Complexity & clutter        (Shannon entropy, Feature-Congestion, Subband-Entropy)
│   │   └── 1.1.6 Holistic spatial envelope   (GIST descriptor — bridges to mid-level)
│   │
│   ├── 1.2 Low-level dynamic (per-frame-pair motion signal)
│   │   ├── 1.2.1 Optical flow                (dense u,v field; mean/median magnitude, orientation histogram, divergence/curl)
│   │   ├── 1.2.2 Camera vs object motion     (global-motion estimate + residual object-motion magnitude)
│   │   ├── 1.2.3 Motion energy               (pymoten spatiotemporal Gabor-channel energies for fMRI encoding)
│   │   └── 1.2.4 Shot/cut boundaries & visual change  (cut probabilities, scene segments, frame-to-frame novelty score)
│   │
│   ├── 1.3 Mid-level perceptual attention & geometry
│   │   ├── 1.3.1 Visual saliency             (static & spatiotemporal fixation maps; concentration, entropy, peak, salient-area)
│   │   ├── 1.3.2 Monocular depth             (relative/metric depth maps; mean/range depth, foreground fraction, layering)
│   │   └── 1.3.3 Image aesthetics & quality  (aesthetic score, technical IQA, memorability)
│   │
│   ├── 1.4 High-level static semantics (what is depicted, per frame)
│   │   ├── 1.4.1 Vision-language embeddings & probes  (CLIP/SigLIP image embeddings; open-vocab object/scene/attribute scores)
│   │   ├── 1.4.2 Self-supervised dense embeddings     (DINOv2/v3 CLS + patch features for RSA / linear probes)
│   │   ├── 1.4.3 Objects (localized)          (open-vocab detection boxes, labels, counts, positions)
│   │   ├── 1.4.4 Scenes/places/attributes     (scene-category posteriors, indoor/outdoor, scene attributes)
│   │   ├── 1.4.5 Scene composition (panoptic) (per-class area fractions, instance masks/counts, segmentation tracks)
│   │   └── 1.4.6 Frame description (VLM)       (free-text captions + structured VQA fields per keyframe)
│   │
│   ├── 1.5 High-level dynamic semantics (what happens, over time)
│   │   ├── 1.5.1 Action recognition          (Kinetics/SSv2 posteriors + pooled spatiotemporal embeddings, sliding window)
│   │   ├── 1.5.2 Zero-shot open-vocab actions (X-CLIP/InternVideo2 similarity to custom action phrases)
│   │   ├── 1.5.3 Temporal action localization (start–end action segments with confidence)
│   │   └── 1.5.4 Dense event description (video-LLM)  (timestamped open-vocabulary event captions / temporal grounding)
│   │
│   └── 1.6 Person-centric visual (faces, bodies, gaze, expression)
│       ├── 1.6.1 Face detection & identity    (bboxes, landmarks, ArcFace identity embeddings, age/sex)
│       ├── 1.6.2 Head pose & gaze             (yaw/pitch/roll, gaze vectors, gaze targets)
│       ├── 1.6.3 Facial expression            (FACS action units, discrete emotion, valence/arousal — depicted)
│       └── 1.6.4 Body & hand pose             (2D/3D/whole-body keypoints → posture, gesture, orientation)
│
├── 2. AUDIO  — features derived from the soundtrack/waveform
│   │
│   ├── 2.1 Low-level acoustic (per-frame signal descriptors)
│   │   ├── 2.1.1 Loudness & energy           (RMS, loudness in sones, intensity dB, zero-crossing rate)
│   │   ├── 2.1.2 Spectral descriptors        (centroid, rolloff, flux, flatness, bandwidth, contrast, MFCC, mel-spectrogram)
│   │   ├── 2.1.3 Pitch / F0                   (F0 contour, voiced flags, salience — CREPE/pYIN/Praat)
│   │   ├── 2.1.4 Harmonic & psychoacoustic    (harmonicity/HNR, inharmonicity, roughness/dissonance, sharpness, tonality)
│   │   └── 2.1.5 Voice quality                (jitter, shimmer, formants F1–F4, spectral tilt — segment-level)
│   │
│   ├── 2.2 Mid-level musical structure
│   │   ├── 2.2.1 Tonal/harmonic content      (chroma, tonnetz, key/mode + strength, chords)
│   │   └── 2.2.2 Rhythm & meter              (onsets, tempo/BPM, beat & downbeat times, time signature)
│   │
│   └── 2.3 High-level audio semantics
│       ├── 2.3.1 Sound events / tagging       (527-class AudioSet posteriors, framewise SED, embeddings)
│       ├── 2.3.2 Acoustic scene               (scene classification, ambience)
│       ├── 2.3.3 Speech/music/noise segmentation  (time-stamped speech/music/noise; VAD)
│       ├── 2.3.4 Open-vocab audio probes (CLAP)   (per-prompt audio-text similarity time series)
│       └── 2.3.5 Music semantics (MIR)        (genre, mood/theme, instrument tags, danceability, music valence/arousal)
│
├── 3. LANGUAGE  — features derived from the (time-aligned) transcript
│   │
│   ├── 3.1 Speech-to-language interface (paralinguistic bridge from audio)
│   │   ├── 3.1.1 ASR transcript & timing      (word/segment text + timestamps, confidence, speaking rate)
│   │   ├── 3.1.2 Speaker diarization & turns   (who-spoke-when, speaker count, overlap, turn transitions)
│   │   ├── 3.1.3 Prosody contours             (per-second F0/intensity/HNR resampled from acoustic layer)
│   │   └── 3.1.4 Vocal affect                 (dimensional A/V/D + categorical emotion from voice — depicted)
│   │
│   ├── 3.2 Low-level lexical (per-word psycholinguistic features)
│   │   ├── 3.2.1 Token annotations            (lemma, POS, NER, stop/punct flags)
│   │   ├── 3.2.2 Frequency & length           (Zipf/log frequency, contextual diversity, AoA, word length)
│   │   ├── 3.2.3 Semantic norms               (concreteness, imageability, valence/arousal/dominance per word)
│   │   ├── 3.2.4 Emotion lexicons             (NRC EmoLex 8-emotion flags, LIWC affect categories)
│   │   └── 3.2.5 Surprisal & entropy          (per-word LLM surprisal in bits, next-word entropy)
│   │
│   ├── 3.3 Mid-level syntactic structure (per-utterance)
│   │   ├── 3.3.1 Dependency & morphology      (dep relations+heads, UD morph feats: tense/aspect/mood)
│   │   ├── 3.3.2 Constituency trees           (PTB phrase-structure trees, nesting depth)
│   │   ├── 3.3.3 Syntactic complexity         (tree depth, Yngve/Frazier, L2SCA clause/T-unit indices, readability)
│   │   ├── 3.3.4 Coreference                  (mention clusters; who/what each pronoun refers to)
│   │   └── 3.3.5 Dialogue acts                (per-utterance speech-act categories)
│   │
│   └── 3.4 High-level semantics, discourse & narrative
│       ├── 3.4.1 Passage embeddings           (dense sentence/window embeddings — the semantic backbone)
│       ├── 3.4.2 Contextual LLM hidden states (per-word last-layer states — fMRI encoding-model standard)
│       ├── 3.4.3 Coherence/drift/novelty      (sliding-window cosine coherence, drift, segmentation troughs)
│       ├── 3.4.4 Topic structure              (topic assignment + topics-over-time prevalence)
│       ├── 3.4.5 Discourse relations          (RST/EDU segmentation, relation labels, nuclearity)
│       └── 3.4.6 Narrative structure (LLM)    (narrative stage, turning points, structured tags, summaries)
│
├── 4. SOCIAL  — depicted interpersonal content (cross-modal: vision + audio + language)
│   │
│   ├── 4.1 Social primitives (deterministic, per-second tracks)
│   │   ├── 4.1.1 Agent presence & identity     (character count, identity tracks, co-presence matrix)
│   │   ├── 4.1.2 Active speaker / addressee     (who-talks-on-screen, speaker↔face binding, listener roles)
│   │   ├── 4.1.3 Gaze & joint attention         (mutual gaze, shared-target / joint-attention signals)
│   │   └── 4.1.4 Proximity, orientation & touch  (pairwise distance, facing toward/away, contact, approach/retreat)
│   │
│   └── 4.2 Social semantics (VLM/LLM-inferred, per-shot/scene)
│       ├── 4.2.1 Interaction type             (cooperation/conflict, dialogue, joint action categories)
│       ├── 4.2.2 Dominance & affiliation       (relational ratings, social-network edges)
│       └── 4.2.3 Theory-of-mind & intention    (mental-state, goal, and intention inferences)
│
├── 5. SITUATION  — setting, schemas, scripts & event structure (cross-modal)
│   │
│   ├── 5.1 Event segmentation                 (data-driven boundaries on feature timelines: HMM/GSBS; + shot priors)
│   ├── 5.2 Setting & location                 (scene type, indoor/outdoor, time-of-day, spatial layout)
│   ├── 5.3 Scripts & schemas                  (recognized situational scripts / schema labels)
│   └── 5.4 Event-indexing dimensions          (space, time, causation, intention, protagonist per event/segment)
│
└── 6. AFFECT  — emotion & affect (cross-modal; depicted vs elicited kept distinct)
    │
    ├── 6.1 Depicted affect (expressed by characters, by source modality)
    │   ├── 6.1.1 Facial affect                (face valence/arousal + categorical — from 1.6.3)
    │   ├── 6.1.2 Vocal affect                 (voice A/V/D + categorical — from 3.1.4)
    │   ├── 6.1.3 Textual/dialogue affect       (utterance emotion + VAD mapping — from 3.2.3/3.4)
    │   └── 6.1.4 Music/soundtrack affect       (soundtrack valence/arousal, mood — from 2.3.5)
    │
    ├── 6.2 Multimodal fused affect            (MLLM window-level categorical + V/A/D + reasoned justification)
    │
    └── 6.3 Elicited (viewer) affect           (induced V/A timeseries, emotional-impact/fear flags — SEPARATE stream)

Node-by-node rationale (design notes)

Top-level split (1–3 modality, 4–6 cross-modal). Classes 1–3 are pure signal-source modalities: every feature traces to pixels, the waveform, or the transcript. Classes 4–6 are representational targets (social content, situational structure, affect) that are intrinsically multimodal and reuse outputs from 1–3 — so they are organized by what they represent, not by signal source. This keeps each leaf feature in exactly one home while letting cross-modal classes reference contributing leaves (the “from X.Y.Z” pointers).

Level ordering within each modality. Every modality is internally ordered low-level signal → mid-level structure → high-level semantics, and within visual/audio the static-vs-dynamic distinction is preserved as a parallel axis (1.1/1.4 static; 1.2/1.5 dynamic). This is the scoping-review backbone: a reviewer can walk any modality from “what the sensor measures” to “what it means.”

Language as a bridge modality. Subclass 3.1 (Speech-to-language interface) is the deliberate seam between Audio and Language: ASR, diarization, prosody, and vocal affect are computed from the waveform but produce the time-aligned linguistic substrate, so they live at the head of Language. Acoustic prosody itself stays in 2.1 (signal) and is referenced, not duplicated, by 3.1.3.

Depicted vs elicited affect. Class 6 explicitly separates depicted/expressed affect (6.1, organized by contributing modality) from viewer-elicited/induced affect (6.3) because they have different ground truth and must be logged as distinct feature streams — a recurring caution in the affect catalog.

Implications for the output data format

This tree maps directly onto a hierarchical schema. Each leaf is a feature group with a stable dotted path (e.g. visual.lowlevel_static.color, language.semantics.coherence_drift, affect.depicted.vocal) and a uniform record envelope:

• path (semantic address), modality, level (low/mid/high), temporal_unit (frame / window / utterance / event / shot), onset, duration, value(s), plus provenance (extractor, model, version, recommendation_tier).
• Cross-modal classes (4–6) store references to the contributing leaf paths rather than copying values, encoding the “from X.Y.Z” pointers as first-class lineage.
• The level and static/dynamic axes become filterable facets — the same metadata that structures the scoping review structures query and resampling (everything resamples to a common per-second / per-TR movie clock).

Paths worth flagging for the format spec: temporal_unit and a depicted_vs_elicited flag (on class 6) are load-bearing — they are the two fields most likely to be conflated downstream and should be required, not optional.

Visual features

Low-level static visual properties

Per-frame low-level image statistics (luminance, RMS contrast, color in RGB/HSV/CIELAB, colorfulness, spatial-frequency power spectra, edge/orientation energy, entropy, clutter) are a mature, well-understood domain that is best served by classic computer-vision libraries and a handful of vision-science toolboxes rather than learned models. The recommended core is scikit-image + OpenCV + NumPy for the bulk of pixel/color/edge/frequency statistics, a steerable-pyramid library (pyrtools / plenoptic) for biologically motivated orientation-energy and spatial-frequency banks, SHINE for canonical luminance/contrast/Fourier-amplitude measures aligned with the vision-science literature, and the Rosenholtz Feature-Congestion/Subband-Entropy code for visual clutter. GIST provides a compact holistic spatial-envelope descriptor that bridges low- and mid-level. Learned descriptors add little over these analytic features for this specific subclass; the main “modern” gain is wrapping the classic operators in a per-frame pipeline (e.g. pliers) and running them at 1-25 fps over a video. All of these run on CPU and are trivially feasible for a small lab; the practical work is timeline alignment and per-frame batching, not compute.

Recommended (best-in-class): scikit-image + OpenCV + NumPy as the core per-frame engine (luminance, RMS contrast, RGB/HSV/CIELAB color, entropy, edges, FFT power spectrum); pyrtools / plenoptic steerable pyramid for orientation-energy and spatial-frequency banks; SHINE for canonical, literature-matched luminance/contrast/Fourier-amplitude-spectrum definitions; Rosenholtz Feature-Congestion + Subband-Entropy for visual clutter; Hasler-Susstrunk colorfulness formula (implemented directly) for perceptual color; pliers as the timestamping/orchestration layer feeding the movie timeline

Likely redundant:

• OpenCV Gabor-kernel bank is superseded by pyrtools/GIST for principled orientation-energy (keep OpenCV mainly for speed and video decoding)
• Global Shannon entropy and raw edge density are weaker clutter proxies superseded by Rosenholtz Feature-Congestion / Subband-Entropy
• GIST overlaps heavily with pyrtools orientation-energy; only one is needed unless a compact fixed-length holistic vector is specifically wanted
• Learned low-level descriptors / CNN early-layer activations add little over analytic luminance/contrast/color/frequency features for this subclass and are better reserved for mid/high-level visual subclasses
• pliers built-in extractors duplicate scikit-image/OpenCV computations; it is an orchestration convenience, not an independent feature source
• scikit-image and OpenCV themselves overlap substantially in color/edge/FFT ops

High-level static visual: objects, scenes, places, attributes

For per-frame movie/audio-story annotation of objects, scenes, places, and attributes, the field has consolidated around a few foundation-model families. Two complementary backbones cover most needs: contrastive vision-language encoders (OpenCLIP, SigLIP 2) that give open-vocabulary probe scores and zero-shot scene/place/object/attribute labels via text prompts, and self-supervised encoders (DINOv2/DINOv3) that give label-free dense semantic embeddings. For explicit localized labels, open-vocabulary detectors (Grounding DINO 1.5/1.6, YOLO-World) and promptable/panoptic segmenters (SAM 2, Mask2Former) emit per-object boxes/masks. For rich natural-language scene descriptions and attribute/relation reasoning, open VLMs (Qwen2.5-VL, InternVL3, LLaVA-OneVision) caption frames or answer structured VQA. A small lab can run most of these locally on a single 24-48GB GPU; the main tradeoff is fixed-vocabulary classifiers (Places365, ImageNet backbones) being largely superseded by open-vocabulary CLIP-family probing, except where a frozen, reproducible, interpretable category vector is explicitly wanted.

Recommended (best-in-class): SigLIP 2 (or OpenCLIP ViT-L) as the primary open-vocabulary text-probe encoder for object/scene/place/attribute scores; DINOv2/DINOv3 as the complementary label-free dense embedding for RSA and linear probes; Qwen2.5-VL-7B as the primary captioning/VQA VLM for structured scene descriptions; Grounding DINO (1.6 Pro via API for accuracy, or open 1.0 / YOLO-World locally) for open-vocabulary per-object detection; Mask2Former/OneFormer panoptic for scene-composition (thing+stuff area fractions)

Likely redundant:

• OpenCLIP and SigLIP 2 fill the same text-probe role - run one as primary (SigLIP 2 generally stronger), not both
• Qwen2.5-VL, InternVL3, and LLaVA-OneVision are interchangeable captioning/VQA VLMs - pick one primary after benchmarking your prompt schema
• YOLO-World is superseded on accuracy by Grounding DINO 1.6 Pro; keep only when per-frame throughput/density is the constraint
• Places365 CNN and ImageNet-trained backbones are largely superseded by zero-shot CLIP/SigLIP2 probing of the same label sets - keep only for a frozen, reproducible fixed-taxonomy probability vector
• SAM 2 alone produces no labels and overlaps Mask2Former for static panoptic needs; only adds value via video-consistent tracking or as Grounded-SAM

Open questions:

• Frame sampling rate to hit true second-by-second alignment vs compute budget (1 fps probing vs shot-level keyframes for heavy VLMs)
• Whether to derive scene/place labels via zero-shot CLIP/SigLIP2 prompting or keep a frozen Places365 head for reproducibility/citation in fMRI work
• DINOv3 licensing (research/non-commercial terms) may restrict use vs Apache-2.0 DINOv2
• Grounding DINO 1.6 Pro best accuracy is API-only - acceptable for a small lab, or fall back to open 1.0/YOLO-World?
• Need for an aggregation/parsing layer to convert free-text VLM captions into consistent categorical per-second feature vectors (schema + JSON-constrained decoding)

Faces, bodies, gaze, and facial expression

This subclass covers per-frame extraction of where faces and bodies are, who they are, where they look, and what they express. The mature open-source stack is: a detector/tracker (RetinaFace/InsightFace, MediaPipe) feeding identity (ArcFace), landmarks, head pose and gaze (OpenFace 3.0, L2CS-Net), facial action units and discrete/continuous emotion (Py-Feat, OpenFace 3.0, HSEmotion/EmotiEffLib, EmoNet), and body/hand keypoints (MMPose’s RTMPose/RTMW, ViTPose, MediaPipe Holistic). As of 2025, OpenFace 3.0 (arXiv 2506.02891) consolidates landmarks+AUs+gaze+emotion into one lightweight CPU-capable multitask model, and MMPose RTMPose/RTMW dominate real-time 2D/whole-body pose; these are the recommended defaults for a small lab. All major components run on a single consumer GPU (often CPU), making second-by-second annotation of long films feasible.

Recommended (best-in-class): OpenFace 3.0 (landmarks + AUs + gaze + emotion in one CPU-capable multitask model) as the primary face-behavior extractor; InsightFace buffalo_l (RetinaFace + ArcFace) for face detection, tracking, and identity embeddings; MMPose RTMPose / RTMW for real-time 2D and whole-body (body+face+hands+feet) keypoints; MediaPipe Tasks for cheap CPU-only body/hand/face landmarks + blendshapes as a fallback/redundancy layer; HSEmotion/EmotiEffLib for fast continuous valence-arousal and expression, and L2CS-Net when high-accuracy 360-degree gaze is a primary signal

Likely redundant:

• OpenFace 2.0 -> superseded by OpenFace 3.0 (faster, multitask, better accuracy); keep only for reproducing legacy pipelines or its validated AU intensity scale
• Standalone RetinaFace -> redundant with InsightFace, which bundles it; and with detectors inside OpenFace 3.0 / Py-Feat
• ViTPose -> redundant with MMPose RTMPose/RTMW for typical lab use; only worth it for max-accuracy keypoints on a strong GPU
• EmoNet -> valence-arousal largely superseded by HSEmotion (faster, newer, more outputs); keep as a citable cross-check
• L2CS-Net gaze and Py-Feat/HSEmotion emotion overlap OpenFace 3.0’s bundled gaze/emotion -> run separately only when those specific signals need best-in-class accuracy
• MediaPipe body/hand keypoints overlap MMPose (lower accuracy, single-best-effort) -> use MediaPipe for cheap CPU-only or redundancy, MMPose for accuracy/multi-person

Dynamic visual: motion, optical flow, motion energy

This subclass covers low-level dynamic visual signal: dense optical flow (per-pixel motion vectors), aggregate motion energy from spatiotemporal filter banks, camera-vs-object motion separation, shot/cut boundaries, and frame-to-frame visual change/novelty. These are deliberately pre-semantic features (how much/which-direction things move, when the camera cuts) distinct from semantic action recognition. The field is mature and largely open-source: RAFT and its 2024 successor SEA-RAFT dominate dense flow, MemFlow/VideoFlow add multi-frame temporal smoothing, pymoten provides the canonical neuroscience motion-energy pyramid, and TransNetV2 plus PySceneDetect cover shot detection. For a small lab, a practical default pipeline is: PySceneDetect/TransNetV2 for cuts, pymoten for a brain-aligned motion-energy regressor, and SEA-RAFT (or RAFT) for dense flow from which you derive global/mean flow magnitude (proxy for camera motion) and residual flow (object motion).

Recommended (best-in-class): SEA-RAFT (dense optical flow; derive global camera-motion vs residual object-motion and mean flow magnitude); pymoten (neuroscience-standard motion-energy pyramid feature for direct fMRI encoding); TransNetV2 (shot/cut boundary detection incl. gradual transitions); PySceneDetect (fast CPU cut detection + continuous per-frame visual-change/novelty score)

Likely redundant:

• RAFT -> superseded by SEA-RAFT (better accuracy and ~2.3x faster); keep only as easy torchvision fallback
• VideoFlow -> superseded by MemFlow (faster, fewer params, higher accuracy for multi-frame flow)
• AutoShot -> overlaps TransNetV2; only preferable for short-form/social video stimuli
• OpenCV Farneback/DIS -> superseded by SEA-RAFT/RAFT in accuracy; keep only as no-GPU fallback
• GMFlow/FlowFormer family -> covered in spirit by SEA-RAFT as the chosen single dense-flow model; no need to run multiple deep flow nets

Open questions:

• Camera-vs-object motion separation is not a turnkey tool: it must be derived from a dense-flow model (e.g. robust homography/affine global-motion fit on the flow field, then residual = object motion). Decide whether to implement this or adopt a SLAM/egomotion package.
• Choose the temporal aggregation strategy: flow and motion-energy are per-frame, so define how to bin/downsample to per-second or per-TR (mean, RMS, or HRF-convolved) for the neuroimaging timeline.
• For long narrative films, decide between per-pair flow (SEA-RAFT) vs temporally-smoothed multi-frame flow (MemFlow) depending on whether jitter in the second-by-second series is a problem.
• Confirm SEA-RAFT vs MemFlow generalization on your actual stimulus domain (animation, archival film, low light) since flow models are trained mostly on Sintel/KITTI/Spring.

Action and activity recognition (video understanding)

Computational action/activity recognition for a movie/audio-story timeline splits into three usable layers: (1) clip-level recognition backbones that emit Kinetics/Something-Something class posteriors and/or pooled spatiotemporal embeddings over short sliding windows (VideoMAE V2, InternVideo2, V-JEPA 2, TimeSformer, VideoSwin, SlowFast), best run at a fixed window stride (e.g. 16 frames / 1-2 s) to produce per-second features; (2) temporal action localization (TAL) heads (ActionFormer / OpenTAD, typically fed InternVideo2 or VideoMAE features) that turn those per-window embeddings into start-end action segments with confidence; and (3) video-LLMs (Qwen3-VL, VideoLLaMA3, InternVL3, Video-LLaVA) that produce open-vocabulary dense event descriptions with timestamps, valuable for naturalistic film where a fixed label set is too restrictive. For a small lab the most leverage comes from: VideoMAE V2 / InternVideo2 as frozen feature extractors on a sliding window, X-CLIP for zero-shot open-vocabulary action probabilities, and one video-LLM for dense captioning. The pure clip classifiers (TimeSformer/VideoSwin/SlowFast) are now largely superseded as standalone choices but remain valuable as the standard, well-documented MMAction2/PySlowFast backbones and as TAL feature sources.

Recommended (best-in-class): VideoMAE V2 (ViT-L/g) or InternVideo2-1B as the frozen sliding-window feature extractor + Kinetics classifier; X-CLIP for lightweight zero-shot open-vocabulary action probabilities over a custom film vocabulary; ActionFormer/OpenTAD over InternVideo2/VideoMAE features for timestamped action segments; Qwen3-VL (8B) as the primary video-LLM for open-vocabulary dense, timestamped event description

Likely redundant:

• TimeSformer / VideoSwin / SlowFast as standalone recognizers -> superseded by VideoMAE V2 and InternVideo2 (keep only for the convenient MMAction2/PySlowFast pipeline or as legacy TAL features)
• InternVL3 (video) and VideoLLaMA3 -> overlap with Qwen3-VL for dense captioning/temporal grounding; pick one video-LLM as primary (Qwen3-VL has the strongest native timestamp grounding)
• X-CLIP zero-shot vs InternVideo2 zero-shot -> same open-vocabulary role; InternVideo2 is more accurate but far heavier, X-CLIP is the cheap default
• V-JEPA 2 vs VideoMAE V2 -> both self-supervised backbones; V-JEPA 2 is the more motion/temporal-biased complement, not a strict duplicate

Open questions:

• Licensing: VideoMAE V2 and InternVideo2 weights are non-commercial/research only; confirm this is acceptable for the project’s use and publication plans.
• Window stride and temporal resolution: clip backbones natively output per-16-frame (~0.5-1 s) features; the target second-by-second cadence requires choosing stride/overlap and a pooling rule that needs to be standardized across features.
• Label taxonomy: Kinetics-400/700 and Something-Something classes are everyday/atomic actions and may map poorly onto narrative-film events; decide whether to rely on TAL with a custom taxonomy vs open-vocabulary X-CLIP/video-LLM outputs.
• Video-LLM timestamp reliability: Qwen3-VL/VideoLLaMA3 emit timestamps but their second-level accuracy on long naturalistic films is not well validated; may need calibration against TAL or manual spot-checks.
• Compute budget: InternVideo2-6B and Qwen3-VL-30B+ exceed a single-GPU small-lab setup; confirm whether 1B/8B-class models suffice for the required annotation quality.

Visual saliency, attention, aesthetics, depth

This subclass produces per-frame attention/complexity proxies: predicted human fixation/saliency maps (static and video), monocular depth maps, and scalar image-level scores for aesthetics, technical quality, and memorability. For a movie/audio-story timeline, the most timeline-relevant outputs are spatiotemporal video-saliency maps (one map per frame, summarizable to scalars like saliency concentration/entropy and number-of-salient-regions) and depth maps (mean/range/foreground-fraction per frame). The field is mature and overwhelmingly open-source: DeepGaze IIE/III for static fixation, ViNet/TASED-Net/UNISAL for video saliency, Depth-Anything-V2 (and Apple Depth Pro for metric) for depth, and CLIP/LMM-based scorers (Q-Align, LAION-Aesthetics, ResMem) for aesthetics/quality/memorability. All run on a single consumer GPU; most also run on CPU slowly. Recommend one video-saliency model + one depth model + one aesthetics/quality scorer + memorability as the core, with static-image saliency only as an alternative.

Recommended (best-in-class): ViNet (or ViNet-A for audio-visual) for per-frame video saliency maps -> saliency concentration/entropy, peak, salient-area, temporal shift; Depth-Anything-V2 for per-frame relative depth (add Apple Depth Pro only if metric/meters depth is needed) -> mean/range depth, foreground-fraction, depth complexity; Q-Align/OneAlign for combined per-frame aesthetic + technical-quality scalars (subsample to ~1 fps), with IQA-PyTorch (MUSIQ/NIMA/LAION-AES) as a lightweight fallback; ResMem for per-frame image-memorability scalar

Likely redundant:

• MiDaS / DPT / ZoeDepth — superseded by Depth-Anything-V2 (relative) and Apple Depth Pro (metric)
• TASED-Net and UNISAL — same video-saliency output as ViNet; pick one (ViNet best accuracy, UNISAL lightest)
• DeepGaze IIE/III (static fixation) — for video timelines superseded by motion-aware video-saliency models; keep only as a pure static-image attention proxy
• LAION-Aesthetics V2 and CLIP-IQA/QualiCLIP — subsumed by Q-Align’s IAA/IQA branches (use them only as lightweight substitutes when the 7B LMM is too heavy)
• MemNet / ResMem-Net — older or less-standardized memorability predictors superseded by the maintained, pip-installable ResMem

Audio features

Low-level acoustic features

Per-frame low-level acoustic descriptors (loudness/RMS, F0/pitch, spectral centroid/rolloff/flux, MFCCs, chroma, onsets, tempo/beat, harmonicity, roughness, voice-quality jitter/shimmer/HNR) are produced by a mature, mostly open-source ecosystem. The core trio is librosa (most flexible general MIR/DSP in pure Python), openSMILE (the standardized affective-computing feature sets eGeMAPS/ComParE used heavily in speech/emotion neuroscience), and Praat via Parselmouth (the reference for voice-quality and F0). Essentia adds a fast C++ engine with a richer set of music/psychoacoustic descriptors (dissonance/roughness, beat tracking) and torchaudio provides GPU-batched, differentiable versions of standard transforms. For pitch specifically, neural CREPE and probabilistic pYIN exceed classic autocorrelation/cepstral trackers in accuracy. All are CPU-friendly and trivially deployable in a small lab; only CREPE benefits from (but does not require) a GPU. For a movie/audio-timeline pipeline, a small overlapping set (librosa + openSMILE eGeMAPS + Parselmouth, optionally Essentia for music-rich roughness/beat and CREPE for high-accuracy F0) covers the entire subclass with minimal redundancy.

Recommended (best-in-class): librosa (primary general spectral/MFCC/chroma/onset/tempo + pyin F0); openSMILE eGeMAPS/ComParE via opensmile-python (standardized speech/affect LLDs, timeline-indexed); Parselmouth/Praat (gold-standard F0, formants, jitter/shimmer/HNR voice quality); Essentia (add for music-rich roughness/dissonance, inharmonicity, robust beat tracking); CREPE/torchcrepe (add for high-accuracy monophonic F0 on speech/vocals)

Likely redundant:

• torchaudio — spectral/MFCC fully superseded by librosa for a small lab; keep only if GPU-batched or differentiable front-ends are needed
• torchaudio.detect_pitch_frequency and librosa.pyin F0 — superseded by CREPE/torchcrepe (accuracy) or Praat (voice science) for the F0 feature
• openSMILE jitter/shimmer/HNR/formant LLDs overlap Parselmouth — run both only for cross-validation; otherwise Parselmouth is canonical and openSMILE provides the standardized 88-dim summary
• Essentia spectral/MFCC/chroma/onset overlap librosa — choose one as the primary MIR engine (librosa for ease, Essentia for speed/roughness), not both for the same descriptors
• Essentia ‘dissonance’ vs mosqito roughness — overlapping roughness measures; mosqito is the standards-based (asper) choice if perceptual fidelity matters

Open questions:

• Will the pipeline analyze the full mixed soundtrack (speech+music+SFX) or source-separated stems? Monophonic F0 tools (CREPE, Praat) and voice-quality measures degrade badly on mixes — may need Demucs/source separation upstream.
• Target output rate: true per-frame (~10-23 ms) vs aggregated per-second for fMRI/timeline alignment? This sets hop sizes and whether to keep LLDs or functionals.
• Is openSMILE’s source-available (non-OSI) license and Essentia’s AGPL acceptable for the project’s distribution/sharing model?
• Are standardized perceptual units (sones/asper for loudness/roughness via mosqito) required, or are relative librosa/Essentia descriptors sufficient?
• Sample rate harmonization: openSMILE/Praat expect specific rates; movie audio is often 48 kHz — confirm resampling strategy to keep features comparable across tools.

High-level audio: events, scenes, music, speech/non-speech

This subclass covers time-resolved semantic audio annotation: (1) audio tagging / sound event detection over the 527-class AudioSet ontology, (2) acoustic scene classification, (3) speech vs music vs noise segmentation, and (4) music information retrieval (tempo, key/mode, beat/downbeat, instrumentation, mood/emotion). The field is mature and largely open-source. For a movie/audio-story timeline, the recommended stack is a transformer audio tagger (BEATs or PaSST/AST) run on a sliding window for AudioSet event posteriors, CLAP for open-vocabulary/zero-shot event labels and audio-text similarity, inaSpeechSegmenter or pyannote VAD for speech/music/noise segmentation, and Essentia + madmom (or Beat This!) for MIR descriptors. Most tools run comfortably on a single consumer GPU or even CPU; nothing here requires heavy multi-GPU compute. Key design choice: window size sets temporal granularity (AudioSet models are trained on ~10 s clips but can be run on ~1 s hops for per-second annotation, with the caveat that very short windows degrade accuracy on slow-onset events).

Recommended (best-in-class): BEATs (or PaSST) for AudioSet event/scene tagging on a sliding window; CLAP (laion/larger_clap_general + music_and_speech) for open-vocabulary zero-shot audio dimensions and audio-text similarity time series; inaSpeechSegmenter for clean speech/music/noise segmentation (add pyannote VAD/diarization if speaker turns needed); Essentia + TF models for MIR scalars and music mood/genre/instrument tags; madmom or Beat This! for beat/downbeat/tempo structure

Likely redundant:

• AST is largely superseded by BEATs/PaSST on AudioSet mAP (keep only for its trivial HF deployment)
• PANNs CNN14 superseded by transformer taggers, but retain for framewise SED and panns_inference simplicity
• YAMNet superseded on accuracy by all transformer taggers; keep only for CPU/real-time/edge constraints
• Choose ONE primary AudioSet tagger (BEATs or PaSST) rather than running AST+PANNs+YAMNet+BEATs together, since their 527-class outputs are highly correlated
• Essentia tempo estimator is redundant with madmom/Beat This! beat tracking (prefer the latter when precise beat/downbeat timing matters)
• CLAP music tags partially overlap Essentia genre/mood models; use CLAP for novel/custom prompts and Essentia for calibrated valence-arousal/tempo/key

Speech: ASR, diarization, prosody, vocal affect

This subclass converts the speech audio of movies/stories into a time-aligned linguistic and paralinguistic layer: word/segment-level transcripts with timestamps (ASR + forced alignment), who-spoke-when (diarization + VAD), low-level prosody (pitch/intensity/voice-quality contours), and emotion-from-voice (categorical and dimensional A/V/D). The open ecosystem is mature and almost entirely locally deployable on a single consumer GPU. A practical pipeline is faster-whisper/WhisperX for ASR+word timestamps, pyannote (community-1) for diarization, Parselmouth/openSMILE for prosody, and an audEERING wav2vec2-dim or emotion2vec model for vocal affect; most other tools are redundant variants of these. Key design choice is per-word vs per-second resampling, since ASR/SER are utterance-level while prosody is natively frame-level (10 ms).

Recommended (best-in-class): faster-whisper (large-v3) for ASR; WhisperX for word-level forced alignment + speaker-attributed transcript; pyannote speaker-diarization-community-1 for diarization/VAD/overlap; Parselmouth (Praat) for prosody and voice-quality contours; audEERING wav2vec2-large-robust-12-ft-emotion-msp-dim for dimensional vocal affect (arousal/valence/dominance)

Likely redundant:

• openai/whisper, insanely-fast-whisper, whisper.cpp -> superseded by faster-whisper (same weights, faster CTranslate2 runtime)
• pyannote speaker-diarization-3.1 -> superseded by community-1 (better speaker counting, same segmentation)
• WebRTC VAD / audiotok -> superseded by Silero VAD and pyannote VAD
• librosa pitch (pyin/piptrack) and standalone MFCC extraction -> superseded by Parselmouth (F0/voice-quality) + openSMILE (spectral LLDs)
• Generic HF wav2vec2/HuBERT IEMOCAP categorical SER checkpoints -> superseded by emotion2vec+ (categorical) and audEERING wav2vec2-dim (dimensional)
• NeMo ASR/diarization and emotion2vec+ -> overlap with the Whisper/pyannote/audEERING defaults; keep only as accuracy or multilingual/cross-domain alternatives

Open questions:

• Target temporal grid: per-word (event-locked) vs fixed per-second resampling - ASR/SER are utterance-level and need windowing/interpolation, whereas Parselmouth/openSMILE are natively 10 ms.
• Movie audio is mixed with music/SFX/overlapping speakers; do we need source separation (e.g. Demucs/Bandit) or a music-VAD gate before ASR/SER to avoid contaminating pitch/jitter/shimmer and emotion estimates?
• Vocal affect should be computed per diarized speaker segment, not on mixed audio - confirm diarization-first ordering and how to handle overlapped speech for SER/prosody.
• Licensing: openSMILE eGeMAPS and audEERING wav2vec2-dim are research-only; confirm acceptable for the study or substitute emotion2vec+/Parselmouth-only features.
• Whether to add a paralinguistics/laughter-cry/non-verbal-vocalization detector (e.g. emotion2vec vocal events or a dedicated laughter model) as a separate feature stream.

Language features

Low-level lexical & word-level features

Per-word lexical/psycholinguistic annotation is a mature, mostly CPU-friendly pipeline problem rather than a deep-learning frontier. The backbone is a fast tokenizer/POS/NER/dependency engine (spaCy) that segments the transcript and tags each word, followed by table lookups against well-established norm databases (SUBTLEX/wordfreq for frequency; Brysbaert for concreteness; Kuperman for AoA; Warriner/NRC-VAD for valence/arousal/dominance; NRC-EmoLex/LIWC for affect categories). The one genuinely model-dependent feature is word-level surprisal/entropy, computed from an incremental LLM (GPT-2 through larger autoregressive models) via the minicons or surprisal packages. For time-alignment, all of these are per-token features that inherit timestamps from forced alignment of the transcript to the audio (e.g. WhisperX/Montreal Forced Aligner) — that alignment step is upstream and out of scope here but is the mechanism that turns per-word features into a per-second movie timeline. The main design decisions for a small lab are: (1) which norm databases to license/merge, (2) how to handle out-of-vocabulary words and lemmatization for lookups, and (3) which LLM to use for surprisal given the size/fit tradeoff.

Recommended (best-in-class): spaCy (en_core_web_trf/lg) as the tokenizer/POS/NER/dependency backbone; wordfreq + SUBTLEX-US for frequency; Brysbaert concreteness + Kuperman AoA + Warriner/NRC-VAD norm joins for psycholinguistic per-word features; NRC EmoLex for discrete emotion channels; minicons (or surprisal) with GPT-2-medium/Pythia for word surprisal & entropy

Likely redundant:

• NLTK — superseded by spaCy for tokenization/POS/NER (keep only for WordNet synset access)
• surprisal package — direct overlap with minicons; pick one surprisal library
• ANEW (1,034 words) — superseded by Warriner VAD (13,915) and NRC-VAD (20,000)
• MRC Psycholinguistic Database — legacy imageability/concreteness, superseded by Brysbaert and Glasgow norms
• LIWC-22 affect categories — most per-word affect content is reproducible with free NRC EmoLex + Warriner VAD + spaCy POS; LIWC is justified mainly for its validated pronoun/person-reference and function-word category scheme
• Glasgow Norms — overlaps Brysbaert/Kuperman/Warriner on the same dimensions at smaller coverage; useful only as a single-source merge or for its imageability/familiarity/semantic-size/gender scales
• VADER — sentence/social-media-tuned sentiment, redundant with Warriner+NRC for per-word valence

Open questions:

• Time-alignment is upstream: per-word features are only as good as the forced alignment (WhisperX / Montreal Forced Aligner) that assigns each token a start/end timestamp — confirm which aligner the pipeline uses before binning to per-second.
• Lemmatization vs surface-form lookup: norm databases key on lemmas/base forms while surprisal and frequency can be surface-form; need a consistent OOV and morphology strategy (e.g. fall back lemma->surface->subword).
• Whether to merge Warriner and NRC-VAD (and with what priority) for valence/arousal/dominance, vs picking one to avoid double-counting correlated features.
• Choice of surprisal LM: smaller GPT-2/Pythia models often fit human reading-time/neural data better than the largest LLMs — decide whether the goal is best psycholinguistic fit (small model) or best raw language modeling (large model).
• Coverage/OOV handling for proper nouns and disfluencies in spoken-narrative transcripts, which are common in movies/audio stories but absent from norm databases.
• LLM-generated norm estimates (2024-2026 work) could fill OOV gaps for concreteness/valence/AoA, but require validation/fine-tuning against the human tables before use.

Syntactic & grammatical structure

Sentence/utterance-level syntactic annotation for movie/audio narratives rests on a mature stack: a few NLP pipelines (spaCy, Stanza, Trankit) emit dependency parses, POS/morphology (tense/aspect/mood as UD feats), and lemmas per token; dedicated constituency parsers (benepar, supar, Stanza/CoreNLP conparse) yield phrase-structure trees from which complexity metrics (tree depth, Yngve/Frazier scores, L2SCA clause/T-unit ratios) are derived; coreference (Maverick, fastcoref, Stanza-coref) links mentions across the timeline; and lightweight libraries (textstat, readability) plus DA classifiers handle readability and dialogue acts. All core tools are open-source and CPU-runnable; transformer variants want a modest GPU. The main workflow is: parse transcript utterances time-aligned to the movie/audio timeline (via forced alignment elsewhere), then compute per-utterance/per-second syntactic features. Most complexity, readability, and Yngve/Frazier metrics are derived post-hoc from parser output rather than being separate models.

Recommended (best-in-class): spaCy 3.8 (en_core_web_trf) for dependency/POS/morphology; Stanza 1.10 for UD-scheme parses + built-in constituency + coref in one tool; benepar for constituency trees feeding complexity metrics; Maverick for coreference resolution; L2SCA (+ Yngve/Frazier from trees) as the derived syntactic-complexity layer

Likely redundant:

• Trankit — superseded by spaCy/Stanza for English; keep only for multilingual or superior sentence-segmentation needs
• supar — overlaps benepar/Stanza; use only when you need joint dep+constituency or CRF marginals
• fastcoref/F-coref — superseded by Maverick on accuracy+efficiency; keep for high-throughput batch coref
• textstat — overlaps py-readability-metrics; surface-only, redundant with L2SCA’s deeper syntactic complexity
• Stanford CoreNLP constituency/coref — largely superseded by Stanza’s neural pipeline and Maverick

High-level semantics, discourse & narrative

This subclass covers meaning at the sentence/passage/scene level: dense semantic embeddings, topic structure, semantic coherence/similarity over time, discourse relations, and narrative structure (arcs, turning points, summaries, LLM-derived tags). For a movie/audio-story timeline the practical pipeline is: align a transcript (with word/utterance timestamps from ASR), window it into rolling segments (e.g. per-sentence, per-utterance, or sliding N-second windows), then emit per-segment features. The dominant, most flexible feature source in 2024-2026 is dense passage embeddings from instruction-tuned encoders (Qwen3-Embedding, E5/multilingual-e5, GTE, BGE, plus API options like OpenAI/Voyage/Gemini), from which you derive coherence (cosine similarity between adjacent windows), drift, and topic clusters. BERTopic gives interpretable topics-over-time. Instruction-tuned LLMs (Llama-3, Qwen2.5, GPT-4o-class) used as zero/few-shot feature extractors now cover the harder, more abstract signals — turning points, discourse relations, event/scene segmentation, narrative tags, and structured summaries — that no specialized supervised model does robustly. Note: in this neuroimaging context, the LAST-token/last-layer hidden states of autoregressive LLMs (GPT-2 through Llama-3) are themselves a primary per-word/per-TR feature and the current standard for naturalistic language-comprehension encoding models.

Recommended (best-in-class): Qwen3-Embedding-0.6B (lab default open encoder; scale to 4B/8B if GPU allows) as the backbone for all semantic features; Sliding-window cosine coherence/drift pipeline on top of those embeddings to produce per-second time-resolved scalars; BERTopic with topics-over-time for interpretable topic timelines (reuses the same embeddings); Autoregressive LLM hidden-state + surprisal extractor (GPT-2 / Llama-3.1-8B) for word-level / per-TR comprehension features (the neuroimaging standard); Instruction-tuned LLM (Llama-3.1-8B-Instruct / Qwen2.5-7B-Instruct local, or GPT-4o/Claude API) as a zero/few-shot extractor for narrative stage, turning points, discourse relations, and structured tags

Likely redundant:

• multilingual-e5, GTE, BGE — pick ONE primary open encoder; the others are interchangeable alternatives to Qwen3-Embedding, not separate features
• OpenAI/Voyage/Gemini embedding APIs — superseded for most needs by open encoders (Qwen3/E5); use only if a leaderboard-topping API embedding is required and off-site transcript transfer is acceptable
• Supervised RST parser (DMRST) and supervised turning-point models — largely subsumed by the instruction-tuned LLM extractor, which is more flexible on spoken transcripts; keep the supervised ones only when standardized, reproducible discourse labels are specifically required
• Standalone raw-embedding clustering — superseded by BERTopic, which wraps clustering with labeling and dynamic topics-over-time

Social features

Social & interpersonal features

Depicted social content for a movie/audio timeline decomposes into (a) low-level perceptual primitives that are reliably extractable today with dedicated open models – number/identity of agents (face detection + recognition), who-speaks-when (diarization), who-is-actively-talking (audio-visual ASD), gaze targets / joint attention, and body pose for proximity/touch – and (b) high-level social semantics (interaction type, cooperation/conflict, dominance/affiliation, theory-of-mind, intentions, joint attention construal) that have no robust dedicated detector and are best realized by prompting a strong video VLM (Qwen2.5-VL, InternVL3, or a frontier API) against schemas derived from social-reasoning benchmarks (Social-IQ 2.0, Social Genome, SIV-Bench). The recommended architecture is a two-layer pipeline: run the cheap deterministic primitive extractors per-second to build a character/speaker track and proximity/gaze graph, then feed those tracks plus sampled frames into a VLM with structured prompts to label interaction type and mental-state content per shot/scene. The dedicated social-reasoning models in the literature are mostly benchmarks, not deployable annotators, so the VLM-prompting layer is unavoidable for the genuinely “social” dimensions.

Recommended (best-in-class): pyannote-audio 3.1 (who-speaks-when); Light-ASD/LR-ASD (active speaker / who-talks-on-screen); InsightFace SCRFD+ArcFace (agent count + character identity + co-presence); Gaze-LLE (gaze target / joint attention / mutual gaze); RTMPose or ViTPose (pose for proximity/touch/orientation); Qwen2.5-VL (primary local VLM for interaction-type, dominance/affiliation, ToM/intention labeling)

Likely redundant:

• TalkNet-ASD — superseded by Light-ASD/LR-ASD (same task, ~23x more params for equal mAP)
• OpenPose — legacy, superseded by RTMPose/ViTPose for multi-person pose
• InternVL3 — redundant with Qwen2.5-VL as the primary local social-reasoning VLM; keep at most one primary + optional agreement check
• Frontier APIs (GPT-4o/Gemini/Claude) — redundant with local VLMs for routine annotation; reserve for gold-label validation/distillation
• DeSIQ / VEGAS / Human Behavior Atlas / SIV-Bench — overlapping social benchmarks, redundant with Social-IQ 2.0 + Social Genome as the chosen schema source
• Dedicated HOI/dyadic-interaction detectors (HOTR, PRTT, ViPLO) — image-centric, no robust video social-interaction-type SOTA; the VLM layer supersedes them for movie-timeline use

Situation features

Situations, schemas, scripts & event segmentation

This subclass spans two complementary tool families: (1) data-driven event/state boundary detection that segments a continuous feature timeline (neural data, or stimulus-derived embeddings such as visual/semantic features) into discrete events with stable patterns, and (2) situational tagging that labels each segment or timepoint with setting, location, time-of-day, scripts/schemas, and the Event-Indexing situational dimensions (space, time, causation, intention, protagonist). For boundary detection, BrainIAK’s HMM (Baldassano 2017) and Geerligs’ GSBS (statesegmentation) are the canonical, locally deployable algorithms; both run on any feature matrix, not just fMRI, making them directly usable to segment movie-derived embedding timelines. Shot/scene cut detection (PySceneDetect, TransNetV2) gives a fast low-level event-boundary prior. For situational content, scene-recognition CNNs (Places365), movie-tuned CLIP (MovieCLIP), and zero-shot CLIP cover location/setting/time-of-day, while modern open-weight video-LLMs (Qwen2.5-VL, VideoLLaMA3) and text/transcript LLMs (GPT-4-class via prompting) provide human-like event segmentation plus rich, queryable situational dimensions (goals, causes, protagonist, schema/script labels) with second-level temporal grounding. A small lab can run the algorithmic tools and 7B-class VLMs on a single modern GPU.

Recommended (best-in-class): GSBS / statesegmentation (primary data-driven event-boundary detector on stimulus feature timelines); BrainIAK EventSegment HMM (alternative boundary detector + soft event posteriors); TransNetV2 (best shot-boundary prior for video); Qwen2.5-VL-7B (open video-LLM for second-level event segmentation + situational dimensions); Zero-shot OpenCLIP (flexible per-frame location/time-of-day/setting/script attributes); Places365-ResNet50 (calibrated fixed-taxonomy location/setting + scene attributes); GPT-4-class / Llama-3-70B LLM (transcript-based human-like event segmentation + Event-Indexing situational tagging, esp. audio stories)

Likely redundant:

• PySceneDetect -> superseded for accuracy by TransNetV2 (keep only as a fast CPU prior)
• MovieCLIP -> overlaps zero-shot CLIP/Places365; use only if its 179 movie-scene taxonomy is a better fit and checkpoints are accessible
• VideoLLaMA3 -> redundant with Qwen2.5-VL as the situational video-LLM; pick one primary
• BrainIAK HMM vs GSBS -> overlapping boundary detectors; GSBS generally preferred (auto-K, faster), keep HMM mainly for soft event posteriors / recall-reactivation modeling
• Indexter -> superseded as a runnable tool by LLM situational tagging; retain only as the target ontology/schema definition

Open questions:

• Ground-truth/validation: which event-boundary timeline (human dense-rating norms, e.g., per-second boundary agreement) will be used to validate algorithmic vs LLM/VLM boundaries on your specific movies/audio stimuli?
• Pipeline order: run shot detection (TransNetV2) first and tag shots, vs run GSBS/HMM on a continuous embedding timeline (from CLIP/VLM/text embeddings) to get semantic events independent of camera cuts? These give different granularities.
• For audio-only stories, the visual tools do not apply; need to confirm WhisperX-style word-level alignment to project transcript LLM event boundaries and situational tags onto the audio timeline.
• Structured output schema: define a fixed JSON schema for the five Event-Indexing dimensions (space/time/causation/intention/protagonist) + script/schema label so VLM/LLM outputs are consistently parseable per second/segment.
• Compute budget: is a single 24GB GPU available for 7B video-LLM inference, or should situational tagging fall back to API (GPT-4-class) for the visual stream?
• Which feature space feeds GSBS/HMM for stimulus segmentation (raw CLIP frame embeddings, VLM hidden states, semantic transcript embeddings) materially changes the events recovered and should be decided/benchmarked.

Affect features

Emotion & affect (multimodal)

Time-resolved emotion annotation for movies/audio splits cleanly by signal source: per-frame face models (EmoNet, HSEmotion/EmotiEffLib), per-utterance/windowed voice models (audeering wav2vec2-dim), per-window text/dialogue models (RoBERTa-GoEmotions, EmoBank-VAD regressors), per-window music models (MERT-based regressors), and clip/window multimodal fusion or MLLM reasoning systems (Emotion-LLaMA, AffectGPT, Qwen2.5-VL/GPT-4o judges). A crucial distinction: most of these models predict character-DEPICTED emotion (the expressed affect of a speaker/face), whereas viewer-ELICITED emotion requires separate models trained on induced-affect ground truth (LIRIS-ACCEDE / MediaEval continuous, MuSe/SEWA benchmarks) — the two are correlated but not interchangeable and should be logged as distinct feature streams. For a small lab, the strongest open, locally deployable defaults are audeering wav2vec2 (voice VAD), HSEmotion EfficientNet (face VA + categorical), RoBERTa-GoEmotions (dialogue), MERT regressor (music), with an MLLM (Qwen2.5-VL or GPT-4o) as a window-level fusion/captioning layer and EmoNet/Emotion-LLaMA as alternatives.

Recommended (best-in-class): audeering wav2vec2-large-robust-12-ft-emotion-msp-dim (voice VAD); HSEmotion / EmotiEffLib EfficientNet (face valence/arousal + categorical); RoBERTa-GoEmotions + NRC-VAD mapping (dialogue affect); MERT-based V/A regressor (soundtrack/music affect, if relevant); Qwen2.5-VL-7B or GPT-4o as window-level multimodal fusion/annotator; LIRIS-ACCEDE/MuSe-trained continuous regressor for viewer-ELICITED valence/arousal (kept as a separate stream from all depicted-emotion models)

Likely redundant:

• EmoNet is redundant with HSEmotion for face valence/arousal+categorical — keep HSEmotion as primary (faster, lighter, more permissive license, active ABAW results), EmoNet for cross-validation only
• AffectGPT is redundant with Emotion-LLaMA as the open MLLM fusion/reasoning layer — pick one (AffectGPT for open-vocabulary/descriptive output, Emotion-LLaMA for turnkey benchmarked labels)
• Emotion-LLaMA/AffectGPT overlap with a general MLLM (GPT-4o/Qwen2.5-VL) — the specialized MLLMs add benchmarked depicted-emotion accuracy, the general MLLM adds schema flexibility and elicited-vs-depicted prompting; running all three is wasteful
• EmoBank/NRC-VAD dimensional text regressor partially duplicates RoBERTa-GoEmotions (same text, dimensional vs categorical view) — include both only if cross-modal VAD alignment is needed
• Per-modality stack (wav2vec2 + HSEmotion + GoEmotions) overlaps the internal branches of any fusion MLLM; the interpretable per-modality signals are generally preferable as primary features, with the MLLM as an added fused/reasoned layer rather than a replacement

Open questions:

• Does the project need depicted (character-expressed) emotion, viewer-elicited (induced) emotion, or both? This determines whether the LIRIS-ACCEDE/MuSe-style regressor is mandatory (it is the only viewer-elicited source) versus the depicted-emotion stack.
• Target temporal resolution: true per-second requires sliding-window inference and decisions on hop size and label smoothing; MLLM-per-shot annotation is cheaper but coarser and costlier per second of film.
• Licensing for downstream use: audeering wav2vec2 (CC-BY-NC-SA), EmoNet, and the MLLM-specialized models are non-commercial/research only; if any commercial redistribution is planned, prefer Apache/MIT options (HSEmotion, GoEmotions, Qwen2.5-VL).
• Soundtrack handling: do you need source separation to split musical-score affect (MERT) from speech affect (wav2vec2) when they are mixed in the film audio?
• Ground-truth/validation plan: which human-rated benchmark (SEND, SEWA, MuSe, LIRIS-ACCEDE) will be used to calibrate and cross-check the automated streams, and how will face/voice/text streams be fused (late fusion vs MLLM)?
• Face pipeline dependency: HSEmotion/EmoNet need an upstream detector+tracker (RetinaFace/MediaPipe) and a policy for multi-face scenes (which/aggregate) before producing a scene-level signal.

Redundancy & Parsimony Analysis

Redundancy & Parsimony Analysis: Computational Narrative Feature Extraction

This analysis identifies redundant or highly-correlated feature extractors across the 16 catalogs, recommends a single best-in-class option per redundancy group, flags cases where keeping two is justified by genuinely distinct information, and ends with a MINIMAL tractable set vs. a FULL set.

A. WITHIN-SUBCLASS REDUNDANCY

These are pairs/groups producing near-identical outputs. Keep one; the rest are interchangeable variants.

B. CROSS-SUBCLASS REDUNDANCY

These span catalogs and are the easiest savings to miss. The same underlying model or signal appears in multiple subclasses.

C. CASES WHERE KEEPING TWO IS JUSTIFIED (distinct information)

These are NOT redundant despite surface similarity — log as separate streams.

D. RECOMMENDED FEATURE SETS

MINIMAL TRACTABLE SET (single consumer GPU + CPU; one-pass-per-modality)

The design principle: one extractor per signal, one shared VLM pass, maximal cross-subclass reuse.

Visual — low-level

• scikit-image + OpenCV core (luminance, RMS contrast, RGB/HSV/CIELAB color, entropy, edges, FFT slope, Hasler-Susstrunk colorfulness)
• pyrtools steerable pyramid (orientation × scale energy)

Visual — dynamic

• SEA-RAFT (flow magnitude, camera-vs-object decomposition)
• pymoten (brain-aligned motion energy regressor)
• TransNetV2 + PySceneDetect (shot boundaries — shared with Situation/Action)

Visual — high-level / faces / action / saliency

• SigLIP 2 (object/scene/place/attribute probes — shared with Situation)
• DINOv2 (label-free embedding for RSA)
• InsightFace (face count/identity — shared with Social)
• OpenFace 3.0 (landmarks + AUs + gaze + emotion in one model — shared with Affect/Social)
• MMPose RTMPose (pose — shared with Social)
• VideoMAE V2 (sliding-window action posteriors + embeddings) + X-CLIP (zero-shot action)
• ViNet (video saliency) + Depth-Anything-V2 (depth)

Audio

• librosa (spectral/MFCC/chroma/onset/tempo)
• openSMILE eGeMAPS (standardized 88-dim LLDs)
• Parselmouth (F0, formants, jitter/shimmer/HNR)
• BEATs (AudioSet events) + CLAP (open-vocab prompts)
• inaSpeechSegmenter (speech/music/noise)
• faster-whisper + WhisperX (ASR + word alignment)
• pyannote community-1 (diarization — shared with Social)
• audEERING wav2vec2-dim (vocal A/V/D — shared with Affect)

Language

• spaCy trf (POS/dep/NER/morph)
• wordfreq + Brysbaert + Kuperman + Warriner/NRC-VAD + NRC-EmoLex (lexical norms — shared with Affect)
• minicons GPT-2/Llama (surprisal + hidden states — shared with Semantics)
• benepar + L2SCA (syntactic complexity)
• Maverick (coreference)
• Qwen3-Embedding + sliding-window coherence/drift + BERTopic (semantics)
• RoBERTa-GoEmotions (dialogue affect — shared with Affect)

Situation / Social / Affect (mostly via shared passes)

• GSBS (data-driven event segmentation on the embedding timeline)
• Light-ASD (active speaker)
• Gaze-LLE (joint/mutual attention)
• HSEmotion (face V/A — or reuse OpenFace 3.0 emotion)
• One Qwen2.5/3-VL pass per shot with a combined JSON schema covering: dense action/event description + situational dimensions (space/time/causation/intention/protagonist) + social interaction-type/dominance/affiliation/ToM + depicted emotion. This single pass replaces four separate VLM deployments.
• LIRIS-ACCEDE/MuSe regressor (viewer-ELICITED valence/arousal — kept separate from all depicted-emotion streams)

This set covers every subclass, runs on one GPU, and eliminates all Section A/B redundancy.

FULL SET (add when resources / specific accuracy needs justify)

Add to the minimal set, only where a distinct signal or best-in-class accuracy is wanted:

• Visual low-level: SHINE (literature-matched canonical defs), Rosenholtz clutter, GIST (compact holistic vector)
• Visual high-level: Grounding DINO (object boxes/counts), Mask2Former (panoptic composition), SAM 2 (video-tracked masks/Grounded-SAM), Places365 (frozen fixed taxonomy)
• Dynamic: MemFlow (temporally smooth flow)
• Action: InternVideo2-1B (stronger zero-shot/features), V-JEPA 2 (motion-biased complement), ActionFormer/OpenTAD (timestamped TAL segments), VideoLLaMA3 (alt video-LLM)
• Saliency/depth: Q-Align (aesthetic + quality), ResMem (memorability), Apple Depth Pro (metric depth), DeepGaze IIE (static fixation)
• Faces: Py-Feat (validated AU detector set), L2CS-Net (best-in-class 360° gaze), MediaPipe (CPU redundancy layer), InsightFace age/sex
• Audio: Essentia + madmom/Beat This! (roughness/beat/MIR + calibrated mood/genre/valence-arousal), CREPE (robust F0), mosqito (standards-based psychoacoustic roughness/loudness/sharpness), MERT music-emotion regressor
• Speech: emotion2vec+ (categorical SER), NeMo Parakeet/Canary (multilingual ASR cross-check)
• Language: Stanza (true UD scheme), GLiNER (custom entity types), LIWC-22 (validated pronoun/function-word scheme), DMRST (reproducible RST discourse labels), supervised TRIPOD turning-points, textstat (cheap readability), dialogue-act classifier
• Semantics: API embeddings (OpenAI/Voyage/Gemini) if leaderboard-topping required
• Situation: BrainIAK HMM (soft event posteriors / reactivation), MovieCLIP (cinematic-taxonomy settings), Llama-3-70B/GPT-4 transcript event segmentation
• Social: InternVL3 or a frontier API as a second VLM for agreement/gold-label validation; Social-IQ 2.0 / Social Genome schemas for prompt construction
• Affect: Emotion-LLaMA OR AffectGPT (benchmarked / open-vocab depicted-emotion reasoning), EmoNet (cross-check), EmoBank-VAD text regressor, AttendAffectNet (fused induced V/A)

E. KEY TAKEAWAYS

1. The single largest saving is consolidating all high-level reasoning into ONE video-LLM pass (C9). Action, situation, social, and depicted-emotion semantics are all currently realized by Qwen-VL-class models — run it once per shot with a unified multi-field prompt rather than four times.
2. Cross-subclass sharing (Section B) saves more than within-subclass pruning. Face emotion, gaze, pose, diarization, prosody, shot boundaries, surprisal, and lexical-VAD each appear in 2-4 catalogs — extract once, route everywhere.
3. Foundation models supersede most fixed-taxonomy CNNs (Places365, ImageNet, YAMNet, AST) — keep the latter only when a frozen, calibrated, reproducible probability vector is explicitly required.
4. The one non-negotiable “keep two” is depicted vs. elicited emotion — these are different constructs and conflating them is a scientific error, not a redundancy.

Recommendations — Best-in-Class Feature Set

This section translates the feature catalogs and redundancy analysis into actionable per-class recommendations. For each feature class (and its subclasses), it names the best-in-class tool(s), the key features to extract, whether the tool runs locally, and a priority tier: core (the minimal tractable set — one extractor per signal, maximal cross-subclass reuse, single consumer GPU + CPU) or extended (added only when resources or a specific accuracy/distinct-signal need justifies it).

Recommendation table

Tradeoffs

Cross-subclass reuse dominates within-subclass pruning. Several signals appear in multiple classes and must be extracted once and routed everywhere, not re-run per catalog. The clearest cases: face detection/identity (InsightFace) feeds Faces, Social, and High-level visual; face emotion (HSEmotion/OpenFace 3.0) feeds Faces, Social, and Affect; body pose (MMPose RTMPose) feeds Faces, Social, and Dynamic visual; diarization (pyannote) feeds Speech and Social; prosody (Parselmouth + openSMILE) feeds Speech, Low-level acoustic, and Affect; shot boundaries (TransNetV2/PySceneDetect) feed Dynamic visual, Situation, and Action; vocal affect (audEERING wav2vec2-dim) feeds Speech and Affect; LLM surprisal and hidden states (minicons) feed Lexical and Semantics; and lexical-VAD/GoEmotions feed Lexical and Affect. Wiring the pipeline around shared passes, rather than per-class extractors, is what makes the whole set tractable on a single consumer GPU.

The single largest compute saving is consolidating high-level reasoning into one video-LLM pass. Action description, situational dimensions (space/time/causation/intention/protagonist), social interaction-type/dominance/ToM, and depicted emotion are all realized by Qwen2.5/3-VL-class models. Running one Qwen-VL pass per shot with a unified multi-field JSON schema replaces four separate VLM deployments. InternVL3, VideoLLaMA3, LLaVA-OneVision, Emotion-LLaMA, and AffectGPT are interchangeable for this role; keep at most one primary plus, optionally, one frontier-API judge reserved for gold-label validation rather than routine annotation.

Foundation models supersede most fixed-taxonomy CNNs, but not always. SigLIP 2 zero-shot probing covers the object/scene/place/attribute role that Places365 and ImageNet backbones (and YAMNet/AST in audio) used to fill. Keep the fixed-taxonomy models only when a frozen, calibrated, reproducible probability vector is explicitly required for interpretability. Likewise, classic flow (RAFT/Farneback), depth (MiDaS/ZoeDepth), saliency (TASED-Net/UNISAL), and shot detection (AutoShot) are superseded by SEA-RAFT, Depth-Anything-V2, ViNet, and TransNetV2 respectively; the older tools survive only as no-GPU fallbacks.

Some surface-similar pairs are genuinely distinct and must both be kept. DINOv2 (label-free dense embedding for RSA) versus SigLIP 2 (text-probed interpretable scores) are different representations. pymoten (brain-aligned motion-energy regressor) versus SEA-RAFT (interpretable camera-vs-object flow), and semantic action recognition (VideoMAE) versus pre-semantic motion (SEA-RAFT/pymoten), are deliberately separate layers. BEATs (calibrated 527-class) versus CLAP (arbitrary open-vocabulary prompts), inaSpeechSegmenter (speech/music/noise partition) versus pyannote (who-spoke-when), Mask2Former (area-fraction composition) versus Grounding DINO (object counts/positions), and CREPE (robust pitch contour) versus Parselmouth (canonical voice-quality) all carry complementary information.

The one non-negotiable “keep two” is depicted versus elicited emotion. Face/voice/text emotion models predict character-expressed (depicted) affect, whereas LIRIS-ACCEDE/MuSe-trained regressors predict viewer-induced (elicited) affect. These are correlated but not interchangeable; logging them as a single stream would be a scientific error, not a parsimony gain. Similarly, interpretable per-modality affect signals (wav2vec2 + HSEmotion + GoEmotions) should remain the primary features, with any fusion MLLM as an added reasoned layer rather than a replacement.

Local-first is feasible throughout. Every core recommendation runs locally; most low-level audio and lexical tools are CPU-only, and the heaviest core components (Qwen-VL, VideoMAE, SigLIP 2, DINOv2, pyannote, Whisper) fit on a single 24-48 GB GPU. API embeddings and frontier-API VLM judges are the only API-only entries and are confined to the extended tier, used only when leaderboard-topping accuracy is required and off-site transfer of transcripts/frames is acceptable.

Annotation Output Data Format

Status: v0.2 design (supersedes the schema/annotation_schema.json v0.1 draft; reconciles it with the Phase 1 catalogs). Scope: the on-disk format the Phase 2 pipeline emits for one stimulus (one movie or one audio story), and the MATLAB reader interface that loads it.

This document defines: (1) the container format and why; (2) the timeline / alignment convention; (3) the hierarchical feature layout; (4) per-feature (per-channel) metadata; (5) how nulls and reserved human-annotation slots are represented; (6) a worked 2-timepoint example; (7) the MATLAB reader interface. It is the authoritative spec; schema/annotation_schema.json is the machine-checkable encoding of it, and matlab/*.m is the reference reader.

1. Format recommendation: HDF5 canonical + JSON sidecar manifest

1.1 The requirement, sized

A single feature-rich movie produces, per second:

• ~hundreds of scalar channels (luminance, RMS contrast, F0, flow magnitude, valence, surprisal …);
• several high-dimensional vector channels — AudioSet tags (527-dim), CLIP/SigLIP probe sets (tens–hundreds of prompts), Places365 (365-dim), DINO/CLIP/LLM embeddings (384–4096-dim), pymoten motion energy (hundreds–thousands-dim), pose keypoints (17–133 × 3);
• categorical / label channels (scene category, speaker id, topic id, dialogue act, narrative stage);
• sparse-event channels (cuts, beats, turning points) carried as both a per-bin flag and an onset list.

A 2-hour movie at 1 Hz is 7200 timepoints. With a few thousand embedding dimensions per timepoint across several encoders, the dense payload is tens of MB to low GB per stimulus. Inline-array JSON (the v0.1 draft) is unworkable at that size: it is ~3–5× larger than binary, parses slowly, holds everything in memory, and has no native typed n-D array. So JSON stays the readable metadata layer, not the bulk container.

1.2 Decision

Canonical container: HDF5 (.h5), one file per stimulus. Sidecar: a human-readable JSON manifest (.manifest.json) holding the same hierarchy and all metadata but no bulk numeric arrays — pointers (data_ref) into the .h5 instead.

Rationale, against the requirements:

Parquet is a strong alternative for the scalar/tabular slice (columnar, great with pandas/Arrow), but it is awkward for ragged high-dimensional per-timepoint vectors and variable-length event lists, and MATLAB support (parquetread) is newer and tabular-only. We therefore choose HDF5 as the single canonical container and optionally emit a flat *_scalars.parquet projection of just the scalar channels as a convenience export (see §3.5).

Pure-JSON profile (small/demo only). For tiny clips, tests, and documentation, the entire payload may be inlined into the JSON file (the v0.1 layout). The reader auto-detects: if a channel has inline value, it uses it; if it has data_ref only, it reads the .h5. This keeps the v0.1 example valid while making the dense case feasible.

1.3 On-disk artifacts per stimulus

annotations/output/<stimulus_id>/
  <stimulus_id>.h5             # canonical: time axis + all feature datasets + attributes
  <stimulus_id>.manifest.json  # readable hierarchy + metadata + data_ref pointers (no bulk arrays)
  <stimulus_id>_scalars.parquet  # OPTIONAL flat export of scalar channels (convenience)

Self-contained alternative: a single .h5 is sufficient on its own (it carries all metadata as attributes); the sidecar JSON exists for grep/diff/readability and for the pure-JSON profile.

2. Timeline & alignment convention

2.1 The common grid

There is exactly one common time grid per file, defined by three numbers and materialized as one dataset:

/time/rate_hz        scalar   e.g. 1.0   (configurable; default 1 Hz)
/time/t_start_sec    scalar   e.g. 0.0   (time of the *center* of sample 0; see §2.2)
/time/n_samples      scalar   e.g. 7200
/time/time_sec       [n_samples] float64   center timestamp of each grid bin (always written, even
                                           though derivable, so MATLAB/Python never recompute it)

Invariant: time_sec[i] == t_start_sec + i / rate_hz. Every feature dataset has leading dimension n_samples and shares this axis — that is what makes the whole file a single aligned matrix family.

A file may also declare an alternate grid (e.g. a 0.5 Hz coarse grid, or a per-TR grid for a specific fMRI study) under /time/alt/<name>/…; channels resampled to an alternate grid point at it via a grid attribute. Default and common case: a single grid.

2.2 Bin convention (center-referenced, half-open)

Grid bin i covers the continuous interval

[ t_start_sec + (i - 0.5)/rate_hz ,  t_start_sec + (i + 0.5)/rate_hz )

and time_sec[i] is its center. All resampling (§2.4) maps native samples into these bins. Center- referencing (not left-edge) keeps a feature’s grid time aligned to the middle of the interval it summarizes, which is the convention most neuroimaging analyses expect and avoids a systematic half-bin lag. The choice is recorded once in /time/bin_reference = "center".

2.3 Native rates feeding the grid

Each extractor has a native rate (from the Phase 1 catalogs), one of:

The native rate is preserved as metadata on every channel (native_rate_hz, which may be a number or one of the strings above), and the raw pre-resampling output is cached (per DEPLOYMENT_FEASIBILITY §2.6) so the grid can be regenerated at a different rate without re-running models.

2.4 Resampling onto the grid (per dtype)

One resampling rule per channel, chosen by dtype and recorded in the resample attribute. These match DEPLOYMENT_FEASIBILITY §2.7:

A native sample with no bin (out of range) is dropped; a bin with no native sample is NaN (continuous) / empty-label (categorical) / 0 (count) — see §5.

2.5 Why a single grid (not per-feature native arrays in the file)

Keeping every channel on one shared, regular grid is what makes the file a coherent design matrix: MATLAB loads it straight into a timetable, cross-feature correlations and PCA (Phase 4) need no re-interpolation, and “inapplicable = NaN of the right length” gives a constant output shape across all stimuli (the README’s core principle). Native-rate arrays still exist — in the cache, and optionally archived under /native/<channel>/… for power users — but the grid is the contract.

3. Hierarchical feature layout

3.1 Top-level structure (HDF5 groups == hierarchy)

/                                      (root; global attributes = schema_version, ids, provenance)
├── time/                              the common grid (§2)
├── stimulus/                          (attrs: id, title, modality, duration_sec, source, media_file, sha256)
├── features/                          MACHINE annotations, mirroring the Phase 1 semantic hierarchy
│   ├── visual/
│   │   ├── low_level_static/          luminance, rms_contrast, colorfulness, edge_density, fft_slope, clutter…
│   │   ├── high_level_static/         scene_category, object_presence, clip_probe…, places365, panoptic_fractions
│   │   ├── faces_bodies_gaze/         n_faces, au_intensity, gaze_yaw/pitch, head_pose, pose_keypoints, expression
│   │   ├── dynamic_motion/            flow_magnitude, camera_motion, residual_motion, motion_energy, cut (event)
│   │   ├── action/                    action_posteriors, action_probe, action_segment (event)
│   │   └── saliency_aesthetics_depth/ saliency_entropy, depth_mean, fg_fraction, aesthetic, quality, memorability
│   ├── audio/
│   │   ├── low_level/                 rms, loudness, spectral_centroid, mfcc, chroma, f0, onset_strength, tempo
│   │   ├── high_level/                audioset_tags, clap_probe, scene, speech_music_noise, key, beat (event)
│   │   └── speech/                    asr_text, speaker_id, vad, speech_rate, prosody_f0, ser_arousal/valence/dominance
│   ├── language/
│   │   ├── lexical/                   word, lemma, pos, freq_zipf, concreteness, aoa, valence_norm, emolex_*, surprisal
│   │   ├── syntax/                    dep_depth, clauses_per_tunit, tree_depth, coref_chain_len, dialogue_act, readability
│   │   └── semantics_discourse/       embedding, coherence, drift, novelty, topic_id, topic_vector, narrative_stage, turning_point (event)
│   ├── social/                        n_agents, characters_present, active_speaker, mutual_gaze, joint_attention,
│   │                                  proximity, interaction_type, dominance, affiliation, tom_intention
│   ├── situation/                     location, time_of_day, indoor_outdoor, script_label, event_id, event_boundary (event),
│   │                                  space/time/causation/intention/protagonist (Event-Indexing dims)
│   └── affect/
│       ├── depicted/                  face_valence/arousal, voice_v/a/d, text_valence, music_v/a, categorical_emotion
│       └── elicited/                  induced_valence, induced_arousal, fear_flag   (separate stream from depicted)
├── human/                             RESERVED for later human annotation (§5.4); same shape contract; empty at emit
│   ├── visual/ … affect/              (mirrors features/ subgroups; created empty)
│   └── _free/                         free-form human channels not in the machine taxonomy
└── provenance/                        per-class model registry, env locks, params hashes (§4.3)

The top-level classes and subclasses are exactly the Phase 1 hierarchy (Visual / Audio / Language / Social / Situation / Affect). A program traverses groups; a human reads the JSON sidecar, which has the identical tree.

3.2 A leaf = one channel = one HDF5 dataset + attributes

Each leaf feature is one dataset named by the channel, shaped [n_samples] (scalar/label/bool/event) or [n_samples × D] (vector/embedding) or [n_samples × K × C] (e.g. pose K keypoints × C coords), with metadata carried as HDF5 attributes on that dataset (§4). The dataset’s HDF5 path is its hierarchical path, e.g. /features/visual/low_level_static/luminance.

3.3 Dtype encodings

Categorical channels store integer codes + a categories attribute (the label vocabulary) so MATLAB reconstructs a categorical array directly and analyses stay numeric; label/text channels store strings directly for the free-vocabulary cases (ASR words, LLM free-text tags).

3.4 Vector channels carry a component axis

A [n × D] vector dataset gets a components attribute: a length-D string array naming each column (e.g. AudioSet class names, CLIP prompt strings, MFCC indices, keypoint names). Embeddings whose dimensions are not individually meaningful set components = [] and rely on dim/model metadata. This makes every vector self-describing without an external codebook.

3.5 Optional flat scalar export

For quick tabular work, the pipeline may also emit <id>_scalars.parquet: one row per timepoint, one column per scalar channel, column names = the slash-path with /→__ (visual__low_level_static__luminance), plus a time_sec column. This is a lossy convenience projection (no vectors/labels/events); the .h5 remains authoritative.

4. Per-feature (per-channel) metadata

4.1 Required + optional attributes on every channel

Carried as HDF5 attributes on the dataset (and as JSON keys in the sidecar). Required marked R.

4.2 Companion channels (uncertainty & dispersion)

Where a model exposes it, a channel X may have siblings:

• X_conf — per-timepoint confidence/probability (e.g. ASR avg_logprob, detection score, voicing prob);
• X_std, X_max — within-bin dispersion when many native samples collapsed into one grid bin;
• X_change — boundary/onset flag for categorical/label channels.

These are ordinary channels with their own metadata; the parent points to them via confidence_ref.

4.3 Global provenance (/provenance + root attributes)

/  (root attributes)
   schema_version   = "0.2"
   pipeline_version = "<git-describe>"
   generated_utc    = "2026-06-13T12:00:00Z"
   common_grid_rate_hz = 1.0

/provenance/
   models/        attr-per-model: name → {version, tier, env, params_hash, citation}
   env_locks/     names of the locked envs used (core-cv, torch-vision, speech, llm, …)
   stimulus_sha256
   transcript_source = "whisperx" | "supplied"

This satisfies the README’s “reproducible provenance”: every channel ties to a model entry, which ties to a locked environment and a params hash.

5. Nulls, applicability, and reserved human slots

5.1 Three distinct “missing” states — never conflated

The applicable flag plus the element-wise fill value lets downstream analysis distinguish “not measured” from “measured zero” — the README’s explicit requirement. Float channels use IEEE NaN (reads into MATLAB as NaN natively); categorical uses code -1; labels/text use ""; events use 0.

5.2 Inapplicable features are still present, full-length, all-null

When a stimulus is audio-only, every visual/* channel is still emitted with the correct [n_samples × …] shape, filled with NaN/""/-1, and applicable=false. This guarantees a constant output shape and hierarchy across the whole corpus, so cross-stimulus matrices, PCA, and the Phase-4 design tool never have to special-case which classes a given stimulus has.

Modality → applicable-class gate (from IMPLEMENTATION_PLAN “applicability rule”):

(*social/situation degrade to the subset derivable from the available modalities; per-channel applicable flags carry the exact truth.)

5.3 Sparse events

An event channel is stored as a per-bin count dataset plus an onset group holding exact times:

/features/visual/dynamic_motion/cut            int32 [n]   per-bin cut count (resample="count")
/features/visual/dynamic_motion/cut__onsets/
    time_sec     [m] float64      exact onset times (m = total events)
    value        [m] (optional)   per-event payload (e.g. transition type, confidence)

So an event is both griddable (the [n] count/flag, usable as a regressor) and exact (the onset list, for precise timing). Beats, turning points, scene boundaries, action segments follow the same pattern; segment-type events add a paired __offsets list.

5.4 Reserved human-annotation slots (/human/)

A top-level /human/ group mirrors the /features/ subgroup tree but is created empty at machine emit time — no datasets, just the group skeleton — plus a /human/_free/ group for human channels that don’t fit the machine taxonomy. Properties:

• Same channel contract. A later human (or human-in-the-loop tool) writes a channel into /human/<class>/<sub>/<name> using the identical schema: [n_samples] aligned to /time, with dtype, applicable, model="human", version=<rater id / protocol>, resample, notes.
• Never auto-populated. Machine runs only (re)create the empty skeleton; they never write under /human/, so re-running the pipeline never clobbers human work. (Writers should append, and the pipeline must not overwrite an existing .h5’s /human/ group — emit a new file or merge.)
• Discoverable. /human has an attribute populated = false until a human channel is added; the reader exposes ann.human as an (initially empty) struct of the same shape as ann.features.
• Provenance for humans. A human channel records rater id, instructions/protocol version, and date in its attributes exactly as a model channel records model/version, so human and machine annotations are first-class and equally traceable.

In the JSON sidecar / pure-JSON profile, this is the human_annotations object: empty {} at emit, later filled with the same hierarchical shape as features.

6. Worked example (~2 timepoints)

Below is the JSON sidecar view (the readable layer) for an audiovisual clip at 1 Hz, showing two grid timepoints and a representative channel of each kind. Bulk vectors point into the .h5 via data_ref; small channels inline value for readability. A null at t=0 for a language feature shows “applicable but not measured” (no speech yet); the whole affect/elicited example shows a separate stream from depicted affect.

{
  "schema_version": "0.2",
  "stimulus": {
    "id": "demo_movie_001", "title": "Demo clip", "modality": "audiovisual",
    "duration_sec": 2.0, "source": "user-supplied",
    "media_file": "data/movies/demo_movie_001.mp4", "sha256": "ab12…"
  },
  "time": {
    "rate_hz": 1.0, "t_start_sec": 0.0, "n_samples": 2,
    "bin_reference": "center", "time_sec": [0.0, 1.0]
  },
  "features": {
    "visual": {
      "low_level_static": {
        "luminance": {
          "dtype": "scalar", "applicable": true, "units": "0-1",
          "model": "scikit-image", "version": "0.24.0",
          "native_rate_hz": 25, "resample": "mean",
          "value": [0.41, 0.43]
        }
      },
      "high_level_static": {
        "scene_category": {
          "dtype": "categorical", "applicable": true,
          "model": "SigLIP2-so400m", "version": "siglip2-so400m-patch14-384",
          "native_rate_hz": "frame", "resample": "mode",
          "categories": ["kitchen", "hallway", "street"],
          "value": [0, 1]                        // codes -> "kitchen", "hallway"
        },
        "clip_probe": {
          "dtype": "vector", "applicable": true, "dim": 3,
          "model": "SigLIP2-so400m", "version": "siglip2-so400m-patch14-384",
          "native_rate_hz": "frame", "resample": "mean",
          "components": ["a kitchen", "a person cooking", "an empty hallway"],
          "data_ref": "/features/visual/high_level_static/clip_probe"   // [2 x 3] in .h5
        }
      },
      "dynamic_motion": {
        "cut": {
          "dtype": "event", "applicable": true, "units": "count",
          "model": "TransNetV2", "version": "1.0",
          "native_rate_hz": "frame", "resample": "count",
          "value": [0, 1],                         // one cut landed in bin t=1
          "onsets": { "time_sec": [1.04], "value": ["hard"] }
        }
      }
    },
    "audio": {
      "low_level": {
        "loudness": {
          "dtype": "scalar", "applicable": true, "units": "LUFS-rel",
          "model": "librosa", "version": "0.10.2",
          "native_rate_hz": 100, "resample": "mean",
          "value": [0.62, 0.71]
        }
      },
      "high_level": {
        "audioset_tags": {
          "dtype": "vector", "applicable": true, "dim": 527,
          "model": "BEATs", "version": "iter3+",
          "native_rate_hz": "window", "resample": "mean",
          "components": ["Speech", "Music", "..."],         // 527 names (truncated)
          "data_ref": "/features/audio/high_level/audioset_tags"   // [2 x 527] in .h5
        }
      }
    },
    "language": {
      "lexical": {
        "valence_norm": {
          "dtype": "scalar", "applicable": true, "units": "1-9 (Warriner)",
          "model": "Warriner-norms", "version": "2013",
          "native_rate_hz": "word", "resample": "mean",
          "value": [null, 6.2],                    // t=0 NaN: no speech in that bin (not measured)
          "notes": "null where no word onset falls in the bin"
        }
      },
      "semantics_discourse": {
        "embedding": {
          "dtype": "vector", "applicable": true, "dim": 1024,
          "model": "Qwen3-Embedding-0.6B", "version": "0.6B",
          "native_rate_hz": "utterance", "resample": "mean",
          "components": [],                         // opaque embedding dims
          "data_ref": "/features/language/semantics_discourse/embedding"  // [2 x 1024]
        },
        "narrative_stage": {
          "dtype": "categorical", "applicable": true,
          "model": "Llama-3.1-8B-Instruct", "version": "4bit",
          "native_rate_hz": "scene", "resample": "mode",
          "categories": ["setup", "conflict", "climax", "resolution"],
          "value": [0, 0]                           // both bins "setup"
        }
      }
    },
    "social": {
      "n_agents": {
        "dtype": "scalar", "applicable": true, "units": "count",
        "model": "InsightFace-buffalo_l", "version": "0.7",
        "native_rate_hz": "frame", "resample": "mean",
        "value": [2.0, 2.0]                          // measured 2 (not null)
      }
    },
    "affect": {
      "depicted": {
        "face_valence": {
          "dtype": "scalar", "applicable": true, "units": "-1..1",
          "model": "HSEmotion-EfficientNet-B2", "version": "va_mtl",
          "native_rate_hz": "frame", "resample": "mean",
          "value": [0.10, 0.30]
        }
      },
      "elicited": {
        "induced_valence": {
          "dtype": "scalar", "applicable": true, "units": "-1..1",
          "model": "LIRIS-ACCEDE-regressor", "version": "1.0",
          "native_rate_hz": "second", "resample": "mean",
          "notes": "viewer-elicited; distinct stream from affect.depicted",
          "value": [0.05, 0.12]
        }
      }
    }
  },
  "human_annotations": {},     // empty skeleton mirrors features/ (see §5.4)
  "provenance": {
    "pipeline_version": "0.2.0", "generated_utc": "2026-06-13T12:00:00Z",
    "models": {
      "SigLIP2-so400m": { "version": "siglip2-so400m-patch14-384", "tier": "gpu", "env": "torch-vision" },
      "BEATs": { "version": "iter3+", "tier": "gpu", "env": "audio-tag" }
      /* … */
    }
  }
}

For an audio-only story, the entire features.visual subtree is still present and full-length, with every channel applicable=false and value all-NaN (scalars) / -1 codes (categoricals) / "" (labels). Shape is identical to the audiovisual file; only applicable and the fill values differ.

7. MATLAB reader interface

MATLAB reads HDF5 natively (h5read, h5info, h5readatt) and JSON via jsondecode — no toolbox required for the core reader. The reference implementation lives in matlab/.

Implementation status. IMPLEMENTED and verified: readAnnotations (.h5, folder, or JSON), getFeature, featuresToTimetable(ann) (no name-value options; scalar/bool/event channels only, not-applicable channels as NaN), plus readAnnotationCorpus, analyzeCorpus, selectStimulusSet, refreshAnalysis, annotationMovieViewer. The signatures below marked (planned) — listFeatures, getFeatureMatrix, writeHumanChannel, the "Lazy" option, and featuresToTimetable’s name-value options — are design targets, not yet implemented.

High-level signatures:

% --- Load -----------------------------------------------------------------
ann = readAnnotations(path)
%   READANNOTATIONS  Load one annotation into a MATLAB struct.
%   PATH may be the .h5, the .manifest.json, or the stimulus folder.
%   If given JSON-with-data_ref, reads bulk arrays from the sibling .h5;
%   if given pure-JSON, uses inline values; if given the .h5, reads everything
%   from it (metadata from attributes). Returns:
%     ann.stimulus  struct   (id, title, modality, duration_sec, source, sha256)
%     ann.time      struct   (rate_hz, t_start_sec, n_samples, bin_reference)
%     ann.time_sec  double[n] column   (the common grid; always materialized)
%     ann.features  struct   nested groups mirroring the hierarchy; each leaf is
%                            a channel struct (see below)
%     ann.human     struct   same shape as features (empty until humans populate)
%     ann.provenance struct
%   JSON null / HDF5 fill -> NaN (numeric), "" (label/text), <undefined> (categorical).

% A leaf channel struct:
%   ch.value         numeric [n] | [n x D] | categorical [n] | string [n] | int [n]
%   ch.dtype, ch.applicable, ch.units, ch.model, ch.version,
%   ch.native_rate_hz, ch.resample, ch.components (string[]), ch.categories,
%   ch.notes, ch.onsets (struct .time_sec/.value for event dtypes)

% --- Navigate / select ----------------------------------------------------
ch = getFeature(ann, "visual/low_level_static/luminance")
%   GETFEATURE  Retrieve one channel by hierarchical (slash) path. Errors if the
%   path is not a leaf channel. Works for features/ and human/ paths.

paths = listFeatures(ann, namevalue)
%   LISTFEATURES  Return the slash-paths of all channels, filterable by:
%     "Class","visual" | "Dtype","scalar" | "Applicable",true |
%     "Modality",... | "Pattern","*valence*"
%   Use to discover what a file contains without manual traversal.

% --- Reshape for analysis -------------------------------------------------
tt = featuresToTimetable(ann, namevalue)
%   FEATURESTOTIMETABLE  Collect channels into a MATLAB timetable on the common
%   grid (RowTimes = seconds(ann.time_sec)). Options:
%     "Dtypes",["scalar","bool"]   which kinds to include (default scalars)
%     "Class","audio"              restrict to one branch
%     "ExpandVectors",true         explode [n x D] vectors into D named columns
%     "IncludeNaNApplicable",false drop all-NaN inapplicable channels
%   Variable names are the slash-path with "/"->"__". Vector/label/event channels
%   are skipped unless ExpandVectors / explicit Dtypes request them.

M = getFeatureMatrix(ann, paths)
%   GETFEATUREMATRIX  Stack a list of channel paths into one [n x P] numeric
%   matrix (scalars) for PCA / correlation / the Phase-4 design tool; returns the
%   column->path map and propagates NaN.

% --- Corpus level (Phase 4) ----------------------------------------------
C = readAnnotationCorpus(folder, namevalue)
%   READANNOTATIONCORPUS  Load many annotation files into a struct array (or a
%   stacked timetable keyed by stimulus_id), enforcing the constant-shape
%   contract so cross-stimulus matrices align. Options mirror featuresToTimetable.

% --- Write-back (human-in-the-loop) --------------------------------------
writeHumanChannel(path, "social/interaction_type", values, namevalue)
%   WRITEHUMANCHANNEL  Append a human annotation channel under /human/ in the .h5
%   (and update the JSON sidecar), validating length == n_samples and stamping
%   model="human", version=<RaterId>, plus protocol/date attributes. Never
%   touches /features/.

Design notes for the reader:

• No toolbox dependency for read: h5read/h5info/h5readatt + jsondecode are base MATLAB. timetable/categorical are base MATLAB. (Parquet export read needs no extra toolbox via parquetread, which is base since R2019a.)
• Lazy option. readAnnotations(...,"Lazy",true) reads only metadata + the time axis and defers bulk dataset reads until getFeature/getFeatureMatrix touches them — important for the multi-GB embedding channels.
• Categorical round-trips. Integer-code + categories attribute reconstructs a MATLAB categorical directly, so code -1 becomes <undefined> and grid math stays numeric.
• Constant shape guarantee is what lets readAnnotationCorpus stack heterogeneous stimuli without per-file branching: inapplicable channels are present and NaN, so the corpus matrix is rectangular.

8. Summary of decisions

1. HDF5 canonical, one file per stimulus, with a readable JSON sidecar manifest; optional flat *_scalars.parquet export; a pure-JSON profile for small/demo files. All load into MATLAB with no toolbox.
2. One shared, configurable common grid (default 1 Hz), center-referenced half-open bins, materialized time_sec; every channel shares the n_samples leading axis. Native rates preserved as metadata; raw native outputs cached for re-gridding.
3. Hierarchical groups mirroring the Phase 1 semantic hierarchy (visual / audio / language / social / situation / affect, with subclasses); each leaf is one dataset + self-describing attributes.
4. Per-channel metadata: model, version, native rate, resample op, units, components/categories, tier, params hash; global provenance ties channels to locked environments.
5. Three explicit missing-states — not-applicable (applicable=false, all-null), not-measured (element NaN/""/-1/0), measured-zero (real value) — never conflated.
6. Reserved /human/ group mirroring the machine tree, empty at emit, never auto-clobbered, with identical channel contract and human-rater provenance.
7. MATLAB reader (readAnnotations, getFeature, listFeatures, featuresToTimetable, getFeatureMatrix, readAnnotationCorpus, writeHumanChannel) loading the file into structs and timetables on the common grid.

Phase 2 Core-Set Proposal — freeze sheet

Purpose: the minimal, tractable, best-in-class feature set to build in Phase 2. Derived from docs/scoping_review/09_recommendations.md. Mark each row keep ✓ / drop ✗ / → extended and edit freely; your sign-off at the bottom freezes the Phase 2 scope. Everything here runs locally on Tier A (CPU) + Tier B (one 24–48 GB GPU). Default answer for every row is keep.

Note: keep all i didn’t answer. x = selected (i.e., x by keep means keep it. x by change means change)

1. Global parameters

2. Core extraction passes (extract once → route to all classes)

The core set is ~18 model passes, not 146 tools, because signals are shared. Each pass below is one deployable extractor; “feeds” shows which feature classes consume it.

Visual

Audio / speech

Language (consumes the transcript)

High-level reasoning (one consolidated VLM/LLM pass)

Situation / affect (algorithmic + dedicated)

3. Deliberately keep-both (distinct signals, not redundant)

DINOv2 (RSA) vs SigLIP 2 (interpretable); pymoten (brain-aligned) vs SEA-RAFT (interpretable) vs VideoMAE (semantic action); BEATs (fixed 527) vs CLAP (open-vocab); inaSpeechSegmenter (partition) vs pyannote (who-spoke); depicted vs elicited emotion. ☐ accept all ☐ edit: ____

4. Notable items left to the extended tier (off by default)

Grounding-DINO/Mask2Former/SAM2 (object boxes/masks), Places365 (calibrated taxonomy), Q-Align (aesthetics), Essentia/madmom (music key/beat/mood), DMRST (RST discourse), turning-point model, API embeddings, frontier-API VLM judge (gold-label validation). ☐ ok ☐ promote to core: ____

5. Modality applicability (auto-null rule)

Audio-only story → all visual channels applicable=false (NaN). Text-only → visual+audio null, language/social/situation/affect run on text. ☐ accept

Sign-off

• Approved by: Tor Wager Date: 6/19/2026
• Common grid rate frozen at: 1 Hz
• Hosted models: ☐ none x enabled for: OpenAI ChatGPT models__________________
• Notes / edits: ________________________________________________

On sign-off, Phase 2 builds one Extractor (src/extractors/<class>/) per kept pass behind the base.py interface, wired around the shared passes above.

Phase 2 — Build Status

The pipeline package lives in src/nfe/. It ingests a stimulus, runs the applicable extractors, resamples every signal onto the common grid, and emits the canonical HDF5 + JSON-sidecar annotation (docs/design/ANNOTATION_FORMAT.md).

Quickstart

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
PYTHONPATH=src .venv/bin/python -m nfe.run path/to/movie.mp4 --out annotations/output --rate 1.0

Output: annotations/output/<id>/<id>.h5 (+ <id>.manifest.json). Load in MATLAB:

ann = readAnnotations("annotations/output/<id>");   % folder, .h5, or .json
tt  = featuresToTimetable(ann);                       % scalars on the common grid
ch  = getFeature(ann, "audio/low_level/mfcc");        % one channel + metadata

Pipeline core — DONE & verified

• base.py — types, common-grid TimeGrid (center-referenced bins), FeatureChannel, and the native-rate→grid resampler (mean/max/sum/nearest/mode/count + event onsets).
• ingest.py — PyAV decode (downscaled RGB frames at an analysis fps) + audio extraction via the imageio-ffmpeg static binary. No system ffmpeg, no cv2 (avoids the PyAV/OpenCV ffmpeg-dylib clash on macOS).
• emit.py — HDF5 writer (hierarchical groups, per-channel attrs, reserved /human/ group, /provenance model registry) + JSON sidecar manifest with data_ref pointers.
• pipeline.py / run.py — orchestration + CLI, per-extractor error isolation, modality-driven applicability.
• MATLAB reader extended to the canonical .h5 path (matlab/readAnnotations.m), verified end-to-end on generated output (scalars, vectors, event onsets, timetable).

Extractors

Constant-shape contract — DONE

Every stimulus can yield an identical channel set via an auto-generated channel template (schema/channel_template.json, built by tools/build_channel_template.py from a real full run — no hand-maintained spec lists). Run with --template schema/channel_template.json: channels not produced (class inapplicable to the modality, or pass disabled) are filled as applicable=false, all-NaN skeletons with the right dtype/dim/components. Verified: a CPU-only run + template = the same 95-channel hierarchy as the full stack (e.g. kungfury: 78 measured + 8 skeleton; a CPU-only run has correspondingly more skeleton channels). This unblocks Phase 3/4 stacking.

Code-review fixes (2026-07)

A verified review pass fixed: event_id dtype consistency across the degenerate GSBS branch (constant-shape); extractor failures now tracked (n_failed/failed in the summary + status=partial in corpus_index.csv) instead of being silently relabelled “not applicable”; faces_present/pose_present computed as any-in-bin with NaN for unmeasured bins (were derived from the mean, erasing missingness); temp workdirs cleaned up; zero-length media/text now errors instead of writing an empty “ok” file; the nearest resampler returns nearest-to-bin-center; MATLAB reader always orients vectors time-first; featuresToTimetable excludes categorical class-code channels from the analysis matrix; analyzeCorpus tolerates constant channels; and the batch index merges across filtered runs. Shared frame decode DONE: Ingest caches decoded frames (capped for very long films) so the ~9 visual passes share one decode instead of re-decoding.

Second review pass (2026-07-08)

Fixed: skeleton event/bool fill values no longer leak into analyses as real data (featuresToTimetable + the search-index builder treat applicable=false as NaN/absent); the design tool excludes candidate segments dominated by imputed/missing cells (they could previously win the objective because data were missing); refreshAnalysis now produces the full documented artifact set in one command (full-corpus + AV-subset stats/figures, NaN-safe class lookups, crash-guarded contingency split); web search fixes (rounded-zero std wipeout, High/Low toggle ignored on check, text-only play dead-end, index-like channels excluded, missing-feature coverage shown); viewer works for audio/text-only stimuli with clean timer teardown; walkthrough root detection; SigLIP2/CLAP text towers precomputed (verified bit-identical to the combined forward; visual-semantic pass ~3× faster).

Known follow-ups

• F0 (pyin) is the runtime bottleneck (~0.85× audio duration); make optional / coarser.
• CLAP is fed 22 kHz audio (loses >11 kHz); optionally extract a 48 kHz stream for it.
• Replace CPU substitutes (histogram cuts) with the frozen GPU picks.
• Planned MATLAB APIs not yet built: listFeatures, getFeatureMatrix, writeHumanChannel, lazy reads (see ANNOTATION_FORMAT §7 status note).

Deployment Feasibility

How the recommended Phase 1 feature set is realistically deployed by a small lab. Grounded in the scoping-review catalogs (docs/scoping_review/) and aligned with the extractor contract in IMPLEMENTATION_PLAN.md and the output format in ANNOTATION_FORMAT.md. Design anchors: local-first, explicit NaN nulls, one common time grid, reproducible provenance.

1. Hardware tiers

Four tiers, derived from each tool’s catalog deployability tag (yes-cpu / yes-gpu / yes-heavy-gpu / api-only) and sanity-checked against real VRAM:

• Tier A — CPU backbone (no GPU). All low-level visual and audio statistics, lexical and syntactic norms, and event-segmentation algorithms: librosa, openSMILE, Praat/parselmouth, scikit-image/OpenCV, spaCy. Runs on any laptop/server.
• Tier B — single 24 GB GPU workhorse. SigLIP / DINOv2, SEA-RAFT (flow), Whisper (ASR), BEATs (audio tagging), Qwen3-Embedding, RTMPose, Depth-Anything, and a 7B VLM (Qwen2.5-VL-7B at 4-bit). This tier covers the great majority of high-level features.
• Tier C — 48 GB+ GPU. Heavier models (InternVideo2, ViTPose-G, Q-Align, 70B/72B LLMs). Each is paired with a Tier-B substitute that should be preferred unless the accuracy gain is needed.
• Tier D — API-only (optional). Frontier embeddings and frontier multimodal judges. None are required; each has a local default and is used as an accuracy ceiling / validation oracle.

2. Environment & orchestration

The real blocker is incompatible dependency worlds (TensorFlow vs PyTorch vs CTranslate2 vs mmcv, plus MATLAB-origin SHINE/Rosenholtz code). Prescription:

• 5–6 locked uv/conda environments, each with a resident GPU worker behind the uniform Extractor interface (src/nfe/base.py).
• A Snakemake/Prefect DAG orchestrating per-stimulus extraction, with transcript-as-hub ordering (ASR runs early because language/social/situational models consume the timestamped transcript).
• Ingest specifics: frame sampling at ~2 fps plus per-shot keyframes; audio extracted to 16 kHz (speech) and 44.1 kHz (music/acoustic) WAVs.
• Batching rules per model family; content-addressed caching at stage boundaries for idempotency and provenance (raw native-rate output cached so the common grid can be regenerated without re-running models).
• A single common-grid resampler with per-feature-type rules (continuous / probability / categorical / per-word / sparse-event) emitting constant-shape, NaN-padded output (see ANNOTATION_FORMAT.md §2.4).

3. Runtime & storage

• Per-hour-of-stimulus real-time factor (RTF): a core config runs ~0.5–1× real-time — a 2-hour movie annotates overnight on Tier A+B.
• Cost drivers: the per-shot VLM, dense per-frame depth/saliency/segmentation, pymoten motion energy, and benepar constituency parsing.
• Storage budget: low tens of GB per hour of stimulus (dominated by dense embedding/vector channels; HDF5 chunked+gzip keeps this in check).

4. Features that benefit from hosted large models

Only 3–4 classes realistically gain from large hosted models, each with a local include default:

1. High-level social semantics / theory-of-mind tagging.
2. Transcript-level narrative / Event-Indexing situational tagging.
3. Multimodal affect with rationale.
4. (Optional) top-end embeddings.

Interface guidance: a thin API client behind the same Extractor interface; JSON-schema-validated structured output; per-shot/per-scene granularity; batch + cache + retry; an opt-in allow_hosted privacy gate; and using APIs mainly to validate/distill into the local 7–8B model rather than as a runtime dependency. See the claude-api skill reference for Anthropic model IDs, structured output, and pricing if wiring a hosted judge.

5. Recommended default configs

• Core (zero-API): Tier A + Tier B only. Fully local, overnight per feature-length film, covers the recommended core feature set.
• Extended: add Tier C models where the accuracy gain is justified.
• Hosted (opt-in): enable Tier D for the 3–4 classes above, gated by allow_hosted.

Implementation Plan — Narrative Feature Extraction

This document is the master plan. It is intentionally concrete about Phase 1–2 (buildable now) and directional about Phase 3–4 (depend on human selection, real movies, and Phase 1 outcomes).

Phase 1 — Scoping review (automated; in progress)

Deliverable: a complete scoping review of computational annotation tools, organized in a logical semantic hierarchy, with descriptions and recommendations (best-in-class vs likely-redundant).

Method: a fan-out of research agents, one per feature subclass, each grounding a structured catalog of current (2023–2026) tools in web sources, followed by cross-cutting analyses (hierarchy, redundancy, output schema, deployment feasibility) and synthesis.

Feature classes covered

• Visual
  • Low-level static (luminance, contrast, color, spatial frequency, edges)
  • High-level static (objects, scenes/places, attributes)
  • Faces, bodies, gaze, facial expression
  • Dynamic / motion (optical flow, motion energy, cut detection)
  • Action & event recognition (video understanding)
  • Saliency, aesthetics, depth
• Audio
  • Low-level acoustic (loudness, pitch, spectral, MFCC, onsets, tempo)
  • High-level audio (sound events, scenes, music vs speech, MIR)
  • Speech (ASR + timestamps, diarization, prosody, vocal affect)
• Language / semantic
  • Low-level lexical & word-level (frequency, concreteness, affect norms, surprisal, POS, NER)
  • Syntax & grammar (parsing, complexity, coreference, dialogue acts)
  • High-level semantics, discourse & narrative (embeddings, topics, coherence, story arc)
• Social / situational / affective
  • Social & interpersonal (agents, who-talks-to-whom, interaction type, theory of mind)
  • Situations, schemas, scripts & event segmentation (event boundaries, situation models)
  • Emotion & affect (multimodal valence/arousal + categorical; depicted vs elicited)

Outputs (written under docs/scoping_review/): 00_overview.md, 01_hierarchy.md, per-class catalog sections, 08_redundancy.md, 09_recommendations.md. Schema & deployment outputs land in schema/ and docs/design/.

Phase 2 — Pipeline build (pending Phase 1 human sign-off)

1. Feature selection checkpoint — human developer reviews Phase 1 recommendations and freezes the final feature/model set (core vs extended tiers).
2. Model deployment — for each selected model: download + pin locally (HF/torch), or build a thin API client where local is infeasible. Each lives in src/nfe/extractors/ behind a uniform interface: extract(stimulus, time_grid) -> {feature: array_over_time, metadata}.
3. Ingest — accept a movie or story; demux video frames (configurable sampling) and audio; generate a timestamped transcript (ASR or supplied) that feeds language/social/situational models.
4. Common-grid resampling — map every native-rate feature onto the configured grid (default 1 Hz), recording the native rate and resampling method per feature.
5. Emit — write one hierarchical annotation file per stimulus (see schema/), with NaN for inapplicable features and reserved slots for later human annotations.
6. MATLAB reader — matlab/readAnnotations.m + helpers load annotations into MATLAB structs/tables.

Applicability rule: stimulus modality (video / audio / text) determines which classes run; the rest emit NaN so the output shape is constant across stimuli.

Phase 3 — Corpus annotation

• Assemble a corpus from user-supplied media and findable/downloadable sources (respecting licensing).
• Maintain a stimulus manifest (id, source, modality, duration, rights).
• Run the Phase 2 pipeline over the corpus; cache per-model outputs; store annotations in annotations/output/.

Phase 4 — Analysis & dissemination

• MATLAB analysis loading annotations across all stimuli: descriptive structure, cross-feature correlations conditioned on stimulus type, principal components of the annotation set, and a network graph of how feature classes relate over time, with visualizations.
• Review paper documenting the models/algorithms and the empirical relationships among measures.
• Interactive web interface to search for movie/story segments high in particular features.
• Experimental-design tool: select stimulus segments that maximize variance across the major annotation principal components while maximizing independence of feature time series — formally, an optimal-design objective (e.g. maximize the determinant / D-optimality of the leading components over the selected set), applied to the corpus to produce a high-variance, low-redundancy stimulus set.

Open decisions for the human developer (Phase 1 → 2 gate)

• Common grid rate (default 1 Hz) and whether multiple rates are emitted.
• Core vs extended feature tier membership.
• Output container format (recommendation produced in schema/).
• Budget/appetite for API-only large models vs local-only constraint.
• Licensing policy for Phase 3 corpus sourcing.

Phase 3 — Corpus Annotation

Assemble the stimulus corpus and run annotations across it, producing one constant-shape annotation file per stimulus for the Phase 4 analysis.

Corpus & manifest

tools/build_manifest.py scans data/movies/** and data/stories/** and writes data/manifest.csv (+ .json): id, path, source, modality, duration_sec, fps, width, height, has_video, has_audio, rights. Rebuild any time media is added.

Current corpus: 83 stimuli / ~470.6 min (~7.8 h) — 49 lab spacetop clips + 4 short films (3 CC-BY Blender open films + Kung Fury) + 29 Narratives spoken-story audio clips (data/stories/narratives/, ~5.3 h) + 1 pure-text sample story (53 audiovisual, 29 audio-only, 1 text-only). To grow it: drop files under data/movies/<source>/ (movies/audio) or data/stories/<source>/ (audio/text stories) and re-run the manifest (see ../ADDING_MOVIES.md; credentialed sets in ../EXTERNAL_STIMULI.md).

Batch runner

python -m nfe.batch annotates manifest entries:

PYTHONPATH=src .venv/bin/python -m nfe.batch \
  --manifest data/manifest.csv --out annotations/corpus \
  --template schema/channel_template.json \
  --vision --audio-hl --events --source spacetop --max-dur 65

• Constant shape: --template schema/channel_template.json → every file has the same 95-channel hierarchy (un-run/inapplicable channels are applicable=false NaN), so Phase 4 stacks them into rectangular matrices.
• Resumable: skips a stimulus whose <id>.h5 already exists (--no-skip to force).
• Crash-safe: rewrites annotations/corpus/corpus_index.csv (id, status, n_measured/n_skeleton, seconds, error) after every stimulus.
• Isolated: one stimulus erroring doesn’t stop the batch.
• Filters: --source, --ids, --max-dur, --limit. Passes: --vision --audio-hl --reason --events. --fps (default 2 for batch).

Output: annotations/corpus/<id>/<id>.h5 (+ .manifest.json) per stimulus.

Runtime guidance (this M1 Ultra, MPS)

Per-pass cost (≈, at 2 fps): CPU low-level + ASR + language are cheap; pose and the frame VLM/depth/action passes dominate; --reason (Qwen2.5-VL) is ~50 s/window and is impractical for long films here. Recommended tiers:

Scale-up: run the full corpus in the background (overnight); the index + skip-existing make it safely resumable. For production-scale --reason, use the 7B on a CUDA GPU or shot-sampled windows.

Adding Movies and Stories to the Dataset

How to grow the corpus and refresh everything that depends on the specific set of stimuli (annotations, search index, analysis figures, and the corpus-specific numbers in the docs). The extractor code, schema/template, scoping review, and format spec do not change when you only add media.

For the user

1. Add the media

Drop files under a source subfolder of data/movies/:

data/movies/<source>/...        e.g. data/movies/myset/clip01.mp4     (movies / audio)
data/stories/<source>/...       e.g. data/stories/myset/story01.txt   (stories)

• Movies: any video container (.mp4 .mkv .mov .avi .webm .m4v). Modality is detected automatically (audiovisual / video-only).
• Audio stories: audio files (.wav .mp3 .m4a .flac .aac .ogg) — detected as audio-only; visual channels are emitted as NaN (not applicable).
• Pure-text stories: plain-text files (.txt .text .story) — detected as text-only. The text is placed on an estimated reading-rate timeline (~3 words/s) so it shares the 1 Hz grid; the language, text-emotion, surprisal, and event passes run, and visual/audio channels are NaN. (Markdown .md is treated as documentation, not a story.) Put stories under data/stories/<source>/ (or data/movies/ — both are scanned).
• Use one subfolder per source (keeps provenance/rights tidy). If the material has licensing terms, add a SOURCES.md in that subfolder (see data/movies/open/SOURCES.md).
• Credentialed neuroimaging stimuli (HCP 7T movies, CamCAN Hitchcock clip): these need a data-access request — see EXTERNAL_STIMULI.md. Placeholder folders data/movies/hcp/ and data/movies/camcan/ are ready for them.

2. Refresh — two options

Option A (easiest): tell Claude “I added media under data/movies/<source>/, refresh the dataset and docs”. Claude runs the runbook below and updates everything.

Option B (do it yourself):

# 1) manifest + annotate NEW stimuli (resumable) + rebuild search index
tools/refresh_corpus.sh                 # add --reason for the VLM pass; --no-skip to redo all
# 2) regenerate figures + the corpus stats the docs use (in MATLAB)
matlab -batch "addpath matlab; refreshAnalysis('annotations/corpus')"
# 3) update the corpus-specific numbers in the docs from analysis/corpus_stats.json
#    (ask Claude, or edit the files listed in the runbook below)

3. Verify

• annotations/corpus/corpus_index.csv — one row per stimulus, status = ok/skipped/error.
• analysis/figures/*.png and analysis/corpus_stats.json — regenerated.
• Web search: rebuilt automatically by step 1; reopen analysis/web/index.html.
• Spot-check a new stimulus in the viewer: annotationMovieViewer(movie, annDir).

Notes

• Resumable: existing annotations are skipped; only new stimuli are processed. Long films take a while (especially --reason); run overnight if needed.
• Template is stable: adding media does not change the 95-channel template. Only rebuild it (tools/build_channel_template.py) if you change the extractor set — then re-annotate everything with --no-skip.
• Hardware/runtime guidance: docs/design/PHASE3_CORPUS.md, design/DEPLOYMENT_FEASIBILITY.md.

Agent runbook (for Claude)

When asked to refresh after media was added:

1. Annotate + index — bash tools/refresh_corpus.sh (forward --reason / --source / --no-skip as requested). Resumable. For many/long stimuli run it in the background and monitor annotations/corpus/corpus_index.csv (count */*.h5, watch for status=error). This rebuilds data/manifest.csv and analysis/web/segments.json.

2. Analysis + stats — in MATLAB (via MCP): addpath matlab; cd <root>; refreshAnalysis("annotations/corpus"). Regenerates analysis/figures/*.png and writes analysis/corpus_stats.json.

3. Update corpus-dependent docs from analysis/corpus_stats.json (key → where). refreshAnalysis exports full-corpus stats under full.*, audiovisual-subset stats under av_subset.* (the paper’s structural figures are AV-subset), and NaN for any class that drops out of the analysis — a NaN here means “not enough data”, not “update the doc to NaN”.

   stat key	update in
   n_stimuli, total_minutes, by_source, by_modality	README.md (Phase 3 row), docs/CONTENTS.md (Datasets + folder map count), docs/REVIEW_PAPER.md §5, docs/design/PHASE3_CORPUS.md
   n_timepoints, full.n_channels_analyzed, av_subset.n_channels_analyzed	REVIEW_PAPER.md §5/§6
   full.pcs_to_80, av_subset.pcs_to_80 (+ pc1_5_pct)	REVIEW_PAPER.md §6.2, PHASE4_ANALYSIS.md
   class_visual_social, class_audio_social, class_audio_visual	REVIEW_PAPER.md §6.3, PHASE4_ANALYSIS.md
   contingency.visual_social_*, contingency.audio_social_*	REVIEW_PAPER.md §6.4
   design_greedy, design_random, design_k, design_n_source_stimuli	REVIEW_PAPER.md §7, PHASE4_ANALYSIS.md
   searchable segment/channel counts	from build_search_index.py stdout → PHASE4_ANALYSIS.md

4. Surface results — SendUserFile the regenerated analysis/figures/*.png if useful.

5. Memory — update the project memory with the new corpus size and date.

6. Sanity checks — corpus_index.csv has 0 errors; readAnnotationCorpus loads the expected N; the constant 95-channel shape still holds across all files; if a new SOURCES.md/rights are needed for the added source, create it.

Edge cases

• New modality (audio-only / text-only): supported. Confirm inapplicable classes are present as applicable=false NaN skeletons (the template fill handles this). Text stories run the language/affect/event passes on an estimated reading-rate timeline (--audio-hl --events reaches text-emotion + surprisal; lexical/syntax are always on).
• Extractor set changed (not just new media): rebuild the template (tools/build_channel_template.py on a full-stack short clip) → re-annotate all with --no-skip → then steps 2–6.
• Failures: per-stimulus errors are isolated and logged in corpus_index.csv (status=error, message); investigate individually.

Does NOT need updating when only media is added

src/nfe/**, matlab/**, schema/** (template stable), docs/scoping_review/**, docs/design/ANNOTATION_FORMAT.md, docs/design/PHASE2_*, and this file.

Obtaining external neuroimaging stimuli (Narratives, HCP, CamCAN)

Three widely-used naturalistic-fMRI stimulus sets that enrich this corpus and enable comparison with a large body of published brain data. The Narratives spoken-story collection is openly available and has already been downloaded and integrated; the HCP 7T movie clips and the CamCAN Hitchcock clip require data-access requests. This page gives provenance/licensing for each and how to (re)obtain and integrate them.

1. Narratives spoken-story stimuli (obtained)

What it is. 29 naturalistic spoken-story audio clips (~3–56 min, ~5.3 h total) from the “Narratives” fMRI dataset (Nastase et al., 2021), OpenNeuro ds002345 — natural stories (Pie Man, Tunnel, Slumlord, Milky Way, Sherlock, Merlin, …), several scrambled/paraphrased control variants, and a few short TV/film audio clips. Integrated here as audio-only stimuli under data/stories/narratives/ (ASR transcribes them; audio/language/affect/event passes run; visual channels are NaN).

How it was obtained (reproducible). Downloaded from OpenNeuro’s public S3 bucket:

base="https://s3.amazonaws.com/openneuro.org/ds002345/stimuli"
for s in pieman tunnel slumlordreach ... ; do curl -fsSL -o "data/stories/narratives/${s}_audio.wav" "$base/${s}_audio.wav"; done

(The full file list is on S3 under ds002345/stimuli/; word/phoneme transcripts and Gentle alignments are also in ds002345 if you want ground-truth transcripts instead of ASR.)

License / terms. The dataset is CC0, but the audio clips are not public domain — they are shared for non-profit, non-commercial scholarly research under “fair use” (some are copyrighted TV/film audio); do not redistribute. Attribution + details: data/stories/narratives/SOURCES.md. Rights holders may request removal via the dataset’s contact.

Status. Downloaded and catalogued in the manifest; annotation runs via the standard audio config (--audio-hl --events --template).

2. HCP 7T movie-watching stimuli

What it is. Four movie runs (MOVIE1–MOVIE4), each a ~12–14 min .mp4 concatenating short (1–4.3 min) clips from two sources:

• Creative-Commons Vimeo independent films (filenames marked CC) — genuinely open,
• Hollywood film excerpts (the set published by Cutting et al.) — copyrighted, plus a repeated Vimeo “validation” clip shown across runs. Metadata files HCP_7T_Movie_Info.csv (clip origins) and HCP_7T_Movie_Clip_Timing.csv (start/stop of each clip and REST block) accompany the stimuli.

How to obtain (official).

1. Register for a free account at ConnectomeDB — https://db.humanconnectome.org.
2. Accept the HCP Open Access Data Use Terms.
3. Open the WU-Minn HCP 1200 Subjects (S1200) project page and find the “7T Movie Resources” — download the 7T movie stimulus zip plus HCP_7T_Movie_Info.csv and HCP_7T_Movie_Clip_Timing.csv. Direct project link: https://db.humanconnectome.org/data/projects/HCP_1200.
4. Open subset (optional): if you only need the freely-licensed portions, the CC-named clips are CC Vimeo films; HCP_7T_Movie_Info.csv lists each clip’s origin, so those can be fetched individually. (This will not reproduce the exact HCP stimulus, which also includes the copyrighted Hollywood excerpts and specific concatenation/timing.)

Licensing. The Hollywood excerpts are copyrighted; HCP shares them under its data-use terms for research use — do not redistribute. The CC clips carry their Creative-Commons terms (attribution).

Integrate. Put the .mp4s in data/movies/hcp/ and refresh (below).

3. CamCAN movie-watching stimulus

What it is. An 8-minute edited version of Alfred Hitchcock’s “Bang! You’re Dead” (Alfred Hitchcock Presents, S7E2, 1961), condensed from the ~25–30 min original while preserving the plot. It is copyrighted (Universal / the Hitchcock estate). The specific 8-min CamCAN edit — not just the original episode — is what matches the published data and its extensive existing annotations.

How to obtain (official).

1. Apply through the Cam-CAN data access portal — https://camcan-archive.mrc-cbu.cam.ac.uk/dataaccess/.
2. Provide a valid academic affiliation and a specific hypothesis, and request the movie-watching data/stimulus. Notes from Cam-CAN: they reject vague “all data” requests and student-submitted applications — a supervisor/PI must apply, selecting only the data types justified by the proposal.
3. On approval you receive sftp download instructions; the Hitchcock stimulus is included.

Licensing. Copyrighted TV footage provided for research under the Cam-CAN agreement — do not redistribute. (The full original episode airs on some streaming services, e.g. Roku, but that is not the 8-min Cam-CAN edit.)

Integrate. Put the clip in data/movies/camcan/ and refresh (below).

Why these weren’t downloaded automatically

Both require a personal account and a data-use agreement that you must accept, and both contain copyrighted Hollywood/Hitchcock footage. I have no such accounts, and pulling the copyrighted content from unofficial sources would not be appropriate. Obtain them through the official channels above; then integration is one step.

After you obtain them

data/movies/hcp/       ← HCP 7T MOVIE1–4 .mp4 files
data/movies/camcan/    ← the 8-min Hitchcock clip

Then run the standard refresh (see ADDING_MOVIES.md) — or just tell Claude “refresh the dataset”:

tools/refresh_corpus.sh                                  # annotates the new stimuli
matlab -batch "addpath matlab; refreshAnalysis('annotations/corpus')"

The new stimuli are annotated on the same 95-channel constant-shape template and folded into all analyses and the search interface automatically. If you also have the HCP clip-timing CSV, tell me and I can align annotations to the per-clip boundaries.

Sources. HCP: 7T imaging protocol, ConnectomeDB S1200, HCP-users: movie stimuli, 7T movie clip-info wiki. CamCAN: data access portal, Cam-CAN protocol (Shafto et al. 2014, BMC Neurology), Bang! You’re Dead (Hitchcock wiki).

Phase 4 — Analysis & Dissemination

MATLAB tools that load the annotated corpus and analyze the structure of the annotations across stimuli, plus interactive viewing. Builds on the constant-shape corpus from Phase 3 (annotations/corpus/).

Done

readAnnotationCorpus(folder)

Loads every <id>/<id>.h5 under folder into one analysis structure:

• C.ids / C.ann — stimulus ids and full annotation structs
• C.channels — shared scalar-channel set (constant-shape → identical across stimuli)
• C.X [sumT x P] — all timepoints of all stimuli stacked into one rectangular matrix
• C.stim, C.time_sec, C.nT — stimulus id and within-stimulus time per row of X

Robust to partially-written files (skips + warns), so it can run while the Phase 3 batch is still filling the corpus. X is the direct input to correlation / PCA / the design tool. Example:

C = readAnnotationCorpus("annotations/corpus");
R = corr(C.X, 'rows','pairwise');          % cross-feature correlation across the corpus
[coeff,score,~,~,expl] = pca(normalize(C.X)); % principal components of the annotation set

annotationMovieViewer(movieFile, annPath)

Plays a movie with its annotation time series synced below and a red marker on each series tracking the playback position. Play/Pause + scrub slider; "Channels" and "Speed" name-value options. Example:

m = "data/movies/spacetop/videos/ses-01/ses-01_run-01_order-04_content-parkour.mp4";
annotationMovieViewer(m, "annotations/corpus/ses-01_run-01_order-04_content-parkour")

analyzeCorpus(C)

Structure analysis over the stacked corpus matrix → figures in analysis/figures/: clustered cross-feature correlation heatmap, PCA scree/cumulative variance, a channel correlation network (nodes colored by feature class, edge |r|≥thresh), and a feature-class network (edge = mean |r| between classes). Returns R, cluster order, PCA coeff/score/explained, and classR. NaN-aware z-scoring before PCA. NaN-aware. On the 83-stimulus modality-mixed corpus, 12 PCs reach 80% over the 26 audio/language channels shared across modalities; on the 53-stimulus audiovisual subset, 18 PCs reach 80%, with modest class couplings — strongest affect↔social (mean |r| 0.15), then visual↔social and audio↔social (0.11) — roughly stable across content. (The depicted-affect, social, and situational scalars come from the VLM reasoning pass, now run across the whole audiovisual subset; the language class stays under-represented in the dialogue-light AV clips; for the visual/social structure analyze the audiovisual subset — see REVIEW_PAPER §5–6.)

selectStimulusSet(C)

Experimental-design / stimulus-selection tool. Splits every stimulus into fixed-length candidate segments and greedily maximizes log det(cov) of the concatenated annotation time series (projected onto the leading PCs) — jointly rewarding high variance across the major feature dimensions and independence of the feature time series (D-optimality over the annotation space). Returns sel.table (selected segments) and objTrace vs a random baseline. Candidate segments dominated by missing/imputed data are excluded (they cannot win the objective by being blank). On the 83-stimulus corpus, selecting 20 × 10 s segments reaches log det(cov) 12.8 vs 2.5 for random — well over 5× the generalized variance — drawing from 16 distinct stimuli across modalities.

refreshAnalysis(corpusFolder)

One-command refresh matching the docs: full-corpus stats + design figure, audiovisual-subset structural figures (Figs 1–5 of the review paper) and class couplings, the speech-rich vs speech-sparse comparison, and analysis/corpus_stats.json with both full and AV-subset numbers (missing classes export as NaN, never empty).

See also ../CONTENTS.md (full guide) and ../walkthrough.m.

Interactive web search — analysis/web/

Static browser app over a precomputed segment index (tools/build_search_index.py → analysis/web/segments.json; 5684 × 5 s segments, 174 searchable channels — index-like channels and not-applicable skeleton fills are excluded, and interpretable vector channels are expanded into per-component columns so e.g. EmoNet’s 20 emotion categories and the 8 facial expressions are individually searchable). Pick any combination of features, toggle High/Low, and segments are ranked by mean z-score (with a features-covered count when a segment lacks some selected channels); ▶ Play seeks the actual clip to that segment (audio stories play as sound; text-only has no playback). Serve from the project root (python3 tools/serve.py → analysis/web/index.html; the bundled server adds HTTP Range support so video seeking works). Rankings validate well — high flow surfaces action scenes, high word-rate the dialogue clips, EmoNet “Aesthetic Appreciation” the beach-sunset clips, etc. Launch + rebuild notes: ../../analysis/web/app_readme.md.

Review paper

../REVIEW_PAPER.md — draft review covering the models/algorithms behind each annotation and the empirical structure of the annotation space (redundancy, dimensionality: 12–18 PCs→80%, class networks — couplings are modest, strongest affect↔social ≈ 0.15, and roughly stable across speech density — and the D-optimal design tool). The group-conditioned figure is analysis/figures/class_coupling_by_group.png.

Phase 4 complete

Corpus reader, synced viewer, correlation/PCA/network analysis, experimental-design tool, web search interface, and the draft review paper are all in place. Future work (per the paper’s §9): swap in production models, scale the VLM reasoning pass, add elicited-affect + diarization, validate against human ratings / brain data.

Computational Annotation of Movies and Stories: Models, a Second-by-Second Annotation Pipeline, and the Structure of Naturalistic Feature Spaces

Draft review paper — Narrative Feature Extraction project. Figures referenced here are generated by matlab/analyzeCorpus.m and matlab/selectStimulusSet.m into analysis/figures/.

Abstract

Naturalistic stimuli — movies and spoken stories — are increasingly used to study perception, language, social cognition, and emotion under ecologically rich conditions. Their analytic value depends on dense, well-organized annotations of what the stimulus contains at each moment. We describe an infrastructure that converts an arbitrary movie or audio story into second-by-second computational annotations spanning visual, auditory, linguistic, social, situational, and affective features, produced by a curated set of 23 best-in-class (or best-available-locally) models. The annotations are emitted in a single constant-shape, hierarchical format that loads natively into MATLAB, with explicit nulls where a feature does not apply. We annotate an 83-stimulus, ~7.8-hour corpus spanning three modalities — audiovisual clips and films, spoken-story audio (the open Narratives collection), and a text story — and analyze the structure of the resulting feature space. The annotations are highly multidimensional (12 principal components explain 80% of the variance in the modality-shared audio/language channels; 18 in the audiovisual subset that adds the visual/social/situational/affective channels), organized into interpretable redundancy blocks. Cross-class couplings are modest — the strongest is affect↔social (mean |r| ≈ 0.15), followed by visual↔social and audio↔social (≈ 0.11) — and roughly stable across content; language-class scalar features are defined only by dialogue not present throughout, so that class is under-represented in the corpus-wide analysis, while the spoken stories make the corpus richly language-annotated. Finally, we use the annotation space to drive an experimental-design tool that selects high-variance, low-redundancy stimulus segments. All tools, derivatives, and an interactive viewer and web search interface are released with the project.

1. Introduction

A central methodological problem in naturalistic neuroimaging and cognitive science is annotation: to relate brain or behavioral responses to a movie or story, one needs a time-resolved description of the stimulus across many feature classes simultaneously. Hand annotation is slow, low-dimensional, and hard to standardize. Recent advances in computer vision, audio understanding, speech recognition, and language and vision-language models make it feasible to annotate naturalistic stimuli automatically across a wide range of features, from low-level image statistics to high-level social and situational content.

This project builds that infrastructure in four phases: (1) a scoping review of computational tools per feature class (docs/scoping_review/); (2) a pipeline of selected models producing a hierarchical, second-by-second annotation; (3) annotation of a corpus; and (4) analysis of the annotation structure plus dissemination tools. This paper documents the models and algorithms, the annotation format, and the empirical structure of the annotations over the corpus.

2. Feature taxonomy and annotation hierarchy

Annotations are organized into a semantic hierarchy of six top-level classes, each with subclasses (full tree: docs/scoping_review/01_hierarchy.md):

• Visual — low-level static (luminance, contrast, color, edges, spatial-frequency slope, entropy), high-level static (object/scene semantics), faces/bodies/gaze, dynamic motion, action, and saliency/depth.
• Audio — low-level acoustics (loudness, spectral shape, MFCC, chroma, pitch, onset, tempo), high-level audio events/scenes, and speech.
• Language — lexical/word-level, syntactic/grammatical, and high-level semantic features computed from the transcript.
• Social — agents present, interaction type, dominance/affiliation, proximity.
• Situation — setting, event boundaries, and situational dimensions.
• Affect — depicted (expressed) and elicited emotion across face, voice, text, and fusion.

Each leaf of the hierarchy is a channel — a value series sampled on a common time grid (default 1 Hz). This taxonomy is the organizing backbone of both the scoping review and the on-disk format.

3. Models and algorithms

The pipeline implements 23 extractors. For each we name the model actually deployed and, where a lighter local stand-in was used, the best-in-class model it substitutes for (production targets). Local-first deployment runs on a single Apple-Silicon GPU (Metal/ MPS) or CPU; a small number of high-level reasoning features can optionally be routed to hosted large models.

3.1 Visual

• Low-level static (scikit-image / NumPy): per-frame luminance, RMS contrast, RGB/ HSV color means, Hasler–Süsstrunk colorfulness, Canny edge density, Shannon entropy, and the slope of the radially averaged Fourier power spectrum.
• High-level semantics: SigLIP 2 (sigmoid language–image pretraining) provides a per-frame image embedding and zero-shot probe scores over a text prompt set; DINOv2 provides a label-free self-supervised embedding. These are kept as distinct representations (interpretable text-probed vs. representational-similarity).
• Motion: dense optical flow via RAFT (substituting SEA-RAFT) → flow magnitude, a global-median “camera motion” term, and a residual “object motion” term.
• Depth: Depth-Anything-V2 → per-frame relative-depth statistics (mean, range, foreground fraction, depth entropy).
• Action: VideoMAE over sliding 16-frame clips → Kinetics-400 action posteriors and the top action label.
• Faces: MTCNN (facenet-pytorch; substituting InsightFace/OpenFace) → face count, presence, largest-face size, detection confidence; also feeds Social (agents present).
• Pose: torchvision Keypoint R-CNN (substituting MMPose RTMPose) → person count, mean keypoint confidence, and nearest-pair distance (a proximity proxy feeding Social).
• Saliency: spectral-residual saliency (substituting ViNet) → saliency mean/peak/ entropy and salient-area fraction.
• Shots: a color-histogram cut detector (substituting TransNetV2 + PySceneDetect) → cut events and a running shot index, feeding Visual, Situation, and Action.

3.2 Audio and speech

• Low-level acoustics (librosa): RMS, spectral centroid/bandwidth/rolloff/flatness, zero-crossing rate, onset strength, pitch (pYIN), tempo, MFCC and chroma vectors.
• Audio events/scenes: AST (Audio Spectrogram Transformer; substituting BEATs) → AudioSet-527 tag posteriors and top tag.
• Open-vocabulary audio: CLAP → an audio embedding and zero-shot scores over a prompt set (speech / music / noise / laughter / …), kept distinct from AST’s fixed taxonomy.
• Speech (the transcript hub): faster-whisper transcribes with word- and segment-level timestamps. The resulting transcript is attached to the pipeline and consumed by all language, social, situational, and text-affect passes — speech-present, word-rate, and per-bin text are emitted directly.
• Vocal affect: the audEERING wav2vec2 dimensional model → continuous voice arousal / dominance / valence (a depicted affect stream).

3.3 Language (from the transcript)

• Lexical (spaCy + wordfreq + optional psycholinguistic norms): per-word Zipf frequency, length, and — when the standard norm tables are supplied — valence, arousal, dominance, concreteness, and age-of-acquisition.
• Syntax (spaCy): per-utterance dependency-tree depth, mean dependency distance, content-word fraction, and noun/verb fractions.
• Surprisal: a GPT-2 language model → per-segment surprisal and entropy (bits) — the standard predictability regressors.

3.4 Social, situational, and affective; event structure

• Consolidated reasoning: a single Qwen2.5-VL vision-language pass per time window emits a structured JSON that populates Social (interaction type, dominance), Situation (setting, indoor/outdoor, scene description), and Affect (depicted emotion, valence, arousal) at once — replacing several separate high-level models with one pass.
• Text emotion: RoBERTa-GoEmotions → 28-way emotion scores over the transcript, a depicted (content) affect stream distinct from voice/face/elicited affect.
• Text sentiment: CardiffNLP twitter-roberta (trained on ~124M tweets + TweetEval) → negative/neutral/positive posteriors and a signed polarity scalar over the transcript, a coarse valence-polarity complement to the fine-grained GoEmotions categories.
• Image emotion (EmoNet): the Kragel et al. (2019) AlexNet-based model → a 20-way distribution over normative emotion categories for each whole frame (“emotion schemas embedded in the visual system”), a validated image-level depicted-affect signal.
• Facial affect: HSEmotion (EfficientNet trained on AffectNet, state-of-the-art facial-expression recognition) reads each MTCNN-detected face → eight expression posteriors plus continuous facial valence and arousal, averaged over faces per frame.
• Event segmentation: GSBS (Greedy State Boundary Search) runs as a post-pass over the assembled feature matrix, yielding data-driven event states and boundaries.

Affect is thus captured from five complementary sources — image (EmoNet), face (HSEmotion), voice (audEERING), text emotion (GoEmotions) and text sentiment (CardiffNLP), plus the scene-level VLM stream. A deliberate design principle is to keep complementary signals separate rather than collapse them — e.g., interpretable vs. representational embeddings, fixed-taxonomy vs. open-vocabulary audio tagging, and depicted vs. elicited affect.

4. Pipeline and output format

Media are decoded with PyAV (frames + audio) without a system ffmpeg dependency. Each extractor produces channels at its native rate; a common-grid resampler maps every channel onto the shared 1 Hz grid (center-referenced bins) with a per-channel reduction rule. Output is a single HDF5 file per stimulus with hierarchical groups mirroring the taxonomy, per-channel metadata (model, version, native rate, units, resample op), a reserved /human/ group for later human annotation, and a readable JSON sidecar manifest (full spec: docs/design/ANNOTATION_FORMAT.md).

A central feature is the constant-shape contract: an auto-generated channel template ensures every stimulus yields the same channel set; features that do not apply to a stimulus (e.g., visual features for an audio-only story, or a pass that was not run) are present with applicable=false and all-NaN. This makes the corpus directly stackable into rectangular matrices for analysis, and distinguishes not-applicable from not-measured from measured-zero. A lightweight MATLAB reader loads any annotation into structs and timetables.

5. Corpus

The analyzed corpus comprises 83 stimuli totaling ~470.6 minutes (~7.8 hours) across three modalities: 49 short audiovisual clips from a naturalistic-fMRI stimulus set, 4 short films (3 Creative-Commons Blender open films — Big Buck Bunny, Sintel, Tears of Steel — and Kung Fury), 29 spoken-story audio stimuli (~5.3 h) from the openly-released Narratives collection (Nastase et al., 2021; OpenNeuro ds002345), and 1 pure-text sample story — i.e. 53 audiovisual, 29 audio-only, and 1 text-only stimuli. Every stimulus was annotated under the same constant 95-channel template; classes that do not apply to a modality (e.g. visual features for an audio story) are present as applicable=false null skeletons. Stacked, the corpus yields 28,237 one-second timepoints, demonstrating that stimuli of very different modalities share one common analysis format.

Because the corpus is modality-mixed and now dominated (in timepoints) by audio stories, the channels populated across all stimuli are the audio, speech, and language features; the visual/social/situational channels are — correctly — undefined for the audio and text stimuli. Corpus-wide analysis of the shared scalar channels therefore reflects this audio/language core (26 channels survive a ≤40%-missing filter), while modality-specific structure is analyzed on the 53-stimulus audiovisual subset (below; 51 channels). Adding the spoken stories also transformed the corpus from dialogue-light to language-rich: word-level surprisal is defined in 91% of story timepoints, versus sparse coverage in the audiovisual clips.

The vision-language reasoning pass was run across the entire audiovisual subset, so its depicted-affect, social-interaction, and situational scalars are populated throughout that subset and enter the cross-class analysis below. Affect is in fact the most richly sourced class — image-level (EmoNet), face-level (HSEmotion), voice (audEERING), text emotion and sentiment, and the scene-level VLM stream — but the scalar cross-class analysis below is dominated by the dense VLM affect scalars: the face-level valence/arousal and dialogue-level sentiment are only defined when a face or speech is present, so across the intermittently-peopled, dialogue-light audiovisual clips they exceed the ≤40%-missing filter and drop out of the aggregate matrix (they remain fully available per-timepoint as fMRI regressors, and EmoNet’s dense 20-way image-emotion distribution is retained as a feature vector rather than a single scalar). One class remains under-represented in the scalar analysis for the same reason: language is richly populated by the audio stories but sparse in the dialogue-light audiovisual clips, so it survives the missing-data filter in the full corpus but not in the audiovisual subset. Interpret the couplings below accordingly.

6. The structure of the annotation space

6.1 Redundancy structure

The clustered cross-feature correlation matrix (Fig. 1, audiovisual subset) shows interpretable redundancy blocks — for example, the brightness channels (luminance, R/G/B and value means) are nearly collinear, and groups of spectral audio descriptors co-vary. These blocks are expected and motivate keeping a parsimonious core set plus complementary extras (cf. docs/scoping_review/08_redundancy.md).

6.2 Dimensionality

The annotation set is highly multidimensional. Across the full 83-stimulus corpus, the 26 audio/language channels shared by all modalities require 12 principal components to reach 80% of the variance (PC1–5 ≈ 50%). On the 53-stimulus audiovisual subset (Fig. 2), which adds the visual/social/situational/affective channels, 51 channels require 18 PCs to reach 80%. Either way, naturalistic feature space has many semi-independent axes to sample — encouraging for stimulus design.

6.3 Cross-feature and cross-class networks

On the audiovisual subset, thresholding the correlation matrix (|r| ≥ 0.30) gives a feature network whose nodes cluster by class (Fig. 3). After excluding categorical class-code channels (which must not be treated as magnitudes), the classes with valid scalar features here are affect, visual, audio, social, and situation (language drops out for the reason in §5). Aggregating to the class level (mean |r| between classes; Fig. 4), the cross-class couplings are modest: the strongest is affect ↔ social (0.15) — depicted emotion tracks the social-interaction structure — followed by visual ↔ social and audio ↔ social (both 0.11), then audio↔visual (0.07). Affect is otherwise weakly coupled to visual, audio, and situational features (|r| ≤ 0.06), and the situational features are nearly independent of the others (|r| ≤ 0.04). In the full corpus the audio stories carry no visual/social/affective channels, so the corpus-wide network is dominated by the audio↔language block — a direct consequence of the modality mix. (The affect↔social value here comes from the vision-language reasoning pass run across the audiovisual subset; an earlier draft reported a larger 0.26, which a code review found was inflated by treating the text_emotion_top class-index channel as a numeric feature — the 0.15 above uses only the model’s continuous valence/arousal scalars.)

6.4 Couplings are modest and roughly stable across content

Splitting the audiovisual subset by speech density (per-stimulus mean word rate, median split; Fig. 5) shows that the valid cross-class couplings are small and fairly stable across content — e.g. visual↔social is 0.13 in speech-rich vs 0.13 in speech-sparse stimuli. The annotation classes are thus largely non-redundant with one another regardless of stimulus type, which is favourable for stimulus design (independent axes to sample). Complementarily, adding the 29 spoken stories makes the corpus richly language-annotated (word-level surprisal defined in 91% of story timepoints) where the audiovisual clips were dialogue-light.

7. Experimental design from the annotation space

A direct application is principled stimulus selection. We split every stimulus into fixed-length candidate segments and greedily select a subset that maximizes the log-determinant of the covariance of the concatenated annotation time series projected onto the leading principal components. Maximizing log det(cov) simultaneously rewards high variance across the major feature dimensions and statistical independence (low cross-correlation) among the feature time series — a D-optimal design over the annotation space. Candidate segments dominated by missing (mean-imputed) data are excluded, so a segment cannot win the objective because its features are undefined. Over the full 83-stimulus corpus, selecting 20 ten-second segments yields a log det(cov) of 12.8 versus 2.5 for random selection (Fig. 6) — about 5× the generalized variance — while drawing segments from 16 distinct stimuli across modalities. This tool turns the corpus into an optimized stimulus set for an experiment targeting the annotation feature space.

8. Tools and reproducibility

The project ships the full pipeline (Python), a MATLAB analysis suite (readAnnotations, readAnnotationCorpus, analyzeCorpus, selectStimulusSet), an interactive movie-with-annotations viewer (plays a stimulus with its time series and a synced playback marker), and an interactive web segment-search interface (rank segments by any combination of features and play the matching moment). A runnable walkthrough (docs/walkthrough.m) and a full contents guide (docs/CONTENTS.md) document everyday use.

9. Limitations and future directions

• Local substitutions. Several passes use lighter local models (RAFT, MTCNN, Keypoint R-CNN, spectral-residual saliency, AST, histogram shot detection) in place of the best-in-class targets (SEA-RAFT, InsightFace/OpenFace, RTMPose, ViNet, BEATs, TransNetV2). Swapping in the production models is a configuration change.
• High-level reasoning coverage. The vision-language reasoning pass now covers the full audiovisual subset (depicted-affect, social, and situational scalars are populated throughout), but it is computationally heavy and runs at a 5 s cadence on a local model; scaling it (a larger model on dedicated hardware, finer or shot-sampled windows) would sharpen these channels and extend them to any future long-film additions.
• Elicited affect and diarization are not yet included (no off-the-shelf induced- affect model; speaker diarization needs gated model access).
• Modality balance and under-represented classes. Adding the 29 spoken stories made the corpus language-rich but also audio-heavy, so corpus-wide scalar analyses are dominated by the audio/language channels and visual/social/situational/affective structure is only defined on the audiovisual subset. One class remains under-represented in the scalar cross-class analysis: language (sparse in the dialogue-light audiovisual clips). Adding dialogue-rich audiovisual material would rebalance it. (The depicted-affect class, previously under-represented, is now populated across the audiovisual subset by the VLM pass; a code review separately removed a spurious categorical channel that had inflated the affect↔social coupling — see §6.3.)
• Validation. The annotations are validated here by face validity and internal structure; external validation against human ratings and brain data is future work.

10. Availability

All code, the channel template, the annotated corpus derivatives, analysis figures, the viewer, and the web interface are in the project repository. Start at docs/CONTENTS.md; the scoping review with full model citations is in docs/scoping_review/; the format specification is in docs/design/ANNOTATION_FORMAT.md.

Interactive segment browser

A static, in-browser tool that ranks every stimulus segment by any combination of annotation features and plays the matching moment. It reads a precomputed index (analysis/web/segments.json) and runs entirely client-side — no server logic.

▶ Open the segment browser

Pick any set of features, toggle each High or Low, and segments are ranked by their mean z-score across the chosen features (with a features-covered count when a segment lacks some channels). Interpretable vector channels are expanded per component, so — for example — EmoNet’s 20 emotion categories and the eight facial expressions are individually searchable. Rankings are face-valid: high optical flow surfaces action scenes, high word-rate the dialogue clips, EmoNet “Aesthetic Appreciation” the beach-sunset clips.

Playback & media. The ranking interface works anywhere. Actually playing a clip needs the source media served from the same origin — so playback works when you serve the project locally, but not on the public GitHub Pages site, where the licensed media are not hosted. To use it fully, serve the repository root with python3 tools/serve.py (a small Range-capable server, so video seeking works — the built-in python -m http.server cannot seek) and open analysis/web/index.html.

Notes from the app README

Interactive browser tool to rank movie/story segments by any combination of computational annotations and play the matching moment.

Launch

Serve from the project root with the bundled server, then open the page:

cd <project-root>
python3 tools/serve.py            # http://localhost:8000  (pass a port to change it)
# then open:  http://localhost:8000/analysis/web/index.html

Use tools/serve.py, not python -m http.server: the built-in server ignores HTTP Range requests, so <video> cannot seek — the scrub bar and “play segment” won’t jump. tools/serve.py adds byte-range support so seeking works. Serving from the root is also what lets the player load videos from /data/movies/...; opening index.html via file:// shows rankings but cannot play clips.

Use

• Filter/scroll the feature list (grouped by class); tick features to search on.
• Toggle each picked feature High / Low.
• Optionally filter by source and choose how many results to show.
• Segments are ranked by the mean z-score of the selected features. Each result has ▶ Segment (seeks to its start, auto-stops at its end) and ▶ Clip (plays the whole stimulus); the player repeats these for the loaded segment.
• Selected features are plotted as a time series over the whole clip, with a marker that tracks playback. Click the plot to jump the video to that time; the video scrub bar works too.

Rebuild the index

segments.json is generated from the annotated corpus. Regenerate after annotating more stimuli (or to change the window length):

PYTHONPATH=src .venv/bin/python tools/build_search_index.py --seglen 5

Segment Search — web interface

Interactive browser tool to rank movie/story segments by any combination of computational annotations and play the matching moment.

Launch

Serve from the project root with the bundled server, then open the page:

cd <project-root>
python3 tools/serve.py            # http://localhost:8000  (pass a port to change it)
# then open:  http://localhost:8000/analysis/web/index.html

Use tools/serve.py, not python -m http.server: the built-in server ignores HTTP Range requests, so <video> cannot seek — the scrub bar and “play segment” won’t jump. tools/serve.py adds byte-range support so seeking works. Serving from the root is also what lets the player load videos from /data/movies/...; opening index.html via file:// shows rankings but cannot play clips.

Use

• Filter/scroll the feature list (grouped by class); tick features to search on.
• Toggle each picked feature High / Low.
• Optionally filter by source and choose how many results to show.
• Segments are ranked by the mean z-score of the selected features. Each result has ▶ Segment (seeks to its start, auto-stops at its end) and ▶ Clip (plays the whole stimulus); the player repeats these for the loaded segment.
• Selected features are plotted as a time series over the whole clip, with a marker that tracks playback. Click the plot to jump the video to that time; the video scrub bar works too.

Rebuild the index

segments.json is generated from the annotated corpus. Regenerate after annotating more stimuli (or to change the window length):

PYTHONPATH=src .venv/bin/python tools/build_search_index.py --seglen 5

Deploying the documentation site (GitHub Pages)

The documentation book is published to GitHub Pages from this repository at https://canlab.github.io/narrative_feature_annotations/. A GitHub Actions workflow (.github/workflows/deploy-book.yml) rebuilds and republishes it on every push to main.

What lives in git vs. Dropbox

This project mixes small, shareable text with large, copyrighted media. The split is:

Why the media and annotations stay out of git:

• Media exceed GitHub’s 100 MB per-file limit and are copyrighted (the spacetop clips, the Narratives audio, and Kung Fury are third-party IP). Only the CC-BY Blender open films are freely redistributable.
• Annotations are large derivatives (and their ASR transcripts reproduce film dialogue). They are regenerable from the media, so they are kept in Dropbox and can be published separately (e.g. OSF/Zenodo with a DOI) if a citable dataset is wanted.

The published site does not need either: it is built from the docs, the figures, and the interactive search index, all of which are committed.

One-time setup

cd "<this folder>"

# 1. Initialize git and keep Dropbox out of git's internals (see "Dropbox" below).
git init
xattr -w com.dropbox.ignored 1 .git      # macOS: stop Dropbox syncing the .git folder

# 2. Stage everything; .gitignore already excludes media, annotations, .venv, book build.
git add -A
git status                                 # sanity check: no .mp4/.wav/.h5 should appear

# 3. First commit.
git commit -m "Narrative Feature Extraction: pipeline, docs, and documentation book"

# 4. Create the GitHub repo and push (org: canlab).
git branch -M main
gh repo create canlab/narrative_feature_annotations --source=. --remote=origin --push --public
#   …or, if the repo already exists:
#   git remote add origin https://github.com/canlab/narrative_feature_annotations.git
#   git push -u origin main

Then enable Pages once: repo Settings → Pages → Build and deployment → Source: “GitHub Actions”. The next push (or a manual run of the Deploy documentation book workflow) publishes the site. GitHub Pages requires a public repo on the free plan.

Updating the site

Edit the Markdown under docs/ (or README.md), commit, and push — the workflow regenerates the book and redeploys. To preview locally first:

brew install mdbook                  # one-time; or: cargo install mdbook
python3 tools/build_book.py          # assemble book/src/ from docs/
mdbook serve book                    # live preview at http://localhost:3000

book/src/ and book/book/ are generated (git-ignored); never edit them by hand.

Regenerating the annotations (not in the repo)

Because annotations/ is Dropbox-only, a fresh git clone has the code and docs but not the derivatives. To rebuild them you need the media (also Dropbox-only) in place, then:

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
PYTHONPATH=src .venv/bin/python -m nfe.batch --manifest data/manifest.csv \
    --out annotations/corpus --template schema/channel_template.json \
    --vision --audio-hl --events
# then refresh derivatives the site uses:
PYTHONPATH=src .venv/bin/python tools/build_search_index.py    # analysis/web/segments.json
# (MATLAB) refreshAnalysis("annotations/corpus")               # analysis/figures + stats

See ADDING_MOVIES.md for the full refresh workflow.

Dropbox + git

This working folder lives in Dropbox. That is fine — GitHub Pages does not care where the local folder is — with one caveat: Dropbox continuously syncs the .git folder, which across multiple machines (or if it syncs mid-commit) can create “conflicted copy” files inside .git and corrupt the repo. Mitigations:

• Tell Dropbox to ignore .git (the xattr command above), so Dropbox leaves git alone.
• The arrangement is otherwise complementary: Dropbox holds the large media + annotations; GitHub holds the code + docs. Don’t sync the repo across machines via Dropbox — use git clone/pull/push for that and let Dropbox carry only the media.