Situation features

Situations, schemas, scripts & event segmentation

This subclass spans two complementary tool families: (1) data-driven event/state boundary detection that segments a continuous feature timeline (neural data, or stimulus-derived embeddings such as visual/semantic features) into discrete events with stable patterns, and (2) situational tagging that labels each segment or timepoint with setting, location, time-of-day, scripts/schemas, and the Event-Indexing situational dimensions (space, time, causation, intention, protagonist). For boundary detection, BrainIAK’s HMM (Baldassano 2017) and Geerligs’ GSBS (statesegmentation) are the canonical, locally deployable algorithms; both run on any feature matrix, not just fMRI, making them directly usable to segment movie-derived embedding timelines. Shot/scene cut detection (PySceneDetect, TransNetV2) gives a fast low-level event-boundary prior. For situational content, scene-recognition CNNs (Places365), movie-tuned CLIP (MovieCLIP), and zero-shot CLIP cover location/setting/time-of-day, while modern open-weight video-LLMs (Qwen2.5-VL, VideoLLaMA3) and text/transcript LLMs (GPT-4-class via prompting) provide human-like event segmentation plus rich, queryable situational dimensions (goals, causes, protagonist, schema/script labels) with second-level temporal grounding. A small lab can run the algorithmic tools and 7B-class VLMs on a single modern GPU.

Recommended (best-in-class): GSBS / statesegmentation (primary data-driven event-boundary detector on stimulus feature timelines); BrainIAK EventSegment HMM (alternative boundary detector + soft event posteriors); TransNetV2 (best shot-boundary prior for video); Qwen2.5-VL-7B (open video-LLM for second-level event segmentation + situational dimensions); Zero-shot OpenCLIP (flexible per-frame location/time-of-day/setting/script attributes); Places365-ResNet50 (calibrated fixed-taxonomy location/setting + scene attributes); GPT-4-class / Llama-3-70B LLM (transcript-based human-like event segmentation + Event-Indexing situational tagging, esp. audio stories)

Likely redundant:

• PySceneDetect -> superseded for accuracy by TransNetV2 (keep only as a fast CPU prior)
• MovieCLIP -> overlaps zero-shot CLIP/Places365; use only if its 179 movie-scene taxonomy is a better fit and checkpoints are accessible
• VideoLLaMA3 -> redundant with Qwen2.5-VL as the situational video-LLM; pick one primary
• BrainIAK HMM vs GSBS -> overlapping boundary detectors; GSBS generally preferred (auto-K, faster), keep HMM mainly for soft event posteriors / recall-reactivation modeling
• Indexter -> superseded as a runnable tool by LLM situational tagging; retain only as the target ontology/schema definition

Open questions:

• Ground-truth/validation: which event-boundary timeline (human dense-rating norms, e.g., per-second boundary agreement) will be used to validate algorithmic vs LLM/VLM boundaries on your specific movies/audio stimuli?
• Pipeline order: run shot detection (TransNetV2) first and tag shots, vs run GSBS/HMM on a continuous embedding timeline (from CLIP/VLM/text embeddings) to get semantic events independent of camera cuts? These give different granularities.
• For audio-only stories, the visual tools do not apply; need to confirm WhisperX-style word-level alignment to project transcript LLM event boundaries and situational tags onto the audio timeline.
• Structured output schema: define a fixed JSON schema for the five Event-Indexing dimensions (space/time/causation/intention/protagonist) + script/schema label so VLM/LLM outputs are consistently parseable per second/segment.
• Compute budget: is a single 24GB GPU available for 7B video-LLM inference, or should situational tagging fall back to API (GPT-4-class) for the visual stream?
• Which feature space feeds GSBS/HMM for stimulus segmentation (raw CLIP frame embeddings, VLM hidden states, semantic transcript embeddings) materially changes the events recovered and should be decided/benchmarked.