Audio features

Low-level acoustic features

Per-frame low-level acoustic descriptors (loudness/RMS, F0/pitch, spectral centroid/rolloff/flux, MFCCs, chroma, onsets, tempo/beat, harmonicity, roughness, voice-quality jitter/shimmer/HNR) are produced by a mature, mostly open-source ecosystem. The core trio is librosa (most flexible general MIR/DSP in pure Python), openSMILE (the standardized affective-computing feature sets eGeMAPS/ComParE used heavily in speech/emotion neuroscience), and Praat via Parselmouth (the reference for voice-quality and F0). Essentia adds a fast C++ engine with a richer set of music/psychoacoustic descriptors (dissonance/roughness, beat tracking) and torchaudio provides GPU-batched, differentiable versions of standard transforms. For pitch specifically, neural CREPE and probabilistic pYIN exceed classic autocorrelation/cepstral trackers in accuracy. All are CPU-friendly and trivially deployable in a small lab; only CREPE benefits from (but does not require) a GPU. For a movie/audio-timeline pipeline, a small overlapping set (librosa + openSMILE eGeMAPS + Parselmouth, optionally Essentia for music-rich roughness/beat and CREPE for high-accuracy F0) covers the entire subclass with minimal redundancy.

Recommended (best-in-class): librosa (primary general spectral/MFCC/chroma/onset/tempo + pyin F0); openSMILE eGeMAPS/ComParE via opensmile-python (standardized speech/affect LLDs, timeline-indexed); Parselmouth/Praat (gold-standard F0, formants, jitter/shimmer/HNR voice quality); Essentia (add for music-rich roughness/dissonance, inharmonicity, robust beat tracking); CREPE/torchcrepe (add for high-accuracy monophonic F0 on speech/vocals)

Likely redundant:

• torchaudio — spectral/MFCC fully superseded by librosa for a small lab; keep only if GPU-batched or differentiable front-ends are needed
• torchaudio.detect_pitch_frequency and librosa.pyin F0 — superseded by CREPE/torchcrepe (accuracy) or Praat (voice science) for the F0 feature
• openSMILE jitter/shimmer/HNR/formant LLDs overlap Parselmouth — run both only for cross-validation; otherwise Parselmouth is canonical and openSMILE provides the standardized 88-dim summary
• Essentia spectral/MFCC/chroma/onset overlap librosa — choose one as the primary MIR engine (librosa for ease, Essentia for speed/roughness), not both for the same descriptors
• Essentia ‘dissonance’ vs mosqito roughness — overlapping roughness measures; mosqito is the standards-based (asper) choice if perceptual fidelity matters

Open questions:

• Will the pipeline analyze the full mixed soundtrack (speech+music+SFX) or source-separated stems? Monophonic F0 tools (CREPE, Praat) and voice-quality measures degrade badly on mixes — may need Demucs/source separation upstream.
• Target output rate: true per-frame (~10-23 ms) vs aggregated per-second for fMRI/timeline alignment? This sets hop sizes and whether to keep LLDs or functionals.
• Is openSMILE’s source-available (non-OSI) license and Essentia’s AGPL acceptable for the project’s distribution/sharing model?
• Are standardized perceptual units (sones/asper for loudness/roughness via mosqito) required, or are relative librosa/Essentia descriptors sufficient?
• Sample rate harmonization: openSMILE/Praat expect specific rates; movie audio is often 48 kHz — confirm resampling strategy to keep features comparable across tools.

High-level audio: events, scenes, music, speech/non-speech

This subclass covers time-resolved semantic audio annotation: (1) audio tagging / sound event detection over the 527-class AudioSet ontology, (2) acoustic scene classification, (3) speech vs music vs noise segmentation, and (4) music information retrieval (tempo, key/mode, beat/downbeat, instrumentation, mood/emotion). The field is mature and largely open-source. For a movie/audio-story timeline, the recommended stack is a transformer audio tagger (BEATs or PaSST/AST) run on a sliding window for AudioSet event posteriors, CLAP for open-vocabulary/zero-shot event labels and audio-text similarity, inaSpeechSegmenter or pyannote VAD for speech/music/noise segmentation, and Essentia + madmom (or Beat This!) for MIR descriptors. Most tools run comfortably on a single consumer GPU or even CPU; nothing here requires heavy multi-GPU compute. Key design choice: window size sets temporal granularity (AudioSet models are trained on ~10 s clips but can be run on ~1 s hops for per-second annotation, with the caveat that very short windows degrade accuracy on slow-onset events).

Recommended (best-in-class): BEATs (or PaSST) for AudioSet event/scene tagging on a sliding window; CLAP (laion/larger_clap_general + music_and_speech) for open-vocabulary zero-shot audio dimensions and audio-text similarity time series; inaSpeechSegmenter for clean speech/music/noise segmentation (add pyannote VAD/diarization if speaker turns needed); Essentia + TF models for MIR scalars and music mood/genre/instrument tags; madmom or Beat This! for beat/downbeat/tempo structure

Likely redundant:

• AST is largely superseded by BEATs/PaSST on AudioSet mAP (keep only for its trivial HF deployment)
• PANNs CNN14 superseded by transformer taggers, but retain for framewise SED and panns_inference simplicity
• YAMNet superseded on accuracy by all transformer taggers; keep only for CPU/real-time/edge constraints
• Choose ONE primary AudioSet tagger (BEATs or PaSST) rather than running AST+PANNs+YAMNet+BEATs together, since their 527-class outputs are highly correlated
• Essentia tempo estimator is redundant with madmom/Beat This! beat tracking (prefer the latter when precise beat/downbeat timing matters)
• CLAP music tags partially overlap Essentia genre/mood models; use CLAP for novel/custom prompts and Essentia for calibrated valence-arousal/tempo/key

Speech: ASR, diarization, prosody, vocal affect

This subclass converts the speech audio of movies/stories into a time-aligned linguistic and paralinguistic layer: word/segment-level transcripts with timestamps (ASR + forced alignment), who-spoke-when (diarization + VAD), low-level prosody (pitch/intensity/voice-quality contours), and emotion-from-voice (categorical and dimensional A/V/D). The open ecosystem is mature and almost entirely locally deployable on a single consumer GPU. A practical pipeline is faster-whisper/WhisperX for ASR+word timestamps, pyannote (community-1) for diarization, Parselmouth/openSMILE for prosody, and an audEERING wav2vec2-dim or emotion2vec model for vocal affect; most other tools are redundant variants of these. Key design choice is per-word vs per-second resampling, since ASR/SER are utterance-level while prosody is natively frame-level (10 ms).

Recommended (best-in-class): faster-whisper (large-v3) for ASR; WhisperX for word-level forced alignment + speaker-attributed transcript; pyannote speaker-diarization-community-1 for diarization/VAD/overlap; Parselmouth (Praat) for prosody and voice-quality contours; audEERING wav2vec2-large-robust-12-ft-emotion-msp-dim for dimensional vocal affect (arousal/valence/dominance)

Likely redundant:

• openai/whisper, insanely-fast-whisper, whisper.cpp -> superseded by faster-whisper (same weights, faster CTranslate2 runtime)
• pyannote speaker-diarization-3.1 -> superseded by community-1 (better speaker counting, same segmentation)
• WebRTC VAD / audiotok -> superseded by Silero VAD and pyannote VAD
• librosa pitch (pyin/piptrack) and standalone MFCC extraction -> superseded by Parselmouth (F0/voice-quality) + openSMILE (spectral LLDs)
• Generic HF wav2vec2/HuBERT IEMOCAP categorical SER checkpoints -> superseded by emotion2vec+ (categorical) and audEERING wav2vec2-dim (dimensional)
• NeMo ASR/diarization and emotion2vec+ -> overlap with the Whisper/pyannote/audEERING defaults; keep only as accuracy or multilingual/cross-domain alternatives

Open questions:

• Target temporal grid: per-word (event-locked) vs fixed per-second resampling - ASR/SER are utterance-level and need windowing/interpolation, whereas Parselmouth/openSMILE are natively 10 ms.
• Movie audio is mixed with music/SFX/overlapping speakers; do we need source separation (e.g. Demucs/Bandit) or a music-VAD gate before ASR/SER to avoid contaminating pitch/jitter/shimmer and emotion estimates?
• Vocal affect should be computed per diarized speaker segment, not on mixed audio - confirm diarization-first ordering and how to handle overlapped speech for SER/prosody.
• Licensing: openSMILE eGeMAPS and audEERING wav2vec2-dim are research-only; confirm acceptable for the study or substitute emotion2vec+/Parselmouth-only features.
• Whether to add a paralinguistics/laughter-cry/non-verbal-vocalization detector (e.g. emotion2vec vocal events or a dedicated laughter model) as a separate feature stream.