Labelling news bulletin segments: can cheap models match a frontier one?

The hookLink to this section

Once a bulletin is split into stories, the pipeline needs to know what each stretch is: the anchor reading, a pre-recorded package, a soundbite from someone in the story, an interview, the opening titles. Soundbites especially matter, because a short is usually built around one.

A frontier video model labels this consistently. The question was whether a cheaper one could do the same job. Not yet: the cheap models land about 15 points below the frontier model's agreement with itself.

The setup, in one pictureLink to this section

21 real bulletins (6.96 h; English, Spanish, Russian)
   │ each model watches the whole bulletin with sound
   ▼
consecutive segments, one role each:
anchor · package · soundbite · standup · interview · graphic · titles · other
   ▼
compare on a 10-second grid (2,504 cells)
reference: Gemini 4 Argon, prompt A · ceiling: same model, prompt B
   ▼
accuracy · macro-F1 (95% CI, by bulletin) · tokens per hour of video

What the numbers sayLink to this section

• The frontier model agrees with itself; the cheap ones don't come close. Gemini 4 Argon's two prompts agree on 88.6% of cells (macro-F1 86.5). Gemini 3.5 Flash-Lite reaches 73.5% and Gemini 3.5 Flash 71.4%.
• They fail in different places. Flash misses soundbites (recall 43%); Flash-Lite finds them (82%) but is worse on anchors. Neither finds ads and filler well.
• Open models are input-bound, not dumb. The API can't give them a whole video with sound, so DeepSeek V4.1 Flash got the transcript (64.7%) and Gemma 4 26B the transcript plus keyframe contact sheets (56.4%). Without seeing who is on screen, a soundbite and a package read the same, and their soundbite recall drops to 17–28%.
• Cost follows price per token. Video input is about 330,000–380,000 tokens per hour for every Gemini model, so the bill is set by the model's rate.

The dirty laundryLink to this section

• The reference is one model family. Reference and candidates are all Gemini, so shared blind spots are invisible. A second family was set up and blocked by a zero quota, so the cross-check waits.
• Some roles are too rare to score. Standup had 10 reference cells and graphic 2, so we also report macro-F1 over roles with 50+ cells.
• Timeouts cut off long episodes. A 900 s read timeout killed three calls on 30-minute episodes. Raised to 1,800 s, and failed rows are now retried instead of counted as done.
• A 10-second grid blurs short soundbites at segment edges.

The recipeLink to this section

1. For role labels, use the frontier video model (Gemini 4 Argon here, about 380,000 input tokens per video hour).
2. If you must go cheap, take the cheapest: Flash-Lite isn't worse than Flash overall, and it is better on soundbites.
3. Don't judge open models on a task you've starved of input. Give them the pictures, or expect them to guess.

What's nextLink to this section

A second-opinion labeller from another model family, and a cheaper model that closes the 15-point gap.