Which news stories make good shorts? LLM rankings vs their own ceiling

The hookLink to this section

Which story from tonight's bulletin should become a short, and which 40 seconds of it? Every "AI clipping" tool claims to answer this. We asked four models to rank 174 real stories and pick the cut, and checked first how much the reference model agrees with itself.

It agrees only so-so. The same frontier model, asked with different wording, picks the same top story 58% of the time. Story choice is a judgment call. Where to cut is not: that part needs the video.

The setup, in one pictureLink to this section

19 real bulletins · stage-7 story starts · stage-8 roles
   │ drop titles/teasers/sign-offs and stories under 20 s
   ▼
174 stories (3–15 per bulletin)
   │ rank every story as a standalone short; give a 20–60 s span for the top 3
   ▼
Gemini models: video + sound · open models: transcript (+ keyframes for Gemma 4)
   ▼
Spearman per bulletin · top-1 · top-3 overlap (chance 43.3%) · clip IoU
reference: Gemini 4 Argon, prompt A · ceiling: same model, prompt B

What the numbers sayLink to this section

• The ceiling is low. Gemini 4 Argon agrees with itself at Spearman 0.696 (95% CI 0.53–0.84) and on the top story 57.9% of the time. Silver labels can rank models here only coarsely.
• Open models choose stories as well as the cheaper closed model. Gemma 4 26B (0.660) and DeepSeek V4.1 Flash (0.600) sit at or above Gemini 3.5 Flash (0.564), with overlapping intervals. Choosing a story runs mostly on what is said.
• Cutting needs pictures. Video models agree on clip spans (IoU 0.65–0.77); transcript-only models don't (0.43–0.45), even when they pick the same story. The cut depends on delivery and visuals, not words.
• Clips behaved. Spans stayed inside 20–60 s 93–100% of the time, with medians of 40–54 s.

The dirty laundryLink to this section

• No human or audience truth yet. This stage needs people more than any other so far.
• Same-family reference for the video rows.
• Small bulletins inflate overlap. Russian bulletins have only 3–4 stories, so top-3 overlap is easy there.
• Clip IoU is computed only where both models put the story in their top 3.

The recipeLink to this section

1. Split the task by input. Rank stories on the transcript with a cheap open model (Gemma 4 or DeepSeek V4.1 Flash).
2. Send only the top stories to a video model to pick the cut.
3. Measure your reference's self-agreement before trusting any ranking on a subjective task.

What's nextLink to this section

Human preference data: a small panel, or replay data like Mr. HiSum (see stage 13).