Finding story boundaries in TV news bulletins: LLMs vs TextTiling

The hookLink to this section

A 30-minute bulletin is a dozen stories glued together, and a short should be one of them. So the pipeline's first editorial step is finding where each story starts.

The classic answer is TextTiling: embed the transcript and cut where the topic shifts. On 21 real bulletins it finds about half the boundaries. An LLM reading the same transcript matches the reference about as well as the reference matches itself.

The setup, in one pictureLink to this section

21 real bulletins from Wikimedia Commons · 6.96 h
   (VOA Africa 54, VOA Headlines, Martí Noticias, a Russian local station)
   │ Chirp 3 transcripts with word times, 58,236 words
   ▼
reference: Gemini 4 Argon watches each video, prompt A
ceiling:   the same model, prompt B, worded differently
   ▼
methods from the transcript only: LLMs vs TextTiling (Qwen3-Embedding)
   ▼
boundary F1 within ±5 s and ±15 s · Pk · bootstrap by bulletin

No human story labels exist for these bulletins, so the reference is model-made ("silver"). Running the labeller twice with different prompts gives a ceiling: a method inside that noise can't be ranked below the labeller.

What the numbers sayLink to this section

• The transcript is nearly enough. Gemini 4 Argon on the transcript scores 86.7 F1 at ±15 s, against an agreement ceiling of 87.8 (difference −1.1, 95% CI −5.7 to +3.8).
• Open models are close behind. Gemma 4 26B scores 80.9 and DeepSeek V4.1 Flash 78.6: 6–8 points behind the closed model, and 12+ points ahead of every embedding method.
• TextTiling fails on real news. Told the right number of boundaries, it still finds two-thirds of them (66.7); on its own cutoff it more than doubles the count and scores 53.8.
• Text gets the place, not the second. Gemini loses 7 points going from ±15 s to ±5 s, so boundaries from text should be snapped to nearby pauses and shot changes.

The dirty laundryLink to this section

• Real bulletins are hard for the reference too. Two prompts on the same model agree at 87.8, against 99% on joined synthetic stories. Most disagreement is granularity: is a teaser, a brief or an interview its own story? One Africa 54 episode had 18 stories under one prompt and 10 under the other.
• Same-family bias. The reference and the best method are both Gemini 4 Argon, and Gemma is also Google's. DeepSeek beats Gemma on general benchmarks but scores lower here. A human-checked slice is the way to settle it.
• A transcription job never finished. One Chirp 3 batch never reported done, twice, for 50+ minutes. Polling now gives up after 20 minutes and resubmits.

The recipeLink to this section

1. Segment stories with an LLM on the transcript, not with embedding similarity.
2. Snap the boundaries to pauses (stage 3) and shot changes before cutting.
3. If you build silver labels, run the labeller twice with different prompts and report that agreement as your ceiling.

What's nextLink to this section

A human-checked slice, and a second labeller from another model family.