Can AI catch a news clip that misleads out of context? 10 judges tested

The hookLink to this section

A clip is an excerpt, so every word in it was really said. That is exactly why it can mislead: end a clip one sentence before "...but the court overturned that ruling" and the viewer leaves believing the opposite of the story.

Fact-checking models can't catch this. They ask "is this supported by the source?", and the answer is always yes. We built 162 traps from real news to measure which judges can catch it. The standard support checker finds 2% of them.

The setup, in one pictureLink to this section

150 real bulletin stories (40 s+, 5+ sentences)
   │ generator returns three whole-sentence clips per story:
   │   faithful · trimmed (shorter, still fair) · misleading (ONE edge moved)
   │ 94 stories declined: no honest misleading cut exists
   ▼
54 stories · 162 clips · truth by construction
   ▼
10 judges see the full story + one clip, shuffled, and answer: misleading?
   ▼
macro-F1 (95% CI, by story) · recall on traps · false alarms on fair trims

The "trimmed" clip is the hard negative: missing detail isn't misleading, and a judge that flags every short clip is useless.

What the numbers sayLink to this section

• Gemini 4 Argon leads at macro-F1 0.847 (95% CI 0.785–0.904).
• A good open verifier exists. GLM-5.2 scores 0.814, within about 3 points with overlapping intervals, and has the fewest false alarms on fair trims (7%). It rarely blocks a good clip.
• The fact-checker is blind by design. MiniCheck says the story supports 98% of misleading clips, which is literally true. It catches 2%.
• Judges split into over-flaggers and under-flaggers. gpt-oss-120b catches 83% of traps but flags 43% of fair trims. DeepSeek V4.1 Flash, Gemma 4 26B and Gemini 3.5 Flash-Lite are polite and miss 65–87% of traps.
• Being good at one text task doesn't predict this one. DeepSeek ranked stories well in stage 9 and judges poorly here.

The most common trap was a dropped correction or contrast (26 of 54), then a dropped outcome (10) and a dropped qualifier (9).

The dirty laundryLink to this section

• The generator disagrees with itself. The same model built every trap, yet under a second prompt it calls only 54% of its own misleading clips misleading. Out-of-context distortion is a judgment call, which is why absolute accuracy needs human labels.
• Most stories can't be made misleading by moving one edge: 94 of 150 were declined. Good news for a pipeline that cuts whole sentences inside one story.
• Text only. Judges read transcripts; pictures can also add or remove context.
• Operational traps: gcloud handed back near-expiry tokens and 116 calls got 401s; some calls hung until a 30-minute timeout. Tokens now refresh every 5 minutes and timed-out calls are retried.

The recipeLink to this section

1. Never use a grounding fact-checker to verify clips. It can't see a context cut.
2. Use a reasoning judge that sees the whole story. GLM-5.2 is our open verifier: near the frontier, fewest false alarms.
3. Score false alarms on fair-but-shorter clips, or you'll pick a judge that blocks everything.

What's nextLink to this section

Human-labelled traps, transfer tests on public misinformation sets, and a judge that sees pictures as well as words.