Does translating a news clip change its meaning? Six translators tested

The hookLink to this section

A news short in English can reach far more people with Hindi or Spanish subtitles. But translation is another edit. Drop a "reportedly" or flip a negation and a fair clip starts to mislead; smooth over a contrast and a misleading one looks fine.

Translation is usually scored by how close it is to a reference (chrF, COMET). We asked a different question: does the verdict survive? Is a fair clip still fair after translation, and a misleading one still misleading?

The setup, in one pictureLink to this section

stage-10 clips: 108 faithful + misleading, from 54 real stories
   │ English → Hindi and Spanish · Spanish and Russian → English
   ▼
202 translations per translator
   Cloud Translation NMT · Cloud Translation LLM · Gemini 4 Argon
   Gemini 3.5 Flash · DeepSeek V4.1 Flash · Gemma 4 26B
   ▼
the stage-10 judge re-judges each translated clip against the source story
   ▼
verdict kept % · flips each way · numbers kept · chrF++ vs Argon's translation
noise floor: the same judge re-run on the original clips

The noise floor matters. The judge agrees with itself on 95.4% of the original clips, so a translator near 95% adds almost nothing. Without the floor, every translator would look like it changes 5–9% of verdicts.

What the numbers sayLink to this section

• Translation mostly keeps the verdict. Every translator keeps 90.6–94.1% of verdicts against the 95.4% floor, so translation adds at most about 5 points of flips.
• The open LLMs are closest to the floor. Gemma 4 26B and DeepSeek V4.1 Flash both keep 94.1% of verdicts, and Gemma keeps the most numbers (96.4%).
• Cloud NMT is the riskiest. It turns the most fair clips into misleading ones (12 of 202, about twice the open LLMs) and drops more numbers.
• chrF isn't faithfulness. Cloud Translation LLM scores lowest on chrF yet keeps verdicts as well as Gemini Flash. Part of a low chrF is simply style distance from an Argon reference.

The dirty laundryLink to this section

• One judge, one reference. The chrF reference and the judge are both Gemini 4 Argon, which favours Gemini's style and shares a family with two translators.
• The number check is crude. It misses numbers written as words and counts reformatted times ("1:00") as lost. Read it as relative.
• Small non-English sets, and no COMET or human check in this run.

The recipeLink to this section

1. Translate news subtitles with an open LLM: Gemma 4 26B, with DeepSeek V4.1 Flash as the alternative.
2. Avoid classic NMT for news shorts unless a verifier re-checks the translated clip.
3. Always measure your judge's noise floor before blaming the thing you're testing.

What's nextLink to this section

Human translations for a proper reference, COMET on a GPU run, and more target languages.