The hook

The Open ASR Leaderboard has a clear #1 this month: a 30B mixture-of-experts model. Below it sit models a tenth its size. So, a simple question: point all of them at real news broadcasts (studio anchors, field reporters, press conferences, crowd noise). Does the order hold?

It doesn't. On one rented H100 the seven land within one WER point of each other, and a paired test can't tell any two apart. What does differ is cost.

The setup, in one picture

Wikimedia Commons news (VOA, White House, Al Jazeera)
   │ ffmpeg once → 16 kHz mono Opus (byte-identical audio for every model)
   ▼
human subtitles ──► 15–30 s windows cut at cue boundaries
   ▼
7 models × same windows · 1× H100 · batch 16 · no language hint
   ▼
corpus WER · 95% CI by bootstrapping *videos* · paired test vs the best

Bootstrapping videos, not windows, matters: twelve windows from one briefing share a room and a microphone. Clustered intervals are wider, and honest.

Why it looks different from the leaderboard

Benchmark fitting. On the leaderboard's own private held-out set the #1 slot changes hands. Part of a public lead is training close to the test sets' conventions.
Style match. Subtitles are edited. Verbatim models (Nemotron inserted 628 words, 9% of them stutter repeats) get billed for faithfulness; a "clean" model (parakeet-unified, 333) matches the captioner. Our WER partly measures style.
Resolution. With 31 broadcasts the paired interval is about ±1 point, wider than the leaderboard's top spread. "Tied at this resolution" is the honest claim.
Serving stack. Speed here is plain Transformers. Tuned engines (vLLM, NeMo, NIM) are faster; this is a fair relative comparison, not a ceiling.

The dirty laundry

Whisper memorised White House transcripts. On crosstalk it loops The Press-. The Press-. The Press-., the format of the web transcripts it trained on. It did this in 7 windows; no other model did.
The reference was sometimes wrong and the models right. Speaker labels (The Vice President:) and captioner errors cost every model points. A pre-registered rule flagged 5 of 331 windows; the ranking didn't move.
One long window sank a whole batch. A 30 s+ cue crashed Whisper's batched call and failed all 16 windows in it (WER read 15.5% instead of 7.6%). Retry failed batches item by item.
A subtitle isn't always a transcript. English subtitles over Russian speech are translations; score only windows where the languages match.

The recipe

On English broadcast audio, don't pay for size. Pick by throughput and memory: at ~$3/h for a spot H100, an hour of news costs ~$0.003 with the fastest model and ~$0.06 with the slowest, at the same accuracy.
Pick by language coverage. The most accurate English model here is English-only. Unknown languages need a multilingual model or a language-ID router.
Check your reference before trusting a half-point gap.

What's next

The same windows go to closed models (Gemini 3.5 Transcribe, Chirp 3) and the open models this run couldn't execute. Then the real test for the 30B model: can one model doing transcription, speaker labels and timestamps replace a three-model cascade?

Model	WER	95% CI	Speed	Peak VRAM	Fits on
Parakeet Unified EN 0.6B	7.34	5.2–10.0	597x	8.2 GB	~12 GB GPU
Qwen3-ASR 1.7B	7.58	5.5–9.8	89x	10.7 GB	~12 GB GPU
Whisper large-v3-turbo	7.64	5.1–10.6	115x	5.5 GB	~8 GB GPU
Parakeet TDT 0.6B v3	7.97	5.9–10.4	658x	8.1 GB	~12 GB GPU
Nemotron-3.5-Transcribe 30B-A4B	8.00	5.8–10.5	49x	71.8 GB	~80 GB GPU
Granite Speech 5.0 2B	8.02	5.9–10.5	13x*	4.8 GB	~8 GB GPU
freetranscribe parakeet-ft-v1	8.28	6.1–10.8	1117x	8.4 GB	~12 GB GPU

Seven speech models on real news broadcasts: the leaderboard #1 ties a 0.6B model

Word error rate (%) · lower is better

Speed · times faster than real time (log scale)