Speaker diarization on news video: Nemotron-3 vs pyannote vs DiariZen

The hookLink to this section

On VoxConverse, the standard diarization benchmark, Nemotron-3 is the clear winner. So we pointed it and two other open diarizers at real broadcast-style video: White House briefings and roundtables, where human subtitles already say who is speaking ("The President:", "Secretary Azar:").

On ordinary events, all three are excellent. On panels, the leader quietly falls apart, and the public benchmark had no way to show it.

The setup, in one pictureLink to this section

16 White House videos with speaker-labelled subtitles
   │ keep videos with 3+ labelled changes
   ▼
12 videos · 320 min · 407 labelled speaker changes
   ▼
Nemotron-3 · pyannote community-1 · DiariZen  (1× H100)
   ▼
confusion % of named-speaker time · change recall/precision within 2 s

A label holds until the next one. "The Press" and "Audience Member" count for speaker changes but not identity; "Audience" is mostly applause and is ignored.

What the numbers sayLink to this section

• Up to 8 speakers, everyone is good. On the 9 videos with 8 or fewer named speakers, confusion is at most 1.31%, and Nemotron-3 is lowest at 0.36%.
• Above 8, Nemotron-3 breaks. It never returns more than 8 speakers. On a 17-speaker law-enforcement roundtable it mislabels 27.1% of named speech; pyannote and DiariZen get 0.5% and 0.3% and return 17 speakers each.
• Over all 12 videos pyannote has the lowest confusion (0.98%), DiariZen 1.45%, Nemotron-3 5.54%. The paired intervals against Nemotron-3 still include zero, because only 3 videos trigger the cap.
• Boundaries are everyone's weak spot. Change recall is 71–79%: about a quarter of speaker changes aren't found within 2 s. That matters for clip edges.

The fix we useLink to this section

Run Nemotron-3 first. If it returns exactly 8 speakers, rerun that video with pyannote. Hitting the cap is free to detect. Replayed on this data the rule gives 0.89% confusion on news and keeps most of Nemotron-3's VoxConverse lead (6.83% DER vs 5.93% alone). Both models allow commercial use; DiariZen's weights are non-commercial, which rules it out for a product.

That rule was chosen after seeing these results, so 0.89% is a replay, not a fresh test.

The dirty laundryLink to this section

• Our first score was wrong, and showed the opposite result. The label parser accepted any sentence before a colon ("We also know this:") and counted applause as a speaker change, which put Nemotron-3 last everywhere. We caught it when all three models scored 0 recall on the same videos, fixed the pattern, and reviewed every label by eye.
• Subtitles aren't diarization labels. Timing is coarse and a few names are misspelled, which can inflate speaker counts by one or two. Change precision is understated, because subtitles don't mark every real change.
• White House events aren't newsrooms. No anchors, field reports or music beds.

The recipeLink to this section

1. Don't trust a DER ranking for panels. DER weights talk time, so a speaker cap costs little on VoxConverse and a lot on a roundtable.
2. Nemotron-3 with a pyannote fallback at 8 speakers is our open default for English news.
3. Snap clip edges to pauses, not diarization turns, until boundary recall improves.

What's nextLink to this section

A fresh test of the fallback rule, and newsroom material with anchors and field reports.