Active speaker detection on split screens: specialists near chance

The hookLink to this section

Reframing a 16:9 news shot for a 9:16 phone means choosing which face to keep, and the obvious choice is the person talking. Active speaker detectors do exactly that, and the well-known ones score 92–95% mAP on the AVA-ActiveSpeaker benchmark.

So we built the case news actually throws at them: a split screen with two people, both moving their lips, and one voice. The detectors pick the right person 55–60% of the time. A coin gets 50%.

The setup, in one pictureLink to this section

15 White House videos with speaker-labelled subtitles
   │ 10 s windows inside long single-speaker turns, face check
   ▼
144 clean clips → 120 side-by-side pairs, audio from one side only
   │ drop 4 pairs whose audio side had no sound track
   ▼
116 pairs · truth = the half holding the labelled speaker
   ▼
TalkNet · LR-ASD · Light-ASD · LRX-ASD  vs  Gemini 4 Argon ("left or right?")

Both people are talking, so nothing but lip-to-audio sync can decide.

What the numbers sayLink to this section

• The specialists barely beat chance. TalkNet 59.5%, LR-ASD 57.8%, Light-ASD 56.0%, LRX-ASD 55.2%. Every detector found faces on both halves of every pair, so this isn't a face-detection failure.
• A frontier video model solves it. Gemini 4 Argon gets 94.8% (95% CI 90.5–98.3), 35–40 points ahead of every detector.
• It isn't just recognising famous voices, probably. On the 7 pairs with the same person on both sides, Gemini got all 7, about a 1% chance by guessing. Seven pairs is too few to settle it.

Our reading: AVA rarely has two people talking at once, so the detectors may have learned "mouth moving" more than "mouth matches this audio".

The dirty laundryLink to this section

• "Exactly one face" kept zero clips. White House shots put officials beside the speaker. We switched to the largest face and made the truth the half of the screen, not one face.
• Dependency hell is real. LR-ASD needs numpy below 1.24, current OpenCV needs numpy 2. Pinned numpy 1.23.5 and OpenCV 4.10.
• A results sync flattened each clip's folder, and the first score showed every detector at 0%.
• The comparison isn't symmetric. Gemini was asked a direct two-way question and told both people move their lips; the detectors just output scores. Detector timing includes model loading, so there is no speed result here.

The recipeLink to this section

1. One face on screen: use any detector, or none. The face is the subject.
2. Two or more talking faces: don't trust the specialists. Ask a video model, once per shot, whose voice it is.
3. Treat "who is speaking" as a per-shot question, not a per-frame score, when designing a reframe.

What's nextLink to this section

Cost and latency of the per-shot video-model call per broadcast, and a larger same-person set to test the voice-recognition question properly.