Predicting YouTube 'most replayed' moments on news: position beats content

The hookLink to this section

YouTube's "most replayed" graph is the only large-scale record of which seconds people come back to. If a cheap signal from the video itself (loud audio, a busy frame, a striking sentence) predicted that graph, a clipping pipeline could aim for those moments.

On general YouTube, audio does best. On news, the best predictor is simply where in the video the moment is. And every cheap proxy is weak.

The setup, in one pictureLink to this section

Mr. HiSum replay scores (per second, 50,000+ viewers per video)
   + frozen features: InceptionV3 (visual) · AST (audio) · RoBERTa (text)
   │ no video downloaded
   ▼
train: 2,500 random videos (news excluded)
test:  1,899 official test videos · 117 news videos (held out)
   ▼
proxies: position prior (mean replay by relative time) · ridge per modality · ridge fused
   ▼
Spearman per video · HIT@15% (chance 0.15) · bootstrap by video

What the numbers sayLink to this section

• On news, position beats content. The position prior scores Spearman 0.143 and HIT@15 0.222, ahead of every content proxy on the 117 news videos, though the intervals overlap.
• Audio is the best content signal in general (0.171 on the general test set) but drops most on news (0.117). On generic YouTube, music, laughter and crowds mark the moments; news audio is mostly steady speech.
• Every proxy is weak. All correlations stay under 0.18. A cheap engagement score could break ties at most; it shouldn't choose clips.
• The replay curve has a shape. Replay is highest in the first 5% of a video (0.38), flat at 0.34–0.36 through the middle, and falls to 0.17 in the last 5%.

The dirty laundryLink to this section

• Linear proxies on frozen features. Trained models like TripleSumm do better in general and weren't run on news here.
• Rewatching isn't sharing. Replay heat measures which moments inside a long video people return to, not which short they would pass on.
• The disk was too small. A 25 GB feature file didn't fit next to earlier work, so each file was downloaded, subset and deleted in turn. The fused model ran out of memory at 2,560 dimensions and was refitted on every 6th second.
• 117 news videos, mixed formats: presenters, weather and full programmes.

The recipeLink to this section

1. Don't build a cheap engagement model for news clips. The signal isn't there.
2. If you need a baseline, use position plus audio. It is free and at least as good as anything fancier here.
3. Spend the effort on real preference data for shorts instead (see stage 9).

What's nextLink to this section

A small human panel on news shorts, the preference signal this pipeline actually needs.