Content Depth Matters in Short-Video Recommendation: Rethinking the Attention Economy
Liwei Deng, Jing Jiang, Zhiwei Li, Yang Wang, Guodong Long
cs.AI, cs.IR
2026-08-14
UTS researchers introduce Content Depth Score and a 150K-video benchmark, finding that 13 real recommender systems recommend content barely deeper than random selection.
Short-video recommenders are almost universally optimized for engagement signals like watch time and clicks, which structurally favors content that grabs attention within seconds. Growing evidence links prolonged exposure to that kind of content with weakened sustained attention and worse mental well-being, and regulators in Australia and the UK have already imposed under-16 usage restrictions. But those interventions all target access, not content, for a simple reason: there's never been a way to measure content depth itself, so neither platforms nor regulators have had anything to optimize or evaluate against.
The paper introduces the Content Depth Score (CDS), a seven-level rubric grounded in three established frameworks from cognitive psychology: Dual Process Theory (System 1 intuitive reactions vs. System 2 deliberate processing), the Revised Bloom's Taxonomy (six cognitive levels from Remember to Create), and the SOLO Taxonomy (complexity of demonstrated understanding). The seven levels run from 0 (pure affect or entertainment, no explanatory content) to 6 (a generalizable model or framework transferable across contexts), passing through isolated opinion, single-concept explanation, procedural demonstration, and mechanism/causal analysis along the way.
Rather than processing raw video and audio, the annotation pipeline feeds an LLM three textual signals per video: the caption, the category label, and the ASR transcript, asking it to return a score plus reasoning and confidence. To validate this against human judgment, five annotators with computer science and psychology backgrounds built a gold-standard subset using a Delphi-style process (independent annotation, a second round on disagreements, median aggregation), then the team picked the LLM with the strongest alignment to that gold set (Qwen3.7-Max) to scale annotation to 150,000 videos.
Beyond per-video scores, the paper defines List-wise CDS (LCDS), a position-weighted aggregation of a top-K recommendation list's per-item CDS into a single 0-1 score for evaluating a whole recommender rather than a single item.
| Check | Value |
| Inter-annotator agreement (within 1 level) | 97.94% |
| Best LLM evaluator (Qwen3.7-Max) exact match vs. humans | 78.03% |
| CDS distribution across 150K videos | Low (s=0): 54.04%; Medium (1-3): 19.74%; High (4-6): 0.94%; insufficient info (NaN): 25.37% |
| Content depth of 13 recommenders (A-LCDS/E-LCDS) | All fall in the Low-LCDS band [0, 1/6), most close to random |
Category breakdowns match intuition: dance, comedy, and beauty dominate the low-depth group, while finance, law, history, military, and science dominate medium and high. Two robustness checks rule out obvious shortcuts: swapping a video's category label from a low-depth one (dance) to a high-depth one (history) doesn't inflate its score, so the model isn't just reading the category tag, and padding ASR transcripts with filler doesn't raise scores either, while trimming a high-depth video's transcript does lower its score, indicating the evaluator responds to reasoning density in the content, not text length.
This is the first time content depth becomes something measurable enough to evaluate and optimize a recommender against, rather than staying at the policy level of "should platforms restrict how long minors can use the app." For teams building recommenders, this offers a new evaluation axis that surfaces an uncomfortable truth: the common engineering assumption that better engagement metrics mean better user value doesn't hold here. Thirteen representative algorithms, spanning ID-based and multimodal approaches, all score competitively on engagement while none recommends content meaningfully deeper than random selection. Current recommendation objectives simply encode no signal that rewards depth. Optimizing watch time and clicks won't produce depth as a side effect; something like CDS needs to be added explicitly to the training objective for that to change.
CDS annotation relies purely on text signals (caption, category, ASR), so videos whose depth lives mainly in visuals or audio rather than language get marked as insufficient information (NaN), which affects 25.37% of all videos, not a small share. The paper treats these as lower-bound estimates without validating that assumption. The scoring also depends entirely on an LLM judge; despite the human gold-set validation and robustness checks, the evaluator likely inherits knowledge-domain biases from its training data (more readily recognizing analytical structure in finance or history content, for instance, while potentially undervaluing depth expressed through non-Western cultural framing), a possibility the paper doesn't address. CDS measures whether content has depth, not whether it's accurate, healthy, or actually beneficial: a highly structured video full of misinformation would still score high.