Researcher flags unresolved data transparency issues in compositionality evals
najoungkim · x · 2026-10-27
A researcher notes that while industry effort in this direction is welcome, the data transparency problem in compositionality evals remains unaddressed. She recalls trying to preempt contamination in RExBench, an eval that critically requires the research ideas involved to be novel—data leakage would undermine its validity. She admits becoming a bit jaded after investing in this area, but still finds the industry momentum cool.
More from Research
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- PSNR below 15 dB in papers likely means the metric was misused, researcher warns — kwangmoo_yi · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27