Challenges of Non-Parametric Statistical Tests on Real-World Data
IanArawjo · x · 2026-07-24
The author emphasizes the importance of running on real data. While PPI (Prediction-Powered Inference) corrected tests work well in simulations, real data—especially poorly aligned, terrible quality judge scores—can cause problems for non-normal statistics. They are currently investigating this issue.
Related event: Eliminating LLM Evaluation False Positives with PPI-Corrected Tests(5 posts)→
More from Research
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27
- Marin's dlwh Explains the Scaling Law Fit: Estimating the Loss Floor Early — dlwh · 2026-10-27
- Marin's Open Scaling Law Extrapolates 300× and Predicts Loss Within 0.2% — dlwh · 2026-10-27
- Marin Shares Forecast-vs-Measured Plots: Paloma Hits 0.4% RMS Error — dlwh · 2026-10-27
- AW-0 hits 80.8% on RoboTwin tasks, beating strongest baseline by 16 points — trevordarrell · 2026-10-27
- AW-0's code stream fixes world model prediction: picks mango where baseline grabs apple — trevordarrell · 2026-10-27