Researcher questions hacking eval realism: citrusdrop web app hardly a real attack scenario
NathanpmYoung · x · 2026-11-19
A discussion about an agentic hacking study. The replier argues the most legible flaws are statistical errors everyone agrees on, but the bigger issue is realism: the headline/abstract numbers are based on a sample web app called citrusdrop, and critics ask whether anyone looking at its code would call it a realistic test of web server hacking.
More from Research
- NVIDIA Releases Nemotron-Labs-Sigma: Continuous Diffusion LLMs Generating Token Blocks in Parallel — karsten_kreis · 2026-11-19
- PRIM paper: models extract key math insight at 92% but only discover it themselves 25% of the time — _vztu · 2026-11-19
- Karpathy unveils nanochat miniseries v1, treating LLMs as a scaling-law model family with one compute dial — eliebakouch · 2026-11-19
- Replication problems raised for Palisade's AI self-replication study — NathanpmYoung · 2026-11-19
- Live imaging captures an egg stretching into an ellipse during ovulation in mice — JosephJacks_ · 2026-11-19
- City2Graph: open-source GeoAI library turns urban geospatial data into graphs for GNNs — adnan_hashmi · 2026-11-19