NVIDIA releases 43K instruction-hierarchy rubrics for judging system vs user prompts
kalyan_kpl · x · 2026-10-24
NVIDIA published Nemotron-System-User-Instruction-Hierarchy-v1 on Hugging Face:
- 43K evaluation rubrics for judging whether a model's response obeys system vs user instructions;
- Covers conflicting, supplementary, and user-only instruction settings, scored via LLM-as-judge;
- CC-BY-4.0 licensed, JSON format.
A public eval resource for instruction hierarchy, a key capability for safety and controllability.
More from Research
- KAN author Ziming Liu: LLMs learn alchemy, not the science of AI — ZimingLiu11 · 2026-10-24
- Training-Physics Intuition Compressed Into a Decision Tree Hits 78.2% Out-of-Fold — ZimingLiu11 · 2026-10-24
- LLMs Rank Optimizers at 85% but Are Data-Blind on Architectures, New Benchmark Shows — ZimingLiu11 · 2026-10-24
- Architecture IQ benchmark shows LLMs still lag humans in training intuition, ResNet-style — ZimingLiu11 · 2026-10-24
- François Chollet declares "Symbolic learning works" in terse but weighty statement — fchollet · 2026-10-24
- Harness²: Google's self-improving agent framework adds +34.7 pts without training — Prompt Engineering · 2026-10-24