Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
Hanlin Tian, Minhao Li, Yu Mi, Sihan Zhu, Zhao Yang, Yuxiang Wang, Hongquan Zhu, Qiufei Hu
cs.AI, cs.CL
2026-08-31
DataGrids splits a task instruction from a few-KB private artefact, then leak-audits and executable-checks it. Opus goes from 0% to 68% pass on fifteen tasks with the artefact; seven survive a coarse screen. No training study is claimed.
Professional agent tasks often hang on in-house conventions: thresholds, alias tables, normalisation rules, operators nobody wants to retype. Those facts are missing from public corpora. Benchmarks rarely control whether the agent was given them, so "cannot execute" and "does not know the house rule" get scored as the same failure.
DataGrids treats this as a construction problem, not a story about self-improvement. The curator is asymmetric at write time: an answer can be planted, and checking a structured output against that plant is cheap. The aim is to make access to that knowledge a switch you can audit.
Each unit is a pair. The instruction says what to deliver and never names the artefact. The artefact is a few kilobytes of terminology, private conventions, reference tables, and low-level operators, never an end-to-end solver. Instructions are byte-identical with and without the artefact. They refer to "the established in-house specification" and add a fallback: if no spec is present, apply common conventions. That forces the withheld condition to submit a checkable answer instead of abstaining.
Two gate families. Convention gates plant choices that cannot be derived from the instruction, such as whether a turnover metric drops the opening minutes, or a denominator of 235 rather than 240. Operator gates put brittle algorithms in a utility library, such as Brandes betweenness under a fixed normalisation, or Garman-Klass volatility. Those can in principle be re-derived, but not reliably under time pressure. Parsing, grouping, graph construction, and serialisation stay with the agent.
A static leak audit scans the instruction and environment for gated constants and any mention of the artefact. Nine of fifteen calibration tasks have executable audit scripts; six do not. Structured outputs are checked by independent Python solvers and rule corpora, typically at 1e-4 float tolerance. The protocol also defines all-pass criterion rubrics. Every pass rate in this batch uses the deterministic path. No LLM judge.
Calibration uses two configurations and five trials per cell. The frontier setup is Claude Opus 4.8 through Claude Code. The second is Qwen3.6-Plus through OpenCode, always with the artefact. Runs sit in a Docker sandbox with file I/O. Retention is coarse: frontier with artefact ≥60%, frontier without =0%, second config with artefact ≤40%. One flip moves a cell by 20 points. This is a quality-control heuristic, not a statistical test.
Pooled over fifteen tasks, Opus scores 51/75 (68.0%) with the artefact and 0/75 without it. Qwen with the artefact scores 17/75 (22.7%). Seven tasks survive: legal-clause, paper-index, bom-costing, airline-booking, clinical-variant, mortgage-escrow, dependency-audit. Three of those sit on the 40% second-config ceiling; dependency-audit also sits on the 60% frontier floor.
| Task | Opus -art. | Opus +art. | Qwen +art. |
| Contract clause risk | 0/5 | 4/5 | 0/5 |
| BOM cost rollup | 0/5 | 5/5 | 0/5 |
| Dependency audit | 0/5 | 3/5 | 2/5 |
| Enterprise search / inventory / 13F | 0/5 | 0/5 | 0/5 |
| All | 0/75 | 51/75 | 17/75 |
Eight are dropped. trial-cohort misses the frontier bar. msa-redline, fuzz-secure, and esg-synthesis are too easy for the second config. cpi-tracker is 40% with the artefact. enterprise-search, inventory-rollforward, and sec-13f stay at 0% even with it.
On bom-costing, a perturbed artefact that looks the same but changes three conventions (overhead 12% to 15%, missing leaf price to sibling average, cycle value to 0) yields 0/5 for Opus, matching no artefact; the correct document is 5/5. On bom-costing and legal-clause, Qwen recites every gated convention and scores 100% in a single-turn, no-tool setting with data inlined. Under the full harness those two tasks are 0/5. The miss looks like file discovery, tools, and format compliance, not missing knowledge.
The public repo ships five representative retained tasks. paper-index keeps its SkillsBench vendor-sample provenance. The authors state, repeatedly, that these runs validate the construction contract. They do not show that retained tasks improve post-training.
People who build agent training sets need a way to separate "never saw the house rule" from "cannot orchestrate". This protocol makes that a three-way switch: artefact, no artefact, wrong artefact, plus an executable witness. For RLVR filtering it is directly useful. Tasks that are always 0 or always 1 carry no gradient.
Success here measures whether an agent holds and applies a supplied key. That overlaps professional competence and is not the same thing. The paper says so. On tasks not built for strict gating, skill-availability deltas in concurrent work sit around 18-36 points. Here the planted gap is 68. The comparison is not controlled. The direction matches.
Fifteen tasks and five trials per cell make retention one flip away from reversing. Leak audits cover nine tasks; a static scan cannot rule out paraphrase leakage. The two configs change both model and harness, so Qwen's low score is not a model ranking. The retained set is selected on these same outcomes and cannot be reused as independent proof of better training data. Rubric judging is specified and unused. The public subset is five tasks; the 68% figure cannot be reconstructed from it.