Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Juan S. Santillana
cs.CL
2026-10-02
Matched Spanish security LMs tie on a keyword tool-use score (0.660 vs 0.650) while strict tests show 6/6 vs 0/6 real calls; a 3.3-GPU-hour SFT repairs what 6B tokens could not.
VectraYX is a series of small Spanish-language security models built for air-gapped deployment; the headline capability is native tool calling, emitting a <|toolcall|>-wrapped JSON payload that an MCP server executes. Two siblings share the same decoder implementation, 32K tokenizer, and special-token layout, and on the series' automatic benchmark they look like twins: B4 tool-use scores of 0.660 for the 600M (661.6M parameters, realized pretraining mix roughly 65% code and technical text, no dedicated SFT, permanently frozen at 64% of its schedule) and 0.650 for the 1B (1,109M, a web-heavy multi-phase curriculum with a 6B-token tool-SFT phase). In actual use the two do not overlap at all. The 600M spontaneously emits well-formed bashexec calls; the 1B had never once produced the JSON structure. The B4 scorer credits a response when the expected tool's name shows up anywhere in the text, so a fluent model that talks about tools without calling them farms credit. Keyword harnesses fail open, and this is the series' second documented instance of the failure class. The paper follows the crack to the bottom: confirm the false positive, localize the failure, repair it, then locate where the repair landed.
The core is a ladder of increasingly strict diagnostics, each costing minutes of CPU time:
Greedy decoding removes sampling luck, the training-time forward pass removes serving-stack format drift, and the generalization criterion separates recitation from capability.
| Check | 600M | 1B pre-repair | 1B repaired |
| B4 keyword score | 0.660 | 0.650 | — |
| Verbatim reproduction | 6/6, generalized | 0/4–6 | 4/4, exact copies |
| First-token probability | greedy top-1 | 1e-4–1e-5 | 0.996–0.998 |
| Well-formed on 269 corpus rows | 0.926 | 0.100 | 0.959 |
| Pass rate on 238 novel prompts | 0.428 | 0.000 | 0.536 (p=0.004) |
Scaled to the full corpus, 0.988 of the repaired 1B's well-formed emissions are byte-exact copies, against 0.235 for the 600M. On 238 prompts built over entities absent from training (CVE-2025 identifiers, reserved-range IPs, invented domains), the order flips: the repaired 1B passes 0.536 against 0.428 (paired McNemar p = 0.0039). Axis by axis, the 600M picks the right tool more often (0.776 vs 0.717) and builds more complete argument objects (0.855 vs 0.819); the repaired 1B wins on extracting unseen entities (0.667 vs 0.560) and on command quality (soft miss 0.484 vs 0.767, lower is better). Reciting your own corpus and slot-filling on unseen prompts are different skills, and this pair of experiments separates them.
Trajectory probes over 38 archived checkpoints supply the causal story. At phase-2 step 36K the 1B still held a 0.432 prior; 12,000 steps (3.1B tokens) of prose-dominated mixture erased it by nearly six orders of magnitude, and the dedicated 6B-token phase never brought it back. At the checkpoint where the lenient harness recorded the 1B's best-ever B4, the strict probe sat at its floor.
The embedding-drift check kills the dead-embedding hypothesis. The trigger row moved by cosine 0.999996, no different from 20 arbitrary ordinary rows, and 97.7% of the bf16 table is bit-identical after training: at learning rate 1e-5 a single Adam update sits below the rounding threshold, so the recipe's nominally unfrozen embeddings never acted. What changed is the surrounding network, with mid and late layers learning to route toward a direction that was well-placed all along.
Both models over-trigger. On conceptual questions that mention a CVE or a shell command, the 600M answers without a call on 0.09 of items and the repaired 1B on 0.17.
Three lessons anyone training sub-1B models can use directly. Run a strict structural check before quoting any keyword-scored tool-use benchmark; it costs minutes and fails closed. SFT volume does not predict outcome: 6B tokens failed where 3.3 GPU-hours succeeded, and the difference was a diagnosis-informed recipe. The first-token probability is near-free training telemetry; logging it through training would have exposed the missing prior before any SFT compute was spent. The bf16 finding stands on its own for weight analysis: a nominally unfrozen tensor can be numerically inert, so drift reports need changed-entry fractions rather than nominal configs. The matched pair also links code-heavy pretraining composition to structured output arriving by default, with the caveat that it is a natural experiment.
The paper states most of them itself. The pair is a natural experiment, not an ablation: parameter count, token budget, and curriculum shape are all unmatched, though every unmatched variable runs against the observed outcome. The repair is one recipe on one lineage, and the original checkpoint plus its warm-start parent were lost; every number comes from an end-to-end reproduction. The generalization battery is self-built, covers no unseen tools and no multi-call sequences, and all harness numbers are single-seed. The repair is capability-specific: asked a plain conversational question, the model correctly suppresses the call and returns an empty completion. Two concerns from this reading: the 0.536 vs 0.428 gap rests on a battery the authors designed themselves, and over-triggering is bad enough (0.78 to 0.91 wrong-call rates on near-miss prompts) that neither model is close to deployable.