VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use
Juan S. Santillana
cs.CL
2026-08-09
A 1B Spanish cybersecurity VLM where 16M vision-SFT tokens yield just 0.08 tool-ID (GPT-4o 0.94). Honest negative result, a falsifiable NoPE-by-vision ablation, and a loader bug that mimics training collapse.
Security analysts work through images. A reverse engineer reads an IDA disassembly pane; a SOC analyst triages a Wireshark capture; a forensics examiner scans a Volatility process listing; a pentester interprets Nmap and Metasploit output. The machine tools that help are either text-only LLMs that cannot see the screen, or general vision-language models (VLMs) that are neither specialized for security imagery nor small enough to run locally. Latin America gets two extra layers of pain: these models answer technical questions poorly in Spanish, and their 7B-70B footprints do not fit the commodity hardware of under-resourced teams. Worse, sending screenshots that may carry customer data, malware strings, or internal topology to a third-party cloud VLM is often prohibited by policy or law.
VectraYX-Vision-1B targets that gap: a just-over-1B Spanish/LATAM security VLM that runs offline on CPU, reads security-tool screenshots, emits structured reasoning through native <|think|> tokens, and invokes external tools via native <|toolcall|> tokens over MCP.
The stack is a LLaVA-style three-piece plus a four-phase curriculum.
Architecture. A frozen SigLIP-so400m encoder (27x27 = 729 patch tokens, no CLS) feeds a two-layer MLP projector (13.1M params), which feeds the VectraYX-1B decoder: 22 layers, dmodel=2048, GQA, SwiGLU, RMSNorm, vocab 32768, 1.04B params. Visual tokens enter the sequence via LLaVA-style placeholder substitution and are consumed like text.
Four-phase curriculum. Phases 1-3 are inherited language training (9.2B tokens of Spanish pretraining, a three-block 50B-token mix, 6B tokens of tooling). Phase 4 is vision: 4a alignment (projector only), 4b instruct (projector plus decoder, 60:40 vision/text replay), 4c think+tools (50:50, with reasoning and tool-call traces). The vision phase totals 16M tokens and finishes in 2.2 hours on 2xA100-40GB.
Replay against forgetting. Each vision sub-stage mixes in text replay (40% in 4b, 50% in 4c) so the decoder does not forget its Spanish and tool competence while learning to see.
Two designs are where the real content sits.
First, native tokens. <|think|>, </think>, and <|toolcall|> are special tokens baked into the vocabulary; in 4c the model is trained to open a think block after reading a screenshot, then answer, and emit a tool call when warranted, following its own generation path rather than faking it with prompt formatting. Reasoning and tool use are trained capabilities, not prompting tricks.
Second, the NoPE-by-vision question, which is genuinely open. The decoder backbone does something rare: one of every four layers uses no positional encoding (NoPE), the other three use RoPE. When the 729 visual tokens are injected as a contiguous block, RoPE layers stamp each with a rotary phase by 1D sequence index, but patches form a 2D grid with no canonical linear order, so row-major flattening imposes a spurious "token i precedes token i+1" relation on spatially adjacent patches. NoPE layers impose no such order and are permutation-equivariant over the visual block. Whether NoPE helps or hurts visual attention has never been measured, because no released VLM has combined a NoPE backbone with visual-token injection. The authors turn it into a falsifiable three-variant ablation (V0 keep NoPE, V1 all-RoPE, V2 NoPE plus learned 2D positional embedding) and let the sign of B6(V0) minus B6(V1) on spatially-structured panes discriminate the two competing hypotheses.
Say it plainly first: this model currently has no usable vision. On B6vision (identify the tool and name the key technical fact on 50 held-out screenshots), the best run (v3B, 1900 steps) scores 0.08 tool identification and 0.06 correctness. GPT-4o sits at 0.940 (the ceiling), Phi-4-multimodal (4.5B) at 0.440. The model generates fluent Spanish and mostly ignores the image.
One control rescues the benchmark's legitimacy: with the image removed and only the question text kept, the same backbone drops to 0.02 tool identification and 0.00 correctness, statistically indistinguishable from the with-image score. The items cannot be guessed from text alone, so the model has failed to learn vision rather than the benchmark being easy.
| Model | B6 tool-id | B6 correct |
| GPT-4o (ceiling) | 0.940 | 0.000 |
| Phi-4-multimodal 4.5B | 0.440 | 0.000 |
| VectraYX-Vision-1B (best, v3B) | 0.080 | 0.060 |
| Same backbone, no image (control) | 0.020 | 0.000 |
The text-regression check (Table 4) shows the vision phase did not wreck language and tool competence: phase 4a (decoder frozen) regresses nothing; unfreezing the token embedding table (tokemb) pushes tool selection B4 to 0.110, above the pre-vision high-water mark of 0.075. That pinpoints frozen tokemb as the real cause of v3B's degradation, since the model needs a trainable embedding table to emit the <|toolcall|> sequence reliably.
The efficiency claim holds and is independent of the vision fine-tune: the 1B decoder exports to a 2.2GB GGUF in F16 (0.6 to 0.7GB in 4-bit), the mmproj (encoder plus projector) adds 0.8 to 0.9GB, for under 4GB total versus 14GB-plus for larger VLMs. On a GPU-less commodity CPU box (Azure D8sv3) the F16 decoder serves 18.5 tokens/s with under 10s load time.
The value here is not "another usable security VLM" (it is far from usable). It is three transferable things. First, a diagnostic trap: the authors found early checkpoints all near zero, assumed training had collapsed, and eventually traced it to a checkpoint-loader bug. At export, the loader failed to strip the llm. prefix from the state-dict keys, so no LLM weights matched and the backbone stayed at random initialization in inference (embedding norm 164, versus 630 for an intact backbone). Training itself was correct (the optimizer updated the right parameters), but every metric from affected checkpoints was invalid. This bug applies to any LLaVA-style export pipeline, and embedding norm is a one-line check.
Second, a dosage warning. The standard VLM recipe of alignment plus light instruct is not enough at 16M vision tokens to teach a model to read security screenshots. Anyone replicating a small VLM should not expect hundreds to two thousand steps to produce vision. The authors' remediation: longer SFT, at least 60% replay, a lower decoder learning rate.
Third, the NoPE-by-vision question itself. As NoPE gets adopted in text decoders for length generalization, the standard injection recipe will keep hitting this interaction when those decoders become VLMs. A clean measurement, even a negative one, informs every future VLM built on a NoPE-family backbone.
The authors frame Section 9 heavily enough that "a reviewer should weigh them against the contributions." Item by item.
Negative result pending remediation. The near-zero B6 is real training deficit, not the bug. The remediated headline numbers, the NoPE ablation (V0, V1, V2) results, and the full post-vision B1-to-B5 regression table are not in this preprint; only the designs, checkpoints, and runner are released. The NoPE question, the most interesting part, has no answer yet.
Synthetic-QA validity. Most of the corpus is LLM-generated, with two known risks: answer leakage (the question implies its answer) and factual errors in the answers. No human audit exists yet; publication needs an expert audit of at least 200 items with inter-annotator agreement. This is why the no-image control is mandatory.
Synthetic-to-real image gap. Much of the imagery is programmatically rendered IDA, Ghidra, and Wireshark panes, missing the chrome, fonts, color themes, and layout noise of real tool UIs. Good performance on synthetic renders need not transfer to real captures, and the real-screenshot subset is small with the gap unquantified.
Single seed. The full four-phase curriculum takes about two weeks of dedicated 2xA100 time, unaffordable for a single-author, institution-unaffiliated project, so every number is single-run, compensated only partly by per-checkpoint trajectories and per-item bootstrap intervals. B6 has only 50 items, so seed variance is understated.
On safety (visual prompt injection, tool over-triggering) the authors scope evaluation as future work and have not done it; the tool-calling configuration ships only once the safety numbers exist.