Subliminal Learning in Vision-Language Models
Arman Hatami, Romina Aalishah, Ilya E. Monosov
cs.CV, cs.AI, cs.CL, cs.LG, eess.IV
2026-11-05
Same-base students trained only on a VLM teacher's shape-count numbers pick pandas 9.92 points more often than controls, and their explanations move closer too.
Distillation assumes a student learns only what the teacher's outputs actually say. Language models already break that assumption. Subliminal learning means a teacher hides a preference in samples that never mention it, and a finetuned student still leans the same way. The usual demo tunes a teacher to favor one animal, then has it write number lists, code, or reasoning traces with that animal's name absent. A student that shares the base model picks the animal more often, and shared initialization makes the leak stronger.
A vision-language model reads images and text together, and the preference is about images. Whether it can ride along with "count the shapes, then emit numbers from the bin for that count" does not follow from the text-only result. Finetuning now consumes a lot of labels written by other models. A number string with no animal name passes a keyword filter. If behavior still moves, reading the training text is not enough.
The experiments are from Johns Hopkins. The bases run from 2B to 7B: Qwen2-VL-2B-Instruct, Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, and Phi-4-multimodal-Instruct (5.6B). Each base produces two teachers. The biased teacher always selects the target. The control teacher sees the same images and prompts and selects the target on half the items, chance on a two-way question. Both students are reinitialized from the base, one on the biased teacher's numbers and one on the control teacher's. Images and prompts match.
Animal targets are dolphin, elephant, lion, otter, panda, and wolf, 2,000 two-choice items each. Position, distractor species, and prompt are balanced, and training images are reused across pairs. Evaluation is 1,200 six-way items, one image per species, positions balanced across A-F, so chance is one in six.
Five face attributes come from FaceSynthetics: opaque eyewear, a visible beard, visible headwear, an open mouth, and high hair coverage. Positive and negative faces are matched on the other attributes, again with 2,000 items each. Evaluation shows six faces of which exactly one has the attribute, on images held out from training.
The carrier has to use the picture and still allow many valid answers, so the two teachers can solve one item and write different numbers. Images are 336 by 336 with one to eight colored geometric shapes. The animal carrier asks for exactly eight two-digit integers: 1-2 shapes map to 00-24, 3-4 to 25-49, 5-6 to 50-74, and 7-8 to 75-99. The face carrier asks for five three-digit integers in 000-249, 250-499, 500-749, and 750-999.
A kept item is counted correctly, formatted correctly, and filled with in-range integers, with no animal name or attribute word. Both teachers must pass, which equalizes the distribution of shape counts. Animal runs keep 8,000 of 20,480 candidates. Face runs keep 12,000 of 65,536. Labels are sampled at temperature 1.0 and top-p 1.0, or written by greedy decoding.
Delta T is the biased student's target rate minus the control student's. The account is first-order. If the student update points the same way as the biased teacher's offset from the base, called delta thetaT, the student moves toward that teacher, and the aligned piece pushes the selection rate up. Aligned pieces can add across examples, so more data can strengthen transfer, and a reversed batch can stop it. The same displacement shifts next-token probabilities in other contexts, which is why wording can move. The teacher's reasoning is not copied.
The largest animal effect is Qwen2.5-VL-3B on panda with greedy decoding: 30.50% for the control student, 40.42% for the biased student, a gap of +9.92 points, 95% interval [+7.83, +12.08]. In that same cell, lion is +5.67 and dolphin is +5.58. Temperature sampling leaves panda at +1.58, while lion stays at +5.33, interval [+3.67, +7.08]. On the 7B model the largest gap is dolphin: +6.00 with sampling, interval [+4.33, +7.67], and +3.25 with greedy decoding, interval [+1.17, +5.33]. Animal numbers for the 2B model and Phi-4 are only in the supplement.
| Setting | Metric | Result |
| 3B panda, greedy | selection rate | 30.50% control, 40.42% biased, gap +9.92 |
| 3B lion / dolphin, greedy | delta T | +5.67 / +5.58 |
| 3B panda / lion, sampling | delta T | +1.58 / +5.33 |
| 7B dolphin, sampling / greedy | delta T | +6.00 / +3.25 |
| 3B panda, data size | delta T | +3.17 at 512, +9.92 at 8,000 |
| 2B faces, sampling | delta T | +1.33 to +3.50, headwear highest |
Only one scaling curve is fully reported: 3B, panda, greedy. Nested, range-balanced subsets of 512, 1,024, 2,048, 4,096, and 8,000 examples are each trained for 10 epochs. The gap rises from +3.17 to +9.92 at every size. The supplement contains settings where more data does not help. Epoch count is fixed, so more examples also mean more optimizer steps. Those two changes are tied together.
Face transfer is reported for Qwen2-VL-2B with temperature sampling. All five point estimates are positive, from +1.33 to +3.50, with headwear at +3.50. Some intervals include zero. That is weaker than the strongest animal cells.
Wording is scored after animal names and choice letters are removed. On all 1,200 panda items for the 3B model, token F1 moves from 4.92 for the control student to 8.14 for the biased student, a gap of +3.21, interval [+2.53, +3.90]. On the 334 items where the teacher and both students pick the same image, the gap is +5.63 (8.82 versus 14.45), interval [+4.19, +7.09]. TF-IDF, Jaccard, and ROUGE-L stay positive under all three filters. Of those 334 items, the biased student is closer on 168, the control student on 71, and 95 tie.
Chance among six animals is about 16.7%, and the control student is already at 30.50% on panda. The subliminal claim is the gap.
Shared-base distillation is a normal recipe. Finetune a teacher from the base, then train a student from that same base on the teacher's labels. Here those labels are valid number strings. A filter for animal names keeps all of them, and the panda rate still differs by 9.92 points.
On a six-way choice the move is from about 30% to about 40%. Face gaps are mostly 1 to 3 points, and some intervals include zero. Change the target, the model, or the decoder, and the gap can fall near 2 points. A stable injection method needs more than this.
Wording is the part that hits products which display a reason. After all three models pick the same image, the biased student's sentences are still closer to the teacher, and those sentences were not in the labels. From 512 to 8,000 examples this one curve goes from 3.17 to 9.92 points. The other settings do not have to rise with it.
The paper's own numbers show the instability. Panda is +9.92 under greedy decoding and +1.58 under temperature sampling. More data does not always raise the gap, and the supplement includes cases that do not rise. With 10 epochs fixed, going from 512 to 8,000 examples adds samples and gradient steps together.
The dot product between the student update and the teacher direction is not reported. Accumulation along that direction is an argument. Language-model follow-ups have cut transmission by deleting a few divergence tokens. No such deletion is tried on these number strings.
Every run shares a base. The carrier is synthetic shapes, and both teachers must answer correctly. Captions, visual question answering, and reasoning traces are untested. Detailed wording numbers exist for one cell, 3B panda. The prompts stay fixed: which animal to put in a room, which portrait to put on a character card. How fully the teacher was biased, and whether students can name the animal, is not quantified in the main text. Finance and defense appear as risk examples in the introduction. The measurements are animal choice and face attributes. Token F1 on the 334 shared-image items is 14.45 versus 8.82.