Meta's MetaEncoder Turns a 30B Decoder into One Bi-Encoder for Both Multiple-Choice Decisions and Retrieval

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao

cs.CL, cs.AI

2026-10-08

Meta compares bi-encoders, cross-encoders and constrained decoders, then fine-tunes a 30B decoder into MetaEncoder, which tops 11 of 14 benchmarks.

What problem this solves

A lot of production work is not text generation. It is picking one option from a set, quickly, with a probability you can trust: agent routing, tool selection, content moderation, item ranking. The paper calls models for this job System One models.

There are three ways to score candidates, and nobody had compared them under one controlled setup. A bi-encoder embeds the request and each candidate separately and scores by cosine similarity. A cross-encoder concatenates the request with one candidate and runs full attention, which is accurate but expensive. A constrained decoder puts all options in one prompt labeled A/B/C and reads the next-token probabilities over those labels. The choice trades off accuracy, model size and inference cost, and the evidence was scattered.

Method

The first half is a controlled study. Decoders from 0.6B to 9B parameters are adapted into each of the three scorers, trained on 10M request-candidate pairs and evaluated on 14K multiple-choice questions.

The second half is MetaEncoder, which bets on the bi-encoder because candidate vectors can be cached offline and latency stays flat as the pool grows.

Results

The controlled study gives four findings:

How candidates are written also matters:

Candidate renderingMMLU (v0)MMMU (v0)MMLU (v1)
label only0.65120.49830.4760
description only0.74560.56290.8320
label + description0.77670.57880.8360

The main table compares against OpenJev-27B (text cross-encoder), Jev-Omni (multimodal constrained decoder on Gemma4-12B), WeMM-Embedding (9B) and Qwen3-VL-Embedding (8B). The paper says MetaEncoder v1, the fully fine-tuned model, scores highest on 11 of 14 benchmarks and sets a new state of the art on MMEB-V3 image and video retrieval. The extracted main table is garbled, so per-cell numbers are not quoted here.

Shuffling option order 10 times moves every benchmark by a standard deviation of at most 0.003.

Why it matters

If a product has both small closed-set decisions and retrieval over millions of candidates, this is a concrete argument that one bi-encoder can cover both, with candidate vectors precomputed. The price is somewhat lower closed-set accuracy than a cross-encoder, which the paper concedes.

The packing recipe is useful to anyone converting a large decoder into an encoder. It is an engineering improvement, not a new algorithm.

Limitations

The authors name reasoning as the ceiling. The Muse-Glimmer backbone scores 0.49 on MMMU when it must answer directly and 0.72 with chain-of-thought. Prepending a chain-of-thought trace lifts MetaEncoder v0 from 0.57 to 0.71, and 0.68 with the answer label stripped. Scoring models lack intermediate reasoning, and complex compositional tasks remain a weak spot.

On video, going from 8 to 16 frames adds 0.08, while 64 frames adds nothing at 8x the compute, and higher-level reasoning tracks even regress slightly.

Several things are unverified:

Terms

Source

What people are saying

Related papers

All paper explainers