Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning
Ahmed Elshabrawy, Yongxin Huang, Iryna Gurevych, Alham Fikri Aji
cs.CL
2024-04-19
Statement-Tuning trains RoBERTa as a true/false discriminator; the 355M model hits 75.3 zero-shot average accuracy, above Llama-3-70B's 70.5.
Decoder LLMs can do unseen tasks from a prompt. Encoder models such as BERT and RoBERTa are strong after task-specific fine-tuning, but the classification head is glued to a fixed label set, so a new task is a non-starter. Cloze methods like PET rewrite tasks as fill-in-the-blank so the pretrained MLM can score labels. Zero-shot performance then rides almost entirely on pretraining, not on multi-task instruction following.
MBZUAI and TU Darmstadt propose Statement-Tuning: rewrite any finite-label discriminative task as natural-language true/false statements, and train one binary head to judge whether a statement is true. After multi-task training, a new task is solvable zero-shot if it can be phrased as statements.
Sixteen datasets across nine task types (QQP, MNLI, SQuAD, PiQA, Yelp, and others) are verbalized into statements. Each example yields several statements via label-specific templates: the gold label is True, the rest False. RoBERTa is fine-tuned with a standard sequence-classification head.
At inference, one statement is built per candidate label and the label with the highest True probability wins. Zero-shot uses the multi-task model as-is. K-shot continues fine-tuning on a handful of target-task statements.
Three design choices matter. Statements are more general than text-plus-label concatenation or premise-hypothesis pairs, because not every discriminative task is an entailment problem. Sampling is balanced across tasks, True/False, and original classes, so large corpora do not drown small ones. Multiple templates per dataset push the model toward label semantics rather than surface cues.
Seven evaluation sets. BCOPA, FigQA, StoryCloze, Emotion, and Yahoo Answers are unseen tasks. MRPC and Amazon Polarity reuse trained task types in new domains.
| Model | Params | Zero-shot avg accuracy |
| Llama-3-70B-Instruct | 70B | 70.5 |
| Qwen1.5-7B-Chat | 7B | 70.9 |
| RoBERTa-base (Best) | 125M | 67.1 |
| RoBERTa-large (Best) | 355M | 75.3 |
The 355M RoBERTa-large beats the strongest open LLM, Qwen1.5-7B-Chat, by 4.4 points on average, and Llama-3-70B by 4.8. On FigQA it scores 74.2 against Llama-3's 42.0 (+32.2). On StoryCloze, 92.1 against 82.7 (+9.4). On BCOPA and Amazon Polarity, zero-shot is already close to full fine-tuning.
Data efficiency is high. One thousand statements per training set give RoBERTa-base 63.0 average accuracy, 96% of its best 65.4 at 4k. At 50k, base falls to 58.5; large peaks around 10k then drops, overfitting the training tasks. Raising task diversity from 3 categories to 9 lifts geometric-mean accuracy from 26.0 to 61.2. Extra templates barely move the mean, but they cut run-to-run standard deviation from 6.9 at one template per task to 3.0 at three.
In the extreme few-shot regime, continued statement fine-tuning usually beats fine-tuning vanilla RoBERTa from scratch. Once labeled data is plentiful, ordinary fine-tuning wins.
A 2024 result that still reads cleanly: finite-label NLU does not require a 7B decoder. A 125M-355M encoder, multi-task trained as a statement discriminator, can sit in the same zero-shot band as the open 7B-70B models of that year, especially on commonsense multiple choice. About 16k statements are enough to get moving.
The useful setting is narrow: finite labels, latency budget, no generation. Serving cost scales with the number of labels, one forward pass each.
The authors flag the obvious costs: one forward pass per label; full fine-tuning still wins with enough data; sensitivity to training size and other hyperparameters; English only; encoders cannot do translation or abstractive summarization.
The LLM comparison is shaky. Llama-3 and Qwen may have seen the eval sets; the paper itself mentions contamination. The suite is seven classification or multiple-choice tasks, with no generative reading comprehension and no long context. The 32-point FigQA gap looks more like decoder zero-shot failing at figurative language than like a 355M encoder matching 70B capability. Those 2024 baselines are also dated. Treat the numbers as evidence that the encoder path works, not as a leaderboard.