Efficient Few-Shot Learning Without Prompts
Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, Oren Pereg
cs.CL
2022-09-22
SetFit contrastively fine-tunes a Sentence Transformer, then fits logistic regression, no prompts. At 8 shots a 110M model scores 62.3, matching T-Few 3B (63.4) about 19x cheaper.
Around 2022, few-shot text classification ran on three tracks: in-context learning with GPT-3, PEFT methods such as T-Few, and cloze-style PET / Adapet. All three leaned on billion-parameter models and hand-written prompts or verbalizers. Swap the dataset and the templates have to be rewritten; scores swing with the wording.
What practitioners actually wanted was a small model, no prompt, and a handful of labels per class. SetFit (Sentence Transformer Fine-tuning) splits the job in two: pull same-class sentences together in embedding space, then train a cheap head on those embeddings.
Step one fine-tunes an off-the-shelf Sentence Transformer with contrastive pairs. For each class the recipe samples R positives (two sentences, same class, label 1) and R negatives (different classes, label 0); R defaults to 20. K labeled examples yield far more pairs than K, which is how the method stretches a tiny set. Loss is cosine similarity, learning rate 1e-3, batch 16, max 256 tokens, one epoch.
Step two is thin: encode the original K examples with the tuned ST and fit logistic regression. Inference is embed then linear classify. No masks, no templates, no verbalizer.
Three backbones: paraphrase-MiniLM-L3-v2 (15M), paraphrase-mpnet-base-v2 (110M), all-roberta-large-v1 (355M).
Six English classification sets, ten random splits per size, mean reported. AGNews is dropped from the average because it appeared in T-Few's training data.
| Method | Params | N=8 avg | N=64 avg |
| RoBERTa-Large fine-tune | 355M | 43.0 | 69.7 |
| Perfect | - | 48.7 | 72.7 |
| Adapet | - | 58.3 | 73.8 |
| T-Few 3B | 3B | 63.4 | 70.3 |
| SetFit-MPNet | 110M | 62.3 | 75.3 |
At N=8, SetFit beats standard fine-tuning by 19.3 points and sits next to T-Few 3B's 63.4, with a model more than 27× smaller. At N=64 it leads T-Few by 5 points. The per-task picture is uneven. On Amazon Counterfactual at N=8, SetFit reaches 40.3 against T-Few's 19.0. On five-way SST-5 the order flips: T-Few 55.0, SetFit 43.6. On Customer Reviews, 8 labels per class already hit 88.5, against 92.4 for full-set fine-tuning.
RAFT leaderboard as of 2022-09-05: SetFit-RoBERTa 71.3, above GPT-3 at 62.7 and PET at 69.6, below T-Few 11B at 75.8 and the human baseline at 73.5, beating humans on 7 of 11 tasks. The MPNet variant scores 66.9.
On the Multilingual Amazon Reviews Corpus (5-star MAE×100, lower is better), 8 shots per class: SetFit averages 86.4–88.3, XLM-R fine-tuning 117–122, Adapet 135–152. Training on English only and testing on every language is SetFit's best setting at 86.4. Full 20k-example XLM-R falls to 47.7, so the few-shot gap to the full-data ceiling remains.
Using Kaplan-style FLOPs per token, SetFit-MPNet is about 19× cheaper than T-Few 3B at train and inference. Distilling into MiniLM (15M) is about 123× faster at 60.3 average, a 3.1-point drop. On a p3.2xlarge, N=8 training takes about 30 seconds and $0.025; T-Few 3B wants 40GB of GPU memory, about 700 seconds, and $0.7. Checkpoints are 70MB and 420MB against 11.4GB for T0-3B.
With very little unlabeled data, a SetFit student matching a 110M teacher's cosine similarities beats a plain MiniLM student at N=8 unlabeled by 24.8 / 25.1 / 8.9 points on AGNews, Emotion, and SST-5. At 1,000 unlabeled examples the two meet.
The paper cuts the assumption that few-shot classification needs a giant LM and a prompt. For teams iterating labels on a CPU or a small GPU, 110M plus logistic regression is a recipe you can ship. Code and datasets were released with the paper. Multilingual use is mostly a backbone swap.
This is classification, not generation and not multi-step reasoning. It does not compete with later instruction-tuned LLMs on breadth. Those models cover more tasks with zero labels. SetFit is the better fit when a few gold labels exist and cost plus stability matter.
There is no limitations section. The numbers still show the cracks. Fine-grained five-way sentiment lags T-Few; an embedding plus a linear head struggles with tight label boundaries. AmazonCF at N=8 has a standard deviation of 11.8, so the split matters a lot. Contrastive pairing needs same-class pairs, which gets thin at true 1-shot. Sequence length is capped at 256; long documents were not tested. Hyperparameters were tuned on development sets, so the "no validation set" few-shot protocol is not fully clean. On RAFT the 355M model still trails T-Few 11B by 4.5 points. All comparisons are against 2022 prompt methods, not later small instruction models.