Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts
Joel Jang, Seonghyeon Ye, Minjoon Seo
cs.CL
2022-09-26
On nine tasks, negating the prompt makes OPT and GPT-3 worse from 125M to 175B. Original-plus-negated averages stay near 50%, and the best method trails 13-year-olds by 31.3%.
Zero-shot accuracy, one prompt and no examples, was expected to rise with parameter count. That scaling law was measured on prompts that ask the model to get the item right. This KAIST study flips the prompt and asks for a wrong answer, an inappropriate ending, or an unnatural continuation.
Negation is scarce in pretraining. Asked not to draw a monkey holding a banana, DALL-E-2 still draws one. On COPA, asked for an inappropriate ending after a man turns on a faucet, GPT-3 still chooses water flowing from the spout.
A 540B model had already been used as a robot's decision core, with no separate reinforcement-learning policy. If do and do not collapse into the same instruction, the action comes out backwards.
Nine datasets, three per group: commonsense reasoning (PIQA, ARC-Easy, COPA), sentence completion (HellaSwag, StoryCloze, Lambada), and question answering (WebQuestions, Natural Questions, TriviaQA). Prompts come from Promptsource, a library of ready-made dataset prompts. The lines chosen are ones that already work for OPT, with polarity words flipped by hand. Each item is run twice.
API cost caps every dataset at 300 examples. Where a question has more than two choices, several count as correct under negation. That rule covers ARC-Easy, Lambada, and HellaSwag. Lambada distractors are random words from the passage. Wrong QA options come from training-set answers that do not appear in the test set.
Four comparisons. OPT spans nine sizes from 125M to 175B. GPT-3 is Ada, Babbage, Curie, and Davinci, plotted at 350M, 1.3B, 6.7B, and 175B. Instruction-adapted models are T0 at 3B and 11B, plus InstructGPT. T0 is multitask-finetuned on prompted datasets. InstructGPT is GPT-3 trained further so outputs follow the user instruction. ICL, examples written into the prompt with frozen weights, is tried at 2, 4, and 8 shots on OPT 66B. Fine-tuning is only OPT 125M, 350M, and 1.3B on negated prompts: every training row under 10k, otherwise a 10k cap, 5 epochs, learning rate 1e-3, last checkpoint.
Humans did one dataset per group, COPA, Lambada, and Natural Questions. One hundred items, 50 original and 50 negated, no overlap, three 13-year-olds. The minor study followed advice from the National Human Rights Commission of Korea and had parental consent.
The edit is a word or two. If the model is reading the instruction, a better original score should not sink the negated one.
GPT-3 shows inverse scaling on all three task types. Larger models score worse on negated prompts and better on the original ones. OPT matches that trend in the appendix. The main text does not list per-dataset scores.
Gains on the original prompt are mostly lost on the negated prompt. The average of the two sits on a flat line near 50% for all nine tasks. Prompts that request opposite actions receive the same action.
T0 and InstructGPT do a bit better on negation. How much is not stated. The combined average stays near 50%, except when GPT-3 175B answers questions with in-context examples. On Lambada, 8 shots for that 175B model exceeded the API length limit, so the cell is empty.
In-context examples work only in some settings. Fine-tuning on negated prompts is zero-sum: the negated score rises and the original score falls. The best method still trails the three 13-year-olds by about 31.3 percentage points on the combined average. Humans were tested on only those three tasks.
| Setup | Metric | Result |
| GPT-3 (350M to 175B) | Negated prompts | Worse with scale on all three task types |
| Same models | Original prompts | Better with scale |
| Average of both | Nine tasks | About 50%, roughly flat |
| T0, InstructGPT, most ICL and fine-tunes | Same average | About 50%, except GPT-3 175B ICL on QA |
| Best method vs three 13-year-olds | Same average | About 31.3 points behind |
| Fine-tune on negation (at most 1.3B) | Original task | Falls while the negated task rises |
The cause is marked as a conjecture. Pretraining text leans affirmative, so a large model treats a rare not as a typo and finishes the affirmative sentence. Bigger models are harder to flip with one word. They may also just be fitting conditional probabilities. Neither account is tested.
Do not do X is not a dependable switch on these models. At 175B the affirmative instruction is more stable. InstructGPT and T0 were trained further to generalize to new instructions, and their combined average still hugs 50%.
Few-shot prompts do not repair this across the board. A small model fine-tuned on negated data can solve the negated items while the original task drops. It changes sides. Fine-tuning stops at 1.3B, and the size of that drop is not reported.
The paper is a measurement. Flip the polarity, and scale, instruction training, and in-context examples still leave a gap of about 31 points to a 13-year-old. An alignment method that claims the model now follows instructions needs this flip in the test set.
The corpus story is unchecked. There is no count of affirmative versus negative sentences, and no intervention that changes the mix to see whether inverse scaling remains.
The human side is narrow: three tasks, 100 items each, three 13-year-olds. The 31.3 point gap is not a general deficit against adults. The 300-item cap is an API budget.
The experiment is thinner than understanding a prompt. Negation is mostly a swap from correct to incorrect, with choices written into the prompt. An open-ended order not to delete a span is untested. The monkey example in the introduction has no number attached.
The 50% figure is not a separate ability score. On a two-choice item, a model that ignores negation and picks the same option twice is right once and wrong once, so the average lands near 50%. Inverse scaling is then the mirror of ordinary scaling. With more than two options, several answers count as correct under negation, so chance is higher, yet the combined average is still reported near 50% and the two scores are not released separately. How clean the mirror is cannot be checked.
Fine-tuning helps in every setting they tried, but only from 125M to 1.3B, at a fixed learning rate of 1e-3, for 5 epochs, using the last checkpoint. Zero-sum stays qualitative.