Aging of Prompt Engineering Techniques Across LLM Versions
Anastasiia Rudyk, Julian Oertel, Regina Hebig
cs.SE
2026-08-25
Rostock's ICSME replication on 195 Python tasks finds prompt gains are family-specific: GPT-4o's Few-Shot delta drops 7.4 points vs GPT-3.5; Qwen2.5 still gains +11.5 from CCoT.
Prompt recipes travel with the model. Teams keep Few-Shot, chain-of-thought, and contrastive demos in the same repo and assume a stronger successor will only use them better. This Rostock study checks a narrower claim: after one generation, does the same prompt engineering technique still buy functional correctness on function-level code generation, or does the marginal gain shrink and sometimes go negative.
Khojah et al. (2025) already measured prompt combinations on CodePromptEval. That was a cross-section. What was missing is a version-to-version comparison. If instruction following and reasoning scaffolds are already inside the later checkpoint, extra "think step by step" text is just noise.
The work is a partial replication of Khojah et al., not a full factorial of stacked techniques. CodePromptEval starts at 221 Python function tasks; three fail the few-shot format and drop out, leaving 218. Another 23 tasks yield at least one generation with no extractable code and are removed entirely, so scoring uses 195 tasks. Five templates run in isolation:
Three pairs: GPT-3.5-Turbo / GPT-4o, Qwen2-7B-Instruct / Qwen2.5-7B-Instruct, Mistral-7B-Instruct / Mistral-Large-2407 (123B). Temperature 0.2, three samples per prompt, 1,090 prompts, 19,620 generations, 17,550 scored functions. The only metric is pass@k. Training cut-offs are required to predate CoderEval's February 2024 release, to limit leakage.
On the shared 195-task slice, replicated GPT-4o pass@1 sits about 6.5 points below Khojah's numbers; Few-Shot is the worst gap, 43.6% versus 51.6%. Ranking of raw ability matches the usual order.
| Model | pass@1 | pass@3 |
| Qwen2 | 35.59% | 36.72% |
| Qwen2.5 | 36.41% | 42.36% |
| Mistral-7B | 36.92% | 40.82% |
| Mistral-Large | 42.36% | 44.00% |
| GPT-3.5-Turbo | 42.87% | 44.00% |
| GPT-4o | 43.38% | 44.51% |
Versus Zero-Shot, GPT-3.5 gets about +10 points from Few-Shot and about -7 from PoT. On GPT-4o, Few-Shot shrinks to roughly +1 to +5, while CCoT stays positive (+6.0 / +9.6 / +12.4 across the three runs). Qwen2 mostly loses from Few-Shot and CCoT; Qwen2.5 turns CCoT positive and gains 5.95 points from pass@1 to pass@3, the largest sampling lift. Mistral-7B gains from Few-Shot, CoT, and PoT; Mistral-Large keeps CCoT (about +8.6 to +15.2) and turns CoT and PoT negative.
Table VII reports how each successor's PET delta moves relative to its predecessor: Qwen CCoT +11.5 and Few-Shot +7.9; Mistral CoT -13.8 and PoT -12.1, CCoT still +5.6; GPT Few-Shot -7.4, CCoT only +1.1. GPT-4o rises just 1.13 points from pass@1 to pass@3. Tasks it can solve, it usually solves on the first try.
Aging is family-specific. It is not "newer models ignore prompts."
Treat prompt libraries as perishable. After a family or generation change, rerun a Zero-Shot control before keeping the old recipe. On stronger closed models, start from the shortest prompt. On 7B-class Qwen, Few-Shot and contrastive demos can still return double-digit relative gains. CCoT is the most stable positive signal here because it adds checkable right and wrong traces, not just a request to reason.
This is an empirical update, not a new method. The contribution is turning "prompt engineering gets stale" into numbers you can inspect per family.
The task set is Python function synthesis only, 195 items, three samples per prompt. The Mistral pair is 7B versus 123B, so version effects and scale are tangled; Mistral-7B also wraps prompts in [INST] tags that Large does not use. Replicated GPT-4o sits 6.5 points below the original paper, so the evaluation pipeline is not fully aligned. pass@k ignores readability, security, and efficiency. Adapted PoT is no longer Chen et al.'s original technique, so the negative result does not indict vanilla PoT. GPT-3.5 to GPT-4o is a large jump, and the internalization story is too strong for that pair.