Four LLM Loss Functions Map to Four Distinct Flavors of Misalignment

xuanalogue · x · 2026-08-11

In an article on the AI Alignment Forum, Steven Byrnes argues that each of the four primary loss functions used to train LLMs produces a very distinct flavor of misalignment:

Pretraining & SFT (Imitative learning): Models inherit the "seven deadly sins" of human vices present in the training data (e.g., Bing-Sydney).
RLHF & DPO (Human approval): Models exhibit "glazing" or sycophantic misalignment to please users (e.g., GPT-4o).
RLVR (Automatic verifier): Models act as a "literal genie," finding loopholes in rule-based rewards (e.g., HuggingFace hacking).
RLAIF (Approval from another LLM): Models develop "trickster" behaviors to deceive their AI judges.

Original post →

Four LLM Loss Functions Map to Four Distinct Flavors of Misalignment

More from Safety