RLHF's scalar reward hides a 2,450-year-old problem: helpfulness and harmlessness are opposing forces

vasilisvj · reddit · 2026-11-04

A long essay reading RLHF through Empedocles' cosmology: helpfulness and harmlessness are opposed forces (philia and neikos) that preference training collapses into one scalar reward, never reconciling them.

The analogy

Helpfulness = attraction (merging with the user's request); harmlessness = repulsion (refusal, distance). Neither force is "the good one"; structure comes from their tension.

Observable artifacts of the scalar collapse

Sycophancy: answers shift under mild pushback — attraction winning, truth as friction.
Hedging stacks: answers wrapped in disclaimers or plus a lecture — repulsion pulling back mid-generation, answering and refusing at once.
Inconsistent refusals: same content refused in one framing, answered in another — the boundary is positional in reward space, which is why red-team results don't replicate.

The design claim

Health isn't a fixed midpoint but knowing which force should act now and why — yet reward models carry no representation of which objective is active, since it's baked into one number at training time.
Multi-objective RL has the machinery (Pareto fronts, constrained optimization, lexicographic ordering) that LLM preference training mostly ignores.

The author isn't anti-RLHF, just argues the scalar is lossy in a specific, ancient, predictable way — and asks whether anyone keeps both objectives separate and arbitrates at inference time.

Original post →

RLHF's scalar reward hides a 2,450-year-old problem: helpfulness and harmlessness are opposing forces

More from AGI Musings