A hiring model with 94% accuracy was cheating: SHAP exposes proxy bias at runtime
ChannelLivid · reddit · 2026-08-19
The author demonstrates the blind spot of accuracy-only validation with a synthetic hiring pipeline: a logistic regression model hits 94.2% validation accuracy yet its training labels were deliberately poisoned — constructed directly from a postcode field, simulating a historical process where location decided who advanced.
A shap.LinearExplainer inspection of a single inference exposes the shortcut: technicalscore contributes -0.0007, yearsexperience -0.068, while postcode weighs 3.5031 — the model isn't measuring suitability, it's faithfully reproducing a biased historical decision rule.
Explanation alone makes the problem visible but enforces nothing. The estimator is wrapped in an L2 semantic execution boundary (ramen-mlflow-guard): the app submits features plus SHAP evidence, and the wrapper checks them against a proxy-bias policy before predict() runs — denying the request with GovernanceDeniedException, so the model never executes, while returning remediation steering, statutory anchors (e.g., EU AI Act), and a locally verified Ed25519 receipt.
More from Safety
- Flock's AI Tool Can Identify Drivers and Track Vehicle Patterns, Report Finds — ArtificialOther · 2026-08-19
- Critique of "Zero Regulation" stance on AGI — danfaggella · 2026-08-19
- Why the concern over "rogue AIs"? User argues guardrails should be enough — kaljakin · 2026-08-19
- Endowed chairs proposed to retain academic voices in AI safety — tallinzen · 2026-08-19
- Student Wins Federal Suit Over AI Detection False Positive, Defense Guide — aitrendz_xyz · 2026-08-19
- AI voice cloning used in terrifying home invasion robbery scam — flavioAd · 2026-08-19