PostHog engineer on moving into AI product engineering: learning evals and agent interfaces
rseroter · x · 2026-11-21
PostHog engineer Dylan Martin writes about shifting from traditional product engineering (leading the feature flags team) to AI product engineering.
Key takeaways:
- The transition felt smaller than job descriptions suggest: software engineering itself is changing dramatically, and staying in systems engineering wouldn't have insulated him from it. Understanding systems, failure modes, and proving correctness still matters.
- He already used agents daily on the flags team, knew their failure modes, and had built systems for validating agentic work.
- The perspective shift: from mitigating agent pitfalls as a user to designing products that deliver productivity gains without the traps.
- New skills include writing evals and balancing human vs. agent interfaces.
More from coding & agent
- ENZO open-source self-hosted AI workspace unifies free-tier APIs and local models — anonymous20256 · 2026-11-21
- Codex raises WebSocket limit to 1 GB, big speed gains on large turns — charliermarsh · 2026-11-21
- Codex raises WebSocket request limit 8x to 128MB, making large turns 5x faster — charliermarsh · 2026-11-21
- Neon boosts Antigravity cache hit rate ~4%, raising quota effectiveness by at least 20% — cgarciae88 · 2026-11-21
- ArcFront launches with 4 of 5 frontier labs as customers, targeting agent evals — eptwts · 2026-11-21
- NEJM AI perspective: cardiologist took 3rd in coding contest via vibe coding, but clinical reality demands more — zakkohane · 2026-11-21