TIME Serves AI Crawlers Stripped-Down Markdown With Ads Built In
bibryam · x · 2026-09-02
TIME is delivering different content based on the User-Agent. While browsers receive the full 300KB HTML page, AI crawlers like ClaudeBot, PerplexityBot, and OAI-SearchBot get a 13KB stripped-down Markdown version. Crucially, this Markdown version includes ad code from vendor Mobian. OpenAI's GPTBot and ChatGPT-User are blocked with a 406 status, indicating a selective policy where specific bots are allowed to index a text-only, ad-supported version of the content.
More from Apps
- Open-source OpenShorts undercuts Opus Clip and Descript with free local AI clipping — aigleeson · 2026-09-02
- The Influencer AI: Platform for Generating Consistent Virtual Personas — jonathan_wilke · 2026-09-02
- Dev builds local semantic file search tool FileSeek — revived_soul_37 · 2026-09-02
- 7 Powerful Open-Source AI Projects: Agent Context to Self-Hosted UIs — Shruti_0810 · 2026-09-02
- Fable 5.1 praised for feeling like a human co-founder — MattGarciaEth · 2026-09-02
- Fable 5.1 Update: Removes Controversial Data Retention Policy — Stratechery · 2026-09-02