Raycast v2.2 adds multi-provider AI routing, sparking debate on fair local vs hosted model evals
daani_maas · reddit · 2026-09-10
Raycast v2.2 lets Pro users route AI workflows through OpenAI-compatible providers, Ollama, or OpenRouter — but switching providers also changes tool support, retries, latency, and help-seeking frequency, complicating comparisons.
The author proposes freezing tool schema, permissions, input fixtures, and acceptance checks, then comparing task success on hidden checks, total cost, tool-call counts, recovery after failed calls, and unsupported features.
They ask the community whether anyone has built a provider-neutral eval around one workflow, and which variable was hardest to keep constant.
More from coding & agent
- YC to host 'Make Something Agents Want' hackathon Oct 17-18 in SF as agents become new users — ycombinator · 2026-09-10
- Next AI adoption wave will be computer use: Astra already automates real office workflows, argues thread — zephyr_z9 · 2026-09-10
- Stripe Launches Agentic Treasury in Public Preview, Lets AI Agents Move Money — jeff_weinstein · 2026-09-10
- Meta's Stilla acquisition is a distribution bet, not an agent capability bet — arthaudm · 2026-09-10
- Indie dev builds 15-book multilingual kids' reading app with Fable 5.1 in 3 days — GCWebDesigner · 2026-09-10
- A one-line prompt 'change 10 to 5' got an agent to modify an existing codebase — thisdudelikesAI · 2026-09-10