Perfume comparison app can't reliably scrape Bahrain store prices despite trying every scraping stack
Leading_Layer_546 · reddit · 2026-11-16
A small team building a perfume-vs-perfume comparison app (price, reviews, specs, pros/cons) is stuck on prices: local Bahrain store pages list BHD prices in plain sight, yet their pipeline often returns nothing — Sauvage gets 58.110 BHD while Bleu de Chanel shows "no confirmed price yet."
Current flow (Python): search product, pick likely store pages, fetch, parse JSON-LD/OpenGraph/microdata (plus Shopify/Woo JSON), fall back to an LLM. Each comparison takes 25 seconds.
Tried so far: curlcffi with Chrome impersonation (best results, and free), Playwright, patchright, crawl4ai, Firecrawl, Scrape.do, Zyte, Bright Data, plus Claude with web search on top. Still frequently empty — and pasting product links or uploading bottle photos often times out or returns nothing.
More from coding & agent
- Quant researcher co-writes a short math-heavy book on agentic engineering with LLMs — blaizedsouza · 2026-11-16
- llm-council built into Hermes: multiple models answer, blind-rank, and a chairman decides — Scobleizer · 2026-11-16
- Grok Build v1.0.87 improves multi-agent workflows, MCP panel and session search — XFreeze · 2026-11-16
- Dynamic tool discovery vs loading every MCP schema upfront for agents — Pakiya09 · 2026-11-16
- Swarm orchestration runs 12 parallel agents to build a Blender 3D model, claims tester — venturetwins · 2026-11-16
- AgentWeb lands YC W27 spot building endpoints that let AI agents operate any site — damnGruz · 2026-11-16