Multi-agent shopping crawler comparing products with affiliate URL masking and rotated proxy pools.
E-commerce buyers face extreme fragmentation when comparing prices across major digital storefronts, while developers face scraper bans and rate limiting when querying prices.
We built PricePilot (DealLens AI), an AI-assisted shopping search bot engine.
Sprint 1: Connector integrations & bot orchestration
Sprint 2: Search cache setup
Sprint 3: Bot tracing UI & logs validation
Product sweeps on major shopping marketplaces triggered strict Cloudflare captchas. Solved by integrating headless Playwright workers with rotating User-Agents and jitter query delays.
Querying multiple databases concurrently caused slow search responses. Resolved by setting up OpenSearch index templates and caching responses in Redis.