Large Scale Scraping Project Over 100m URL's

Looking for an engineer and/or team who can help with a very large scale scraping project of over 100m URL's. This could very easily become full time work as we may need this every 30 days. The URL set is already defined as in this is fetch-only, not spidering. Success = the full set crawled to an agreed completion rate, at low cost per page, with results landed cleanly in our warehouse. Skills we're looking for Core scraping: - Run crawls at tens-of-millions-to-billions scale (not small scrapers) - Strong in Scrapy, Crawlee, Python asyncio/aiohttp, or Go Concurrency, queueing, and backpressure Headless browsers (Playwright/Puppeteer) — and knowing when not to use them for cost - Land large volumes into cloud storage / a warehouse (schema, partitioning, dedup) Proxies (Important) - Picks the right proxy tier per target (datacenter vs. residential vs. mobile) instead of overspending - Rotation + sticky sessions, geo-targeting, and auto-dropping dead/banned IPs - Bandwidth-cost aware (residential is priced per GB) Please reply with the largest project you have successfully scraped. Looking to hire immediately.