Senior Web Scraping Engineer — Python, Browser Automation, Production Data Pipelines

We are hiring a senior scraping engineer to join an experienced team building a real estate data platform. This is not a new build starting from scratch. We are approximately 90% of the way to launch, with the core platform already in place and the team pushing through the final stage of development. We are a small team, which means every hire matters. We are looking for an A player who can take ownership of data acquisition, solve difficult problems, and help us get the product across the finish line. The Role You will own data acquisition for our platform. Your primary focus will be building, maintaining, and improving our scraping systems. You will not own frontend development, billing systems, or the broader application backend. However, you must be comfortable with general backend development and understand how your scraping systems connect to the rest of a production application. You may need to assist with integration work, troubleshoot issues across system boundaries, and make backend changes related to the reliable delivery of scraped data. Your job is to: -Scrape the required source data. -Build and maintain reliable production scrapers. -Validate the completeness and accuracy of the results. -Deliver data using the agreed schema and system architecture. -Work with the backend team to integrate the data successfully. -Continue supporting and improving the scraping system after integration. We are looking for someone who genuinely enjoys scraping and is willing to work through difficult websites, changing page structures, browser issues, blocked requests, and incomplete data. This is not a role for someone who only builds simple, one-time scripts. We need someone who can own a production scraping system, diagnose failures independently, improve reliability, and keep the data flowing when source websites change. You will work closely with a Lead Backend Engineer who owns the broader backend platform. Scraping and data acquisition will remain your primary responsibility, but you will be expected to collaborate carefully on integrations and take responsibility for the ongoing performance of what you build. Responsibilities -Build and maintain Python scrapers. -Work with dynamic and JavaScript-heavy websites. -Build and maintain browser automation. -Diagnose blocked, failed, or incomplete runs. -Handle sessions, cookies, proxies, rate limits, and other access challenges. -Update scrapers when source websites change. -Validate data completeness and accuracy. -Prevent duplicate and missing records. -Add logging, retries, monitoring, and alerts. -Deliver data using an agreed schema and architecture. -Collaborate with the backend team on integration and troubleshooting. -Support scraping-related backend components when needed. -Document your work clearly. -Recommend better data-acquisition methods when appropriate. -Take long-term ownership of the reliability of the scraping systems you create. Required Experience -Strong Python experience -General backend development experience -Playwright, Selenium, Scrapy, or similar tools -Production web scraping -Browser automation -HTML parsing and structured data extraction -Proxy and session management -Retry and recovery systems -Monitoring unattended scrapers -Data validation -Docker -GitHub Experience with Cloudflare, CAPTCHA workflows, public records, APIs, Google Cloud Run, MongoDB, or BigQuery is helpful. What Success Looks Like -Scrapers run reliably without constant supervision. -Failures are detected quickly. -Missing and duplicate records are reduced. -Website changes are handled quickly. -Data is delivered accurately and on schedule. -Scraping systems integrate cleanly with the broader platform. -The backend team does not need to take ownership of maintaining the scrapers. -The systems you build remain understandable, supportable, and reliable over time. -We Are Looking for Someone Who enjoys solving difficult scraping problems. -Can work independently and diagnose issues without constant guidance. -Has experience maintaining production scraping systems. -Is comfortable working within a broader backend architecture. -Understands how to keep data pipelines reliable as websites change. -Takes care with architecture, documentation, testing, and maintainability. -Can communicate technical issues clearly and recommend improvements. -Takes ownership and pushes projects across the finish line.