Web Scraping & Data Automation Engineer

We are looking for a highly capable engineer to help us build the data and automation infrastructure behind our platform. The immediate need is someone who can build reliable web scraping systems across a wide variety of websites. But this role goes well beyond scraping. We are looking for someone who can help us connect web scraping, databases, AI, workflows, and automation into one scalable system. Today, many of our processes are built around Excel and manual workflows. Our goal is to move toward an infrastructure where data is continuously collected, processed, enriched, analyzed, and routed through automated systems with significantly less human intervention. This person will play a major role in helping us build that system. Core Responsibilities 1. Web Scraping & Data Collection Build web scrapers and automated data collection systems for a wide variety of websites Develop reusable scraping infrastructure rather than one-off scripts Work with static and JavaScript-rendered websites Work with APIs, browser automation, authentication, pagination, rate limits, and other scraping challenges Build systems that can run continuously or on scheduled intervals Monitor scrapers and build systems for detecting failures or changes to websites Clean, normalize, validate, and deduplicate collected data 2. AI Integration A major part of this role will be determining where AI can be integrated into our data collection and operational workflows. We are not simply looking for someone who knows how to call an AI API. We want someone who can identify opportunities to use AI to make our systems substantially more intelligent and automated. Potential applications include: Using AI to extract structured information from unstructured websites, articles, social media, and other sources Using AI to classify and categorize scraped information Using AI to identify whether two records represent the same person/entity Using AI for data matching, enrichment, and validation Using AI to identify changes or important events in newly collected data Using AI to summarize or interpret information collected by scrapers Connecting AI models directly to databases and internal tools Building AI agents or workflows that can research information and take defined actions Using AI to determine which sources should be researched next Creating automated workflows where AI makes a decision and triggers the next step in a process Using AI to reduce repetitive manual research and data-entry work We are interested in someone who can think creatively about how AI can become part of the infrastructure, rather than simply being an add-on chatbot. 3. Data Infrastructure Help transition existing Excel-based processes into more scalable database-driven systems Design databases and data structures that can support growing volumes of information Connect scrapers, databases, APIs, AI systems, and internal tools Build data pipelines that move information from collection → processing → validation → database → application Create systems for tracking changes to records over time Build automated data-quality checks Develop APIs and integrations where appropriate 4. Workflow Automation We have a number of recurring processes that currently require manual work. The goal is to identify these processes and determine: What can be automated? What should be handled by traditional code? What should be handled by AI? And how should everything connect? You may work on: Automated research workflows Data update workflows Notifications and alerts Record matching Data validation Database updates Reporting Internal administrative tools AI-powered research workflows Integrations between third-party platforms and our database