Principal Data Acquisition and Trust Engineer

# Principal Data Acquisition and Trust Engineer ## About the Role We are hiring a Principal Data Acquisition and Trust Engineer to own the technical integrity of a large-scale public web data acquisition platform. The platform collects structured and semi-structured information from a broad network of public websites and document repositories on a recurring schedule. Sources include static websites, JavaScript-driven applications, PDFs, office documents, and frequently changing page structures. The central challenge is not simply retrieving data. It is determining whether each stage of acquisition and extraction succeeded, detecting when source behavior changes, preserving complete lineage, and ensuring failures can be recovered through targeted reprocessing. This is a principal-level individual contributor role. It is not a people-management position. The successful candidate will be an active builder who also establishes technical direction, validation standards, and operating discipline for the data acquisition function. ## What You Will Own You will own the technical framework that determines: * What data should be acquired * How new sources are discovered and onboarded * How acquisition success is measured * How missing, incomplete, or unexpected content is detected * How source and website changes are identified * How extraction quality is evaluated * How raw captures and transformed outputs are versioned * How records are traced from source through downstream processing * How failures are isolated, retried, replayed, and reprocessed * How architectural changes are validated before replacing existing components You will work within an active production system. You will have latitude to improve or replace major components, but changes must be introduced safely. New approaches must demonstrate measurable improvement, detect their own failures, preserve recoverability, and avoid catastrophic downstream effects. ## Responsibilities * Design and implement the end-to-end validation and observability strategy for web data acquisition. * Establish clear success and failure criteria at each stage of the acquisition and processing lifecycle. * Build checks that distinguish successful execution from successful data acquisition. * Develop systems for detecting website redesigns, missing content, structural drift, semantic changes, blocking, and extraction regressions. * Define and maintain lineage from source URL and raw capture through normalization, extraction, and downstream records. * Preserve sufficient source, execution, code, and configuration history to reproduce or reprocess prior outputs. * Design retry, replay, backfill, and recovery workflows that isolate failures and avoid unnecessary recomputation. * Improve source discovery, onboarding, classification, and coverage tracking. * Define extraction schemas and semantic standards for acquired source content. * Establish validation methods using deterministic checks, statistical analysis, historical comparisons, sampling, and human review. * Improve human-review workflows so review outcomes become reusable quality signals, fixtures, and regression tests. * Build provider-level and source-level health monitoring, dashboards, alerts, and service-level indicators. * Identify patterns of recurring failure and replace source-specific remediation with reusable platform capabilities where practical. * Evaluate architectural changes through controlled comparisons, regression testing, and measurable quality outcomes. * Contribute directly to production code across acquisition, parsing, validation, orchestration, and recovery systems. * Provide technical guidance and review for engineers working on source integrations and extraction workflows. * Lead root-cause analysis for systemic acquisition or data-quality failures. * Define the technical standards used to expand source coverage without reducing data trust. ## What Success Looks Like * Every active source has a clear and current operational state. * Expected non-execution can be distinguished from actual failure. * A completed scraper run does not count as successful unless expected content and outputs are validated. * Provider-wide failures are detected before they silently affect downstream data. * Source, schema, and extraction changes are identified through automated checks. * Production records can be traced back to their source, acquisition event, and processing history. * Failed acquisitions and transformations can be replayed or reprocessed without manual reconstruction. * Architectural changes can be evaluated against historical fixtures and existing production behavior. * Human review produces measurable and reusable validation data. * Source coverage and extraction quality can improve without weakening reliability. ## Required Experience * Significant experience operating large-scale web scraping or public web data acquisition systems. * Experience designing production data-validation frameworks. * Experience building lineage, provenance, metadata, or reproducibility systems. * Strong understanding of website, source, schema, and content drift. * Experience operating distributed data pipelines in AWS or a comparable cloud environment. * Experience owning production failures, root-cause analysis, and recovery. * Strong Python and SQL skills. * Experience working with structured and semi-structured data at scale. * Experience parsing HTML, JSON, PDFs, or office documents. * Experience designing idempotent, replayable, or reprocessable workflows. * Ability to reason about semantic correctness, not only technical schema validity. * Experience introducing architectural changes into active production systems. * Strong written technical communication and system documentation skills. ## Relevant Technical Experience The current environment includes technologies such as: * Python * SQL * TypeScript * AWS Lambda * AWS Glue * AWS Step Functions * Amazon SQS * Amazon S3 * ECS or EKS * Relational databases * Parquet-based storage * Browser automation * HTML and document-processing libraries * Monitoring, dashboarding, and alerting systems Experience with every listed component is not required. Candidates should have operated comparable distributed acquisition and data-processing systems. ## Preferred Experience * Statistical anomaly detection for high-volume data pipelines * Historical drift and distribution monitoring * Data contracts and schema enforcement * Human-in-the-loop validation systems * Source reliability scoring * Geospatial normalization or location-based data * Provider-template or adapter-based scraping architectures * Source-discovery systems * CI testing for data pipelines and extraction logic * Compliance-sensitive, audited, or institutional data environments * Agentic or LLM-assisted developer tooling Interest in agent-assisted scraper recovery is useful. Potential workflows may include gathering failure context, diagnosing source changes, proposing code modifications, validating those modifications against current and historical fixtures, and opening reviewable pull requests. This is not primarily an LLM or entity-extraction role. ## Working Style * You treat successful execution and successful acquisition as separate conditions. * You expect failures and design systems to contain and recover from them. * You require evidence before replacing production components. * You think in terms of complete acquisition funnels rather than isolated scraper jobs. * You care about semantic correctness as well as technical validity. * You convert recurring operational problems into reusable controls. * You remain directly involved in implementation. * You can establish technical authority without relying on people-management responsibility. ## Role Structure This role reports to the CTO and works closely with engineering, product, and data stakeholders. The expected balance is approximately: * 60% implementation and direct technical contribution * 40% architecture, validation strategy, technical standards, and engineering guidance The Principal Data Acquisition and Trust Engineer will provide technical direction to other contributors but will not initially have direct reports. ## Why This Role Matters The reliability of the product depends on the ability to acquire changing public data consistently and explain how every downstream result was produced. This role will establish the controls that make the acquisition platform measurable, recoverable, and trustworthy. It will also define how the system evolves without sacrificing continuity, historical reproducibility, or data quality.