Senior Data Quality Engineer

A major New York City Health organization is seeking a Data Quality Engineer, Data Lakehouse to support Communications, Marketing, Government & Community affairs department. This role will help development and execute a best-in-class data infrastructure to enable a solid framework for data driven decision making, reporting & analytics and marketing automation. In this role, the ideal candidate will take ownership of the quality, reliability, and integrity of our data assets within the Databricks Lakehouse Platform. You must have hands-on experience as a data quality expert, responsible for ensuring that the data powering departmental analytics and data science is accurate, timely, and trustworthy. You will be one of the subject matter experts for data quality within the department and work closely with data engineers and analysts. Your primary focus will be on designing, implementing, and automating a robust data quality framework across our entire data pipeline, from raw data ingestion to curated gold-level datasets, all within the Databricks ecosystem. Key Responsibilities Data Quality Framework & Strategy • Design and implement a data quality framework for our Databricks pipelines, leveraging the Medallion Architecture (Bronze, Silver, Gold) • Enforce data quality standards and validation rules tailored to the business logic • Develop and maintain KPIs, metrics, and dashboards to track quality of key datasets over time Implementation & Automation • Implement automated data quality checks and tests directly into our data pipelines using Databricks-native tools and associated technologies. • Leverage Delta Live Tables (DLT) with declared expectations (CONSTRAINT / ASSERT) to enforce data quality rules and manage pipeline failures. • Build and maintain data validation test suites using tools on Databricks, PySpark, or SQL. • Write custom data quality tests to validate business logic, consistency, and accuracy • Integrate quality checks into CI/CD pipelines and Databricks Workflows Monitoring, Governance & Lineage • Utilize Databricks Unity Catalog for data governance • Implement data quality rules to perform root cause analysis on quality issues • Develop automated monitoring and alerting systems to proactively detect and notify stakeholders of data anomalies, pipeline failures, or schema drifts. • Identify potential quality risks before they are integrated into production pipelines • Maintain a Data Quality Issue Log, tracking incidents documenting the root cause Qualifications • 5+ years experience in a data quality role or data engineering that features QA. • Must have Databricks Lakehouse Platform exp, (Spark, Delta Lake, and Databricks SQL) • Proficient in Python (especially PySpark) and advanced SQL • Proven experience implementing data quality frameworks and automated testing • One major data quality or testing framework (e.g., dbt, Great Expectations, Deequ) • Solid understanding of modern data engineering principles, including CI/CD, Git, and data orchestration (e.g., Databricks Workflows, Airflow) • Excellent problem-solving skills and the ability to perform complex root cause analysis on data issues. • Direct experience with Delta Live Tables (DLT) and its expectation syntax. • Hands-on experience using Databricks Unity Catalog for governance, monitoring. Collaboration & Domain Expertise • Partner closely with data engineers to embed quality checks into development workflows and promote a "quality-first" engineering culture. • Collaborate with data analyst team to understand their data requirements, translate business rules into technical validation logic, and build trust in the data. • Act as the primary point of contact for all data quality inquiries related to the department's data assets. • Educate and train team members on data quality best practices, tools, and processes.