You will get a data quality checker for your warehouse with automated CI/CD checks
Project details
Bad data breaks dashboards and reports silently — a null slips through, duplicate rows inflate totals, an orphaned key drops records out of a join, and nobody notices until a stakeholder asks why the numbers look wrong. I build automated quality checks that catch this before it reaches your warehouse or BI tool.
My checker validates against a proper Kimball-modeled star schema and runs configurable checks for nulls, duplicates, referential integrity, freshness, and schema drift, outputting a structured JSON report you can plug into any alerting system. Infrastructure is provisioned with Terraform, and a GitHub Actions pipeline enforces the checks automatically
on every push — this isn't a one-off script, it's built to run in a real CI/CD pipeline. Code is public on GitHub so you can review it before you buy.
My checker validates against a proper Kimball-modeled star schema and runs configurable checks for nulls, duplicates, referential integrity, freshness, and schema drift, outputting a structured JSON report you can plug into any alerting system. Infrastructure is provisioned with Terraform, and a GitHub Actions pipeline enforces the checks automatically
on every push — this isn't a one-off script, it's built to run in a real CI/CD pipeline. Code is public on GitHub so you can review it before you buy.
Database Type
SQLiteWhat's included
| Service Tiers |
Starter
$350
|
Standard
$600
|
Advanced
$900
|
|---|---|---|---|
| Delivery Time | 3 days | 5 days | 7 days |
Number of Revisions | 1 | 2 | 3 |
Source Code |
Optional add-ons
You can add these on the next page.
Extra table validated
(+ 2 Days)
+$150Frequently asked questions
About Krupa
Data Engineer | ETL Pipelines, PostgreSQL, Python, dbt, Kafka
Ahmedabad, India - 8:07 am local time
I'm a data engineer specialising in ETL/ELT pipeline design, real-time data ingestion, and analytics infrastructure. I work with Python, SQL, Apache Kafka, dbt, PostgreSQL, BigQuery, and PySpark to build data systems that run in production — not just demos.
What I can do for you:
Design and build ELT pipelines using Airflow, dbt, and BigQuery – staging, intermediate, and mart layers with incremental models and schema tests
Real-time data ingestion and streaming with Apache Kafka and PySpark Structured Streaming — micro-batch processing, JSON parsing, and time-series storage
RAG chatbots and document Q&A systems using LangChain, ChromaDB, and Streamlit — semantic search, source citations, multi-turn memory
SQL analytics and query optimisation across PostgreSQL, BigQuery, and MySQL — window functions, CTEs, recursive queries, stored procedures
Grafana dashboards for live operational monitoring connected to TimescaleDB hypertables
Python automation for data ingestion, transformation, and pipeline orchestration using Pandas, NumPy, and SQLAlchemy
Tableau and Power BI dashboards for KPI tracking and business reporting
Data quality management: validation, anomaly detection, schema enforcement built into pipelines from day one
Automated data quality pipelines with CI/CD — Kimball-modeled warehouses, null/duplicate/referential-integrity/freshness checks, Terraform-provisioned AWS infrastructure, and GitHub Actions enforcing quality on every push
Dimensional modeling and data warehouse design for scalable analytics
What makes my pipelines different:
I don't just write scripts — I build systems with failure handling, retry logic, schema validation, and documentation. Every project I deliver includes a README with architecture decisions, how to run it locally, and why I made the technical choices I did. Clients don't get black boxes.
Recent projects (all on GitHub):
→ Real-Time Sales Pipeline — Kafka + PySpark + TimescaleDB + Grafana. Python producer sends 2 synthetic orders/sec to Kafka. PySpark Structured Streaming reads, parses JSON, and writes to TimescaleDB hypertables. 3-panel Grafana dashboard: live revenue gauge, orders/min time series, and product breakdown pie chart. 138+ rows confirmed end-to-end. Full-stack
containerised via Docker Compose.
→ E-Commerce ELT Pipeline — Airflow + dbt + BigQuery on 99K+ Brazilian Olist e-commerce orders. Airflow 2.8.1 orchestration, dbt transformations across staging → intermediate → mart layers with incremental models and schema tests, BigQuery warehouse, fully containerised. Production-grade pipeline design with documented architecture.
→ RAG Document Chatbot — LangChain + ChromaDB + Streamlit. Ingests PDFs, chunks semantically using Hugging Face embeddings, stores in ChromaDB vector store, and serves answers via LangChain Q&A chain with source citations and conversation memory. Clean Streamlit UI with adjustable chunk size and retrieval count sliders. Runs locally via Ollama — no
external API dependency.
→ Data Quality Checker — Kimball star schema + AWS S3 + Terraform + GitHub Actions. Validates a data warehouse (DuckDB, dialect-portable to Snowflake/Postgres/BigQuery) with automated checks for nulls, duplicates, referential integrity, freshness, and schema drift. Infrastructure
provisioned as code with Terraform, CI/CD pipeline runs lint + full pytest suite + terraform validate on every push. Seeded with real data quality issues to prove the checks actually catch them.
Tech stack:
Data Engineering: Apache Kafka · PySpark · dbt · Airflow · ETL/ELT · TimescaleDB · HDFS · Spark SQL · DuckDB
Databases: PostgreSQL · MySQL · BigQuery · TimescaleDB
Languages: Python (Pandas, NumPy, SQLAlchemy) · SQL · PL/pgSQL · Spark SQL
Visualisation: Tableau · Power BI · Grafana · Matplotlib
Cloud & Infra: Azure · Google Cloud (BigQuery, GCS) · AWS (S3, boto3) · Terraform (Infrastructure as Code)
Tools: Git · Docker · GitHub Actions (CI/CD) · PowerShell · Dimensional Modeling
Background:
P.G. Diploma in Big Data Solution Architecture (Conestoga College, Canada) and M.E. in Computer Engineering (GTU). International experience — studied and worked in Canada: async-first communication, clear documentation, and on-time delivery.
If you need someone who builds data systems that actually run in production, let's talk.
Steps for completing your project
After purchasing the project, send requirements so Krupa can start the project.
Delivery time starts when Krupa receives requirements from you.
Krupa works on your project following the steps below.
Revisions may occur after the delivery date.
1
Review your schema and requirements, confirm which table(s) and columns to validate
2
Configure and run the quality checks against your data