Azure Data Engineer for Microsoft Fabric

# Azure Data Engineer — Microsoft Fabric, Databricks, Cosmos DB and Azure DevOps ## Project overview We are building the first operational data-ingestion and transformation flow for an insurance technology platform on Azure. We are looking for a mid-senior Azure data engineer with practical experience integrating Microsoft Fabric and Azure Databricks. This is a fixed-price, milestone-based engagement. The first phase will implement the complete orchestration pattern for one source entity, `accounts`, as a reusable vertical slice. Later phases may extend the pattern to products, quotes, orders, payments and policies. We already have a development platform foundation, infrastructure-as-code, Fabric automation, Databricks Asset Bundles, security controls, automated validation and Azure DevOps deployment stages. The successful engineer will extend this foundation rather than redesign or replace it. The detailed architecture and repositories will be shared with shortlisted candidates. The supplied **Recommended MVP Architecture** and its **Orchestration Pattern** are the source of truth for implementation. ## Current architecture - Azure Cosmos DB is the operational system of record. - Microsoft Fabric Data Factory Pipelines own scheduled batch ingestion, ingestion history and successful watermark advancement. - Microsoft OneLake is the persistent analytical storage layer. - Bronze, Silver, Gold, AI Data and Knowledge use separately governed Fabric Lakehouses. - Azure Databricks is stateless engineering compute for transformation, data quality and reconciliation. - Databricks reads and writes approved OneLake locations; it must not create a second analytical storage platform. - Gold data is served through the Fabric Lakehouse SQL analytics endpoint. - Azure resources, Fabric items and Databricks workloads use separate deployment mechanisms and credentials. - Azure DevOps provides validation, protected development deployments and sanitized deployment evidence. This contract is development-only and batch-only. ## Phase 1 objective Implement a production-shaped `accounts` data flow that proves the complete orchestration pattern and can be reused for other operational entities. The flow must: 1. Start from a scheduled Fabric Data Factory Pipeline. 2. Read configuration for the enabled source entity. 3. Support an initial full load and later incremental loads from development Cosmos DB. 4. Use a reliable timestamp-plus-identifier watermark strategy based on the source contract. 5. Write source-aligned Bronze Delta data to OneLake with ingestion metadata. 6. Capture source and Bronze row counts and reconciliation inputs. 7. Trigger a version-controlled Databricks workflow deployed through an Asset Bundle. 8. Transform Bronze data into a conformed Silver account dataset. 9. Run schema, mandatory-field, business-key, timestamp, status, duplicate, relationship, monetary and PII-related checks where applicable. 10. Send invalid records to quarantine with actionable failure reasons. 11. Prevent Gold publication when a critical data-quality check fails. 12. Publish an initial Gold buyer/account data product after successful validation. 13. Make the approved Gold output visible through the existing SQL analytics endpoint. 14. Record successful or failed run metadata. 15. Advance the entity watermark only after the associated ingestion and required downstream checks complete successfully. The implementation must address idempotency, replay safety, late-arriving records, partial failure, retries, observability and sanitized deployment evidence. ## Required data contracts The solution should implement and document the following minimum metadata. ### Bronze ingestion metadata - Ingestion batch ID - Source system and source entity - Source record ID - Source update timestamp - Ingestion timestamp - Fabric pipeline run ID - Source schema version ### Pipeline run metadata - Pipeline and source entity identifiers - Start and completion timestamps - Status - Source, Bronze, Silver and rejected row counts - Watermark range - Sanitized error information ### Data-quality and reconciliation results - Rule identifier, name and severity - Records checked and failed - Result status and execution timestamp - Source, Bronze and Silver counts - Count difference and reconciliation status Exact table and field mappings will be finalized during the first milestone and must remain compatible with the approved architecture. ## Deliverables and fixed-price milestones ### Milestone 1 — Discovery and contract validation Deliver: - Review of the architecture, accepted decisions, account source schema, current infrastructure, identities and pipelines. - Source-to-Bronze-to-Silver-to-Gold mapping for `accounts`. - Incremental extraction and compound-watermark design. - Data-quality, reconciliation and quarantine rule catalogue. - PII classification and proposed Gold exclusions or transformations. - Implementation sequence, risks, dependencies and required platform access. Acceptance: - Design follows the supplied architecture without introducing an alternative orchestration or storage platform. - Mapping and failure behavior are sufficiently detailed for implementation. - Security and PII decisions are explicit and reviewable. ### Milestone 2 — Fabric ingestion and operational controls Deliver: - Configuration-driven full and incremental Cosmos DB extraction for `accounts`. - Source-aligned Bronze Delta writes with required ingestion metadata. - Per-entity watermark handling that advances only after success. - Pipeline-run and reconciliation-input recording. - Retry, failure, rerun and late-arriving-record behavior. - Idempotent Fabric item deployment using the repository's existing automation approach. Acceptance: - Full and incremental development runs succeed without unrestricted analytical access to Cosmos DB. - Failed or partial runs do not advance the watermark. - Replaying the same batch does not create duplicate trusted records. - Source and Bronze counts are captured and reconcilable. ### Milestone 3 — Databricks transformation, quality and Gold publication Deliver: - Asset Bundle-managed Bronze-to-Silver transformation. - Schema enforcement, standardization and deterministic deduplication. - Data-quality checks and machine-readable results. - Quarantine output with rule identifiers and failure reasons. - Silver-to-Gold task gated by critical data-quality results. - Initial Gold buyer/account data product written to the approved OneLake location. - Reconciliation between source, Bronze and Silver. Acceptance: - Databricks remains stateless compute and does not create permanent duplicate storage. - Critical failures prevent Gold publication. - Valid and invalid test records follow the expected Silver/quarantine paths. - Approved Gold output is queryable through the existing Fabric SQL analytics endpoint. ### Milestone 4 — Automated testing and Azure DevOps integration Deliver: - Python unit and contract tests. - Transformation, schema and data-quality tests. - Tests for full load, incremental load, compound watermark boundaries, rerun safety and late-arriving records. - Tests for quarantine, critical-quality failure, failed-run metadata and watermark non-advancement. - Reconciliation and Gold-visibility tests. - Mocked Fabric and Databricks integrations for local validation. - Authenticated development smoke/integration tests kept separate from local tests. - Azure DevOps integration extending the existing validation and protected deployment stages. - Sanitized build and deployment evidence. Acceptance: - The repository's validation command and complete automated test suite pass. - Azure, Fabric, Databricks and source credentials remain separately scoped. - Pipeline changes preserve existing approvals, environments and deployment boundaries. - A failed validation or critical integration test prevents promotion. ### Milestone 5 — Documentation and handover Deliver: - Deployment, execution, monitoring, replay and troubleshooting runbooks. - Data contracts, DQ rules and metric definitions. - Sanitized evidence from a successful development run and controlled failure test. - Knowledge-transfer walkthrough for our engineering team. - Prioritized recommendations for extending the pattern to products, quotes, orders, payments and policies. Acceptance: - Another engineer can deploy, operate, diagnose and safely rerun the solution from the documentation. - No required operational knowledge exists only in the contractor's personal environment. Each milestone will be reviewed and accepted before the associated fixed-price payment is released. ## Required experience Please apply only if you can demonstrate hands-on experience with most of the following: - Microsoft Fabric Data Factory Pipelines, Lakehouses, OneLake and Delta tables. - Fabric connections, workspace items, SQL analytics endpoints and supported deployment APIs. - Azure Databricks, PySpark, SQL, Delta Lake and Databricks Workflows. - Databricks Asset Bundles and serverless/classic compute boundaries. - Reading and writing approved OneLake locations from Databricks. - Azure Cosmos DB batch extraction and reliable incremental watermark patterns. - Medallion architecture with separate Bronze, Silver and Gold responsibilities. - Data-quality gates, quarantine, reconciliation and processing metadata. - Azure DevOps YAML pipelines, protected environments, service connections, artifacts and approvals. - Python automated testing and mocking of external platform calls. - Azure RBAC, workload identities/service principals, Key Vault and least-privilege access. - PII classification, masking or exclusion from unrestricted analytical datasets. Insurance experience is useful but not mandatory. Strong Azure data-platform implementation experience is more important. ## Non-negotiable engineering requirements - Cosmos DB remains the operational system of record. - Fabric owns ingestion and watermark advancement. - Databricks owns complex transformation, data quality and reconciliation. - OneLake owns permanent Bronze, Silver and Gold data. - Existing infrastructure-as-code, Fabric automation, Databricks Asset Bundles and Azure DevOps stages must be preserved and extended. - Automation must be deterministic, environment-parameterized, non-interactive and idempotent where supported. - External Fabric and Databricks calls must be mocked in local tests. - Secrets, tokens, customer data, generated deployment output and local state must not be committed. - Bronze and Silver must remain engineering-restricted. - PII must be classified and excluded, masked or transformed before unrestricted Gold consumption. - No backend business-logic changes may be made without prior approval. We can assist with Azure DevOps permissions, service connections, protected environments and access coordination where required. The contractor must still be capable of designing and implementing the pipeline integration. ## Out of scope for this contract - Production, test/UAT or disaster-recovery environments - Streaming, Event Hubs, Service Bus or queue-based ingestion - Airflow or a standalone Azure Data Factory resource - Replacing Fabric orchestration with Databricks ingestion - Databricks-owned permanent Bronze, Silver or Gold storage - Fabric Warehouse - Microsoft Foundry, Azure AI Search or Fabric Data Agents - Power BI dashboard or semantic-model design - Expanding ingestion to all operational entities - New application or backend business features Power BI readiness should be validated by confirming the approved Gold output can support later semantic-model consumption. Dashboard implementation is not included. ## What to include in your proposal Please answer each item directly: 1. Describe one or two comparable solutions where you integrated Microsoft Fabric and Azure Databricks. State what you personally implemented. 2. How would you design an incremental Cosmos DB extraction using `updatedAt` plus a stable identifier so records sharing the same timestamp are neither skipped nor duplicated? 3. How would you prevent a watermark from advancing after a partial ingestion, Databricks failure or critical DQ failure? 4. How would you implement quarantine and reconciliation across source, Bronze and Silver? 5. How would you test Fabric and Databricks automation locally and in Azure DevOps without exposing credentials? 6. Confirm that you can extend the supplied architecture without substituting Airflow, standalone ADF, streaming, Databricks-owned storage or another orchestration platform. 7. Provide a fixed-price estimate for each of the five milestones. 8. Provide your estimated timeline, weekly availability, earliest start date, assumptions and required access. 9. Identify any part of the scope that you believe should be changed, with a concise technical reason. Generic proposals that do not address the architecture and failure scenarios above are unlikely to be shortlisted. ## Engagement details - Contract type: fixed-price, milestone-based - Initial environment: development only - Initial entity: `accounts` - Working arrangement: remote - Documentation and communication: written English and recorded or live technical walkthroughs - Potential follow-on work: additional operational entities and later analytical products, subject to successful Phase 1 delivery Please submit your milestone estimate rather than assuming a published budget.