Lead/Principal Data Engineer – DBT & Databricks

About the role

We are looking for a Lead/Principal Data Engineer to design, build and operate DBT models on Azure Databricks. The role focuses on delivering production‑grade gold‑layer data domains in regulated enterprise environments while leveraging AI‑assisted development tools.

Key responsibilities

• Design, build, and maintain DBT models on Azure Databricks using Delta Lake, Unity Catalog, and Medallion Architecture principles.
• Deliver curated gold‑layer data domains to production standards in regulated enterprise settings.
• Utilize AI‑assisted development tools such as Claude Code for model generation, refactoring, documentation, debugging, and engineering acceleration.
• Establish comprehensive DBT testing strategies, including schema validation, data quality testing, unit testing, and automated quality controls.
• Operate in fast‑paced CI/CD environments with pull‑request workflows and automated quality gates.
• Curate datasets, semantic definitions, and business terminology to support Databricks Genie and natural‑language analytics initiatives.
• Define and enforce data contracts, domain ownership, and governance standards.
• Implement data observability, monitoring, and root‑cause analysis across data platforms.
• Drive modernization and migration initiatives through AI‑assisted data engineering approaches.
• Contribute to engineering playbooks, standards, and enablement programs related to DBT, Databricks, and AI‑powered development.
• Measure and improve AI‑generated code adoption, quality, productivity, and delivery outcomes.

Required profile

• Minimum 5 years of data engineering experience.
• At least 2 years of hands‑on experience building and operating production‑grade DBT projects, including models, tests, macros, snapshots, and large‑scale refactoring.
• Strong SQL skills and solid Python programming experience.
• Hands‑on experience with Azure Databricks, Delta Lake, Unity Catalog, and Medallion Architecture.
• Experience with DBT testing frameworks, data validation, and quality‑assurance best practices (dbt‑expectations, Great Expectations, Delta Live Tables expectations).
• Familiarity with Git‑based pull‑request workflows, code reviews, and CI/CD deployment pipelines.
• Experience using AI‑powered development tools such as Claude Code, GitHub Copilot, Cursor, or similar solutions.
• Ability to independently own data domains and collaborate directly with business stakeholders.
• Strong communication, analytical, and problem‑solving skills.

Required skills

• DBT
• Azure Databricks
• Delta Lake
• Unity Catalog
• Medallion Architecture
• SQL
• Python
• PySpark
• Git
• CI/CD pipelines
• Pull‑request workflows
• dbt‑expectations
• Great Expectations
• Delta Live Tables expectations
• AI‑assisted development tools (Claude Code, GitHub Copilot, Cursor)