Senior Data Platform Engineer (Databricks, Python)

We are looking for a Senior Data Platform Engineer to help build, run, and evolve our Azure Databricks data platform, which ingests, validates, and protects data for the business domain data engineering teams.

We run the platform like a software product: domain teams onboard through configuration, shared Python frameworks handle ingestion, data quality, monitoring, and data protection, and every change ships through code review, automated testing, and CI/CD.

This is not a pure data pipeline role. We want an engineer with software engineering and DevOps discipline and enough data engineering experience to understand what our users need. As the team shifts toward operations, experience running production systems is highly valued.

***We will consider remote in the US for this role

Responsibilities

• Platform Development: Build and maintain the shared Python libraries and configuration-driven frameworks that let domain teams onboard and run their pipelines with minimal platform help.
• CI/CD & Infrastructure as Code: Own the GitHub Actions pipelines and release process that promote Databricks Asset Bundles from sandbox to production, and extend Terraform coverage of our Azure and Databricks infrastructure.
• Operations, Reliability & Recovery: Support the production platform, build enhanced observability, triage incidents, lead root cause analysis, and turn lessons learned into runbooks, automation, and permanent fixes. Track performance, reliability and delivery metrics from a data and platform perspective.
• Security & Governance: Maintain necessary platform and data controls with Unity Catalog, and support sensitive data protection and governance in a regulated healthcare environment.
• Domain Enablement: Partner with domain teams to onboard data sources, resolve escalated issues, and turn recurring needs into self-service features.

3+ years progressive in software, platform, DevOps/SRE, or data engineering, including:

• 2+ years building or operating cloud data platforms or production data systems.
• Software Engineering & CI/CD: Strong Python skills (packaging, modular and testable code, pytest), Git-based code review, and proven experience building CI/CD pipelines (GitHub Actions preferred) for multi-environment deployments.
• Infrastructure as Code: Hands-on Terraform experience managing Azure and Databricks resources.
• Data Engineering: Production experience with Spark/PySpark and SQL, including batch and incremental ingestion, schema evolution, medallion data modeling, and data quality validation.
• Databricks & Azure: Practical experience with Delta Lake, Unity Catalog, Jobs, and Delta Live Tables / Lakeflow (Asset Bundles and system tables a plus), and working knowledge of ADLS Gen2, Entra ID, Key Vault, and cloud networking.
• Operations: Experience supporting production systems in a DevOps, SRE, or platform role: monitoring and alerting, incident response, on-call, root cause analysis, runbooks, and disaster recovery planning.
• Communication: Clear communication and documentation, and the ability to work well on a remote team with technical and business stakeholders.

• 6+ years progressive in software, platform, DevOps/SRE, or data engineering,
• Disciplined use of AI to accelerate development and analysis, with the same engineering rigor and ownership of AI-assisted work as any other work shipped
• Familiarity with healthcare payer data (X12 834/837, HL7) and HIPAA; HITRUST experience a plus
• Databricks or Azure certifications