Databricks Lakeflow

Databricks Lakeflow Developer Jobs

4 jobs found

2d ago

Senior CE Data Engineer

Eli Lilly Indianapolis, IN

Indianapolis, IN

2d ago

$63k - $150k/yr Full time Senior Big Data Bigdata

Apply Now

Job Description

The Senior CE Data Engineer will design, build, and operationalize Eli Lilly’s LillyDirect Covered Entity (CE) Pharmacy data platform on Databricks. This role focuses on governed, auditable data at scale, including medallion pipelines, identity crosswalk logic, and PHI isolation controls.

What You’ll Do

Design and build scalable, efficient Databricks pipelines that implement the CE Architect’s canonical data models across the medallion layers (bronze → silver → gold), scoped within the CE trust boundary.
Select and apply appropriate Databricks capabilities and integration patterns, including Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, and ingestion connectors, aligned to performance, cost, and scalability constraints.
Implement identity crosswalk and PHI classification logic defined by the CE Architect, using systems such as Reltio MDM, Auth0/Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, and Transcend, including merge and split integrity in code.
Build and enforce ODCS data contracts through pipeline logic, covering schema, quality, freshness/SLA, and lineage requirements.
Implement row and column-level security controls, along with masking and tokenization boundaries, ensuring PHI isolation at the platform layer in coordination with the Policy-as-Code Engineer’s OPA/Rego policies.
Own the end-to-end pipeline lifecycle for CE data products, from requirements and prototyping to production deployment and ongoing maintenance, in partnership with Analytics Engineers and the DataHub Product Owner.
Develop and maintain CI/CD pipelines using Git-based promotion across dev → test → prod for CE data, contract, policy, and agent artifacts (including GitHub Actions).
Automate data ingestion and CE product creation to reduce manual pipeline maintenance and shorten onboarding time for new CE data sources.
Collaborate with the CE Data Architect on reference architecture and patterns, providing implementation feedback to keep designs buildable and performant.
Build pipelines that support CE Skills and agents consuming governed data, with PHI classification and consent traveling through agentic consumption paths.

Required Qualifications

Bachelor’s degree in Computer Science, Engineering, Information Technology, or a similar degree.
Experience in data engineering, specifically building production data pipelines and data products.
Advanced SQL and Python, with hands-on fluency in Databricks and Unity Catalog (notebooks, Delta Lake, catalog/schema/grants, Workflows).
Qualified applicants must be authorized to work in the United States on a full-time basis. Lilly will not provide support for or sponsor work authorization or visas for this role.

Technologies and Tools

Databricks, Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, ingestion connectors
Reltio MDM, Auth0, Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, Transcend, ODCS
OPA/Rego, GitHub Actions, Git, Python, SQL
RBAC, ABAC, row- and column-level security, tokenization, masking
CI/CD, PySpark, DLT, pytest, DQX, Great Expectations, Terraform
MDM, identity resolution, Jira, Agile/Scrum, HIPAA, PHI
AI Skills, agents

Compensation

USD 63,000 - 149,600 per year.

Benefits

Company bonus (depending, in part, on company and individual performance)
Eligibility to participate in a company-sponsored 401(k)
Pension
Vacation benefits
Eligibility for medical, dental, vision, and prescription drug benefits
Flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts)
Life insurance and death benefits
Certain time off and leave of absence benefits
Well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities)

Location and Education

Location: Indianapolis, IN (onsite)
Education: Bachelor’s degree in Computer Science, Engineering, Information Technology, or similar degree

View full job page →

19d ago

Data Engineer

Pyx Health Inc Remote 🇺🇸 Remote

Remote

19d ago

Full time Python Apache Airflow Azure

Apply Now

Job Description

Pyx Health Inc is building healthcare-focused data capabilities on Azure, where reliable pipelines and governed data platforms are essential for downstream reporting and analytics. As a Data Engineer, you will help design, build, and maintain the infrastructure and workflows that ingest, transform, and serve healthcare data using modern Azure tooling.

This is a remote role supporting data workflows that rely on Databricks, Airflow, Python, and SQL to keep healthcare solutions moving with dependable data engineering practices.

What you will do

Build and maintain data pipelines on Azure for ingesting, transforming, and cleaning healthcare data using Databricks, PySpark, and Delta Lake
Implement pipeline logic based on defined specifications, with architectural decisions guided by senior engineers
Create reusable testing frameworks to improve pipeline reliability
Monitor pipelines for failures and performance issues, escalating complex problems as needed
Develop and maintain Airflow DAGs including error handling and retry logic
Support pipeline deployment and configuration via Astronomer on Azure using the Astro CLI
Improve reliability and reduce manual intervention through ongoing pipeline enhancements
Implement data models in Delta Lake, including merge/upsert patterns
Design and optimize Delta Lake tables for analytic workloads
Use Unity Catalog to support data governance, organization, and security
Apply Change Data Capture (CDC) patterns under senior team direction
Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and related ETL support
Work across the Azure ecosystem, including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps
Follow established security and scalability standards when building data infrastructure
Support infrastructure-related tasks with architecture guidance from senior engineers
Ensure scripts and datasets are well documented to support enterprise data governance
Design, implement, and continuously improve automated data quality monitoring, reconciliation, and alerting for reliable downstream reporting
Troubleshoot pipeline failures, documenting root causes and resolutions
Collaborate with data scientists, analysts, and business stakeholders to understand reporting requirements, troubleshoot issues, and improve data assets
Participate in code reviews, incorporating feedback to improve code quality
Document pipelines, processes, and implementation decisions clearly and consistently
Stay current with data engineering technologies, especially in the healthcare space

Required qualifications

2–4 years of experience as a Data Engineer or in a closely related data role
Hands-on experience with Azure cloud services (including ADLS, Databricks, or similar)
Working knowledge of SQL for scripting and data modeling, including T-SQL, Spark, and Databricks SQL
Ability to contribute to technical projects with moderate oversight
Strong communication skills, comfortable asking questions and providing updates to cross-functional partners
Familiarity with CI/CD concepts, Git, and version control workflows
Solid problem-solving skills with a systematic approach to debugging
Familiarity with healthcare data standards and regulations is a plus (HIPAA, HL7, etc.)
Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience)

Core technical requirements

Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts
Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience
Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations
Airflow Python Development (Mid): Ability to write and maintain DAGs, including understanding of error handling and retry patterns
Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage
Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps
T-SQL (Foundational–Mid): Basic stored procedures, SQL Server querying, and data manipulation

Nice to have

Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers
Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, version control best practices, and deploying production data pipelines using CI/CD workflows

Tech stack

Azure, Databricks, Airflow, Python, SQL
PySpark, Delta Lake, Astronomer, Astro CLI, Unity Catalog
T-SQL, Databricks SQL, ETL
ADLS Gen2, Azure DevOps, Key Vaults, Logic Apps
Git, CI/CD, Spark, Databricks Spark Notebooks
SQL Server, Delta tables, Change Data Capture (CDC)
Azure Data Factory, Git + Azure DevOps CI/CD

Location: Remote

Experience: 2+ years

View full job page →

1mo ago

Machine Learning Engineer

Interwell Health Remote 🌍 Remote

Remote

1mo ago Closed

Full time Ai Ml Ai Workflows Analytics

This position is no longer accepting applications (closed Aug 31, 2026).

Job Description

Interwell Health is seeking a Machine Learning Engineer to own the end-to-end lifecycle of machine learning initiatives, including development, MLOps, deployment, and ongoing monitoring. The role covers both traditional ML workflows and large language model based capabilities, across cloud-based environments with a focus on cross-functional collaboration. This is a remote position.

Responsibilities

Plan, design, and deliver comprehensive ML solutions from data preparation to model serving, establishing monitoring, logging, and maintenance workflows.
Work closely with engineers, product managers, clinicians, and other stakeholders to create new ML products and enhance existing systems.
Architect and implement MLOps frameworks, including pipeline development, CI/CD integration, drift detection, retraining workflows, and rollback strategies.
Monitor model performance in production, identify issues, propose remediation steps, and maintain strong test coverage to ensure system reliability.
Apply modern software engineering practices to build scalable, secure, and maintainable AI/ML systems.
Develop and tailor API integrations to enable seamless connectivity between cloud platforms and ML services.
Participate in architectural discussions to ensure ML platforms meet compliance, performance, and scalability standards.

Requirements

Bachelor's degree in Computer Science, Data Analytics, Software/Computer Engineering, Computational Statistics, Mathematics, or a related field.
3+ years of end-to-end ML development in production, including data preparation, feature engineering, modeling, calibration, deployment, monitoring, and maintenance.
3+ years of MLOps experience building production pipelines (CI/CD, model registry, feature store), with monitoring, drift detection, and automated retraining.
3+ years of Python for production ML (testing, packaging, type hints, linting) and SQL for analytical and production workloads; knowledge of Scala is a plus.
2+ years working with distributed compute and cloud ML environments (for example Spark/Databricks on Azure/AWS/GCP) and modern data ecosystems (data lakes, DBMS).
Strong debugging and optimization skills across data and ML workflows.
Proven ownership and problem-solving track record, delivering measurable impact under evolving requirements and ambiguity.
Ability to articulate technical decisions clearly and contribute to documentation and design discussions.
Demonstrated system design and architecture skills for scalable, high-performance ML services and batch or streaming workflows; familiarity with API design and service integration patterns.
Understanding of tradeoffs related to latency, cost, performance, and compliance.

Technologies

Python
SQL
Scala
Spark
Databricks
Azure
AWS
GCP

Preferred

1+ year of Databricks experience plus some exposure to infrastructure and networking
1+ year implementing LLM based solutions in production, including prompt/response design, evaluation frameworks, guardrails and latency or cost optimization
1+ year designing compliant ML platforms (HIPAA, SOC 2) and working with PHI/PII governance, access controls, and auditability

View full job page →

1mo ago

Sr Databricks Data Engineer

Deloitte Pittsburgh, PA

Pittsburgh, PA

1mo ago Closed

$116k - $229k/yr Full time Apache Airflow Automation Big Data

This position is no longer accepting applications (closed Aug 19, 2026).

Job Description

Senior Databricks Data Engineer in Pittsburgh, onsite with Deloitte, offering a salary range of USD 116,200 to 229,100 per year.

Responsibilities

Champion best in class approaches for data architecture, integration, and modeling, and document them for organizational adoption.
Own the design, development, and ongoing maintenance of scalable data pipelines and architectures that support enterprise data needs.
Drive initiatives to improve data quality, enhance operational efficiency, and scale data processes.
Provide team and technology leadership by evaluating and piloting new big data and analytics tools, coaching data engineers and architects, and ensuring successful project delivery.
Design and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems.
Translate technical concepts and business value for executives, business leads, and technology teams.
Oversee DevOps practices and automation, enabling CI/CD with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell.

Requirements

Bachelor's degree in Computer Science, Engineering, or a related field
5+ years of hands-on data engineering experience with a focus on Databricks across AWS, Microsoft Azure, or Google Cloud Platform (GCP)
Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
Experience with data warehousing, third normal form (3NF), dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
1+ year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
Ability to travel 50%, on average, based on the work and client engagements
Limited immigration sponsorship may be available

Technologies

Databricks
AWS, Microsoft Azure, Google Cloud Platform (GCP)
Apache Spark
Delta Lake
Unity Catalog, Delta Live Tables, Autoloader
Structured Streaming, Databricks Workflows
Apache Airflow
CI/CD tools: Azure DevOps, AWS Code Pipeline, Jenkins, TFS, PowerShell
PySpark
Databricks Lakeflow
Experience with AI and machine learning solutions

Benefits

Discretionary annual incentive program
Benefits package aligned with Core Talent Model

Qualifications Required

Bachelor's degree in Computer Science, Engineering, or a related field
5+ years of hands-on data engineering experience with Databricks on AWS, Azure, or GCP
Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
1+ year leading complex, cross-functional data projects and technical teams, including experience with Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
Willingness to travel 50 percent on average, depending on client needs
Limited immigration sponsorship may be available

Preferred

Master's degree in Computer Science, Engineering, or a related field
Experience in one or more of AWS, Azure, and GCP cloud ecosystems and associated big data services
Experience tuning and optimizing performance in Databricks and Apache Spark environments
Experience with Databricks Lakeflow
Experience with artificial intelligence and machine learning solutions

View full job page →

Databricks Lakeflow Developer Jobs

Senior CE Data Engineer

Data Engineer

Machine Learning Engineer

Sr Databricks Data Engineer

Get Databricks Lakeflow Developer Jobs alerts