Senior CE Data Engineer

The Senior CE Data Engineer will design, build, and operationalize Eli Lilly’s LillyDirect Covered Entity (CE) Pharmacy data platform on Databricks. This role focuses on governed, auditable data at scale, including medallion pipelines, identity crosswalk logic, and PHI isolation controls.

What You’ll Do

• Design and build scalable, efficient Databricks pipelines that implement the CE Architect’s canonical data models across the medallion layers (bronze → silver → gold), scoped within the CE trust boundary.
• Select and apply appropriate Databricks capabilities and integration patterns, including Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, and ingestion connectors, aligned to performance, cost, and scalability constraints.
• Implement identity crosswalk and PHI classification logic defined by the CE Architect, using systems such as Reltio MDM, Auth0/Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, and Transcend, including merge and split integrity in code.
• Build and enforce ODCS data contracts through pipeline logic, covering schema, quality, freshness/SLA, and lineage requirements.
• Implement row and column-level security controls, along with masking and tokenization boundaries, ensuring PHI isolation at the platform layer in coordination with the Policy-as-Code Engineer’s OPA/Rego policies.
• Own the end-to-end pipeline lifecycle for CE data products, from requirements and prototyping to production deployment and ongoing maintenance, in partnership with Analytics Engineers and the DataHub Product Owner.
• Develop and maintain CI/CD pipelines using Git-based promotion across dev → test → prod for CE data, contract, policy, and agent artifacts (including GitHub Actions).
• Automate data ingestion and CE product creation to reduce manual pipeline maintenance and shorten onboarding time for new CE data sources.
• Collaborate with the CE Data Architect on reference architecture and patterns, providing implementation feedback to keep designs buildable and performant.
• Build pipelines that support CE Skills and agents consuming governed data, with PHI classification and consent traveling through agentic consumption paths.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a similar degree.
• Experience in data engineering, specifically building production data pipelines and data products.
• Advanced SQL and Python, with hands-on fluency in Databricks and Unity Catalog (notebooks, Delta Lake, catalog/schema/grants, Workflows).
• Qualified applicants must be authorized to work in the United States on a full-time basis. Lilly will not provide support for or sponsor work authorization or visas for this role.

Technologies and Tools

• Databricks, Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, ingestion connectors
• Reltio MDM, Auth0, Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, Transcend, ODCS
• OPA/Rego, GitHub Actions, Git, Python, SQL
• RBAC, ABAC, row- and column-level security, tokenization, masking
• CI/CD, PySpark, DLT, pytest, DQX, Great Expectations, Terraform
• MDM, identity resolution, Jira, Agile/Scrum, HIPAA, PHI
• AI Skills, agents

Compensation

USD 63,000 - 149,600 per year.

Benefits

• Company bonus (depending, in part, on company and individual performance)
• Eligibility to participate in a company-sponsored 401(k)
• Pension
• Vacation benefits
• Eligibility for medical, dental, vision, and prescription drug benefits
• Flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts)
• Life insurance and death benefits
• Certain time off and leave of absence benefits
• Well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities)

Location and Education

• Location: Indianapolis, IN (onsite)
• Education: Bachelor’s degree in Computer Science, Engineering, Information Technology, or similar degree