Lead Clinical Data Engineer

Yale University is hiring a Lead Clinical Data Engineer to serve as a senior technical specialist supporting biomedical research across the Yale School of Medicine and the Yale New Haven Health System. In this remote role based in New Haven, CT, you will lead the design and development of the Aligned Data Warehouse (ADW), expand its capabilities, and build the data pipelines that enable analytics and research data sharing.

This position focuses on end-to-end data integration, from ETL and data management processes to performance tuning, monitoring, and troubleshooting. You will also contribute to engineering standards and documentation while ensuring patient privacy and the security of healthcare data across systems and databases.

What You’ll Do

• Design, build, test, maintain, and control data pipelines and ETL jobs that integrate data into the ADW and other research databases.
• Contribute to DevOps and documentation, helping develop standards and procedures for database and data pipeline design, operational management, and ongoing maintenance.
• Design, build, test, maintain, and control data management processes, including management and loading of flat files in formats such as csv, tsv, pipe-delimited, XML, and JSON.
• Facilitate design sessions, code walkthroughs, and peer reviews; produce technical documentation, SOPs, and KBAs.
• Performance tune database objects, stored procedures, ETL jobs, and related scripts to optimize both customer queries and pipeline processing.
• Monitor scheduled ETL jobs and related processes to ensure expected functioning and uptime.
• Lead and/or coordinate timely troubleshooting and remediation of ETL job failures.
• Provide world-class customer service across interactions with customers, stakeholders, and other teams.
• Maintain patient privacy and the integrity and security of healthcare data, complying with applicable laws, regulations, and institutional policies related to IRB, patient privacy, and IT cybersecurity.
• Independently research and stay current on new and emerging technologies relevant to Scientific Computing & Data to improve efficiency, quality, and/or reduce operating costs.
• Lead projects and data stewardship strategies.
• Work closely with the DBA team and vendors to optimize data pipelines.
• Assist with the design, build, test, and performance tuning of reports and dashboards in SSRS, Microsoft Power BI, or similar BI tools.
• Perform related duties as assigned or requested.

Required Qualifications

• Extensive experience developing, automating, and optimizing data integration workflows with a focus on Microsoft Fabric Data Pipelines for large-scale ingestion, transformation, and integration.
• Proven ability to design, develop, and optimize scalable data models, semantic layers, and storage architectures for enterprise reporting, self-service analytics, operational intelligence, and advanced analytics.
• Experience developing and maintaining Fabric Notebooks utilizing PySpark, Spark SQL, and related technologies for transformation, enrichment, validation, and automation.
• Demonstrated experience designing and implementing Microsoft Fabric Lakehouses, Warehouses, and enterprise data organization strategies supporting governance, performance, security, and scalability.
• Ability to collaborate with business stakeholders, analysts, architects, and technical teams to translate complex requirements into scalable data engineering solutions.
• Requires a bachelor’s degree in a related area and a minimum of four years of experience in the field or a related area.

Technologies

• Microsoft SQL Server, T-SQL, SSIS, SQL Agent
• Microsoft Fabric, Metadata-driven dynamic SQL
• Linux, Windows
• OMOP Common Data Model, OHDSI (Observational Medical Outcomes Partnership)
• SSRS, Microsoft Power BI
• PySpark, Spark SQL, Fabric Notebooks, Microsoft Fabric Data Pipelines, Microsoft Fabric Lakehouses, Microsoft Fabric Warehouses
• Large Language Models, GPU-based software
• Data formats: csv, tsv, pipe-delimited, XML, JSON
• Epic data warehouse, Epic’s Clarity & Caboodle databases
• PCORnet, OnCore, IRES - IRB

Education

• Bachelor’s degree in a related area

Benefits

A general description of benefits for this Managerial & Professional position is available at: Yale Benefits.

Additional Information

• Job posting date: 10/13/2026
• Job category: Professional
• Work model: Remote
• Time type: Full time
• Duration type: Staff
• Location: New Haven, CT (remote)
• Salary: USD 90,000 - 165,750 per yearly (salary offers depend on qualifications, experience, skills, education, grade profile, and market conditions)

Background, Health, and Posting Notes

• Candidates must complete pre-employment background screening, which may include motor vehicle, DOT certification, drug testing, and credit checks depending on job requirements. Offers are contingent on successful completion.
• Some positions have health requirements based on job responsibilities.
• Yale University is a tobacco-free campus.