AI Data Engineer

The Howard Hughes Medical Institute (HHMI) is seeking an AI Data Engineer to help build and run the data-engineering foundation for HHMI’s AI Accelerator. You will turn institutional content into governed, AI-ready data by implementing pipelines, medallion-style curation, workflow orchestration, governance, and retrieval-supporting infrastructure under the direction of the Principal Knowledge & Data Architect.

This hybrid role supports AI developers and product teams by delivering the data layer they need for AI use cases, while also partnering with Operations Capabilities on source-system contracts and data quality outcomes.

Responsibilities

• Build AI-facing data pipelines that ingest landing-zone content from Operations Capabilities, transform it through raw bronze, silver, and gold stages, and serve governed, AI-ready datasets for downstream consumption.
• Implement the medallion architecture by translating design patterns from the K&D Architect into working pipelines, tables, and materialization schedules.
• Design and operate Delta Lake tables with partitioning, optimization, and evolution in mind.
• Own workflow orchestration using Databricks Workflows and Delta Live Tables, including retry policies, alerting routes, run history, and cost tagging.
• Implement governance patterns for AI-facing assets using Unity Catalog, including sensitivity classification, access control, and audit.
• Build and operate retrieval-supporting infrastructure such as embedding pipelines, vector store maintenance, reindexing when models upgrade, and retrieval evaluation frameworks.
• Partner with Operations Capabilities on source-system contracts, defining what the AI Fabric consumes at the landing zone, including schema, cadence, SLA, and quality thresholds.
• Own the platform-side portion of those source-system contracts.
• Design and operate data quality and observability, including data-quality checks, freshness monitoring, drift detection, and alerting that surfaces issues before AI users are impacted.
• Support AI Developer velocity by providing targeted data engineering support when AI developers deploy into product teams.
• Contribute to and consume the reference-pattern library, including reusable pipeline patterns, code templates, and standards maintained in the shared platform layer.

Requirements

• At least 4 years of hands-on production data engineering experience designing, building, and operating production data pipelines.
• Depth with Databricks and Spark, including Delta Lake, medallion architecture, Delta Live Tables, Databricks Workflows, Databricks SQL, and Unity Catalog.
• Fluency in Python and SQL, including PySpark, ETL patterns, and SQL that runs at scale.
• Working familiarity with Git, CI/CD for data pipelines, and infrastructure-as-code using Terraform.
• AI-adjacent data engineering experience, including building the data foundation for AI use cases such as embedding pipelines, vector stores, chunking strategies, and retrieval evaluation.
• Practical workflow orchestration experience with Databricks Workflows or Airflow in production, including retry semantics, dependency management, and failure handling.
• Data quality and observability experience with Great Expectations, Databricks data-quality monitors, or comparable tools.
• Governance discipline, including working with Unity Catalog structures and sensitivity classification, with access control and audit in mind from the first commit.
• AWS foundations at the level needed for Databricks-on-AWS work, including IAM, S3, and KMS.
• Strong communication skills, including collaboration with the K&D Architect, AI Developers, and Operations Capabilities, and the ability to explain data-engineering trade-offs to non-engineers.
• Bachelor’s degree or equivalent, plus meaningful exposure to AI or knowledge-management use cases alongside at least four years of hands-on data engineering.

Technologies

• Databricks Workflows, Delta Live Tables, Delta Lake, Databricks SQL, Unity Catalog
• Python, PySpark, SQL
• Git, CI/CD, Terraform
• Databricks data-quality monitors, Great Expectations
• Embedding pipelines, vector store maintenance, retrieval evaluation frameworks
• Databricks-on-AWS: IAM, S3, KMS
• Airflow

Benefits

• Competitive pay
• Exceptional health benefits
• Retirement plans
• Time off
• Recognition and wellness programs

Practical Details

• Location: Chevy Chase, MD (hybrid)
• Schedule: three days per week in-person at HHMI’s offices in Chevy Chase, Maryland
• Reports to: Director of AI Enablement
• Compensation: $128,816.80 - $161,021.00 annually (pay type: annual)
• Visa: HHMI is not able to sponsor a visa for this position at this time

Physical Requirements

• Ability to remain in a normal seated or standing position for extended periods of time
• Ability to reach and grasp by extending hands or arms
• Dexterity to manipulate objects with fingers, such as using a keyboard
• Communication skills using the spoken word
• Ability to see and hear within normal parameters
• Mobility to move about the workspace
• Ability to move materials weighing up to several pounds, such as a laptop computer or tablet

Note: This description outlines principal duties, responsibilities, and requirements and is not an exhaustive statement. Unless they begin with the word “may,” essential duties and responsibilities listed above are essential functions as defined by the Americans with Disabilities Act. HHMI is an equal opportunity employer and uses E-Verify to confirm identity and employment eligibility of all new hires.