Principal Data Engineer
Artificial Intelligence
Aws Data Engineer
Aws Data Engineering
Big Data
Cloud Data Engineering
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Integration
Data Integration Tools
Data Pipeline
Data Platform
Data Processing
Data Science
Database
Databases
Engineering
ETL
Informatica
Integration
Lead Data Engineering
Software Development
SQL
Technical Lead
Job Description
Lead the architecture and evolution of a production data platform supporting human genetics, multi-omics, and high-content imaging workflows.
Responsibilities
- Act as technical owner and architect for the AWS environment, including account structure, IAM, networking, compute, and storage.
- Own and extend cost governance across the platform, including budgets, alerting, tagging enforcement, rightsizing, and storage lifecycle and tiering.
- Define and drive data platform developer operations, covering infrastructure as code, CI/CD, container builds, and secrets and environment management.
- Upgrade and modernize the data lake and related components: managed orchestration, metadata, and ingestion workflows to scale with future production and improve lab-to-cloud data transfer.
- Partner with ML and computational biology leaders to define the interface between the platform and modelling efforts.
Requirements
- 10+ years building and operating production data infrastructure, with demonstrated end-to-end platform ownership and architecture.
- Deep AWS expertise, including S3, IAM, EC2, Batch, Lambda, and Spark (EMR or equivalent).
- Experience as a technical owner of production cloud environments, including multi-account architecture and cross-account access.
- Strong Python and SQL, with production-grade pipeline development experience.
- Hands-on production experience with workflow orchestration, such as Nextflow, Airflow, Temporal, Step Functions, Redun, or similar, including judgement to evaluate and select tools.
- Proficiency in infrastructure as code (CloudFormation/CDK), CI/CD, containers, and observability; DevOps is a core competency for the role.
- Ability to design data modelling and warehouse or lakehouse solutions, including metadata management, lineage, and data quality practices.
- Proven cost governance at scale, including budgets, tagging, rightsizing, and storage lifecycle management across hundreds of terabytes.
- Fluency in scientific data semantics (studies, samples, donors, cell lines, assays) and experience collaborating directly with scientists to translate questions into data models.
Technologies
- AWS, S3, IAM, EC2, Batch, Lambda, Spark, EMR
- Python, SQL
- Nextflow, Airflow, Temporal, Step Functions, Redun
- CloudFormation, CDK, CI/CD, containers, observability
- Databricks, OpenMetadata, Snowflake
- SageMaker, MLflow, Weights & Biases, feature stores
- LLM, MCP servers, tool APIs
- Infrastructure as code
Benefits
- Healthcare coverage
- Annual incentive program
- Retirement benefits
- A broad range of other benefits
Preferred Qualifications
- Familiarity with genetics and genomics data.
- Experience with microscopy or high-content screening data, including large binary object stores, transfer optimization, and image streaming for model training.
- Production experience with Databricks, OpenMetadata, or a comparable catalog.
- Experience with Snowflake, especially governed access patterns over cloud object storage.
- ML infrastructure experience: SageMaker or Databricks ML, MLflow or Weights & Biases, and feature stores.
- Experience serving data to LLM or agent clients (MCP servers, tool APIs).
- Experience leading data platform architecture as a senior or principal individual contributor, ideally in biotech, pharma R&D, or a research institute.
Location: Cambridge, MA (onsite)
Salary: USD 179,000 - 236,000 per yearly
Similar Jobs
