IT Data Engineer – BB4352
Big Data
Bigdata
CI/CD
Cloud
Cloud Infrastructure
Cloud Native
Cloud Platform
Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Architecture
Data Engineer
Data Governance
Data Integration
Data Lake
Data Lakehouse
Data Management
Data Operations
Data Pipeline
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Workflows
Delta Lake
DevOps
DevSecOps
Engineering
ETL
Informatica
Information Technology (IT)
Infrastructure As Code
Kubernetes
Platform Engineering
Pyspark
Software Development
Spark
SQL
Job Description
The IT Data Engineer position supports Discovery R&D and data science initiatives by building scalable, Databricks-based data platforms and automated bioinformatics and data science pipelines. The role emphasizes cloud-native (AWS) engineering, orchestration via Seqera (Nextflow Tower), and scientific data processing workflows.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and cloud-native technologies.
- Build and optimize ETL/ELT processes for structured, semi-structured, and unstructured data.
- Develop data ingestion frameworks for research, laboratory, clinical, and external scientific datasets.
- Implement data quality, validation, monitoring, and governance processes.
- Support enterprise data lakehouse architecture and data platform modernization initiatives.
- Develop and maintain Databricks assets, including notebooks, workflows, Delta Live Tables, and Jobs.
- Create optimized Spark-based transformations and data processing solutions.
- Apply Medallion Architecture (Bronze, Silver, Gold) to manage data lifecycle stages.
- Manage Delta Lake environments, optimizing performance, scalability, and cost.
- Integrate Databricks with cloud-native services and enterprise applications.
- Deploy, configure, and support Seqera Platform (formerly Nextflow Tower).
- Develop and maintain Nextflow pipelines for bioinformatics, genomics, imaging, AI/ML, and scientific computing workloads.
- Integrate Seqera workflows with AWS cloud infrastructure and compute environments.
- Support containerized workflows using Docker and Kubernetes.
- Enable reproducible, scalable, and compliant scientific data processing workflows.
- Design and implement cloud-based data solutions in AWS, including S3 and data lifecycle policies.
- Develop Infrastructure-as-Code solutions using Terraform or CloudFormation.
- Implement security controls and access management aligned with enterprise IT standards.
- Partner with data scientists, researchers, bioinformaticians, and business stakeholders to understand data requirements.
- Troubleshoot pipeline failures, performance issues, and workflow bottlenecks.
- Contribute to platform roadmaps and continuous improvement initiatives.
- Maintain technical documentation, SOPs, and knowledge articles.
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, Data Engineering, Bioinformatics, or a related technical field.
- 5+ years of experience in data engineering, cloud engineering, or analytics platform development.
- 3+ years hands-on experience with Databricks and Apache Spark.
- 2+ years experience with Seqera Platform (Nextflow Tower) and Nextflow workflows.
- Experience supporting scientific research, life sciences, pharmaceutical, biotech, or healthcare environments is preferred.
Preferred / Relevant Skills & Tools
- Databricks Lakehouse Platform and Databricks & Data Engineering practices
- Apache Spark (PySpark, Spark SQL)
- Delta Lake and Delta Live Tables (DLT)
- Databricks Workflows
- Unity Catalog
- SQL and Python
- Nextflow pipeline development and bioinformatics workflow automation
- Docker and container technologies
- Kubernetes orchestration
- High-performance computing environments
- AWS (required): S3, IAM, EC2, VPC, Lambda
- Terraform or CloudFormation
- Cloud monitoring and logging tools
- Data lake and lakehouse architectures, ETL/ELT frameworks, data modeling
- Data cataloging and governance, API integrations, data quality frameworks
- GitHub/GitLab, CI/CD pipelines, Jenkins or GitHub Actions (or similar)
- Infrastructure as Code
- Agile and DevOps methodologies
Technology Stack
Databricks, Seqera Platform (Nextflow Tower), Nextflow, Apache Spark, PySpark, Spark SQL, Delta Lake, Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, Medallion Architecture (Bronze, Silver, Gold), AWS, S3, IAM, EC2, VPC, Lambda, Terraform, CloudFormation, Docker, Kubernetes, SQL, Python, GitHub, GitLab, CI/CD, Jenkins, GitHub Actions.
Location and Work Details
- Work location: In person
- Site: Cambridge, MA (onsite)
- Job type: Contract
- Experience required: 2+ years
Compensation
- Salary: USD 70 - 95 per hourly
- Pay range (hourly): $70.00 - $95.00 per hour
Benefits
- Dental insurance
- Health insurance
Application Questions
- Where are you currently located?
- What hourly rate are you seeking on a W2?
Similar Jobs
