Lakehouse

Lakehouse Developer Jobs

7 jobs found

1d ago

Lead Data Engineer

INSPYR Solutions Doraville, GA 🇺🇸 Remote

Doraville, GA

1d ago

$60 - $70/hr Contract Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

Lead Data Engineer at INSPYR Solutions will architect and optimize distributed data pipelines, establish Lakehouse architecture, enable ML workflows, ensure data quality, and improve production reliability.

Responsibilities

Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment.
Design and maintain enterprise Lakehouse infrastructure with Delta Lake, ensuring ACID compliance, lineage, auditability, and governance.
Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points.
Prepare feature-ready datasets and establish reproducible machine learning deployment patterns to enable ML workflows.
Lead platform-wide data quality, access control, and cataloging frameworks.
Implement cost optimization, cluster tuning, and performance engineering strategies.
Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
Own production reliability, troubleshooting, and root cause analysis for data and ML pipelines.

Requirements

7+ years of experience in advanced data engineering with distributed compute technologies.
Expert Spark engineering experience including performance tuning, cluster configuration, partition strategies, and large dataset optimization.
Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
Strong Python and SQL skills for large-scale data transformations.
Experience supporting machine learning pipelines or model operationalization.
Proven track record architecting cloud-native data platforms on Azure, AWS, or GCP.
Experience integrating diverse, complex data sources at enterprise scale.
Demonstrated ability to own mission-critical production systems.
Experience with distributed streaming frameworks such as Kafka or Event Hubs.
Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
Background in data security, compliance controls, or audit-ready governance.
Experience automating data operations with CI/CD and infrastructure as code.

Technologies

Databricks, Python, SQL
Apache Spark, Delta Lake
MLflow, Notebooks
Hugging Face Transformers, LangChain, LlamaIndex
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
Kafka, Event Hubs, ADLS, S3, GCS
Git, GitHub, GitLab, Azure Repos, Databricks Repos
GitHub Actions, Azure DevOps
Unity Catalog, RBAC, REST APIs

Benefits

Comprehensive medical benefits
Competitive pay
401(k) retirement plan

Core Tools

Databricks (Spark, Delta Lake, MLflow, Notebooks)
Python & SQL
Apache Spark via Databricks
Delta Lake for lakehouse architecture

Cloud Platforms

Azure, AWS, or GCP
Cloud storage options: ADLS, S3, GCS

Data Integration

Kafka or Event Hubs for streaming data
Auto Loader for Databricks file ingestion
REST APIs
AI/ML and MLflow for model tracking and deployment
Hugging Face Transformers, LangChain, LlamaIndex for LLM integration
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini

DevOps

Git-based version control (GitHub, GitLab, Azure Repos)
Databricks Repos for notebooks and code
CI/CD: GitHub Actions, Azure DevOps

Security & Governance

Unity Catalog for data governance
RBAC to manage access controls

View full job page →

10d ago

Data Engineer

Major League Baseball New York, NY

New York, NY

10d ago

$115k - $140k/yr Full time Apache Airflow Automation Big Data

Apply Now

Job Description

Major League Baseball is seeking a hands-on Data Engineer to join the LAI team and shape production data pipelines that power league analytics. This onsite role in New York, NY centers on building and operating a GCP lakehouse with Airflow and dbt to deliver trusted data for analysts and engineers. You will own the end-to-end data workflow from ingestion to production monitoring.

Responsibilities

Develop production-grade pipelines with Airflow and dbt to orchestrate batch and streaming transformations across Google Cloud Platform, ensuring downstream analysts and engineers can rely on the data without tracing wiring issues.
Design layered data models (staging, intermediate, mart) as the single source of truth for league analytics, applying dbt best practices for materialization, testing, and documentation.
Manage ingestion through Pub/Sub, GCS, Dataflow, and Knowledge Catalog DataPlex to land batch and streaming sources into the lakehouse with clean, consistent landings.
Establish observability and monitoring standards to surface data quality issues before stakeholders notice them.
Maintain code with GitHub-based CI/CD and contribute to deployment workflows that keep the platform reliable and changes safe.
Follow data governance practices to keep proprietary baseball data secure and compliant.

Requirements

2 to 4 years of hands-on production data engineering experience.
Advanced SQL skills, able to craft complex queries (subqueries, nested logic, window functions) and review others' queries for issues.
Proficient Python for data processing, scripting, and automation.
Hands-on dbt experience building models across staging, intermediate, and mart layers, with tests and production deployment.
Production Airflow experience including DAG authoring, dependency management, and debugging failed runs.
Deep familiarity with Google Cloud Platform (BigQuery, GCS, Pub/Sub) or equivalent depth in AWS/Azure with willingness to convert.
Experience with Git-based development workflows including branching, pull requests, and code reviews as a daily practice.
Clear communicator with both engineers and non-engineers, receptive to feedback, and constructive in responses.
Execution-oriented mindset, able to own projects from requirements to deployment with minimal oversight.

Technologies

Airflow
dbt
BigQuery
GCS
Pub/Sub
Dataflow
Knowledge Catalog DataPlex
GitHub

Benefits

Competitive Benefits Package
Company Contributed 401K Plan
Paid Time Off and Holidays
Paid Parental Leave
Access to Free Tickets to Baseball Games & MLB.TV
Discounts at MLB Store | MLBShop.com
Employee Assistance Programs (EAP)
Onsite/Online Training & Development Programs
Tuition Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Pet Insurance

Nice-to-have

A degree in Computer Science, Engineering, or a related field — or non-traditional background with equivalent practical experience
Experience with Terraform or other Infrastructure-as-Code tools
Experience with AI-assisted development or enterprise AI tooling (Gemini Enterprise, Vertex AI)
A passion for baseball or prior experience in sports, media, or entertainment
Ability to build creative solutions for unusual problems

Salary

USD 115,000 - 140,000 per year, plus bonus.

Why MLB?

Major League Baseball is the historic centerpiece of professional sports in North America, known for a culture that emphasizes growth, teamwork, and professionalism. Employees thrive by taking initiative, solving problems, and putting the team first. MLB aims to empower its workforce to excel by offering experiences that position them for success.

View full job page →

11d ago

Data Engineer (Senior Consultant)

Guidehouse Washington, DC

Washington, DC

11d ago

$85k - $141k/yr Full time Senior Big Data Consultant

Apply Now

Job Description

Guidehouse seeks a Data Engineer (Senior Consultant) in Washington, DC (onsite) with a salary range of USD 85,000 - 141,000 per year to support CMS data modernization through scalable pipelines and lakehouse architectures for secure federal health data processing.

Responsibilities

Design, develop, and maintain end-to-end ETL/ELT pipelines within Databricks-based environments
Implement and manage medallion (multi-layered lakehouse) data architectures (bronze, silver, gold)
Build scalable data pipelines to process large, complex datasets, including structured and semi-structured data
Collaborate with cross-functional teams (data scientists, analysts, SMEs, and CMS stakeholders) to translate business requirements into technical solutions
Ensure data quality, lineage, and governance through validation, monitoring, and documentation
Optimize performance and efficiency of data processing workflows in cloud-based environments
Support data integration and modernization efforts aligned with CMS EIPM objectives
Participate in Agile development practices, including sprint planning and backlog management

Requirements

U.S. Citizenship and ability to obtain a Public Trust clearance
Bachelor’s degree in Computer Science, Data Science, Information Systems, or related field
A minimum of FOUR (4) years of experience in data engineering, software engineering, or related technical roles
Proficiency in Python and SQL for data engineering and transformation workloads
Hands-on experience with Databricks and large-scale data processing environments
Experience developing and maintaining ETL/ELT data pipelines
Experience working with medallion/lakehouse architectures (bronze, silver, gold)
Experience working with relational and/or distributed data systems, including data modeling
Experience with CI/CD practices and tools (e.g., Git-based workflows)
Familiarity with version control and collaboration tools, including Git, Jira, and Confluence
Experience working in Agile development environments
Ability to work independently in a fast-paced environment
Strong analytical and troubleshooting skills
Excellent verbal and written communication skills, with the ability to clearly communicate complex technical concepts to diverse audiences

Technologies

Databricks
Python
SQL
Git
Jira
Confluence

Benefits

Medical, Rx, Dental & Vision Insurance
Personal and Family Sick Time & Company Paid Holidays
Position may be eligible for a discretionary variable incentive bonus
Parental Leave and Adoption Assistance
401(k) Retirement Plan
Basic Life & Supplemental Life
Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
Short-Term & Long-Term Disability
Student Loan PayDown
Tuition Reimbursement, Personal Development & Learning Opportunities
Skills Development & Certifications
Employee Referral Program
Corporate Sponsored Events & Community Outreach
Emergency Back-Up Childcare Program
Mobility Stipend

Job Family

Data Engineering & Architecture Consulting

Travel Required

Up to 10%

Clearance Required

Ability to Obtain Public Trust

Nice to Have

Experience working with healthcare data, such as claims, eligibility, or clinical datasets
Experience supporting CMS, HHS, or other federal health programs
Experience working with state government data programs
Familiarity with data governance, privacy, and compliance frameworks (e.g., HIPAA)
Experience supporting data modernization or cloud migration initiatives

View full job page →

17d ago

Senior Data Engineer

Space Telescope Science Institute Baltimore, MD

Baltimore, MD

17d ago

$125k - $150k/yr Full time Senior Big Data Cloud Operations

Apply Now

Job Description

Based in Baltimore, MD with a hybrid work option, Space Telescope Science Institute seeks a Senior Data Engineer to manage backend data pipelines and MPP database systems for the Mikulski Archive for Space Telescopes (MAST). The role supports high performance data access for scientific research and comes with residency and ITAR requirements. The salary range is USD 125,000 to 150,000 per year. Candidates should have eight or more years of experience and hold a Bachelor’s or Master’s degree in computer science, information technology, or a related discipline.

Responsibilities

Oversee performance and operations for PostgreSQL and MPP platforms
Design, develop, deploy, monitor, and troubleshoot complex Airflow pipelines to process, transform, and load large-scale datasets
Build, maintain, and continuously improve data systems supporting scientific research, including relational databases and cloud-based Lakehouse environments
Ensure data accuracy, accessibility, observability, and reliability through proactive monitoring, alerting, and incident response
Collaborate with scientists, data engineers, and cross-functional teams to translate requirements into robust, scalable platform solutions

Requirements

Advanced expertise in PostgreSQL and Greenplum MPP
Strong proficiency with Apache Airflow
Hands-on experience with AWS cloud services
Strong Python programming skills and proficiency in SQL and SQL performance tuning
Experience designing, building, and optimizing data pipelines at scale

Technologies

PostgreSQL
Greenplum
Apache Airflow
AWS
Python
SQL
Trino
Apache Iceberg
Lakehouse

Benefits

Employer retirement contribution — STScI contributes 10% of salary from day one
12 days sick leave
Up to 24 days of vacation
10 paid holidays
Flexible work schedule with a healthy work life balance
Comprehensive medical, dental, vision, prescription plans, and more

You’ll have the opportunity to grow into

Specialized technologies such as Trino for distributed querying
Apache Iceberg for lakehouse management
Greenplum or other MPP systems and large-scale performance tuning
Handling high-volume scientific data with modern data infrastructure

View full job page →

25d ago

Senior Data Engineer (Chinese Mandarin Speaker)

Bitus Labs Irvine, CA

Irvine, CA

25d ago

$130k+/yr Full time Python Senior AWS

Apply Now

Job Description

Bitus Labs invites a Senior Data Engineer who speaks Chinese Mandarin to join onsite in Irvine, CA. This role focuses on architecting and scaling an AWS based data lakehouse, crafting production grade pipelines in Java and Python, and steering data quality, governance, and platform decisions while mentoring engineers. The position offers a salary of USD 130,000 per year and a benefits package designed to support long term growth and stability.

Location: Irvine, California (onsite). Language requirement: Chinese Mandarin.

Benefits

401(k)
401(k) matching
Dental insurance
Health insurance
Life insurance
Paid time off
Parental leave
Retirement plan
Vision insurance

Responsibilities

Design and implement scalable medallion pattern data lakehouses (Bronze, Silver, Gold) on AWS S3 using Apache Iceberg.
Develop and maintain high throughput ETL and ELT pipelines with AWS Glue, EMR (Spark), and Lambda.
Apply schema evolution, partitioning strategies, and Iceberg table compaction to optimize storage and query performance.
Produce production quality pipeline code in Java and Python, choosing the language based on performance and maintainability needs.
Build and operate event driven pipelines using Kinesis Data Streams, Kinesis Firehose, or Apache Kafka (MSK).
Design exactly-once and at-least-once processing semantics for streaming workloads with Flink or Spark Structured Streaming on EMR.
Manage infrastructure as code with AWS CDK or Terraform for repeatable deployments.
Optimize cost and performance across AWS services including S3, Glue, Athena, Redshift Spectrum, EMR, Lambda, Step Functions, and EventBridge.
Enforce data security best practices: IAM least-privilege, KMS encryption, VPC networking, and Lake Formation access control.
Build and maintain CI/CD pipelines for data workloads using AWS CodePipeline, GitHub Actions, or equivalent tools.
Implement data quality frameworks such as Great Expectations or Deequ and integrate validation into orchestration.
Define and enforce data contracts between producing and consuming systems.
Contribute to data cataloging and lineage tracking via AWS Glue Data Catalog or Apache Atlas.
Collaborate with data scientists, ML engineers, and analysts to deliver performant, well documented datasets.
Mentor mid level and junior engineers through code reviews, design discussions, and pair programming.
Document architecture decisions (ADRs) and contribute to the internal knowledge base.

Requirements

5+ years of professional data engineering experience, including at least 3 years on AWS cloud platforms.
Proven track record delivering production data pipelines at scale (TB+ datasets, high throughput SLAs).
Experience with data lakehouse architectures — medallion pattern, open table formats (Iceberg preferred; Delta Lake or Hudi acceptable).
Java: strong command of Java 8+ for Spark jobs, Iceberg connectors, and performance critical components; familiarity with Maven or Gradle.
Python: proficient in Python 3 for AWS Glue scripts, orchestration logic, data quality checks, and automation tooling; experience with pandas, PySpark, boto3, and packaging practices.
Storage & Compute: S3, Glue (jobs, crawlers, Data Catalog), EMR (Spark/Flink), Lambda, EC2.
Streaming: Kinesis Data Streams, Kinesis Firehose, or MSK.
Orchestration: Step Functions, MWAA (Managed Airflow), or EventBridge Scheduler.
Querying: Athena, Redshift, or Redshift Spectrum.
Security & Governance: IAM, KMS, Lake Formation, Secrets Manager, VPC.
DevOps: AWS CDK or CloudFormation; CodePipeline or equivalent CI/CD tools.
Apache Spark (PySpark and or Spark Java API) with distributed transformations and performance tuning.
Apache Iceberg — table maintenance, time travel, snapshot management, partition evolution.
SQL — advanced SQL including window functions, CTEs, and query optimization.

Tech stack at a glance

Languages: Java (8+), Python 3
Cloud Platform: AWS (S3, Glue, EMR, Kinesis, Athena, Lambda, Step Functions, Lake Formation, CDK)
Processing: Apache Spark, Apache Flink, Spark Structured Streaming
Table Format: Iceberg (primary), familiarity with Delta Lake or Hudi
Streaming: Kinesis, MSK, Kinesis Firehose
Orchestration: MWAA, Step Functions, Airflow
IaC & CI/CD: AWS CDK, Terraform, GitHub Actions, CodePipeline
Related Tools: Maven, Gradle, PySpark, boto3, Apache Atlas, Great Expectations, Deequ
Additional: dbt, SageMaker Feature Store, MLflow, Druid, ClickHouse, Apache Airflow

View full job page →

1mo ago

Senior Scientific Data Engineer

Lawrence Berkeley National Laboratory San Francisco Bay Area, CA

San Francisco Bay Area, CA

1mo ago Closed

$139k - $174k/yr Full time Python Senior Data Engineer

This position is no longer accepting applications (closed Jul 12, 2026).

Job Description

Lawrence Berkeley National Laboratory's Joint Genome Institute invites qualified candidates to apply for a Senior Scientific Data Engineer. This role is based in the San Francisco Bay Area with a hybrid work arrangement, supporting genomic data generation, metadata management, and data platforms across scientific operations. You will help design and sustain the LMS/LIMS, data warehouse and lakehouse capabilities, and related workflows, with opportunities to lead major technical initiatives.

Responsibilities

Translate complex scientific, operational, and user requirements into functional specifications, system designs, and implementation plans for system enhancements, integrations, and shared data platforms.
Design, develop, deploy, and support core systems, APIs, and workflows—including the Laboratory Information Management System (LMS/LIMS), Data Warehouse/Data Lakehouse, and Proposal/Project Management platforms—that enable genomic data generation, metadata management, and laboratory operations.
Lead the resolution of complex technical challenges, drive system improvements, and ensure production platforms are reliable, scalable, interoperable, and high-performing.
Promote engineering best practices through technical reviews, documentation, mentorship, and continuous improvement of development and operational processes.

Requirements

A Bachelor’s Degree (or equivalent knowledge/training) in Computer Science or a related field and a minimum of 8 years of related professional experience developing, integrating, deploying, and supporting production software applications and data systems that enable metadata management, workflow orchestration, data lifecycle operations, and broad user data access to scientific and operational data, or an equivalent combination of education and professional experience.
Experience with data engineering and event-driven technologies such as Airflow, Kafka, or related tools.
Experience working with relational databases, object storage platforms, and systems supporting structured, semi-structured, and large-scale datasets.
Strong knowledge of software and data engineering fundamentals for large-scale production systems, including system design, APIs, testing methodologies, concurrency, reliability, scalability, interoperability, and performance optimization.
Proficiency in Python and experience with one or more additional programming languages.
Experience using AI-assisted development tools, with demonstrated sound judgment in evaluating and validating generated code for production suitability.
Demonstrated ability to provide technical leadership in shaping system architecture and technical direction across cross-functional engineering groups.
Excellent communication skills, including experience organizing and presenting complex technical information to internal teams and stakeholders.
Demonstrated experience collaborating with stakeholders to understand project goals and translate complex scientific, operational, and user requirements into automated systems, technical specifications, and implementation plans.

Technologies

Python
Airflow
Kafka
Laboratory Information Management System (LMS/LIMS)
Data Warehouse/Data Lakehouse
AI-assisted development tools

Benefits

Exceptional health benefits
Generous paid time off, sick time off, and holidays
A culture where you’ll belong - we are invested in our teams
Relocation assistance
Hybrid work schedule eligibility

Why join Berkeley Lab?

Exceptional health benefits.
Generous paid time off, sick time off, and holidays.
A culture where you’ll belong - we are invested in our teams!

Additional Information

Application Date: Priority consideration for resumes and cover letters by June 22, 2026. Applications accepted until the posting is removed.
Appointment Type: Full-time, exempt from overtime pay (monthly paid), two-year term appointment with benefits eligibility, with the possibility of extension or conversion to a Career appointment based on performance, funds, and ongoing needs.
Salary Range: Budgeted salary range of $139,440 - $174,312 annually, within the full range of $139,440 - $235,308 for job code C71.3. Final salary will reflect the candidate’s qualifications and experience.
Background Check: Position is subject to a background check. Convictions will be evaluated for relevance to the responsibilities and requirements. A conviction history does not automatically disqualify an applicant.
Work Modality: Hybrid work schedule combining telework and on-site duties at 1 Cyclotron Road, Berkeley, CA. Residency within 150 miles is required; schedules depend on business needs. In rare cases, full-time telework or remote work may be considered. A REAL ID or other acceptable ID is required.
Relocation Assistance: Eligible.
Work Authorization: Applicants must be legally authorized to work in the United States. Berkeley Lab does not sponsor visas for this position.

View full job page →

3mo ago

Databricks Developer

Coda Staffing Chicago, IL

Chicago, IL

3mo ago Closed

$60 - $70/hr Full time Python Senior AWS

This position is no longer accepting applications (closed May 10, 2026).

Job Description

At Coda Staffing, the Databricks Developer will architect and optimize data pipelines in Azure, leveraging Databricks PySpark and Python to turn healthcare data into reliable, governable insights. The role emphasizes modern data lakehouse patterns, governance, and CI/CD practices to support compliant analytics in healthcare environments.

Responsibilities

Design, implement, and maintain scalable data pipelines using Databricks PySpark and Python.
Build and optimize ETL/ELT processes within Azure cloud environments.
Implement data models following Data Lakehouse principles, including Medallion architecture.
Ensure data quality, consistency, and performance across ingestion, staging, and curated layers.
Collaborate with data architects, analysts, and business stakeholders to translate healthcare data requirements into technical solutions.
Develop reusable data transformation logic and modular processing components.
Support deployment processes aligned with CI/CD and DevOps best practices.
Monitor and optimize data workflows for performance, scalability, and reliability.
Contribute to data governance, security, and compliance practices relevant to healthcare environments.

Requirements

Current knowledge of modern data tools such as Databricks, FiveTran, Data Fabric and others; core experience with data architecture, data integrations, data warehousing, and ETL/ELT processes.
Hands-on experience developing and deploying custom wheel packages or in-session notebook scripts for parallel execution across executor and worker nodes.
Experience with SQL, stored procedures, and PySpark based on area of data platform specialization.
Strong knowledge of cloud and hybrid relational database systems, such as MS SQL Server, PostgreSQL, Oracle, Azure SQL, AWS RDS, Aurora, or a comparable engine.
Strong experience with batch and streaming data processing techniques and file compaction strategies.
Hands-on experience with Databricks in Azure environments.
Advanced proficiency in Python and PySpark for distributed data processing.
Experience building and optimizing data pipelines in Azure (Azure Data Factory, Azure SQL, Data Lake Storage, etc.).
Solid understanding of data warehousing, data lakehouse concepts, and ETL/ELT frameworks.
Experience working with relational databases such as SQL Server, PostgreSQL, Oracle, or similar.
Knowledge of batch and streaming data processing patterns.
Experience working with large, complex datasets in cloud-based distributed environments.
Strong analytical and problem-solving skills; ability to work effectively in cross-functional and distributed teams.
Clear communication skills, with the ability to explain technical concepts to non-technical stakeholders.
Proactive mindset with a strong sense of ownership and a commitment to delivering high-quality, reliable data solutions.

Technologies

Databricks
PySpark
Python
Azure
Azure Data Factory
Data Lake Storage
FiveTran
Data Fabric
Medallion architecture
SQL
Stored Procedures
MS SQL Server
PostgreSQL
Oracle
Azure SQL
AWS RDS
Aurora

Location

Chicago, IL (onsite). Location options include Chicago or Cape Girardeau.

Work location

Hybrid remote in Chicago, IL 60617.

Compensation

USD 60 - 70 per hour.

View full job page →

Lakehouse Developer Jobs

Lead Data Engineer

Data Engineer

Data Engineer (Senior Consultant)

Senior Data Engineer

Senior Data Engineer (Chinese Mandarin Speaker)

Senior Scientific Data Engineer

Databricks Developer

Get Lakehouse Developer Jobs alerts