Azure Data Lakehouse

Azure Data Lakehouse Developer Jobs

3 jobs found

Today

Data Engineer

Pyx Health Inc Remote 🇺🇸 Remote

Remote

Today

Full time Python Apache Airflow Azure

Apply Now

Job Description

Pyx Health Inc is building healthcare-focused data capabilities on Azure, where reliable pipelines and governed data platforms are essential for downstream reporting and analytics. As a Data Engineer, you will help design, build, and maintain the infrastructure and workflows that ingest, transform, and serve healthcare data using modern Azure tooling.

This is a remote role supporting data workflows that rely on Databricks, Airflow, Python, and SQL to keep healthcare solutions moving with dependable data engineering practices.

What you will do

Build and maintain data pipelines on Azure for ingesting, transforming, and cleaning healthcare data using Databricks, PySpark, and Delta Lake
Implement pipeline logic based on defined specifications, with architectural decisions guided by senior engineers
Create reusable testing frameworks to improve pipeline reliability
Monitor pipelines for failures and performance issues, escalating complex problems as needed
Develop and maintain Airflow DAGs including error handling and retry logic
Support pipeline deployment and configuration via Astronomer on Azure using the Astro CLI
Improve reliability and reduce manual intervention through ongoing pipeline enhancements
Implement data models in Delta Lake, including merge/upsert patterns
Design and optimize Delta Lake tables for analytic workloads
Use Unity Catalog to support data governance, organization, and security
Apply Change Data Capture (CDC) patterns under senior team direction
Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and related ETL support
Work across the Azure ecosystem, including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps
Follow established security and scalability standards when building data infrastructure
Support infrastructure-related tasks with architecture guidance from senior engineers
Ensure scripts and datasets are well documented to support enterprise data governance
Design, implement, and continuously improve automated data quality monitoring, reconciliation, and alerting for reliable downstream reporting
Troubleshoot pipeline failures, documenting root causes and resolutions
Collaborate with data scientists, analysts, and business stakeholders to understand reporting requirements, troubleshoot issues, and improve data assets
Participate in code reviews, incorporating feedback to improve code quality
Document pipelines, processes, and implementation decisions clearly and consistently
Stay current with data engineering technologies, especially in the healthcare space

Required qualifications

2–4 years of experience as a Data Engineer or in a closely related data role
Hands-on experience with Azure cloud services (including ADLS, Databricks, or similar)
Working knowledge of SQL for scripting and data modeling, including T-SQL, Spark, and Databricks SQL
Ability to contribute to technical projects with moderate oversight
Strong communication skills, comfortable asking questions and providing updates to cross-functional partners
Familiarity with CI/CD concepts, Git, and version control workflows
Solid problem-solving skills with a systematic approach to debugging
Familiarity with healthcare data standards and regulations is a plus (HIPAA, HL7, etc.)
Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience)

Core technical requirements

Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts
Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience
Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations
Airflow Python Development (Mid): Ability to write and maintain DAGs, including understanding of error handling and retry patterns
Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage
Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps
T-SQL (Foundational–Mid): Basic stored procedures, SQL Server querying, and data manipulation

Nice to have

Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers
Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, version control best practices, and deploying production data pipelines using CI/CD workflows

Tech stack

Azure, Databricks, Airflow, Python, SQL
PySpark, Delta Lake, Astronomer, Astro CLI, Unity Catalog
T-SQL, Databricks SQL, ETL
ADLS Gen2, Azure DevOps, Key Vaults, Logic Apps
Git, CI/CD, Spark, Databricks Spark Notebooks
SQL Server, Delta tables, Change Data Capture (CDC)
Azure Data Factory, Git + Azure DevOps CI/CD

Location: Remote

Experience: 2+ years

View full job page →

Today

Data Engineer - Business Analytics

Primoris Services Corporation Texas 📍 United States Remote

Texas

Today

$135k - $155k/yr Full time Analytics Azure Data Lakehouse Big Data

Apply Now

Job Description

Primoris Services Corporation is hiring a Data Engineer - Business Analytics for a Texas (remote) role. You will design and optimize analytics-ready datasets in a modern Azure + Databricks data platform, with the curated semantic layer delivered in Databricks as part of a medallion architecture (Silver and Gold).

What you’ll do

Design, build, and maintain scalable data pipelines and curated datasets in the Databricks Lakehouse using Silver and Gold layers.
Develop and manage analytics-ready data models and semantic structures in Databricks that align to enterprise reporting needs.
Transform raw data into business-consumable datasets with standardized definitions, KPIs, and metrics.
Apply dimensional modeling and other modeling techniques within a distributed data environment.
Own the semantic layer within Databricks to support consistent definitions across reporting and analytics use cases.
Enable downstream consumption by visualization and analytics tools such as Power BI, with no reliance on embedded semantic modeling.
Partner with business analysts and reporting teams to improve usability, performance, and scalability of datasets.
Optimize data models and datasets for performance, including query efficiency for enterprise reporting workloads.
Ensure data quality, integrity, and consistency throughout the full data lifecycle.
Implement and enforce data governance including data security, masking, access controls, and cost controls.
Translate business requirements into scalable data solutions and align structures to the enterprise data dictionary and KPI definitions.
Communicate dataset structures and data model designs clearly to both technical and non-technical stakeholders.
Support the evolution of the enterprise analytics platform within Azure and Databricks.
Advance best practices for CI/CD, deployment automation, and data engineering standards.
Support advancing data capabilities, including AI/ML use cases where applicable.

What you bring

Bachelor’s degree in Information Technology, Computer Science, Business Analytics, or a related field.
6+ years of experience in data engineering, data modeling, or analytics engineering roles.
Strong expertise in Databricks Unity Catalog, SQL, and Python for transformation and pipeline development.
Proven experience designing and implementing data models within modern data platforms such as a Lakehouse architecture.
Experience with medallion architecture (Bronze/Silver/Gold layers).
Understanding of BI tools like Power BI, emphasizing dataset consumption rather than ownership of the semantic layer.
Experience with CI/CD pipelines, version control, and deployment processes.
Familiarity with Agile/Scrum delivery methodologies.

Tools you’ll work with

Databricks, Databricks Lakehouse, Databricks Unity Catalog
SQL, Python
Azure, Power BI
CI/CD pipelines, Agile/Scrum

Compensation & benefits

Pay range: USD 135,000 - 155,000 per year
Health/Dental/Vision insurance plans
401k with employer match
Paid time off
10 paid holidays
Stock purchase plan
Pay equity compensation range: USD 135,000 - 155,000

Working conditions

Occasional travel may be required.
Flexible work hours may be necessary to meet project deadlines.

View full job page →

4d ago

Sr Data Engineer, Data Analytics & Intelligence, NA

Vantage Data Centers Management Company LLC Denver, CO

Denver, CO

4d ago

$130k - $155k/yr Full time Senior Artificial Intelligence Azure Data Factory

Apply Now

Job Description

Join Vantage Data Centers Management Company LLC to build and scale governed data foundations for Operations across North America.

Responsibilities

Design, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform.
Develop and operate batch and incremental pipelines using Azure Data Factory for orchestration and Azure Data Lake Storage Gen2 as the primary data store.
Build and maintain curated lakehouse / gold-layer datasets and semantic-model inputs for governed operational insights and AI-enabled consumption.
Implement SQL- and Spark-based transformations to produce curated datasets supporting enterprise reporting, analytics, and downstream AI-enabled insight preparation.
Own assigned pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support.
Use Azure Synapse, Microsoft Fabric / Lakehouse patterns (where applicable), and related Azure analytics services for analytical workloads and data consumption patterns.
Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, reliability constraints, known quality limitations, and data dictionary definitions.
Support source visibility, confidence context, and Data Reliability & Trust Indicator integration where applicable so downstream analytics and AI outputs can be understood and trusted.
Contribute to ontology, taxonomy, semantic model, and data dictionary alignment to connect operational context, KPIs, incidents, work orders, and other enterprise domains.
Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to turn requirements into working data solutions.
Apply data governance, security, access control, data classification, and engineering standards for compliant, maintainable, scalable solutions.
Identify, document, and route data quality issues to accountable owners to improve source correction rather than hiding defects downstream.
Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives.
Proactively identify data quality issues, pipeline risks, platform dependencies, and improvement opportunities, and communicate clearly in a fast-paced environment.
Develop and maintain PySpark notebooks and jobs to ingest, transform, validate, and curate data within the enterprise data platform.
Build and modify Azure Data Factory pipelines for batch and incremental data ingestion.
Implement Spark-based transformations to write curated datasets to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established folder structures, naming conventions, and governance standards.
Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs for analytics, operational intelligence, and AI-enabled consumption.
Prepare datasets for Fabric Data Agent / AI agent use cases by documenting business rules, joins, grain, quality limitations, source lineage, and operational definitions.
Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with clear root-cause analysis and remediation steps.
Perform performance tuning of Spark jobs and SQL workloads, including partitioning, filtering, incremental logic, query optimization, and resource-aware design within architectural patterns.
Validate outputs with business partners, operations SMEs, and data stewards, and support documented correction paths for defects or discrepancies.
Support observability, logging, and auditability practices for data pipelines and AI-consumable datasets where applicable.
Commit code using Git, follow branching standards, participate in pull request reviews, and support CI/CD using GitHub, Azure DevOps, or similar tools.
Update documentation for pipelines, datasets, data contracts, data dictionaries, business rules, and operational runbooks as changes are made.
Execute assigned backlog items within sprint timelines; raise risks, dependencies, and blockers early.
Additional duties as assigned by management.

Requirements

Bachelor’s degree in Engineering, Computer Science, Data Analytics, or a related field, or equivalent experience.
5–8 years of experience in data engineering, analytics engineering, or a closely related technical data role.
Proficiency in Python for data pipelines, automation, and data processing workflows, including PySpark-based transformations.
Proficiency in SQL for querying, transformation, analytical processing, model validation, and data quality checks.
Solid understanding of ETL/ELT pipelines, transformation patterns, data integration, incremental processing, and production support practices.
Experience analyzing enterprise data sources to identify relationships, transformations, business rules, grain, ownership, and quality constraints.
Experience building solutions on Microsoft Azure including exposure to Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, Microsoft Fabric / Lakehouse patterns, and related analytics services.
Working knowledge of data modeling fundamentals including fact and dimension tables and semantic models.
Experience supporting governed data products, including metadata, lineage, issue documentation, access-control awareness, data-quality validation, and operational runbooks.
Experience with source control and CI/CD workflows using tools such as GitHub or Azure DevOps.
Strong communication and collaboration skills across IT Global, business SMEs, data governance partners, platform teams, and operations stakeholders.
Experience working in Agile environments using collaboration or project tracking tools such as Jira or similar tools.
Travel required is expected to be up to 10%, but may increase over time as the business evolves.

Technologies

Python, PySpark
Microsoft Azure, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Synapse
Microsoft Fabric, Fabric Lakehouse, Microsoft Fabric / Lakehouse patterns
SQL
Git, GitHub, Azure DevOps
Jira, CI/CD, Spark

Benefits

Medical, dental, and vision coverage
Life and AD&D
Short and long-term disability coverage
Paid time off
Employee assistance
Participation in a 401k program that includes company match
Above market total compensation package
Comprehensive suite of health and welfare, retirement, and paid leave benefits

Physical Demands and Special Requirements

Reasonable accommodations may be made for individuals with disabilities to perform essential functions.
Occasionally required to stand; walk; sit; use hands to handle or feel objects; reach with hands and arms; climb stairs; balance; stoop or kneel; talk and hear.
Occasionally lift and/or move up to 25 pounds.

Salary

$130,000 - $155,000 per year
Range is based on Colorado market data and may vary in other locations
Compensation may vary based on qualifications, skills, competencies, and experience; may fall outside the shown range

Desired Qualifications

Experience working with distributed data processing frameworks, including Apache Spark.
Experience preparing governed data products for AI-enabled use cases, including Microsoft Fabric Lakehouse, semantic models, Fabric/Data Agent patterns, and ontology or taxonomy alignment for explainable AI outputs.
Familiarity with data observability, metadata management, lineage, data contracts, reliability indicators, and operational best practices in production environments.
Familiarity with additional Azure services such as Azure Functions or Logic Apps in support of data workflows.
Experience supporting data platform enhancement, refactoring, modernization, or reusable architecture initiatives.
Experience working with structured and unstructured operational sources such as enterprise applications, operational workflows, documents, dashboards, and knowledge assets.
Experience working in a scaling or fast-paced organization where priorities evolve quickly and practical delivery discipline is required.

Location and Work Model

Denver, CO (onsite)
Based on-site in alignment with the flexible work policy: 3 days on site required and 2 days flexible

View full job page →

Azure Data Lakehouse Developer Jobs

Data Engineer

Data Engineer - Business Analytics

Sr Data Engineer, Data Analytics & Intelligence, NA

Get Azure Data Lakehouse Developer Jobs alerts