Databricks Developer

Responsibilities

• Design, implement, and maintain scalable data pipelines using Databricks PySpark and Python.
• Build and optimize ETL/ELT processes within Azure cloud environments.
• Implement data models following Data Lakehouse principles, including Medallion architecture.
• Ensure data quality, consistency, and performance across ingestion, staging, and curated layers.
• Collaborate with data architects, analysts, and business stakeholders to translate healthcare data requirements into technical solutions.
• Develop reusable data transformation logic and modular processing components.
• Support deployment processes aligned with CI/CD and DevOps best practices.
• Monitor and optimize data workflows for performance, scalability, and reliability.
• Contribute to data governance, security, and compliance practices relevant to healthcare environments.

Requirements

• Current knowledge of modern data tools such as Databricks, FiveTran, Data Fabric and others; core experience with data architecture, data integrations, data warehousing, and ETL/ELT processes.
• Hands-on experience developing and deploying custom wheel packages or in-session notebook scripts for parallel execution across executor and worker nodes.
• Experience with SQL, stored procedures, and PySpark based on area of data platform specialization.
• Strong knowledge of cloud and hybrid relational database systems, such as MS SQL Server, PostgreSQL, Oracle, Azure SQL, AWS RDS, Aurora, or a comparable engine.
• Strong experience with batch and streaming data processing techniques and file compaction strategies.
• Hands-on experience with Databricks in Azure environments.
• Advanced proficiency in Python and PySpark for distributed data processing.
• Experience building and optimizing data pipelines in Azure (Azure Data Factory, Azure SQL, Data Lake Storage, etc.).
• Solid understanding of data warehousing, data lakehouse concepts, and ETL/ELT frameworks.
• Experience working with relational databases such as SQL Server, PostgreSQL, Oracle, or similar.
• Knowledge of batch and streaming data processing patterns.
• Experience working with large, complex datasets in cloud-based distributed environments.
• Strong analytical and problem-solving skills; ability to work effectively in cross-functional and distributed teams.
• Clear communication skills, with the ability to explain technical concepts to non-technical stakeholders.
• Proactive mindset with a strong sense of ownership and a commitment to delivering high-quality, reliable data solutions.

Technologies

• Databricks
• PySpark
• Python
• Azure
• Azure Data Factory
• Data Lake Storage
• FiveTran
• Data Fabric
• Medallion architecture
• SQL
• Stored Procedures
• MS SQL Server
• PostgreSQL
• Oracle
• Azure SQL
• AWS RDS
• Aurora

Location

Chicago, IL (onsite). Location options include Chicago or Cape Girardeau.

Work location

Hybrid remote in Chicago, IL 60617.

Compensation

USD 60 - 70 per hour.