Senior Data Engineer

The contract-to-hire Senior Data Engineer focuses on real-time streaming data pipelines for connected medical devices, operating in a hybrid environment in Philadelphia. The role leverages Azure Databricks, Azure services, and ML/AI data workflows. Compensation ranges from USD 60 to 70 per hour.

Responsibilities

• Design and implement batch and streaming data pipelines on Azure and Databricks, with emphasis on real-time IoT telemetry within a Medallion architecture.
• Create ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
• Develop and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
• Deploy real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, focusing on cost-effective throughput, proper partitioning, and downstream delivery to Databricks.
• Optimize production Databricks pipelines with PySpark, Spark SQL, and Delta Lake, including tuning for performance, reliability, and cost.
• Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective actions.
• Collaborate with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
• Apply data quality, validation, and privacy-first practices, delivering reliable pipelines through software engineering standards, documentation, testing, and CI/CD.

Requirements

• Bachelor's degree in Computer Science, Mathematics, Engineering, or a related technical field with 6+ years of professional experience in data engineering, analytics, or warehousing; or Master's degree in a related field with 3+ years of experience.
• 5+ years designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments.
• 5+ years applying DevOps and CI/CD practices to data and analytics workloads.
• Production experience building data services, APIs, or microservices for downstream data consumption.
• Strong hands-on experience designing and building production-grade data pipelines on Databricks.
• Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design.
• Strong experience with Azure cloud services for data engineering and streaming workloads.
• Data quality, validation, and privacy-aware handling for regulated or sensitive data.

Technologies

• Databricks
• Spark
• PySpark
• Spark SQL
• Delta Lake
• Azure
• Azure Event Hubs
• Azure Stream Analytics
• Azure Databricks
• Medallion architecture

Outcomes

• Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming data pipelines.
• Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics consumption within the Medallion architecture.
• Design and deliver data services or consumption patterns that enable business and ML teams to access near real-time telemetry data reliably, securely, and at scale.

Work Hours and Travel Requirements

• Willingness to assist in troubleshooting and analysis during off-hours production issues as needed.
• The IT team operates in a hybrid environment requiring a minimum of two days per week in the downtown Philadelphia office.