Azure Event Hubs

Azure Event Hubs Developer Jobs

2 jobs found

2d ago

Senior Data Engineer

firstPROPhiladelphia Philadelphia, PA

Philadelphia, PA

2d ago

$60 - $70/hr Contract Senior Analytics Automation

Apply Now

Job Description

The contract-to-hire Senior Data Engineer focuses on real-time streaming data pipelines for connected medical devices, operating in a hybrid environment in Philadelphia. The role leverages Azure Databricks, Azure services, and ML/AI data workflows. Compensation ranges from USD 60 to 70 per hour.

Responsibilities

Design and implement batch and streaming data pipelines on Azure and Databricks, with emphasis on real-time IoT telemetry within a Medallion architecture.
Create ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
Develop and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
Deploy real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, focusing on cost-effective throughput, proper partitioning, and downstream delivery to Databricks.
Optimize production Databricks pipelines with PySpark, Spark SQL, and Delta Lake, including tuning for performance, reliability, and cost.
Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective actions.
Collaborate with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
Apply data quality, validation, and privacy-first practices, delivering reliable pipelines through software engineering standards, documentation, testing, and CI/CD.

Requirements

Bachelor's degree in Computer Science, Mathematics, Engineering, or a related technical field with 6+ years of professional experience in data engineering, analytics, or warehousing; or Master's degree in a related field with 3+ years of experience.
5+ years designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments.
5+ years applying DevOps and CI/CD practices to data and analytics workloads.
Production experience building data services, APIs, or microservices for downstream data consumption.
Strong hands-on experience designing and building production-grade data pipelines on Databricks.
Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design.
Strong experience with Azure cloud services for data engineering and streaming workloads.
Data quality, validation, and privacy-aware handling for regulated or sensitive data.

Technologies

Databricks
Spark
PySpark
Spark SQL
Delta Lake
Azure
Azure Event Hubs
Azure Stream Analytics
Azure Databricks
Medallion architecture

Outcomes

Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming data pipelines.
Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics consumption within the Medallion architecture.
Design and deliver data services or consumption patterns that enable business and ML teams to access near real-time telemetry data reliably, securely, and at scale.

Work Hours and Travel Requirements

Willingness to assist in troubleshooting and analysis during off-hours production issues as needed.
The IT team operates in a hybrid environment requiring a minimum of two days per week in the downtown Philadelphia office.

View full job page →

15d ago

Senior Data Engineer

Deloitte Morristown, NJ

Morristown, NJ

15d ago

$95k - $150k/yr Full time Senior Adls Gen2 Apache Airflow

Apply Now

Job Description

At Deloitte in Morristown, New Jersey, the Senior Data Engineer plays a key role in shaping client data platforms by designing, developing, and optimizing end-to-end solutions and ETL/ELT pipelines with Azure Data Factory, Databricks, and PySpark. The role requires collaborating with engagement teams and client architects to deliver scalable data architectures across development, testing, and production environments, with a focus on governance, performance, and cost efficiency.

Compensation

USD 95,000 - 150,000 per year

Responsibilities

Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and / or technical teams, escalating matters that require additional attention from engagement management
Design, develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks
Write and tune PySpark / Spark SQL notebooks for large-scale data transformation
Architect end-to-end data solutions across dev UAT prod environments using Unity Catalog
Lead and drive design discussions with client architects and other counterparts
Collaborate with different teams on data contracts and schema agreements
Lead design and optimization of high-volume data pipeline
Define and enforce data engineering standards - naming conventions, partitioning strategies, cluster configurations, Spark tuning
Drive performance optimization - AQE tuning, liquid clustering, broadcast joins, shuffle partition management
Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
Conduct root cause analysis on production incidents and implement permanent fixes
Mentor junior and mid-level engineers through code reviews and pair programming
Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs)

Requirements

Python, PySpark, Spark SQL, SQL Server
Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
Databricks (Delta Lake, Unity Catalog, Workflows)
Apache Airflow
Git / Azure DevOps
Deep Spark internals (DAG optimization, spill analysis, skew handling)
Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
Unity Catalog governance (row/column security, external locations, system tables)
IaC - Terraform, Azure ARM templates
Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
Limited immigration sponsorship may be available
Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve

Technologies

Python, PySpark, Spark SQL, SQL Server
Azure Data Factory (ADF), ADLS Gen2, Key Vault, Azure Monitor
Databricks, Delta Lake, Unity Catalog, Workflows
Apache Airflow
Git, Azure DevOps
DABs, DLT, Auto Loader, Serverless Compute, Azure Event Hubs
Terraform, Azure ARM templates

Additional Information

Information for applicants with a need for accommodation: Deloitte assistance for disabled applicants

View full job page →

Azure Event Hubs Developer Jobs

Senior Data Engineer

Senior Data Engineer

Get Azure Event Hubs Developer Jobs alerts