Event Streaming Developer Jobs

1 job found

14d ago

Senior Data Engineer

Deloitte Jersey City, NJ

Jersey City, NJ

14d ago

$95k - $150k/yr Full time Senior Azure Azure Data Factory

Apply Now

Job Description

Deloitte is seeking a Senior Data Engineer to design, build, and optimize end-to-end data pipelines with Azure Data Factory, Databricks, and PySpark. This onsite role in Jersey City, NJ operates within a Project Delivery Model that prioritizes design leadership, cross-team collaboration, and mentoring of colleagues. The position offers a salary range of USD 95,000 to 150,000 per year and requires a bachelor’s degree.

Responsibilities

Coordinate regularly with engagement managers (Directors), project teams, and cross-functional technical stakeholders, escalating issues that require higher-level engagement management review.
Design, build, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations.
Architect end-to-end data solutions across development, UAT, and production environments leveraging Unity Catalog.
Lead design discussions with client architects and other counterparts.
Collaborate with multiple teams to establish data contracts and agreed-upon schemas.
Lead the design and optimization of high-volume data pipelines.
Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
Drive performance optimization through AQE tuning, efficient clustering, broadcast joins, and shuffle partition management.
Design Databricks cluster policies, autoscaling configurations, and cost-optimization strategies.
Conduct root-cause analysis on production incidents and implement durable fixes.
Mentor junior and mid-level engineers through code reviews and pair programming.
Evaluate new technologies and advise on adoption (examples include DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs).

Requirements

Strong skills in Python, PySpark, Spark SQL, and SQL Server.
Experience with Azure components such as Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
Databricks experience including Delta Lake, Unity Catalog, and Workflows.
Apache Airflow experience.
Git or Azure DevOps for version control.
Deep knowledge of Spark internals including DAG optimization, spill analysis, and skew handling.
Delta Lake advanced features such as time travel, deletion vectors, and predictive I/O.
Unity Catalog governance covering row/column security, external locations, and system tables.
Infrastructure as code with Terraform and Azure ARM templates.
Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience.
Limited immigration sponsorship may be available.
Ability to travel about 10% on average, depending on client assignments.

Technologies

Python
PySpark
Spark SQL
SQL Server
Azure Data Factory (ADF)
ADLS Gen2
Key Vault
Azure Monitor
Databricks
Delta Lake
Unity Catalog
Workflows
Apache Airflow
Git
Azure DevOps
Deep Spark internals
DAG optimization
spill analysis
skew handling
Delta Lake time travel
deletion vectors
predictive I/O
Unity Catalog governance
Terraform
Azure ARM templates
DABs
DLT
Auto Loader
Serverless Compute
Event Hubs

Additional information

Accommodation information for applicants: https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applicants.html

View full job page →

Get Event Streaming Developer Jobs alerts

New Event Streaming Developer Jobs, daily.