Senior Data Engineer

Deloitte is seeking a Senior Data Engineer to design, build, and optimize end-to-end data pipelines with Azure Data Factory, Databricks, and PySpark. This onsite role in Jersey City, NJ operates within a Project Delivery Model that prioritizes design leadership, cross-team collaboration, and mentoring of colleagues. The position offers a salary range of USD 95,000 to 150,000 per year and requires a bachelor’s degree.

Responsibilities

• Coordinate regularly with engagement managers (Directors), project teams, and cross-functional technical stakeholders, escalating issues that require higher-level engagement management review.
• Design, build, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations.
• Architect end-to-end data solutions across development, UAT, and production environments leveraging Unity Catalog.
• Lead design discussions with client architects and other counterparts.
• Collaborate with multiple teams to establish data contracts and agreed-upon schemas.
• Lead the design and optimization of high-volume data pipelines.
• Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
• Drive performance optimization through AQE tuning, efficient clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling configurations, and cost-optimization strategies.
• Conduct root-cause analysis on production incidents and implement durable fixes.
• Mentor junior and mid-level engineers through code reviews and pair programming.
• Evaluate new technologies and advise on adoption (examples include DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs).

Requirements

• Strong skills in Python, PySpark, Spark SQL, and SQL Server.
• Experience with Azure components such as Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
• Databricks experience including Delta Lake, Unity Catalog, and Workflows.
• Apache Airflow experience.
• Git or Azure DevOps for version control.
• Deep knowledge of Spark internals including DAG optimization, spill analysis, and skew handling.
• Delta Lake advanced features such as time travel, deletion vectors, and predictive I/O.
• Unity Catalog governance covering row/column security, external locations, and system tables.
• Infrastructure as code with Terraform and Azure ARM templates.
• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience.
• Limited immigration sponsorship may be available.
• Ability to travel about 10% on average, depending on client assignments.

Technologies

• Python
• PySpark
• Spark SQL
• SQL Server
• Azure Data Factory (ADF)
• ADLS Gen2
• Key Vault
• Azure Monitor
• Databricks
• Delta Lake
• Unity Catalog
• Workflows
• Apache Airflow
• Git
• Azure DevOps
• Deep Spark internals
• DAG optimization
• spill analysis
• skew handling
• Delta Lake time travel
• deletion vectors
• predictive I/O
• Unity Catalog governance
• Terraform
• Azure ARM templates
• DABs
• DLT
• Auto Loader
• Serverless Compute
• Event Hubs

Additional information

Accommodation information for applicants: https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applicants.html