Senior Data Engineer

Join Deloitte as a Senior Data Engineer onsite in Arlington Heights, IL. This role offers a competitive annual compensation of USD 95,000 - 150,000 and the opportunity to design, develop, and optimize ETL and ELT pipelines on Azure Databricks. You will collaborate with client and engagement teams and help set data engineering standards within Deloitte's Project Delivery Model. The position spans development, testing, and production environments using Unity Catalog, with opportunities to mentor peers and influence data architecture across engagements. A Bachelor's degree is required.

Responsibilities

• Maintain regular communication with Engagement Managers and project teams, coordinating with functional and technical stakeholders and escalating issues when needed.
• Design, build, and optimize ETL/ELT pipelines with Azure Data Factory and Databricks.
• Develop and tune PySpark and Spark SQL notebooks for large‑scale transformations.
• Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
• Lead design discussions with client architects and counterparts.
• Collaborate across teams to establish data contracts and schema agreements.
• Lead the design and optimization of high‑volume data pipelines.
• Define and enforce data engineering standards, including naming conventions, partitioning, cluster configurations, and Spark tuning.
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling configurations, and cost optimization approaches.
• Perform root cause analyses on production incidents and implement lasting fixes.
• Mentor junior and mid‑level engineers via code reviews and pair programming.
• Evaluate new technologies and advise on adoption, including DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs.

Requirements

• Python, PySpark, Spark SQL, SQL Server
• Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
• Databricks (Delta Lake, Unity Catalog, Workflows)
• Apache Airflow
• Git / Azure DevOps
• Deep Spark internals (DAG optimization, spill analysis, skew handling)
• Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
• Unity Catalog governance (row/column security, external locations, system tables)
• IaC - Terraform, Azure ARM templates
• Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel 10%, on average, based on the work you do and the clients and industries you serve

Technologies

• Python
• PySpark
• Spark SQL
• SQL Server
• Azure Data Factory
• ADF
• ADLS Gen2
• Key Vault
• Azure Monitor
• Databricks
• Delta Lake
• Unity Catalog
• Workflows
• Apache Airflow
• Git
• Azure DevOps
• Terraform
• Azure ARM templates
• Auto Loader
• Serverless Compute
• DABs
• DLT
• Event Hubs

Accommodation

Information for applicants with a need for accommodation: Join Deloitte assistance for disabled applicants.