Senior Data Engineer
Senior
Adls Gen2
Apache Airflow
Azure
Azure Arm Templates
Azure Data Factory
Azure Data Lake Storage
Azure Databricks
Azure Event Hubs
Azure Monitor
Big Data
Bigdata
Cloud
Cloud Platforms
Data
Data Architecture
Data Engineer
Data Governance
Data Integration
Data Lake
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Database
Databricks
DevOps
ETL
Infrastructure As Code
Key Vault
Microsoft Azure
Serverless Compute
Spark
SQL
Job Description
At Deloitte in Morristown, New Jersey, the Senior Data Engineer plays a key role in shaping client data platforms by designing, developing, and optimizing end-to-end solutions and ETL/ELT pipelines with Azure Data Factory, Databricks, and PySpark. The role requires collaborating with engagement teams and client architects to deliver scalable data architectures across development, testing, and production environments, with a focus on governance, performance, and cost efficiency.
Compensation
USD 95,000 - 150,000 per year
Responsibilities
- Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and / or technical teams, escalating matters that require additional attention from engagement management
- Design, develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks
- Write and tune PySpark / Spark SQL notebooks for large-scale data transformation
- Architect end-to-end data solutions across dev UAT prod environments using Unity Catalog
- Lead and drive design discussions with client architects and other counterparts
- Collaborate with different teams on data contracts and schema agreements
- Lead design and optimization of high-volume data pipeline
- Define and enforce data engineering standards - naming conventions, partitioning strategies, cluster configurations, Spark tuning
- Drive performance optimization - AQE tuning, liquid clustering, broadcast joins, shuffle partition management
- Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
- Conduct root cause analysis on production incidents and implement permanent fixes
- Mentor junior and mid-level engineers through code reviews and pair programming
- Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs)
Requirements
- Python, PySpark, Spark SQL, SQL Server
- Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
- Databricks (Delta Lake, Unity Catalog, Workflows)
- Apache Airflow
- Git / Azure DevOps
- Deep Spark internals (DAG optimization, spill analysis, skew handling)
- Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
- Unity Catalog governance (row/column security, external locations, system tables)
- IaC - Terraform, Azure ARM templates
- Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
- Limited immigration sponsorship may be available
- Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve
Technologies
- Python, PySpark, Spark SQL, SQL Server
- Azure Data Factory (ADF), ADLS Gen2, Key Vault, Azure Monitor
- Databricks, Delta Lake, Unity Catalog, Workflows
- Apache Airflow
- Git, Azure DevOps
- DABs, DLT, Auto Loader, Serverless Compute, Azure Event Hubs
- Terraform, Azure ARM templates
Additional Information
Information for applicants with a need for accommodation: Deloitte assistance for disabled applicants
Similar Jobs