Senior Data Engineer

Deloitte’s Arlington area is seeking a Senior Data Engineer to help design, build, and optimize data platforms that empower teams to make informed decisions. The role centers on creating end-to-end data pipelines and products with Azure Databricks, Azure Data Factory, and PySpark, spanning development, UAT, and production environments, while collaborating across cross-functional groups to ensure robust data solutions.

Location

Arlington, Virginia — onsite

Salary

USD 95,000 - 150,000 per year

Education

Bachelor's degree required

Responsibilities

• Maintain regular communication with engagement managers (directors), project teams, and cross-functional technical stakeholders, escalating issues when engagement leadership should weigh in
• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks
• Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations
• Architect end-to-end data solutions across development, UAT, and production using Unity Catalog
• Lead design discussions with client architects and other counterparts
• Collaborate with diverse teams on data contracts and schema agreements
• Lead the design and optimization of high-volume data pipelines
• Define and enforce data engineering standards, including naming conventions, partitioning, cluster configurations, and Spark tuning
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management
• Design Databricks cluster policies, autoscaling configurations, and cost-optimization strategies
• Perform root-cause analysis on production incidents and implement permanent fixes
• Mentor junior and mid-level engineers via code reviews and pair programming
• Evaluate new technologies and advise on adoption, such as DABs, Delta Live Tables (DLT), Auto Loader, Serverless Compute, and Event Hubs

Requirements

• Proficiency in Python, PySpark, Spark SQL, and SQL Server
• Experience with Azure services including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor
• Databricks platform knowledge (Delta Lake, Unity Catalog, Workflows)
• Apache Airflow expertise
• Git or Azure DevOps for version control and CI/CD
• Deep understanding of Spark internals (DAG optimization, spill analysis, skew handling)
• Advanced Delta Lake features (time travel, deletion vectors, predictive I/O)
• Unity Catalog governance knowledge (row/column security, external locations, system tables)
• Infrastructure as Code experience with Terraform and Azure ARM templates
• Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel approximately 10% on average, depending on client needs

Technologies

• Python
• PySpark
• Spark SQL
• SQL Server
• Azure Data Factory
• Azure Data Lake Storage Gen2
• Key Vault
• Azure Monitor
• Databricks
• Delta Lake
• Unity Catalog
• Workflows
• Apache Airflow
• Git
• Azure DevOps
• Terraform
• Azure ARM templates
• DABs
• Delta Live Tables
• Auto Loader
• Serverless Compute
• Azure Event Hubs

The Team

The AI and Engineering group at Deloitte leverages advanced engineering capabilities to build, deploy, and operate integrated solutions across software, data, AI, network, and hybrid cloud infrastructure. This team delivers mission-critical outcomes, helping clients modernize technology and data platforms while enabling engineering teams to stay ahead of rapid advances.

Accommodations

Information for applicants who need accommodation is available at Join Deloitte assistance for disabled applicants.