The Senior Data Engineer role at Deloitte in Boston centers on designing, building, and refining end-to-end data pipelines and solutions. The position emphasizes hands-on work with Azure Data Factory, Databricks, and PySpark, coupled with client collaboration and mentorship responsibilities.

Location

Boston, MA onsite

Compensation

Salary range: USD 95,000 – 150,000 per year

Responsibilities

Maintain regular communication with Engagement Managers (Directors), project teams, and representatives from diverse functional and technical groups, escalating issues that require engagement management input.
Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
Author and tune PySpark and Spark SQL notebooks for large-scale data transformations.
Architect end-to-end data solutions across development, UAT, and production environments leveraging Unity Catalog.
Lead design discussions with client architects and other stakeholders.
Collaborate with multiple teams to align data contracts and schema agreements.
Drive design and optimization of high-volume data pipelines.
Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
Advance performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
Design Databricks cluster policies, autoscaling configurations, and cost optimization approaches.
Perform root-cause analysis on production incidents and implement permanent fixes.
Mentor junior and mid-level engineers via code reviews and pair programming.
Evaluate emerging technologies and advise on adoption (for example, DABs, DLT, Auto Loader, Serverless Compute, event hubs).

Requirements

Proficiency in Python, PySpark, Spark SQL, and SQL Server.
Experience with Azure components including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
Solid working knowledge of Databricks, Delta Lake, Unity Catalog, and Workflows.
Experience with Apache Airflow for workflow orchestration.
Git and Azure DevOps version control and CI/CD workflows.
Deep understanding of Spark internals such as DAG optimization, spill analysis, and skew handling.
Advanced Delta Lake features including time travel, deletion vectors, and predictive I/O.
Unity Catalog governance covering row/column security, external locations, and system tables.
Infrastructure as Code experience with Terraform and Azure ARM templates.
Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or related IT field, or equivalent experience.
Limited immigration sponsorship may be available.
Ability to travel approximately 10 percent, depending on client assignments and engagements.

Technologies

Python, PySpark, Spark SQL, SQL Server
Azure: Data Factory, Data Lake Storage Gen2, Key Vault, Monitor
Databricks: Delta Lake, Unity Catalog, Workflows
Apache Airflow
Git, Azure DevOps
Terraform, Azure ARM templates
DABs, Delta Live Tables (DLT), Auto Loader, Serverless Compute, Event Hubs

The Team

AI and Engineering teams apply advanced engineering capabilities to build, deploy, and operate integrated sector solutions across software, data, AI, networking, and hybrid cloud infrastructure. These efforts are aimed at transforming mission-critical operations and enabling clients to stay ahead by modernizing technology and data platforms through tailored delivery models.

Additional information

Information for applicants needing accommodation: accommodations for disabled applicants.

Senior Data Engineer

Job Description