DeveloperJobs.io
← Back to all jobs
Deloitte

Senior Data Engineer

Deloitte
Austin, TX $95k - $150k/yr Full time Posted 7d ago
Senior Apache Airflow Azure Data Factory Azure Databricks Big Data Data Architecture Data Engineer Data Governance Data Integration Data Lake Data Pipeline Data Platform Data Security Database Databricks Databricks Workflows ETL Microsoft Azure Pyspark Spark SQL
Apply Now

Job Description

The Senior Data Engineer role at Deloitte focuses on designing and refining ETL/ELT pipelines, authoring PySpark and Spark SQL notebooks, and leading data engineering efforts within the Project Delivery Model. This onsite position is based in Austin, TX, with an annual salary range of USD 95,000 to 150,000 and requires a Bachelor’s degree.

Details

  • Location: Austin, TX (onsite)
  • Salary: USD 95,000 - 150,000 per year
  • Education: Bachelor’s degree

Responsibilities

  • Regularly communicate with Engagement Managers (Directors), project team members, and representatives from diverse functional and technical teams, escalating issues requiring engagement management review.
  • Design, develop and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
  • Write and tune PySpark and Spark SQL notebooks for large scale data transformations.
  • Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
  • Lead design discussions with client architects and other counterparts.
  • Collaborate with different teams on data contracts and schema agreements.
  • Lead the design and optimization of high-volume data pipelines.
  • Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
  • Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
  • Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
  • Conduct root cause analysis on production incidents and implement permanent fixes.
  • Mentor junior and mid-level engineers through code reviews and pair programming.
  • Evaluate new technologies and recommend adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and event hubs.

Requirements

  • Proficiency in Python, PySpark, Spark SQL, and SQL Server.
  • Experience with Azure services including ADF, ADLS Gen2, Key Vault, and Azure Monitor.
  • Databricks expertise with Delta Lake, Unity Catalog, and Workflows.
  • Knowledge of Apache Airflow.
  • Git and Azure DevOps experience.
  • Deep understanding of Spark internals, including DAG optimization, spill analysis, and skew handling.
  • Delta Lake advanced features such as time travel, deletion vectors, and predictive I/O.
  • Unity Catalog governance including row/column security, external locations, and system tables.
  • Infrastructure as Code experience with Terraform and Azure ARM templates.
  • Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience.
  • Limited immigration sponsorship may be available.
  • Ability to travel approximately 10 percent, on average, depending on client engagements.

Technologies

  • Python, PySpark, Spark SQL, SQL Server
  • Azure, including Azure Data Factory (ADF), ADLS Gen2, Key Vault, Azure Monitor
  • Databricks, Delta Lake, Unity Catalog, Workflows
  • Apache Airflow
  • Git, Azure DevOps
  • DABs, DLT, Auto Loader, Serverless Compute, event hubs
  • Terraform, Azure ARM templates

Similar Jobs