This role focuses on designing, building, and optimizing cloud-based data engineering solutions on Databricks within Deloitte's AI and Data practice. The Senior Databricks Data Engineer collaborates with business and technology leaders to modernize data platforms and enable analytics and AI at enterprise scale.

Responsibilities

Champion Best Practices: Establish, document, and promote best-in-class approaches for data architecture, integration, and modelling.
Pipeline Ownership: Oversee the design, development, and maintenance of robust data pipelines and architectures that support large-scale enterprise data needs.
Drive Excellence: Initiate and manage efforts to improve data quality, operational efficiency, and process scalability.
Team and Technology Lead: Evaluate, pilot, and integrate new big data and analytics technologies, ensuring the organization stays at the cutting edge. Lead, coach, and develop teams of data engineers and architects, fostering technical growth and effective project delivery.
Data Governance: Consult on, design, and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems.
Communication: Communicate technical concepts and business value to executives, business leads, and technology teams.
DevOps and Automation: Oversee the implementation of CI/CD practices with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell for streamlined deployments and operations.
Guidance: Provide clear direction and mentorship across the data engineering discipline.

Requirements

Education: Bachelor's degree in Computer Science, Engineering, or a related field.
Experience: 5+ years of hands-on data engineering with a focus on Databricks on AWS, Azure, or GCP.
Technical background: Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
Data modeling and warehousing: Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.
Leadership: 1+ year leading complex, cross-functional data projects and technical teams, including expertise with Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources.
Travel: Ability to travel 50 percent on average based on client needs.
Sponsorship: Limited immigration sponsorship may be available.

Technologies

Databricks
Azure DevOps
AWS Code Pipeline
Jenkins
TFS
PowerShell
Delta Lake
Apache Spark
PySpark
Delta Live Tables
Autoloader
Structured Streaming
Databricks Workflows
Unity Catalog
Apache Airflow
Databricks Lakeflow
AWS
Azure
GCP

Benefits

The position offers a discretionary annual incentive program, subject to program rules and individual and organizational performance when determining eligibility for awards.

The Team

Deloitte's Core AI & Data practice helps organizations modernize data platforms, strengthen enterprise data foundations, and scale analytics and artificial intelligence capabilities across the business.

The team collaborates with clients to architect, engineer, and deploy cloud-based data solutions that enhance decision making, foster innovation, and support large-scale transformation.

Practitioners work across business and technology functions to address challenges in data modernization, governance, platform engineering, and insight delivery.

Preferred

Education: Master's degree in Computer Science, Engineering, or a related field.
Cloud experience: Experience in one or more cloud ecosystems (AWS, Azure, GCP) and related big data services.
Performance tuning: Experience tuning and optimizing Databricks and Apache Spark environments.
Databricks Lakeflow: Experience with Databricks Lakeflow.
AI/ML: Experience with artificial intelligence and machine learning solutions.

Sr Databricks Data Engineer