Sr Databricks Data Engineer

Join Deloitte's AI and Data practice in Nashville, onsite, as a Senior Databricks Data Engineer. This role focuses on designing, building, and optimizing cloud-based data engineering solutions on Databricks to modernize data platforms and enable analytics and AI use cases. Compensation ranges from USD 137,500 to 193,600 per year.

Benefits

• Discretionary annual incentive program

Responsibilities

• Define and promote best-in-class data architecture, integration, and modelling practices, with clear documentation of standards.
• Own the design, development, and maintenance of robust data pipelines and architectures that support enterprise-scale data needs.
• Lead initiatives to improve data quality, operational efficiency, and process scalability.
• Assess, pilot, and integrate new big data and analytics technologies, while guiding and mentoring teams of data engineers and architects to ensure successful project delivery.
• Advise on governance, security, and compliance strategies tailored to modern cloud data ecosystems and implement them as needed.
• Translate technical concepts and business value for executives, business leads, and technology teams.
• Oversee CI/CD practices with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell to streamline deployments and operations.
• Provide clear guidance to others to support project success.

Requirements

• Ability to work independently and as part of a team.
• Effective written and verbal communication.
• Meticulous attention to detail and a commitment to quality.
• Ability to build and sustain professional relationships.
• Experience leading projects or workstreams.
• Ability to manage and prioritize multiple tasks in a fast-paced environment.
• Strong interpersonal skills and professional demeanor.
• Ability to meet deadlines.
• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of hands-on data engineering experience focusing on Databricks across AWS, Azure, or GCP.
• Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
• Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.
• 1+ year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources.
• Ability to travel 50 percent on average.
• Limited immigration sponsorship may be available.
• Master's degree in Computer Science, Engineering, or a related field.
• Experience in one or more cloud ecosystems (AWS, Azure, or GCP) and associated big data services.
• Experience tuning and optimizing performance in Databricks and Apache Spark environments.
• Experience with Databricks Lakeflow.
• Experience with artificial intelligence and machine learning solutions.

Technologies

• Azure DevOps
• AWS Code Pipeline
• Jenkins
• TFS
• PowerShell
• Databricks
• AWS
• Microsoft Azure
• Google Cloud Platform (GCP)
• Lakehouse architecture
• Apache Spark
• Delta Lake
• cloud-native databases
• storage solutions
• distributed compute platforms
• PySpark
• Delta Live Tables
• Autoloader
• Structured Streaming
• Databricks Workflows
• Apache Airflow
• Unity Catalog
• CI/CD pipelines
• Databricks Lakeflow