Sr Databricks Data Engineer
Apache Airflow
Automation
Big Data
Bigdata
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Lake
Data Lakehouse
Data Pipeline
Data Platform
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Lakeflow
Databricks Workflows
Delta Lake
Delta Live Tables
ETL
Spark
SQL
Structured Streaming
Job Description
Senior Databricks Data Engineer in Pittsburgh, onsite with Deloitte, offering a salary range of USD 116,200 to 229,100 per year.
Responsibilities
- Champion best in class approaches for data architecture, integration, and modeling, and document them for organizational adoption.
- Own the design, development, and ongoing maintenance of scalable data pipelines and architectures that support enterprise data needs.
- Drive initiatives to improve data quality, enhance operational efficiency, and scale data processes.
- Provide team and technology leadership by evaluating and piloting new big data and analytics tools, coaching data engineers and architects, and ensuring successful project delivery.
- Design and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems.
- Translate technical concepts and business value for executives, business leads, and technology teams.
- Oversee DevOps practices and automation, enabling CI/CD with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related field
- 5+ years of hands-on data engineering experience with a focus on Databricks across AWS, Microsoft Azure, or Google Cloud Platform (GCP)
- Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
- Experience with data warehousing, third normal form (3NF), dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
- 1+ year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
- Ability to travel 50%, on average, based on the work and client engagements
- Limited immigration sponsorship may be available
Technologies
- Databricks
- AWS, Microsoft Azure, Google Cloud Platform (GCP)
- Apache Spark
- Delta Lake
- Unity Catalog, Delta Live Tables, Autoloader
- Structured Streaming, Databricks Workflows
- Apache Airflow
- CI/CD tools: Azure DevOps, AWS Code Pipeline, Jenkins, TFS, PowerShell
- PySpark
- Databricks Lakeflow
- Experience with AI and machine learning solutions
Benefits
- Discretionary annual incentive program
- Benefits package aligned with Core Talent Model
Qualifications Required
- Bachelor's degree in Computer Science, Engineering, or a related field
- 5+ years of hands-on data engineering experience with Databricks on AWS, Azure, or GCP
- Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
- Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
- 1+ year leading complex, cross-functional data projects and technical teams, including experience with Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
- Willingness to travel 50 percent on average, depending on client needs
- Limited immigration sponsorship may be available
Preferred
- Master's degree in Computer Science, Engineering, or a related field
- Experience in one or more of AWS, Azure, and GCP cloud ecosystems and associated big data services
- Experience tuning and optimizing performance in Databricks and Apache Spark environments
- Experience with Databricks Lakeflow
- Experience with artificial intelligence and machine learning solutions