Databricks Engineer - Data Engineer III
Azure Databricks
Big Data
Bigdata
CI/CD
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Integration Tools
Data Lakehouse
Data Lineage
Data Management
Data Modeling
Data Pipeline
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Lakehouse
Databricks Pyspark
DevOps
Devops Tools
ETL
Informatica
Information Technology (IT)
Row/column Security
Software Development
Software Engineering
Spark
SQL
Job Description
Join Deloitte’s AI & Engineering team to build and optimize Databricks Lakehouse data pipelines while supporting governance and performance.
Responsibilities
- Develop and optimize data pipelines using Python, PySpark, and SQL on the Databricks Lakehouse platform
- Use Medallion architecture patterns to structure, transform, and manage data across data layers
- Perform performance tuning for Databricks workloads and SQL queries
- Configure row-level and column-level security
- Manage data lineage and data sharing using Unity Catalog
- Implement CI/CD and testing processes for Databricks workflows
Requirements
- Bachelor’s degree
- 2+ years of Databricks engineering experience
- 4+ years experience with Python, PySpark, and SQL
- Ability to obtain and maintain the required clearance for the role
- Legal authorization to work in the United States without employer sponsorship, now or at any time in the future
- Ability to travel 10-15% on average, based on work you do and client/industry needs
Technologies
- Python
- PySpark
- SQL
- Databricks Lakehouse
- Medallion architecture
- Unity Catalog
- CI/CD
What a Successful Candidate Can Demonstrate
- Ability to work independently and collaborate within a team
- Effective written and verbal communication skills
- Meticulous attention to detail and quality of work product
- Ability to build and sustain professional relationships
- Ability to lead projects or workstreams
- Ability to manage and prioritize multiple tasks in a fast-paced, dynamic environment
- Strong interpersonal skills and professional demeanor
- Ability to meet deadlines
- Ability to provide clear guidance to others
Preferred
- Experience using Medallion architecture in the Databricks Lakehouse platform
- Experience configuring row-level and column-level security, data lineage, and data sharing in Unity Catalog
- Experience implementing CI/CD and testing processes for Databricks
- Experience with performance tuning in Databricks
Location: Washington, DC (onsite)
Compensation: USD 107,700 - 179,500 per year