Databricks Engineer - Data Engineer II
Backend Developer
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Integration Tools
Data Lakehouse
Data Management
Data Pipeline
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Lakehouse
ETL
Informatica
Information Technology (IT)
Pyspark
Software Development
Spark
SQL
Job Description
On Deloitte’s AI and Engineering team, this Databricks Engineer II role focuses on building and optimizing data pipelines on the Databricks Lakehouse platform. You will apply Medallion architecture, implement security controls in Unity Catalog, and improve workload performance, while also supporting CI/CD and testing for Databricks workflows.
Responsibilities
- Develop and optimize data pipelines using Python, PySpark, and SQL within the Databricks Lakehouse platform
- Use Medallion architecture patterns to organize, transform, and manage data across data layers
- Carry out performance tuning for Databricks workloads and SQL queries
- Configure row-level and column-level security, and manage data lineage and sharing in Unity Catalog
- Implement CI/CD and testing processes for Databricks workflows
Requirements
- Bachelor’s degree
- Ability to obtain and maintain the required clearance for this role
- 2+ years of Databricks engineering experience
- 2+ years of experience with Python, PySpark, and SQL
- Ability to travel 10-15%, on average, depending on work scope and client or industry/sectors served
- Must be legally authorized to work in the United States without employer sponsorship, now or in the future
Technologies
- Python
- PySpark
- SQL
- Databricks Lakehouse platform
- Unity Catalog
Preferred Qualifications
- Experience using Medallion architecture in the Databricks Lakehouse platform
- Experience configuring row-level and column-level security, data lineage, and data sharing in Unity Catalog
- Experience implementing CI/CD and testing processes for Databricks
- Experience with performance tuning in Databricks
Additional Skills
- Ability to work independently and collaborate as part of a team
- Effective written and verbal communication skills
- Meticulous attention to detail and quality of work product
- Ability to build and sustain professional relationships
- Ability to lead projects or workstreams
- Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment
- Strong interpersonal skills and professional demeanor
- Ability to meet deadlines
- Ability to provide clear guidance to others
Compensation and Location
Location: Washington, DC (onsite). Compensation: USD 97,700 to 162,800 per year.
Similar Jobs
