Senior Data Engineer - Databricks
Apache Spark
Azure Data Engineer
Azure Databricks
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Infrastructure
Cloud Platform
Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Governance
Data Integration
Data Integration Tools
Data Lake
Data Lakehouse
Data Management
Data Modeling
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Dlt
Databricks Lakehouse
Databricks Lakehouse Platform
Databricks Pyspark
Databricks Workflows
Delta Lake
DevOps
DevSecOps
ETL
Facilities Management
Feature Store
Gcp Data Engineer
Informatica
Information Technology (IT)
Infrastructure As Code
Lead Data Engineering
Security Automation
Security Devops
Spark
SQL
Vector Databases
Job Description
Capgemini is hiring a Senior Data Engineer to help design and optimize scalable data solutions on the Databricks lakehouse. This onsite role supports batch and real-time pipelines that power analytics, machine learning, and generative AI use cases, with an emphasis on governance, security, and operational readiness.
Role Focus
In this position, you will build ETL/ELT pipelines and modern lakehouse architectures that support enterprise analytics and ML/AI workflows. The work includes enforcing data quality, enabling lineage and auditing, and operationalizing machine learning workloads in collaboration with Data Science and AI teams.
Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
- Create robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
- Implement data quality checks, monitoring, and pipeline optimization to improve reliability and performance.
- Design and manage Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers).
- Develop and maintain Delta Lake tables while ensuring optimal query performance.
- Conduct performance tuning using Databricks best practices, including Photon Engine, Liquid Clustering, and partitioning.
- Configure and administer Unity Catalog for enterprise data governance.
- Implement security controls across catalog, schema, table, row-level, and column-level access.
- Enable data lineage, auditing, and compliance requirements across the data platform.
- Develop and maintain workflows using Databricks Workflows.
- Manage deployment processes with Databricks Asset Bundles (DABs).
- Implement CI/CD pipelines using GitHub, GitLab, and DevOps best practices.
- Provision and manage infrastructure using Terraform and Infrastructure-as-Code practices.
- Prepare and curate datasets for machine learning and generative AI applications.
- Support feature engineering and feature store implementations.
- Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
- Collaborate with Data Science and AI teams to operationalize ML workloads.
Requirements
- 5–8 years of experience in Data Engineering.
- Strong proficiency in Python (PySpark) and SQL.
- Hands-on expertise with the Databricks Lakehouse Platform.
- Hands-on experience with Delta Lake.
- Hands-on experience with Unity Catalog.
- Hands-on experience with Databricks Workflows.
- Hands-on experience with Structured Streaming.
- Hands-on experience with Auto Loader.
- Experience designing and supporting large-scale enterprise data platforms.
- Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
- Experience implementing CI/CD, version control, and automated testing practices.
- Hands-on experience with Git, GitHub/GitLab, Terraform, and pytest.
- Experience working with at least one major cloud platform: Azure, GCP, or AWS.
- Ability to work onsite at the customer location in Troy, MI at least four (4) days per week.
Key Technologies
- PySpark, SQL, Delta Lake, Auto Loader
- Databricks Lakehouse, Medallion Architecture, Photon Engine, Liquid Clustering
- Unity Catalog, Databricks Workflows, Databricks Asset Bundles (DABs)
- GitHub, GitLab, DevOps, Terraform, Infrastructure-as-Code
- pytest, Azure, GCP, AWS, Structured Streaming
- LLM, vector search, feature store
Compensation
USD 89,400 - 166,398 per year.
Benefits
- Paid time off based on employee grade (A-F): Vacation 12-25 days depending on grade, plus company paid holidays, Personal Days, and Sick Leave
- Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
- Retirement savings plans (401(k) in the U.S., RRSP in Canada)
- Life and disability insurance
- Employee assistance programs
- Other benefits as provided by local policy and eligibility
Job Location
- Troy, MI (Onsite)
- Onsite requirement: minimum four (4) days per week