DeveloperJobs.io
← Back to all jobs

Job Description

Capgemini is hiring a Senior Data Engineer to help design and optimize scalable data solutions on the Databricks lakehouse. This onsite role supports batch and real-time pipelines that power analytics, machine learning, and generative AI use cases, with an emphasis on governance, security, and operational readiness.

Role Focus

In this position, you will build ETL/ELT pipelines and modern lakehouse architectures that support enterprise analytics and ML/AI workflows. The work includes enforcing data quality, enabling lineage and auditing, and operationalizing machine learning workloads in collaboration with Data Science and AI teams.

Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
  • Create robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
  • Implement data quality checks, monitoring, and pipeline optimization to improve reliability and performance.
  • Design and manage Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers).
  • Develop and maintain Delta Lake tables while ensuring optimal query performance.
  • Conduct performance tuning using Databricks best practices, including Photon Engine, Liquid Clustering, and partitioning.
  • Configure and administer Unity Catalog for enterprise data governance.
  • Implement security controls across catalog, schema, table, row-level, and column-level access.
  • Enable data lineage, auditing, and compliance requirements across the data platform.
  • Develop and maintain workflows using Databricks Workflows.
  • Manage deployment processes with Databricks Asset Bundles (DABs).
  • Implement CI/CD pipelines using GitHub, GitLab, and DevOps best practices.
  • Provision and manage infrastructure using Terraform and Infrastructure-as-Code practices.
  • Prepare and curate datasets for machine learning and generative AI applications.
  • Support feature engineering and feature store implementations.
  • Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
  • Collaborate with Data Science and AI teams to operationalize ML workloads.

Requirements

  • 5–8 years of experience in Data Engineering.
  • Strong proficiency in Python (PySpark) and SQL.
  • Hands-on expertise with the Databricks Lakehouse Platform.
  • Hands-on experience with Delta Lake.
  • Hands-on experience with Unity Catalog.
  • Hands-on experience with Databricks Workflows.
  • Hands-on experience with Structured Streaming.
  • Hands-on experience with Auto Loader.
  • Experience designing and supporting large-scale enterprise data platforms.
  • Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
  • Experience implementing CI/CD, version control, and automated testing practices.
  • Hands-on experience with Git, GitHub/GitLab, Terraform, and pytest.
  • Experience working with at least one major cloud platform: Azure, GCP, or AWS.
  • Ability to work onsite at the customer location in Troy, MI at least four (4) days per week.

Key Technologies

  • PySpark, SQL, Delta Lake, Auto Loader
  • Databricks Lakehouse, Medallion Architecture, Photon Engine, Liquid Clustering
  • Unity Catalog, Databricks Workflows, Databricks Asset Bundles (DABs)
  • GitHub, GitLab, DevOps, Terraform, Infrastructure-as-Code
  • pytest, Azure, GCP, AWS, Structured Streaming
  • LLM, vector search, feature store

Compensation

USD 89,400 - 166,398 per year.

Benefits

  • Paid time off based on employee grade (A-F): Vacation 12-25 days depending on grade, plus company paid holidays, Personal Days, and Sick Leave
  • Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
  • Retirement savings plans (401(k) in the U.S., RRSP in Canada)
  • Life and disability insurance
  • Employee assistance programs
  • Other benefits as provided by local policy and eligibility

Job Location

  • Troy, MI (Onsite)
  • Onsite requirement: minimum four (4) days per week

Similar Jobs