DeveloperJobs.io
← Back to all jobs

Job Description

EXL Service is hiring a Lead Databricks Data Engineer for an onsite role in New York, NY. You will lead the design, development, modernization, and optimization of enterprise data platforms built on Databricks Lakehouse architecture, supporting analytics, AI/ML, and business intelligence. This full-time position offers USD 15 - 18 per hour and is suited for experienced engineers who want to set technical standards, drive architectural decisions, and improve operational excellence across data engineering programs.

What you’ll do

  • Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
  • Architect scalable ETL/ELT data pipelines for batch and real-time processing.
  • Define enterprise data models, ingestion frameworks, and transformation strategies.
  • Lead implementation of Delta Lake and Unity Catalog along with data governance frameworks.
  • Build and optimize highly scalable Spark and PySpark solutions for large-volume datasets.
  • Drive cloud data platform modernization and migration initiatives.
  • Act as the technical lead for Databricks and Data Engineering programs, including engineering best practices and architecture guidelines.
  • Perform architecture reviews, code reviews, and solution assessments.
  • Mentor Data Engineers, Technical Leads, and junior team members, and help establish consistent coding standards.
  • Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders to translate data requirements into technical solutions.
  • Design secure, scalable, and resilient cloud-native data solutions on Azure, AWS, or GCP.
  • Optimize cost, performance, reliability, and availability of data platforms.
  • Lead CI/CD and Infrastructure-as-Code for data workloads.
  • Define and implement monitoring, observability, and operational excellence practices.
  • Establish governance, lineage, cataloging, and security controls, ensuring compliance with organizational and regulatory requirements.
  • Implement data quality frameworks and monitoring solutions, and coordinate with security and compliance teams on data protection standards.
  • Drive delivery planning, risk management, and resource coordination, and present architecture decisions and recommendations to leadership.

Skills and qualifications

  • 10-12 years of experience (minimum overall: 12-15 years of overall IT experience).
  • Minimum 6+ years of hands-on Databricks experience and 7+ years in Spark/PySpark-based data engineering.
  • Databricks Lakehouse Platform: Databricks Workflows, Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables (DLT), and Databricks Asset Bundles.
  • Strong experience with Apache Spark, PySpark, and Spark SQL.
  • Real-time streaming architecture experience including Kafka.
  • Enterprise-scale Data Lake and Lakehouse solution design experience, including large data transformation and modernization programs.
  • Ability to handle structured, semi-structured, and unstructured data processing, supporting analytics, AI/ML, and reporting ecosystems.
  • Proven experience managing technical teams and driving architectural decisions, including mentoring.
  • Bachelor’s or Master’s Degree in Computer Science, Engineering, Information Technology, or a related field.
  • Certifications: Databricks Certified Data Engineer Professional, Azure Data Engineer Associate, and/or AWS Data Analytics Specialty, and/or Google Professional Data Engineer (listed).
  • TOGAF or Cloud Architecture certifications are a plus; MLOps and AI Engineering exposure is included in the preferred background.
  • Experience with Snowflake and Databricks integration, Microsoft Fabric exposure, and Data Mesh implementation.
  • Data governance tools such as Collibra, Alation, or Informatica, plus expertise in data quality frameworks like Great Expectations or Deequ.

Tools you’ll work with

  • Databricks & streaming: Delta Live Tables (DLT), Structured Streaming, Databricks SQL, Databricks Asset Bundles
  • Cloud data platforms: Azure Databricks, Azure Data Factory (ADF), ADLS Gen2, Azure Synapse Analytics, Event Hub, Azure Key Vault, S3, EMR, Glue, Redshift, Lambda
  • Infrastructure, automation, and orchestration: Terraform, Docker, Kubernetes, Azure DevOps, GitHub Actions, Jenkins, CI/CD and Infrastructure-as-Code
  • Architecture and modeling: Dimensional Modeling, Star & Snowflake Schema, Data Vault, Master Data Management
  • Data engineering ecosystem: Kafka, Pub/Sub, Dataflow, Composer

Employment type: Full-Time • Location: New York, NY (onsite) • Salary: USD 15 - 18 per hour • Experience: 10-12 years

Similar Jobs