Lead Data Engineer – Azure Databricks

Solvimate is building an enterprise-scale data and AI platform initiative, and this hybrid role is centered on leading the engineering direction end to end. As the Lead Data Engineer, you will own architecture and delivery of modern data engineering solutions, with a focus on Azure Databricks and the surrounding Azure data and AI ecosystem.

Location: Cary, NC (hybrid)
Compensation: USD 140,000 - 145,000 per yearly
Experience: 12+ years

Role scope

• Lead architecture, development, optimization, and delivery of production-grade data engineering solutions for an enterprise data and AI platform.
• Design and implement scalable Lakehouse / Medallion Architecture with Bronze, Silver, and Gold layers.
• Develop production-grade data pipelines using Python, Scala, PySpark, Spark Structured Streaming, and Delta Lake.
• Build metadata-driven and parameterized ingestion frameworks for batch, database extracts, CDC, and streaming data.
• Collaborate across Azure Databricks, ADLS Gen2, Azure Data Factory, Azure Event Hubs, and Kafka.

Data platform and governance responsibilities

• Design Delta Lake tables, including data contracts, partitioning strategies, schema evolution, and data retention processes.
• Troubleshoot and optimize Spark workloads, clusters, pipelines, and overall platform performance.
• Implement data governance, lineage, access control, and PII protection using Unity Catalog.
• Monitor data quality, schema drift, data anomalies, reconciliation issues, and pipeline health.

AI-enabled data engineering

• Develop AI-assisted solutions for data ingestion, canonical mapping, data quality, and reconciliation.
• Design and implement production-grade LLM, RAG, and agentic AI workflows.
• Work with embeddings, vector search, hybrid retrieval, prompt engineering, and tool-calling architectures.
• Develop semantic-layer capabilities and text-to-SQL for enterprise data access.

Engineering practices and delivery

• Establish coding standards, automated testing practices, and conduct code and PR reviews.
• Implement CI/CD pipelines using Azure DevOps, Terraform, and Databricks Asset Bundles.
• Mentor data engineers and provide technical guidance across architecture, development, and best practices.
• Present and defend technical designs to engineering teams, architects, and business stakeholders.

Required experience and skills

• 12–18 years of professional experience in Data Engineering, Data Platforms, or related fields.
• Expert-level proficiency in Python, Scala, and PySpark.
• Strong SQL and data modeling experience, including dimensional and normalized data models.
• Extensive hands-on experience with Azure Databricks and Delta Lake.
• Strong knowledge of Databricks Jobs & Workflows, Unity Catalog, cluster management, and performance tuning.
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Strong experience with Azure Data Factory and metadata-driven pipeline frameworks.
• Experience with Azure Event Hubs, Kafka, or Spark Structured Streaming.
• Proven experience designing and delivering enterprise-scale Lakehouse / Medallion architectures.
• At least 3 years of hands-on experience designing and deploying production LLM/AI systems.
• Strong understanding of RAG, embeddings, vector/hybrid retrieval, prompt engineering, and agentic workflows.
• Experience with LangChain, LlamaIndex, or LangGraph.
• Experience with at least one AI platform such as Azure OpenAI, OpenAI, or Databricks Model Serving.
• Experience implementing AI evaluation frameworks, golden datasets, regression testing, and human-in-the-loop validation.
• Strong understanding of Azure security, including Entra ID, RBAC, Managed Identities, Key Vault, ACLs, and Private Endpoints.
• Experience with Azure DevOps, Terraform, CI/CD, and automated testing.
• Excellent communication, documentation, problem-solving, and stakeholder management skills.

Technologies

• Azure Databricks, PySpark, Scala, Python, Delta Lake
• Azure data services, Lakehouse / Medallion Architecture, Spark Structured Streaming
• Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory, Azure Event Hubs, Kafka
• Azure DevOps, Terraform, Databricks Asset Bundles
• LLM, RAG, Unity Catalog, embeddings, vector search, hybrid retrieval, prompt engineering, tool-calling architectures
• text-to-SQL, LangChain, LlamaIndex, LangGraph
• Azure OpenAI, OpenAI, Databricks Model Serving
• Entra ID, RBAC, Managed Identities, Key Vault, ACLs, Private Endpoints
• Databricks Jobs & Workflows, CI/CD, automated testing

Preferred qualifications

• Experience with Knowledge Graphs and Ontologies.
• Knowledge of RDF, SPARQL, Neo4j, or graph-based data modeling.
• Experience developing enterprise-scale Text-to-SQL or semantic-layer solutions.
• Experience with ML-based anomaly detection for transactional or time-series data.
• Previous experience in Financial Services or Insurance.
• Experience with LLMOps / MLOps, model and prompt versioning, cost management, and observability.
• Experience with dbt, Great Expectations, or similar data quality frameworks.
• Databricks Data Engineer Professional certification.
• Azure DP-203, DP-700, or AZ-305 certification.

Benefits

• Benefits

Application requirements

• Full Name
• Current Location
• Contact Number
• Email Address
• Work Authorization – US Citizen
• LinkedIn Profile
• Availability to Start
• Interview Availability for the Next 3 Days
• Updated Resume

What Solvimate is looking for

• A senior engineer with strong architecture skills and hands-on coding ability.
• Ownership from technical design through implementation, testing, deployment, and production support.
• Recent production code writing or review experience may be required.