Lead Data Engineer

Lead Data Engineer at INSPYR Solutions will architect and optimize distributed data pipelines, establish Lakehouse architecture, enable ML workflows, ensure data quality, and improve production reliability.

Responsibilities

• Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment.
• Design and maintain enterprise Lakehouse infrastructure with Delta Lake, ensuring ACID compliance, lineage, auditability, and governance.
• Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points.
• Prepare feature-ready datasets and establish reproducible machine learning deployment patterns to enable ML workflows.
• Lead platform-wide data quality, access control, and cataloging frameworks.
• Implement cost optimization, cluster tuning, and performance engineering strategies.
• Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
• Own production reliability, troubleshooting, and root cause analysis for data and ML pipelines.

Requirements

• 7+ years of experience in advanced data engineering with distributed compute technologies.
• Expert Spark engineering experience including performance tuning, cluster configuration, partition strategies, and large dataset optimization.
• Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
• Strong Python and SQL skills for large-scale data transformations.
• Experience supporting machine learning pipelines or model operationalization.
• Proven track record architecting cloud-native data platforms on Azure, AWS, or GCP.
• Experience integrating diverse, complex data sources at enterprise scale.
• Demonstrated ability to own mission-critical production systems.
• Experience with distributed streaming frameworks such as Kafka or Event Hubs.
• Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
• Background in data security, compliance controls, or audit-ready governance.
• Experience automating data operations with CI/CD and infrastructure as code.

Technologies

• Databricks, Python, SQL
• Apache Spark, Delta Lake
• MLflow, Notebooks
• Hugging Face Transformers, LangChain, LlamaIndex
• LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
• Kafka, Event Hubs, ADLS, S3, GCS
• Git, GitHub, GitLab, Azure Repos, Databricks Repos
• GitHub Actions, Azure DevOps
• Unity Catalog, RBAC, REST APIs

Benefits

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan

Core Tools

• Databricks (Spark, Delta Lake, MLflow, Notebooks)
• Python & SQL
• Apache Spark via Databricks
• Delta Lake for lakehouse architecture

Cloud Platforms

• Azure, AWS, or GCP
• Cloud storage options: ADLS, S3, GCS

Data Integration

• Kafka or Event Hubs for streaming data
• Auto Loader for Databricks file ingestion
• REST APIs
• AI/ML and MLflow for model tracking and deployment
• Hugging Face Transformers, LangChain, LlamaIndex for LLM integration
• LLMs: Anthropic Claude, Meta LLaMA, Google Gemini

DevOps

• Git-based version control (GitHub, GitLab, Azure Repos)
• Databricks Repos for notebooks and code
• CI/CD: GitHub Actions, Azure DevOps

Security & Governance

• Unity Catalog for data governance
• RBAC to manage access controls