Lead Data Engineer
Artificial Intelligence
Big Data
Bigdata
Cloud Operations
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Lake
Data Lakehouse
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Delta Lake
DevOps
Engineering
Integration
Kafka
Lakehouse
Machine Learning
Machine Learning Engineer
Ml Ops
Mlflow
NLP
Programming Languages
Pyspark
Spark
SQL
Stream Processing
Technical Lead
Job Description
Lead Data Engineer at INSPYR Solutions will architect and optimize distributed data pipelines, establish Lakehouse architecture, enable ML workflows, ensure data quality, and improve production reliability.
Responsibilities
- Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment.
- Design and maintain enterprise Lakehouse infrastructure with Delta Lake, ensuring ACID compliance, lineage, auditability, and governance.
- Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points.
- Prepare feature-ready datasets and establish reproducible machine learning deployment patterns to enable ML workflows.
- Lead platform-wide data quality, access control, and cataloging frameworks.
- Implement cost optimization, cluster tuning, and performance engineering strategies.
- Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
- Own production reliability, troubleshooting, and root cause analysis for data and ML pipelines.
Requirements
- 7+ years of experience in advanced data engineering with distributed compute technologies.
- Expert Spark engineering experience including performance tuning, cluster configuration, partition strategies, and large dataset optimization.
- Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
- Strong Python and SQL skills for large-scale data transformations.
- Experience supporting machine learning pipelines or model operationalization.
- Proven track record architecting cloud-native data platforms on Azure, AWS, or GCP.
- Experience integrating diverse, complex data sources at enterprise scale.
- Demonstrated ability to own mission-critical production systems.
- Experience with distributed streaming frameworks such as Kafka or Event Hubs.
- Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
- Background in data security, compliance controls, or audit-ready governance.
- Experience automating data operations with CI/CD and infrastructure as code.
Technologies
- Databricks, Python, SQL
- Apache Spark, Delta Lake
- MLflow, Notebooks
- Hugging Face Transformers, LangChain, LlamaIndex
- LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
- Kafka, Event Hubs, ADLS, S3, GCS
- Git, GitHub, GitLab, Azure Repos, Databricks Repos
- GitHub Actions, Azure DevOps
- Unity Catalog, RBAC, REST APIs
Benefits
- Comprehensive medical benefits
- Competitive pay
- 401(k) retirement plan
Core Tools
- Databricks (Spark, Delta Lake, MLflow, Notebooks)
- Python & SQL
- Apache Spark via Databricks
- Delta Lake for lakehouse architecture
Cloud Platforms
- Azure, AWS, or GCP
- Cloud storage options: ADLS, S3, GCS
Data Integration
- Kafka or Event Hubs for streaming data
- Auto Loader for Databricks file ingestion
- REST APIs
- AI/ML and MLflow for model tracking and deployment
- Hugging Face Transformers, LangChain, LlamaIndex for LLM integration
- LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
DevOps
- Git-based version control (GitHub, GitLab, Azure Repos)
- Databricks Repos for notebooks and code
- CI/CD: GitHub Actions, Azure DevOps
Security & Governance
- Unity Catalog for data governance
- RBAC to manage access controls
Similar Jobs
