Developer Jobs in Doraville, GA

1 job found

Include remote jobs

1d ago

Lead Data Engineer

INSPYR Solutions Doraville, GA 🇺🇸 Remote

Doraville, GA

1d ago

$60 - $70/hr Contract Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

Lead Data Engineer at INSPYR Solutions will architect and optimize distributed data pipelines, establish Lakehouse architecture, enable ML workflows, ensure data quality, and improve production reliability.

Responsibilities

Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment.
Design and maintain enterprise Lakehouse infrastructure with Delta Lake, ensuring ACID compliance, lineage, auditability, and governance.
Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points.
Prepare feature-ready datasets and establish reproducible machine learning deployment patterns to enable ML workflows.
Lead platform-wide data quality, access control, and cataloging frameworks.
Implement cost optimization, cluster tuning, and performance engineering strategies.
Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
Own production reliability, troubleshooting, and root cause analysis for data and ML pipelines.

Requirements

7+ years of experience in advanced data engineering with distributed compute technologies.
Expert Spark engineering experience including performance tuning, cluster configuration, partition strategies, and large dataset optimization.
Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
Strong Python and SQL skills for large-scale data transformations.
Experience supporting machine learning pipelines or model operationalization.
Proven track record architecting cloud-native data platforms on Azure, AWS, or GCP.
Experience integrating diverse, complex data sources at enterprise scale.
Demonstrated ability to own mission-critical production systems.
Experience with distributed streaming frameworks such as Kafka or Event Hubs.
Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
Background in data security, compliance controls, or audit-ready governance.
Experience automating data operations with CI/CD and infrastructure as code.

Technologies

Databricks, Python, SQL
Apache Spark, Delta Lake
MLflow, Notebooks
Hugging Face Transformers, LangChain, LlamaIndex
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
Kafka, Event Hubs, ADLS, S3, GCS
Git, GitHub, GitLab, Azure Repos, Databricks Repos
GitHub Actions, Azure DevOps
Unity Catalog, RBAC, REST APIs

Benefits

Comprehensive medical benefits
Competitive pay
401(k) retirement plan

Core Tools

Databricks (Spark, Delta Lake, MLflow, Notebooks)
Python & SQL
Apache Spark via Databricks
Delta Lake for lakehouse architecture

Cloud Platforms

Azure, AWS, or GCP
Cloud storage options: ADLS, S3, GCS

Data Integration

Kafka or Event Hubs for streaming data
Auto Loader for Databricks file ingestion
REST APIs
AI/ML and MLflow for model tracking and deployment
Hugging Face Transformers, LangChain, LlamaIndex for LLM integration
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini

DevOps

Git-based version control (GitHub, GitLab, Azure Repos)
Databricks Repos for notebooks and code
CI/CD: GitHub Actions, Azure DevOps

Security & Governance

Unity Catalog for data governance
RBAC to manage access controls

View full job page →

Get Developer Jobs in Doraville, GA alerts

New Developer Jobs in Doraville, GA, daily.