Store AI Agent - Onshore Data Engineer

This on-site contract role in Dallas, TX focuses on designing, building, and optimizing scalable Databricks-based data and machine learning pipelines. You will support analytics and AI/ML initiatives through cloud-native data engineering, data quality, and deployment-ready infrastructure.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build and optimize batch and real-time ingestion frameworks from diverse data sources.
• Develop ETL/ELT solutions to support enterprise data warehousing, analytics, and AI/ML workloads.
• Implement data quality checks, validation frameworks, monitoring, and observability solutions.
• Optimize Databricks jobs, notebooks, clusters, and workflows for performance and cost efficiency.
• Design and maintain Delta Lake architectures, data models, and metadata management solutions.
• Collaborate with Data Scientists and ML Engineers to prepare training datasets and feature stores.
• Support development and deployment of deep learning solutions using PyTorch.
• Develop scalable feature engineering pipelines and data transformation frameworks.
• Enable model training, experimentation, and inference infrastructure on Databricks.
• Integrate ML workflows with MLOps pipelines and CI/CD processes.
• Apply DataOps and MLOps best practices.
• Automate deployments using CI/CD pipelines.
• Work with cloud-native services for storage, security, monitoring, and orchestration.
• Support infrastructure-as-code and environment automation initiatives.
• Partner with business, analytics, and data science teams to understand requirements and translate them into technical solutions.
• Ensure compliance with data governance, privacy, and security standards.
• Participate in architecture reviews and technology evaluations.

Requirements

• Strong experience with Databricks Platform.
• Expertise in Python.
• Strong hands-on experience with PySpark.
• Experience with PyTorch for machine learning and deep learning workloads.
• Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.
• Experience developing ETL/ELT pipelines.
• Strong SQL and data modeling skills.
• Experience with distributed computing and big data architectures.
• Knowledge of REST APIs and data integration patterns.
• Git version control and CI/CD tools experience.
• Experience with one or more: Microsoft Azure (preferred), AWS, GCP.
• Experience with one or more: Spark, Kafka/Event Streaming, Azure Data Factory or equivalent integration tools, Data Lake architectures, Data Warehousing solutions, Feature Stores.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or related field.
• 5+ years of experience in Data Engineering.
• 3+ years of experience with Databricks and cloud-based data platforms.
• Experience supporting ML and AI use cases using PyTorch.

Technologies

• Databricks Platform
• Python
• PySpark
• PyTorch
• Delta Lake
• Unity Catalog
• Databricks Workflows
• ETL/ELT
• SQL
• REST APIs
• Git
• CI/CD
• Microsoft Azure
• AWS
• Google Cloud Platform (GCP)
• Spark
• Kafka/Event Streaming
• Azure Data Factory
• Feature Stores

Preferred Skills

• Experience with GenAI and LLM data pipelines.
• Knowledge of MLflow and MLOps frameworks.
• Hands-on experience with Docker and Kubernetes.
• Experience with real-time streaming architectures.
• Familiarity with vector databases and AI platforms.
• Certifications in Databricks or Cloud platforms.

Nice-to-Have Experience

• Retail, QSR, Restaurant Technology, Digital Commerce, or Customer Analytics domains.
• Large-scale enterprise data modernization initiatives.
• AI/ML and advanced analytics implementations.

Compensation and Benefits

Pay range: $39.30/hour - $61.40/hour (good-faith expected range).

• Medical
• Dental
• Vision
• Retirement benefits

Position type: contract. Location: Dallas, TX, United States (onsite). Application timing: Applications will be accepted on an ongoing basis.