Senior Data Engineer

Delta Airlines is hiring a Senior Data Engineer (onsite) in Atlanta, GA to support IT Service Assurance through the design and delivery of secure, scalable, AI-ready data solutions and trusted data products. This role modernizes AWS data platforms and helps enable operational intelligence and AI-ready architectures across ITSM, CMDB, observability, telemetry, and enterprise platforms.

Responsibilities

• Independently lead complex data initiatives by identifying improvement opportunities, shaping solution approaches, and driving delivery from concept through production.
• Design and modernize secure, scalable AWS data platforms, pipelines, Lakehouse patterns, and reusable data products aligned with enterprise architecture and governance standards.
• Provide senior technical leadership across ingestion, transformation, modeling, metadata, quality, lineage, performance optimization, and production reliability.
• Build and optimize batch, incremental, event-driven, and near-real-time integrations using APIs, databases, files, ServiceNow, CMDB, observability, telemetry, and enterprise platforms.
• Develop maintainable Python and SQL for data pipelines, orchestration, transformations, analytics layers, APIs, automation, and reusable engineering frameworks.
• Establish trusted data products with automated controls for completeness, accuracy, freshness, reconciliation, monitoring, alerting, and exception handling.
• Advance AI-ready architectures by preparing structured and unstructured data for Amazon SageMaker, Amazon Bedrock, LLMs, RAG, vector databases, semantic search, agents, and MCP-enabled solutions.
• Design retrieval, embedding, indexing, enrichment, and context-management patterns that improve usability, discoverability, and responsible AI outcomes.
• Drive operational excellence using CI/CD, Infrastructure as Code, automated testing, source control, code reviews, runbooks, production monitoring, and resilient deployment practices.
• Apply security, privacy, identity, access control, encryption, retention, data classification, and responsible AI requirements across the data lifecycle.
• Collaborate with stakeholders, mentor engineers, promote reusable standards, evaluate emerging technologies, and guide modernization efforts that improve scalability, automation, cost, and business value.

Requirements

• At least 18 years of age and authorized to work in the United States.
• High school diploma, GED, or high school equivalency.
• Embraces diverse people, thinking, and styles.
• Consistently prioritizes safety and security of self, others, and personal data.
• Strong analytical and interpersonal skills, with the ability to build effective relationships and influence without direct authority.
• Superior written and verbal communication skills, along with executive presence to represent the business across the organization.
• 7+ years of IT experience, including 4+ years designing, developing, and supporting enterprise data pipelines, data platforms, lakehouses, or analytical data solutions.
• Demonstrated experience designing and implementing cloud-based data lake, data warehouse, or lakehouse architectures using AWS services.
• Hands-on experience with Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, AWS Lambda, AWS EventBridge, AWS Lake Formation, AWS IAM, and related data services.
• Advanced proficiency in Python and SQL for ingestion, transformation, automation, analysis, data services, and performance optimization.
• Experience designing scalable ETL and ELT pipelines integrating APIs, databases, structured files, semi-structured content, and cloud services.
• Strong experience with dimensional, relational, normalized, denormalized, and analytics-oriented data modeling, including layered raw, curated, published, and analytics architectures.
• Strong understanding of incremental ingestion, change-data capture, watermarking, deduplication, record versioning, historical tracking, late-arriving data, schema evolution, and reconciliation.
• Experience implementing metadata, lineage, cataloging, source-to-target mapping, data-quality testing, freshness monitoring, logging, alerting, and exception handling.
• Working knowledge of Amazon SageMaker, Amazon Bedrock, foundation models, LLM application architectures, RAG, embeddings, vector search, semantic retrieval, and AI-ready data preparation.
• Experience preparing structured and unstructured data through parsing, chunking, enrichment, embedding generation, indexing, access filtering, and context management.
• Strong understanding of data governance, data classification, privacy, encryption, retention, identity, authorization, and secure handling of personal and sensitive information.
• Experience with Amazon Redshift development concepts, including SQL optimization, workload performance, distribution and sort strategies, materialized views, stored procedures, access controls, and columnar formats such as Parquet.
• Experience with Git, automated testing, CI/CD, and Infrastructure as Code such as AWS CDK or CloudFormation.
• Ability to communicate complex technical concepts, data limitations, architecture decisions, AI data requirements, and implementation plans to both technical and business audiences.
• Demonstrated ability to lead technical design, mentor engineers, conduct code and architecture reviews, establish reusable standards, and manage concurrent Agile initiatives.
• Consistent commitment to safety, security, responsible AI, operational reliability, and protection of personal and sensitive data.

Technologies

• AWS, Amazon SageMaker, Amazon Bedrock, LLMs, RAG, vector databases
• Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, AWS Lambda, AWS EventBridge, AWS Lake Formation, AWS IAM
• Python, SQL, ServiceNow, CMDB, Amazon CloudWatch, Dynatrace, PagerDuty, Sumo Logic
• Amazon OpenSearch Service, PostgreSQL, pgvector, OpenID Connect, OAuth 2.0
• CI/CD, Infrastructure as Code, AWS CDK, CloudFormation, Git
• AWS Glue Data Quality, Great Expectations, dbt, Parquet, ETL, ELT, Lakehouse, Lake Formation

Benefits

• Competitive salary, industry-leading profit sharing program, and performance incentives.
• 401(k) with generous company contributions up to 9%.
• Paid time off including vacation, holidays, paid personal time, maternity and parental leave.
• Comprehensive health benefits including medical, dental, vision, short/long term disability, and life benefits.
• Family care assistance including fertility support, surrogacy and adoption assistance, lactation support, subsidized back-up care, and programs supporting loved ones in all stages.
• Holistic Wellbeing programs including an employee assistance program, free financial coaching, and extensive mental health resources.
• Domestic and International space-available flight privileges for employees and eligible family members.
• Career development programs to support long-term career goals.
• World-wide partnerships focused on community service, sustainability, and reducing carbon footprint.
• Business Resource Groups to promote inclusion, shared perspective, and strategy implementation.
• Recognition rewards and awards through Unstoppable Together.
• Access to over 500 discounts and specialty savings through Deltaperks, including voluntary benefits such as car and hotel rentals, auto/home/pet insurance, legal services, and childcare.

What Will Give You a Competitive Edge (Preferred Qualifications)

• AWS Certified Data Engineer, AWS Certified Solutions Architect, AWS Certified Machine Learning Engineer, or another relevant AWS certification.
• Hands-on experience with Amazon SageMaker, Amazon Bedrock, Bedrock Knowledge Bases, Bedrock Agents or AgentCore, Amazon OpenSearch Service, foundation models, embedding models, and production generative AI architectures.
• Experience implementing enterprise RAG solutions including semantic search, hybrid retrieval, reranking, retrieval evaluation, grounding, citations, and AI-powered knowledge repositories.
• Experience designing or integrating vector databases and vector-capable platforms such as Amazon OpenSearch Service, PostgreSQL with pgvector, or comparable technologies.
• Experience with prompt engineering, model evaluation, guardrails, hallucination reduction, responsible AI controls, observability, and secure AI deployment patterns.
• Deep knowledge of ServiceNow data, APIs, table structures, reference fields, ITSM processes, configuration management data, and CMDB relationships.
• Experience integrating observability and operational platforms such as Dynatrace, PagerDuty, Amazon CloudWatch, ServiceNow, Sumo Logic, or similar enterprise services.
• Experience implementing OAuth 2.0, OpenID Connect, identity federation, role-based access controls, and document- or row-level authorization for data and AI retrieval.
• Experience with automated data testing and quality frameworks such as AWS Glue Data Quality, Great Expectations, dbt tests, or comparable technologies.
• Demonstrated history leading data modernization, cloud migration, operational intelligence, platform engineering, generative AI, or AI-enablement initiatives.