Senior Big Data Engineer

Senior Big Data Engineer at SDA, focused on building scalable data ingestion, integration, and analytics solutions using Java, Kafka, Spark, Elasticsearch, and cloud platforms. This full-time role is based in Arlington, VA with a hybrid work arrangement.

Responsibilities

• Design, develop, and maintain enterprise data ingestion, ETL, and data transformation solutions supporting both batch and real-time processing.
• Build and support scalable Kafka-based streaming architectures for processing and distributing high-volume data.
• Develop Java-based applications and services supporting enterprise data integration and transformation workflows.
• Create Spark and Spark Streaming solutions to process, enrich, and analyze large datasets.
• Design and support enterprise search capabilities using Elasticsearch to enable data discovery, investigative workflows, analytics, and operational decision-making.
• Analyze source and target data structures and develop source-to-target mapping specifications.
• Integrate data from multiple structured and semi-structured sources while ensuring data integrity, consistency, traceability, and performance.
• Develop REST APIs and system integration services that enable data exchange across enterprise platforms.
• Support data migration, modernization, and cloud transformation initiatives.
• Perform data validation, reconciliation, profiling, and quality assurance activities.
• Implement and support data quality, governance, lineage, and validation processes across ingestion and transformation pipelines.
• Optimize SQL queries, database processes, and distributed data pipelines.
• Troubleshoot production issues and improve platform reliability, scalability, and performance.
• Collaborate with architects, developers, analysts, and stakeholders to design and implement technical solutions.
• Support onboarding efforts for data consumers and integration partners.
• Evaluate emerging technologies and develop proof-of-concept solutions.
• Mentor junior engineers and contribute to technical leadership initiatives.

Requirements

• U.S. Citizen
• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field (or equivalent experience).
• 5-10+ years of experience in Data Engineering, Software Engineering, ETL Development, or Enterprise Data Integration.
• Strong hands-on Java development experience.
• Experience designing and implementing ETL and data integration solutions.
• Experience building and supporting Apache Kafka producers, consumers, and streaming applications.
• Experience developing applications using Apache Spark and/or Spark Streaming.
• Strong SQL and relational database experience, including Oracle.
• Experience creating and maintaining source-to-target data mappings.
• Experience with data transformation, data migration, and data validation activities.
• Experience developing REST APIs and integration services.
• Experience working with XML, JSON, CSV, and other structured or semi-structured data formats.
• Experience supporting Linux/Unix environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• CBP BI eligibility.

Technologies

• Java
• Kafka
• Spark
• Elasticsearch
• Oracle
• SQL
• ETL
• Data Integration
• Data Mapping
• REST APIs
• MongoDB
• AWS
• Linux
• Scala
• Spring Framework
• Spring Boot
• Apache NiFi
• AWS Glue
• Informatica
• Talend
• Hadoop
• HBase
• Delta Lake
• PostgreSQL
• DocumentDB
• Redshift
• Docker
• Kubernetes
• Azure
• Google Cloud Platform
• CI/CD Pipelines
• Spark Streaming

Technologies you’ll work with

• Java
• Scala
• SQL
• Spring Framework
• Spring Boot
• REST APIs
• ETL Development
• Data Mapping & Transformation
• Data Migration
• Apache NiFi
• AWS Glue
• Informatica
• Talend
• Apache Kafka
• Apache Spark
• Spark Streaming
• Hadoop
• HBase
• Delta Lake
• Elasticsearch
• Oracle
• PostgreSQL
• MongoDB
• DocumentDB
• Redshift
• Relational & NoSQL Databases
• AWS
• Azure
• Google Cloud Platform
• Docker
• Kubernetes
• CI/CD Pipelines

Benefits

• Work with modern Big Data, cloud, and enterprise integration technologies.
• Design and build large-scale data platforms supporting mission-critical operations.
• Collaborate with highly skilled engineers, architects, and data professionals.
• Influence technical direction and solution architecture.
• Competitive compensation and benefits package.
• Flexible hybrid work environment.
• Opportunities for professional growth and technical leadership.

Preferred Qualifications

• Experience with Elasticsearch and enterprise search platforms.
• Experience with Hadoop ecosystem technologies.
• Experience with Scala development.
• Experience with MongoDB, DocumentDB, HBase, Delta Lake, Redshift, or other Big Data technologies.
• Experience with ETL tools such as Apache NiFi, AWS Glue, Talend, Informatica, Pentaho, or SSIS.
• Experience with event-driven architectures and enterprise messaging technologies.
• Experience with Docker and Kubernetes.
• Experience supporting AWS, Azure, or Google Cloud Platform environments.
• Experience supporting enterprise data migration or modernization initiatives.
• Experience working in Agile software development environments.
• Experience supporting large-scale analytics, risk assessment, investigative, intelligence, law enforcement, or operational decision-support systems.

Why join SDA?

• Work with modern Big Data, cloud, and enterprise integration technologies.
• Design and build large-scale data platforms supporting mission-critical operations.
• Collaborate with highly skilled engineers, architects, and data professionals.
• Influence technical direction and solution architecture.
• Competitive compensation and benefits package.
• Flexible hybrid work environment.
• Opportunities for professional growth and technical leadership.