Data Engineer

VTG is seeking an experienced Data Engineer to design, build, and maintain scalable data processing solutions. This onsite role in Chantilly, VA focuses on ETL pipelines and analytics-ready datasets, including integration and transformation of structured and unstructured data in an Agile delivery environment.

Responsibilities

• Design, develop, and maintain scalable data processing and ETL pipelines using Python, PySpark, and Apache Spark.
• Extract, transform, and load data from multiple structured and unstructured sources into optimized, query-ready datasets.
• Create and maintain data mapping, transformation, and integration processes aligned to business and technical requirements.
• Ingest and convert large volumes of operating system logs, application logs, and other data sources into structured reports, metrics, and analytical dashboards.
• Transform disparate data formats into enriched, indexed, and query-friendly datasets.
• Develop and optimize data processing workflows to improve performance, efficiency, and reliability.
• Build and maintain data integration solutions using SQL, MySQL, and PostgreSQL.
• Work with structured and semi-structured data formats, including XML and JSON.
• Perform data extraction, cleansing, transformation, and validation to ensure accuracy and consistency.
• Develop and maintain data models, database queries, and data mapping specifications.
• Use Regular Expressions (RegEx) to parse, extract, validate, and transform complex data.
• Support integration of data from multiple systems, applications, and external sources.
• Maintain ETL design documentation, including source-to-target mappings, transformation rules, and data dictionaries.
• Use development environments and tools such as Jupyter Notebooks and Visual Studio to support data modeling, analysis, and development.
• Implement data validation and quality assurance processes to protect data integrity, accuracy, and completeness.
• Troubleshoot and resolve data processing, integration, and performance issues.
• Maintain technical documentation describing data workflows, processing logic, and integration requirements.

Requirements

• Active TS/SCI with Polygraph.
• Bachelor’s degree in Computer Science, Information Technology, or a related technical field.
• Demonstrated experience developing data processing solutions using Python, PySpark, and Apache Spark.
• Hands-on experience designing, developing, and maintaining ETL pipelines and data transformation workflows.
• Strong experience with data extraction, mapping, transformation, integration, and loading processes.
• Proficiency in SQL and experience working with MySQL and PostgreSQL.
• Experience processing structured, semi-structured, and unstructured datasets.
• Experience working with XML, JSON, and other common data formats.
• Strong knowledge of Regular Expressions (RegEx) for data parsing, extraction, and manipulation.
• Experience analyzing operating system and application logs and converting them into actionable reports and dashboards.
• Experience developing reports and visualizations using AWS CloudWatch, Kibana, or comparable tools.
• Experience performing data profiling, data quality analysis, validation, and troubleshooting.
• Ability to develop detailed ETL documentation, including source-to-target mappings and data dictionaries.
• Experience using Jupyter Notebooks, Visual Studio, or similar development environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Experience collaborating with customers and integration partners to define and clarify technical requirements.
• Familiarity with Agile software development methodologies and practices.

Technologies

• Python
• PySpark
• Apache Spark
• SQL
• MySQL
• PostgreSQL
• XML
• JSON
• Regular Expressions (RegEx)
• Jupyter Notebooks
• Visual Studio
• AWS CloudWatch
• Kibana