Cleared Data Engineer
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Database
Databases
Databricks
ETL
Informatica
Reporting and Analytics
Software Development
Spark
SQL
Job Description
Guidehouse is seeking a Cleared Data Engineer to build and support scalable batch and streaming data pipelines in Databricks. This role supports cybersecurity metrics, risk analysis, and reporting, and requires an Active Secret clearance along with experience across data quality, orchestration, optimization, governance, and stakeholder collaboration.
Location
Arlington, VA (onsite)
Compensation
USD 85,000 - 141,000 per year
Key Responsibilities
- Design, develop, test, deploy, and maintain scalable batch and streaming data pipelines in Databricks using Python, PySpark, SQL, and reusable engineering patterns.
- Transform raw source data into curated, analysis-ready datasets and dimensional models that support cybersecurity metrics, risk analysis, dashboards, and executive reporting.
- Build automated data-quality controls for ETL jobs, including checks for schema, completeness, validity, consistency, uniqueness, timeliness, referential integrity, and source-to-target reconciliation.
- Define and implement handling for failed quality checks, including quarantining invalid records, stopping critical loads, generating alerts, documenting exceptions, and coordinating corrective actions with data owners.
- Schedule and orchestrate production workflows, monitor job status and pipeline performance metrics (freshness, throughput, latency, resource utilization), troubleshoot root causes, and restore processing within agreed service levels.
- Improve pipeline reliability, performance, and cost through efficient transformations, workload configuration, incremental processing, partitioning, and compute-resource management.
- Implement and maintain data governance and quality assurance standards, including access controls, metadata, data classification, ownership, lineage, retention, auditability, and protection of sensitive cybersecurity information.
- Maintain source-to-target mappings, data dictionaries, transformation rules, validation evidence, standard operating procedures, runbooks, and other technical documentation for operations, knowledge transfer, and audit readiness.
- Collaborate with cybersecurity analysts, data owners, system administrators, dashboard developers, and business stakeholders to define data requirements, metrics, thresholds, and acceptance criteria.
- Maintain effective client and internal relationships, communicate delivery risks and dependencies, anticipate stakeholder needs, and address issues in a timely and professional manner.
- Organize and lead client meetings by coordinating schedules, creating and distributing agendas and pre-read materials, facilitating discussions, recording minutes, maintaining required documentation, and tracking action items through closure.
- Participate in peer reviews, testing, release activities, and continuous-improvement initiatives to strengthen engineering practices, reduce recurring defects, and improve data product reliability and usability.
Required Qualifications
- ACTIVE and CURRENT SECRET federal security clearance
- Bachelor’s degree
- SIX (6) years total experience in a data or programming discipline, including THREE (3) years of data engineering experience
- Proficiency in data engineering-related programming languages such as Python, PySpark, and/or SQL
- Experience with scheduling, monitoring, troubleshooting, and optimizing production ETL/ELT pipelines
- Experience with data engineering principles and best practices
- Excellent problem-solving skills and attention to detail
- Selected Candidate must be able to commute to client and Guidehouse offices as needed
Preferred (Nice to Have)
- Master’s degree in computer science or related field
- Experience working with system and application logs
- Familiarity with the Databricks platform
- Strong communication and collaboration skills for effective teamwork
Technology Stack
- Databricks
- Python
- PySpark
- SQL
Benefits
- Medical, Rx, Dental & Vision Insurance
- Personal and Family Sick Time & Company Paid Holidays
- Position may be eligible for a discretionary variable incentive bonus
- Parental Leave and Adoption Assistance
- 401(k) Retirement Plan
- Basic Life & Supplemental Life
- Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
- Short-Term & Long-Term Disability
- Student Loan PayDown
- Tuition Reimbursement, Personal Development & Learning Opportunities
- Skills Development & Certifications
- Employee Referral Program
- Corporate Sponsored Events & Community Outreach
- Emergency Back-Up Childcare Program
- Mobility Stipend
Travel Requirements
Up to 50%
Clearance Required
Active Secret
Minimum Experience
3 years (data engineering)