Data Engineer - Managing Consultant

Guidehouse seeks a Data Engineer - Managing Consultant to design and deliver scalable, governed data pipelines and analytics solutions using Databricks and AWS. The role emphasizes security, observability, and enterprise data availability and is based onsite in McLean, VA.

Responsibilities

• Design, build, and optimize scalable production grade data ingestion, transformation, and analytics ready pipelines using Databricks components (Delta Lake, Delta Live Tables, Auto Loader) and AWS services to provide trusted data access across enterprise use cases.
• Engineer standardized pipelines supporting batch and near real time processing, integrating legacy data sources with modern cloud native services to enhance enterprise data availability and close data access gaps.
• Manage full delivery lifecycle by supporting intake, discovery, source profiling, and technical design while maintaining requirements traceability and aligning solutions with Architecture Review Board governance expectations.
• Implement and maintain governed data pipelines with embedded metadata, lineage, and data quality controls, ensuring compliance with security and documentation requirements prior to production deployment.
• Develop and operationalize data engineering frameworks that incorporate observability, monitoring, alerting, and resilient error handling to maintain platform stability and support a 99.9 percent availability target.
• Collaborate with platform engineering and cloud operations teams to integrate pipelines with AWS services such as S3, Glue, Kafka/Kinesis, and APIs, enabling secure, scalable data movement and cross platform interoperability.
• Enable governed analytics and self service data consumption through curated datasets, semantic layers, and SQL warehouse integration to support enterprise reporting, dashboards, and advanced analytics use cases.
• Apply security and compliance controls aligned to IRS cybersecurity policies, including RBAC/ABAC, data masking, encryption, and audit logging to protect sensitive data and maintain regulatory compliance.

Requirements

• Bachelor’s degree is required
• Minimum eight (8) years of experience in data engineering within cloud based environments
• Minimum four (4) years of hands on Databricks experience designing scalable data pipelines
• Advanced proficiency in Python, PySpark, and SQL for large scale data processing and pipeline development in lakehouse architectures
• Demonstrated experience building and optimizing data pipelines leveraging Delta Lake, medallion architecture, and modern data ingestion frameworks (batch and streaming)
• Strong experience with AWS data platforms and services (e.g., S3, IAM, VPC, Glue, streaming frameworks) and integration with enterprise data ecosystems
• Experience delivering production ready data solutions incorporating metadata management, lineage, data quality, and observability frameworks
• Familiarity with DevSecOps and CI/CD pipeline implementation, including automation, testing, and deployment within cloud data environments
• Knowledge of data governance, security, and compliance requirements within regulated environments, including FISMA and FedRAMP High

Technologies

• Databricks, Delta Lake, Delta Live Tables, Auto Loader
• AWS, S3, Glue, IAM, VPC
• Kafka, Kinesis, APIs
• Python, PySpark, SQL
• Delta Sharing, Unity Catalog, Genie, Clean Rooms
• Informatica EDC/Axon, EventBridge
• Terraform, GitHub Actions
• Databricks Asset Bundles, Photon, Z-ordering, liquid clustering

Benefits

• Medical, Rx, Dental & Vision Insurance
• Personal and Family Sick Time & Company Paid Holidays
• Discretionary variable incentive bonus eligibility
• Parental Leave and Adoption Assistance
• 401(k) Retirement Plan
• Basic Life & Supplemental Life insurance
• Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
• Short-Term & Long-Term Disability
• Student Loan PayDown
• Tuition Reimbursement, Personal Development & Learning Opportunities
• Skills Development & Certifications
• Employee Referral Program
• Corporate Sponsored Events & Community Outreach
• Emergency Back-Up Childcare Program
• Mobility Stipend

Job Family

Data Engineering & Architecture Consulting

Travel Required

Up to 10%

Clearance Required

Ability to Obtain Public Trust

Nice to Have

• Experience supporting federal data platforms or large scale enterprise data modernization efforts, particularly within IRS, Treasury, or similar regulated environments
• Hands on experience with Databricks Unity Catalog, Delta Sharing, Genie, and Clean Rooms for governed data access and collaboration
• Experience implementing streaming and near real time data pipelines using Kafka, Kinesis, EventBridge, or similar technologies
• Familiarity with Informatica EDC/Axon or enterprise metadata/catalog tooling
• Experience with performance optimization techniques (Photon, Z-ordering, liquid clustering) to improve large scale data workloads
• Exposure to CI/CD automation using Terraform, GitHub Actions, and Databricks Asset Bundles for infrastructure and pipeline deployment
• Advanced cloud or Databricks certifications (AWS, Databricks) in good standing