Senior Data Engineer

Red Cedar Consultancy, LLC is hiring a Senior Data Engineer / Databricks Specialist to support a federal project in Washington, DC (onsite). This role focuses on building secure, reliable, high-performance data processing on AWS Databricks, with a strong emphasis on ETL/ELT pipelines, data quality, and production readiness in mission-critical environments.

Compensation for this position is USD 110,000 - 115,000 per year. You will join cross-functional Agile/Scrum teams working in a multi-contractor federal setting, providing technical leadership and production on-call support.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, Gold), ensuring data integrity, completeness, and lineage.
• Administer Databricks workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
• Manage AWS services including S3, Lambda, IAM, Aurora, and Glue; provision infrastructure with Terraform.
• Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing systems.
• Ingest and integrate structured, semi-structured, and unstructured data from APIs and databases.
• Implement security and compliance controls within DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tools, supporting FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
• Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking.
• Collaborate with Agile/Scrum teams and provide technical leadership and production on-call support.

Requirements

• At least 3 years of dedicated, hands-on Databricks development experience.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL, plus working knowledge of REST API-based data ingestion.
• Strong hands-on experience with AWS S3, IAM, Lambda, Glue, and Aurora.
• Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins, GitLab CI, or equivalent.
• Experience working in Agile/Scrum delivery environments.
• Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
• Active or previous High-Risk Public Trust (Tier 4) clearance.

Benefits

• 401(k) matching
• Dental insurance
• Employee discount
• Flexible spending account
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Professional development assistance
• Referral program
• Vision insurance

Preferred Qualifications

• Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
• Experience supporting federal government contracts and FISMA/FedRAMP environments.

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Machine Learning Associate
• AWS Certified Solutions Architect – Associate or Professional
• AWS Certified Data Analytics – Specialty

Security Clearance

• Confidential (Required)

Application question: Do you have at least 3 years of hands-on experience with AWS Databricks, including PySpark, Spark SQL, and Delta Lake?