Senior Data Engineer - Databricks

Steampunk offers a collaborative, client-focused environment for data professionals who value clear communication, strong customer service, and pragmatic problem solving. This onsite role in McLean, VA offers a yearly salary range of 125,000 to 165,000 USD and requires a Master’s Degree with eight or more years of relevant experience. The position sits within a broader benefits package that recognizes total compensation beyond base pay.

Overview

We seek a seasoned technologist who can design and implement enterprise grade data platforms, services, and pipelines while partnering with clients and teams. The ideal candidate combines deep data engineering expertise with excellent communication skills and a genuine interest in solving complex data problems.

Responsibilities

• Lead the design and migration of data environments to optimize performance and reliability.
• Assess ETL jobs, workflows, BI tools, and reports to inform architectural decisions.
• Address technical inquiries related to customization, integration, enterprise architecture, and feature functionality of data products.
• Develop database and data warehouse solutions on Databricks.
• Apply core skills in Databricks, Python, and SQL to deliver robust data solutions.
• Support an Agile software development lifecycle within projects.
• Contribute to the growth of our Data Exploitation Practice.

Requirements

• Ability to hold a position of public trust with the US government.
• 12+ years of experience with a Bachelors Degree or 8+ years with a Master’s Degree.
• 8-12 years of industry experience in data engineering with a passion for solving complex problems.
• 8-12 years direct data engineering experience, including designing and building data pipelines and data products on modern lakehouse platforms such as Databricks and Apache Spark, including Delta Lake, with distributed processing and performance optimization.
• Experience supporting or integrating GenAI/ML workflows, including feature engineering, vector storage, RAW pipelines, or model lifecycle using tools like MLFlow.
• Experience operating in enterprise or regulated cloud environments, working within platform constraints, security controls, and data governance frameworks.
• Experience building batch and streaming data pipelines using distributed processing frameworks within a lakehouse architecture.
• Experience with search and retrieval patterns for analytics or AI use cases, including indexing and vector-based retrieval.
• Strong proficiency in SQL and Python for data engineering, with experience using distributed data processing frameworks such as Apache Spark.
• Advanced SQL knowledge with experience in relational databases, query authoring, and optimization across multiple databases.
• Ability to lead technical discussions with both technical and non-technical stakeholders, translating business needs into actionable data solutions and guiding decisions.
• Proven ability to evaluate feasibility, identify risks and constraints, and make pragmatic recommendations in complex or ambiguous environments.
• Experience navigating cross-team environments and driving alignment through clear communication and structured problem solving.
• Experience constructing complex queries to analyze results in databases or data processing environments.
• Strong judgment balancing technical constraints, business needs, and long-term scalability.
• Experience architecting transactional and data warehouse systems.
• Experience aggregating results for reporting from multiple datasets.
• Experience working in an Agile environment.
• Experience supporting project teams of developers and data scientists who build web-based interfaces, dashboards, reports, and analytics or machine learning models.

Technologies

• Databricks
• Python
• SQL
• Apache Spark
• Delta Lake
• MLFlow
• Apache Spark Structured Streaming

About Steampunk

Steampunk outlines compensation within a broader total rewards package. The projected annual salary for this role is 125,000 to 165,000 USD. The stated range reflects typical pay for this position, while actual total compensation includes additional benefits and incentives. Learn more about the overall Steampunk benefits package on our site.

Identity verification

As part of the application process, you may be required to participate in video interviews and assessments. We reserve the right to capture information and images to verify identity and prevent fraud during the process.