Senior Data Engineer

Handshake is building reliable data movement to support its career marketplace and AI products, and it needs a senior engineer to strengthen the data platform that makes those workflows dependable. In this on-site role in San Francisco, you will lead technical direction across orchestration, pipelines, streaming and warehouse infrastructure, while owning production operations and guiding scalable platform decisions.

Role Summary

Own and evolve the data platform’s infrastructure for workflow orchestration, data ingestion, warehouse and streaming capabilities, and data delivery for analytics and AI use cases. You will also provide hands-on technical leadership and production support for the systems that keep critical workflows running.

Responsibilities

• Set the technical direction and roadmap for the Airflow and Astronomer platform while owning production operations including deployments, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
• Create “paved paths” for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, local and staging environments, and runbooks.
• Design and operate streaming and change data capture pipelines using Pub/Sub, Dataflow or Beam, and Datastream to support analytics and product needs.
• Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, and define targets for latency, freshness, and reliability.
• Improve cloud infrastructure for data workloads using Kubernetes, Terraform, IAM, secrets management, CI/CD, and cost-aware capacity management.
• Lead cross-team decisions on data contracts, interfaces, and operational ownership, mentor engineers, and align application, analytics, ML, and cloud partners on scalable platform approaches.
• Participate in on-call coverage, troubleshoot production failures across systems, and convert incidents into durable improvements.
• Build dependable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

Requirements

• Strong software engineering skills in Python with experience building and operating production data or distributed systems.
• Deep hands-on Airflow experience beyond authoring DAGs, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging.
• Experience with event-driven or streaming infrastructure, including a message broker or managed event bus and a stream processing system.
• Solid understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and production recovery or replay.
• Experience with cloud infrastructure and infrastructure as code, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
• Demonstrated ability to lead ambiguous infrastructure initiatives by setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
• Clear communication and a track record of partnering across teams while owning systems through production support.

Technologies

• Python, Airflow, Astronomer
• Pub/Sub, Dataflow, Beam, Datastream, Datadog
• BigQuery, Kubernetes, Terraform, IAM, CI/CD
• GKE, Cloud Storage, Spacelift, dbt
• Spark, Dataproc

Benefits

• Equity in a fast-growing company
• 401(k) match, competitive compensation, financial coaching
• Paid parental leave, fertility benefits, parental coaching
• Medical, dental, & vision, mental health support, $500 wellness stipend
• $2,000 learning stipend, ongoing development
• Internet, commuting, & free lunch/gym in our SF office
• Flexible PTO, 15 holidays + 2 flex days
• Team outings & referral bonuses

Extra Credit

• GCP services including Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage
• Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc
• Experience supporting both batch and low-latency consumers, including product-facing data services or ML features
• Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access

Compensation

• $166,000 - $207,000 per year
• Handshake uses standard ranges for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offer amounts depend on factors including geographic location and candidate experience and expertise.

Location: San Francisco, CA (On-site)

Employment type: Full time

Department: Engineering