Datastream Developer Jobs

1 job found

Today

Senior Data Engineer

Handshake San Francisco, CA

San Francisco, CA

Today

$166k - $207k/yr Full time Senior Apache Airflow Application Security

Apply Now

Job Description

Handshake is building reliable data movement to support its career marketplace and AI products, and it needs a senior engineer to strengthen the data platform that makes those workflows dependable. In this on-site role in San Francisco, you will lead technical direction across orchestration, pipelines, streaming and warehouse infrastructure, while owning production operations and guiding scalable platform decisions.

Role Summary

Own and evolve the data platform’s infrastructure for workflow orchestration, data ingestion, warehouse and streaming capabilities, and data delivery for analytics and AI use cases. You will also provide hands-on technical leadership and production support for the systems that keep critical workflows running.

Responsibilities

Set the technical direction and roadmap for the Airflow and Astronomer platform while owning production operations including deployments, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
Create “paved paths” for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, local and staging environments, and runbooks.
Design and operate streaming and change data capture pipelines using Pub/Sub, Dataflow or Beam, and Datastream to support analytics and product needs.
Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, and define targets for latency, freshness, and reliability.
Improve cloud infrastructure for data workloads using Kubernetes, Terraform, IAM, secrets management, CI/CD, and cost-aware capacity management.
Lead cross-team decisions on data contracts, interfaces, and operational ownership, mentor engineers, and align application, analytics, ML, and cloud partners on scalable platform approaches.
Participate in on-call coverage, troubleshoot production failures across systems, and convert incidents into durable improvements.
Build dependable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

Requirements

Strong software engineering skills in Python with experience building and operating production data or distributed systems.
Deep hands-on Airflow experience beyond authoring DAGs, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging.
Experience with event-driven or streaming infrastructure, including a message broker or managed event bus and a stream processing system.
Solid understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and production recovery or replay.
Experience with cloud infrastructure and infrastructure as code, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
Demonstrated ability to lead ambiguous infrastructure initiatives by setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
Clear communication and a track record of partnering across teams while owning systems through production support.

Technologies

Python, Airflow, Astronomer
Pub/Sub, Dataflow, Beam, Datastream, Datadog
BigQuery, Kubernetes, Terraform, IAM, CI/CD
GKE, Cloud Storage, Spacelift, dbt
Spark, Dataproc

Benefits

Equity in a fast-growing company
401(k) match, competitive compensation, financial coaching
Paid parental leave, fertility benefits, parental coaching
Medical, dental, & vision, mental health support, $500 wellness stipend
$2,000 learning stipend, ongoing development
Internet, commuting, & free lunch/gym in our SF office
Flexible PTO, 15 holidays + 2 flex days
Team outings & referral bonuses

Extra Credit

GCP services including Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage
Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc
Experience supporting both batch and low-latency consumers, including product-facing data services or ML features
Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access

Compensation

$166,000 - $207,000 per year
Handshake uses standard ranges for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offer amounts depend on factors including geographic location and candidate experience and expertise.

Location: San Francisco, CA (On-site)

Employment type: Full time

Department: Engineering

View full job page →

Get Datastream Developer Jobs alerts

New Datastream Developer Jobs, daily.