Cloud Dataflow

Cloud Dataflow Developer Jobs

4 jobs found

Today

Senior Data Engineer

Handshake San Francisco, CA

San Francisco, CA

Today

$166k - $207k/yr Full time Senior Apache Airflow Application Security

Apply Now

Job Description

Handshake is building reliable data movement to support its career marketplace and AI products, and it needs a senior engineer to strengthen the data platform that makes those workflows dependable. In this on-site role in San Francisco, you will lead technical direction across orchestration, pipelines, streaming and warehouse infrastructure, while owning production operations and guiding scalable platform decisions.

Role Summary

Own and evolve the data platform’s infrastructure for workflow orchestration, data ingestion, warehouse and streaming capabilities, and data delivery for analytics and AI use cases. You will also provide hands-on technical leadership and production support for the systems that keep critical workflows running.

Responsibilities

Set the technical direction and roadmap for the Airflow and Astronomer platform while owning production operations including deployments, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
Create “paved paths” for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, local and staging environments, and runbooks.
Design and operate streaming and change data capture pipelines using Pub/Sub, Dataflow or Beam, and Datastream to support analytics and product needs.
Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, and define targets for latency, freshness, and reliability.
Improve cloud infrastructure for data workloads using Kubernetes, Terraform, IAM, secrets management, CI/CD, and cost-aware capacity management.
Lead cross-team decisions on data contracts, interfaces, and operational ownership, mentor engineers, and align application, analytics, ML, and cloud partners on scalable platform approaches.
Participate in on-call coverage, troubleshoot production failures across systems, and convert incidents into durable improvements.
Build dependable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

Requirements

Strong software engineering skills in Python with experience building and operating production data or distributed systems.
Deep hands-on Airflow experience beyond authoring DAGs, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging.
Experience with event-driven or streaming infrastructure, including a message broker or managed event bus and a stream processing system.
Solid understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and production recovery or replay.
Experience with cloud infrastructure and infrastructure as code, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
Demonstrated ability to lead ambiguous infrastructure initiatives by setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
Clear communication and a track record of partnering across teams while owning systems through production support.

Technologies

Python, Airflow, Astronomer
Pub/Sub, Dataflow, Beam, Datastream, Datadog
BigQuery, Kubernetes, Terraform, IAM, CI/CD
GKE, Cloud Storage, Spacelift, dbt
Spark, Dataproc

Benefits

Equity in a fast-growing company
401(k) match, competitive compensation, financial coaching
Paid parental leave, fertility benefits, parental coaching
Medical, dental, & vision, mental health support, $500 wellness stipend
$2,000 learning stipend, ongoing development
Internet, commuting, & free lunch/gym in our SF office
Flexible PTO, 15 holidays + 2 flex days
Team outings & referral bonuses

Extra Credit

GCP services including Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage
Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc
Experience supporting both batch and low-latency consumers, including product-facing data services or ML features
Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access

Compensation

$166,000 - $207,000 per year
Handshake uses standard ranges for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offer amounts depend on factors including geographic location and candidate experience and expertise.

Location: San Francisco, CA (On-site)

Employment type: Full time

Department: Engineering

View full job page →

2mo ago

Data Engineer

Major League Baseball (MLB) New York, NY

New York, NY

2mo ago Closed

$115k - $140k/yr Full time Python Analytics Apache Airflow

This position is no longer accepting applications (closed Aug 8, 2026).

Job Description

Join MLB's LAI team as a hands-on data engineer shaping production pipelines with Airflow and dbt on GCP to power analytics across clubs and the Commissioner's Office.

Responsibilities

Build production-grade pipelines using Airflow and dbt to orchestrate batch and streaming transformations across GCP, so downstream analysts and engineers can trust the data they query without checking the wiring.
Architect clean, layered data models (staging, intermediate, mart) that serve as the single source of truth for league analytics, applying dbt best practices for materialization, testing, and documentation.
Operate the ingestion layer using Pub/Sub, GCS, Dataflow, and Knowledge Catalog DataPlex to land both batch and streaming sources cleanly into the lakehouse.
Implement observability and monitoring standards so that data quality issues surface before stakeholders notice them.
Manage code through GitHub-based CI/CD, contributing to deployment workflows that keep our platform reliable and our changes safe.
Adhere to data governance practices that keep proprietary baseball data secure and compliant.

Requirements

2–4 years of production data engineering experience.
Expert-level SQL — comfortable writing complex freehand queries (sub-queries, nested logic, window functions) and reading someone else’s to spot issues.
Strong Python for data processing, scripting, and automation.
Hands-on dbt experience — built models across staging, intermediate, and mart layers, written tests, and shipped to production.
Production Airflow experience — DAG authoring, dependency management, debugging failed runs.
Deep familiarity with Google Cloud Platform (BigQuery, GCS, Pub/Sub) or equivalent depth in AWS/Azure with willingness to convert.
Git-based development workflows — branches, PRs, code reviews as a daily practice.
Clear communication with both engineers and non-engineers, receptive to feedback and able to give it kindly.
Execution mindset. You can own a project from requirements to deployment with minimal oversight.

Technologies

Airflow
dbt
Python
SQL
GitHub
BigQuery
GCS
Pub/Sub
Dataflow
Knowledge Catalog DataPlex
Terraform

Benefits

Competitive Benefits Package
Company Contributed 401K Plan
Paid Time Off and Holidays
Paid Parental Leave
Access to Free Tickets to Baseball Games & MLB.TV
Discounts at MLB Store | MLBShop.com
Employee Assistance Programs (EAP)
Onsite/Online Training & Development Programs
Tuition Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Pet Insurance

Salary

Base salary: USD 115,000 - 140,000 per year, plus bonus.

Nice to have

A degree in Computer Science, Engineering, or a related field — or non-traditional background with equivalent practical experience.
Experience with Terraform or other Infrastructure-as-Code tools.
Experience with AI-assisted development or enterprise AI tooling (Gemini Enterprise, Vertex AI).
A passion for baseball, or prior experience in sports, media, or entertainment.
Ability to build creative solutions for unusual problems.

View full job page →

2mo ago

Senior Data Engineer

Advantage Technical Chicago, IL

Chicago, IL

2mo ago Closed

$55 - $64/hr Full time Senior Apache Airflow Big Data

This position is no longer accepting applications (closed Aug 7, 2026).

Job Description

Advantage Technical offers a collaborative, data-driven environment where your work directly informs business decisions. The Senior Data Engineer will build scalable data pipelines on Google Cloud Platform with a focus on BigQuery and SQL, partnering with analytics and data science teams to unlock actionable insights. This role is based in Chicago and onsite four days per week, with an hourly rate of $55 to $64.

You will work with Google Cloud Platform services including BigQuery, Composer, Dataflow, GCS, and DataStream, along with SQL, Python, Airflow, Kafka, Spark, and Pub/Sub.

Benefits

Medical, dental, vision, and pet insurance
401(k) Retirement Savings Plans
Employee Assistance Program (EAP)
Commuter/Parking Accounts
People Perks
Health Savings Account (HSA)
AD&D Insurance

Responsibilities

Design, build, and maintain high-performance data pipelines and ETL workflows
Create and optimize data models, semantic layers, and cloud-based data solutions
Develop REST APIs and support data integration initiatives
Improve pipeline performance, scalability, reliability, and data quality
Establish CI/CD processes for data products and deployments
Troubleshoot complex data engineering and ingestion issues
Document data architecture, lineage, and operational processes
Collaborate with stakeholders across IT, reporting, product, and business teams
Mentor junior data engineers and support Agile development practices

Requirements

Bachelor’s degree in Computer Science, Engineering, or related technical field
6+ years of data engineering experience for Senior-level opportunities
Experience with Google Cloud Platform (GCP), including BigQuery, Composer, Dataflow, GCS, and DataStream
Advanced SQL skills and strong Python programming experience
Experience with Airflow, Kafka, Spark, Pub/Sub, or similar technologies
Strong knowledge of data warehousing, ETL pipelines, and cloud architecture
Excellent communication and problem-solving skills

View full job page →

2mo ago

Data Engineer

Major League Baseball New York, NY

New York, NY

2mo ago Closed

$115k - $140k/yr Full time Apache Airflow Automation Big Data

This position is no longer accepting applications (closed Aug 3, 2026).

Job Description

Major League Baseball is seeking a hands-on Data Engineer to join the LAI team and shape production data pipelines that power league analytics. This onsite role in New York, NY centers on building and operating a GCP lakehouse with Airflow and dbt to deliver trusted data for analysts and engineers. You will own the end-to-end data workflow from ingestion to production monitoring.

Responsibilities

Develop production-grade pipelines with Airflow and dbt to orchestrate batch and streaming transformations across Google Cloud Platform, ensuring downstream analysts and engineers can rely on the data without tracing wiring issues.
Design layered data models (staging, intermediate, mart) as the single source of truth for league analytics, applying dbt best practices for materialization, testing, and documentation.
Manage ingestion through Pub/Sub, GCS, Dataflow, and Knowledge Catalog DataPlex to land batch and streaming sources into the lakehouse with clean, consistent landings.
Establish observability and monitoring standards to surface data quality issues before stakeholders notice them.
Maintain code with GitHub-based CI/CD and contribute to deployment workflows that keep the platform reliable and changes safe.
Follow data governance practices to keep proprietary baseball data secure and compliant.

Requirements

2 to 4 years of hands-on production data engineering experience.
Advanced SQL skills, able to craft complex queries (subqueries, nested logic, window functions) and review others' queries for issues.
Proficient Python for data processing, scripting, and automation.
Hands-on dbt experience building models across staging, intermediate, and mart layers, with tests and production deployment.
Production Airflow experience including DAG authoring, dependency management, and debugging failed runs.
Deep familiarity with Google Cloud Platform (BigQuery, GCS, Pub/Sub) or equivalent depth in AWS/Azure with willingness to convert.
Experience with Git-based development workflows including branching, pull requests, and code reviews as a daily practice.
Clear communicator with both engineers and non-engineers, receptive to feedback, and constructive in responses.
Execution-oriented mindset, able to own projects from requirements to deployment with minimal oversight.

Technologies

Airflow
dbt
BigQuery
GCS
Pub/Sub
Dataflow
Knowledge Catalog DataPlex
GitHub

Benefits

Competitive Benefits Package
Company Contributed 401K Plan
Paid Time Off and Holidays
Paid Parental Leave
Access to Free Tickets to Baseball Games & MLB.TV
Discounts at MLB Store | MLBShop.com
Employee Assistance Programs (EAP)
Onsite/Online Training & Development Programs
Tuition Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Pet Insurance

Nice-to-have

A degree in Computer Science, Engineering, or a related field — or non-traditional background with equivalent practical experience
Experience with Terraform or other Infrastructure-as-Code tools
Experience with AI-assisted development or enterprise AI tooling (Gemini Enterprise, Vertex AI)
A passion for baseball or prior experience in sports, media, or entertainment
Ability to build creative solutions for unusual problems

Salary

USD 115,000 - 140,000 per year, plus bonus.

Why MLB?

Major League Baseball is the historic centerpiece of professional sports in North America, known for a culture that emphasizes growth, teamwork, and professionalism. Employees thrive by taking initiative, solving problems, and putting the team first. MLB aims to empower its workforce to excel by offering experiences that position them for success.

View full job page →

Cloud Dataflow Developer Jobs

Senior Data Engineer

Data Engineer

Senior Data Engineer

Data Engineer

Get Cloud Dataflow Developer Jobs alerts