DeveloperJobs.io
← Back to all jobs

Job Description

Design and evolve a large-scale offline data platform that powers training dataset generation and reliable ML data workflows.

Responsibilities

  • Build infrastructure for batch and stream big data processing using technologies such as Flink, Spark, Ray, and related systems
  • Design and operate large-scale data pipelines that produce training datasets for machine learning training and experimentation
  • Integrate data pipelines with workflow orchestration systems such as Flyte, Airflow, or similar tooling to support reliable multi-stage training workflows
  • Enhance reproducibility and observability through dataset validation, monitoring, and automated testing
  • Optimize performance and resource utilization across distributed compute systems used for data processing
  • Work closely with ML engineers to enable efficient large-scale experimentation and iterative model development
  • Lead architectural improvements to keep offline data pipelines scalable, reliable, and cost-efficient

Requirements

  • Experience with distributed computing frameworks for data processing, including Flink, Spark, and Ray
  • Experience building infrastructure for training data generation, dataset preparation, or ML feature pipelines
  • Proven ability to optimize big data pipelines with a focus on cost efficiency
  • Strong Python programming skills and experience with large-scale distributed workloads
  • Experience with modern data infrastructure including data lakes, warehouses, orchestration systems, and streaming platforms
  • Strong systems thinking and ability to reason about performance, scalability, reliability, and cost tradeoffs in distributed systems
  • Demonstrated track record of leading technical direction and influencing architectural decisions across teams without formal authority

Technologies

  • Flink
  • Spark
  • Ray
  • Flyte
  • Airflow
  • Python

Benefits

  • Comprehensive health, life, and disability insurance
  • Commute subsidy
  • Employee stock ownership
  • Competitive retirement/pension plans
  • Generous vacation and personal days
  • Support for new parents through leave and family-care programs
  • Office food snacks
  • Mental Health and Wellbeing programs and support
  • Employee Resource Groups
  • Global Employee Assistance Program
  • Training and development programs
  • Volunteering and donation matching program
  • Equity awards
  • Participation in company incentive plans (such as annual discretionary bonuses or sales commissions)

Benefits eligibility, specific offerings, and coverage vary based on the country and employment status. Comprehensive health, life, and disability insurance | Commute subsidy | Employee stock ownership | Competitive retirement/pension plans | Generous vacation and personal days | Support for new parents through leave and family-care programs | Office food snacks | Mental Health and Wellbeing programs and support | Employee Resource Groups | Global Employee Assistance Program | Training and development programs

Similar Jobs