Reliability Engineering Developer Jobs

1 job found

Today

Machine Learning Engineer, Reliability

fal - Features & Labels APAC 📍 APAC Remote

APAC

Today

Full time Application Security Artificial Intelligence Automation

Apply Now

Job Description

This hybrid Machine Learning Engineering and Site Reliability Engineering role focuses on reliability, security, and safety for fal’s generative media model API fleet.

Responsibilities

Own availability, latency, and throughput SLOs for a large fleet of generative media model APIs serving production traffic at scale
Design and run monitoring, alerting, and observability to detect ML-specific issues, including output quality degradation, pipeline breakage, and model regressions
Harden model deployment workflows using canary releases, shadow testing, automated rollbacks, and validation gates for safer model version shipping
Improve the security posture of the model fleet with secure model serving, abuse and misuse detection, rate limiting, and protection against adversarial usage patterns
Operationalize safety systems for generative media, including content moderation pipelines, safety classifiers, and guardrails that run reliably at inference time without sacrificing performance
Lead incident response for model API outages and degradations, perform postmortems, and drive engineering changes to prevent recurrence
Strengthen capacity planning, autoscaling, and GPU fleet efficiency for inference workloads with highly variable traffic
Partner with model and infrastructure teams so reliability, security, and safety requirements are built into new model onboarding to the platform

Requirements

5+ years of professional experience
2+ years operating production ML or high-scale API systems, ideally with on-call ownership
Experience working with and supporting diffusion models in production
Strong systems fundamentals across distributed systems, networking, observability, and incident management
Working knowledge of modern generative models (diffusion, transformers) and their production failure modes
Familiarity with security and safety practices for ML systems (abuse prevention, content safety, or trust and safety experience is a plus)
Automation-oriented mindset with emphasis on measurement and blameless postmortems

Tech & Tools

Python
torch
diffusers
Kubernetes
fal Python SDK

Location

Remote - APAC
Role will need to be based in India, Australia, or New Zealand

Employment

Full time
Department: EngineeringML

Working Environment

Access to a massive GPU cluster for inference and evaluation
Work alongside a team focused on quickly iterating on and deploying new AI breakthroughs while maintaining reliability

View full job page →

Get Reliability Engineering Developer Jobs alerts

New Reliability Engineering Developer Jobs, daily.