Machine Learning Engineer (Real-Time Speech Translation)

LILT is building a live translation product, and this role focuses on the core real-time speech translation backend. You’ll take ownership of the end-to-end pipeline from live audio input through streaming ASR and adaptive MT, delivering low-latency translations backed by GPU infrastructure on Kubernetes (Ray Serve).

This onsite position is based in Boston, MA, and the compensation range is USD 120,000 - 161,434 per year. To support production reliability at scale, you’ll work across model serving, streaming systems, and observability, partnering closely with research and engineering teams to keep quality within strict latency budgets.

What you’ll do

• Build and operate services that handle high-throughput, real-time audio and text streaming.
• Manage signal processing, session lifecycles, and concurrency to ensure robust performance under load.
• Integrate and serve streaming speech recognition and machine translation models, collaborating with research to meet latency requirements.
• Implement model-based confidence scoring, route segments for human intervention when needed, and broadcast real-time updates and corrections to end-users.
• Architect and scale production ML infrastructure on GPU-accelerated Kubernetes clusters.
• Apply batching, load balancing, and autoscaling to maintain both performance and cost efficiency.
• Set up comprehensive instrumentation for real-time performance, then use it to identify bottlenecks and reduce end-to-end latency.
• Define technical contracts and interfaces for audio ingestion and downstream service integrations, coordinating with frontend and platform engineering teams.

Requirements

• BS or MS in Computer Science or a related field, or equivalent practical experience.
• 3+ years building production backend or ML serving systems in Python, with strong asyncio skills.
• Hands-on experience with real-time streaming transport such as WebSocket or gRPC bidirectional streaming, including session state, backpressure, and connection lifecycle handling.
• Experience serving ML models in production on GPUs (examples: Ray Serve, Triton, vLLM) using Docker and Kubernetes.
• Experience integrating speech or NLP models into production systems, ideally streaming ASR (partial hypotheses, endpointing, VAD).
• A latency-engineering mindset, including profiling, instrumentation, and optimization with per-stage budget reasoning.
• Effective use of AI coding agents (examples: Claude Code, Codex, or similar) where you can still debug, review, and verify every change.
• US citizenship and residence in the United States (contract requirement).

Preferred qualifications

• Ray Serve experience, especially streaming responses and model multiplexing.
• Familiarity with simultaneous or incremental MT concepts (retranslation, prefix stability, wait-k policies).
• Experience with machine translation quality estimation and confidence estimation (COMET/CometKiwi class models) in production.
• Background with message brokers for real-time fan-out and state distribution (for example RabbitMQ).
• Experience with streaming text-to-speech and optimizing time to first audio.
• Exposure to WebRTC and SFU concepts or voice pipeline frameworks (examples: LiveKit Agents, Pipecat).
• Experience handling CJK and other non-Latin text in NLP pipelines.
• Observability tooling experience for production ML systems (examples: Datadog, Prometheus).

Technologies you may use

Python, asyncio, WebSocket, gRPC, Ray Serve, Triton, vLLM, Docker, Kubernetes, Claude Code, Codex, COMET, CometKiwi, RabbitMQ, Datadog, Prometheus, WebRTC, SFU, LiveKit Agents, Pipecat

Location & eligibility

• Onsite: Boston, MA
• Eligibility: US citizenship and residence in the United States are required.
• Preferred locations: Washington, D.C.; Boston, MA; Indianapolis, IN (ET timezone preferred)