DeveloperJobs.io
← Back to all jobs

Job Description

Iterable Nova is seeking a Senior Machine Learning Engineer to lead evaluation and automatic optimization for Nova’s models and agents. The role focuses on measuring performance, diagnosing failures, and turning results into production-ready improvements across prompts, context, tools, and agent harnesses.

Key Responsibilities

  • Build evaluation systems for models and agents, including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
  • Establish a feedback loop that converts identified failures into fixes, then verifies improvement with measured results.
  • Translate evaluation outputs into automatic optimization for prompts, context, tools, rubrics, and agent harnesses.
  • Develop synthetic data pipelines to support evaluation and post-training, with a focus on quality, privacy, and reproducibility.
  • Apply relevant research methods, including LLM-as-judge, reward modeling, test-time search, and post-training, in production settings.
  • Measure model and agent performance, locate failure modes, and use those findings to automatically improve prompts, context, tools, and agent harnesses.
  • Partner with Nova retrieval and product teams to connect real interactions to measurable improvements.
  • Mentor engineers and help define the technical direction for Nova.

Required Qualifications

  • 5+ years of hands-on experience in machine learning engineering, including production deployment at scale (for example, production ML pipelines or large-scale experimentation platforms).
  • Master’s or PhD in Computer Science, Machine Learning, AI, or a related field.
  • Demonstrated focus on applying recent research with a goal of shipping reliable changes that measurably improve the customer experience.
  • Strong Python skills, especially for readable, modular, and maintainable code, along with experience using modern ML frameworks.
  • Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning.
  • Experience building automated evaluation systems, such as LLM judges, reward models, or benchmark infrastructure.
  • Experience with at least one of: prompt or context optimization, post-training, preference optimization, or reinforcement learning.
  • Strong communication skills and the ability to collaborate across teams.
  • Experience with LLM APIs from multiple providers, including Anthropic, OpenAI, and Google, with familiarity in tool calling and prompt caching.
  • Experience with automatic prompt optimization and error analysis.

Technologies

  • Python
  • LLM-as-judge
  • Reward modeling
  • Test-time search
  • Post-training
  • Anthropic
  • OpenAI
  • Google

Location

San Francisco, CA (onsite). US.

Compensation

USD 150,000 - 250,000 per year.

Similar Jobs