Senior Machine Learning Engineer, Agentic Evals and Self-Improvement
Agentic Ai
Ai Agent
Ai Agent Platform
Ai Ml
Ai Platform
Artificial Intelligence
Artificial Intelligence Engineer
Data Analysis
Data Engineer
Data Pipeline
Data Processing
Data Science
Engineer
Gen Ai Evaluation
Generative AI
Generative Ai Agents
Generative Ai Engineering
Machine Learning Engineering
Machine Learning Evaluation
Machine Learning Optimization
Machine Learning Platform
Machine Learning Systems
Machine Learning Training
Programming
Software Development
Software Engineering
Job Description
Iterable Nova is seeking a Senior Machine Learning Engineer to lead evaluation and automatic optimization for Nova’s models and agents. The role focuses on measuring performance, diagnosing failures, and turning results into production-ready improvements across prompts, context, tools, and agent harnesses.
Key Responsibilities
- Build evaluation systems for models and agents, including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
- Establish a feedback loop that converts identified failures into fixes, then verifies improvement with measured results.
- Translate evaluation outputs into automatic optimization for prompts, context, tools, rubrics, and agent harnesses.
- Develop synthetic data pipelines to support evaluation and post-training, with a focus on quality, privacy, and reproducibility.
- Apply relevant research methods, including LLM-as-judge, reward modeling, test-time search, and post-training, in production settings.
- Measure model and agent performance, locate failure modes, and use those findings to automatically improve prompts, context, tools, and agent harnesses.
- Partner with Nova retrieval and product teams to connect real interactions to measurable improvements.
- Mentor engineers and help define the technical direction for Nova.
Required Qualifications
- 5+ years of hands-on experience in machine learning engineering, including production deployment at scale (for example, production ML pipelines or large-scale experimentation platforms).
- Master’s or PhD in Computer Science, Machine Learning, AI, or a related field.
- Demonstrated focus on applying recent research with a goal of shipping reliable changes that measurably improve the customer experience.
- Strong Python skills, especially for readable, modular, and maintainable code, along with experience using modern ML frameworks.
- Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning.
- Experience building automated evaluation systems, such as LLM judges, reward models, or benchmark infrastructure.
- Experience with at least one of: prompt or context optimization, post-training, preference optimization, or reinforcement learning.
- Strong communication skills and the ability to collaborate across teams.
- Experience with LLM APIs from multiple providers, including Anthropic, OpenAI, and Google, with familiarity in tool calling and prompt caching.
- Experience with automatic prompt optimization and error analysis.
Technologies
- Python
- LLM-as-judge
- Reward modeling
- Test-time search
- Post-training
- Anthropic
- OpenAI
Location
San Francisco, CA (onsite). US.
Compensation
USD 150,000 - 250,000 per year.
Similar Jobs
F