Ml Experimentation Developer Jobs

1 job found

Today

Machine Learning Engineer – Evals

Mango Global Technologies New York, NY

New York, NY

Today

$220k - $300k/yr Full time Artificial Intelligence Data Science Engineer

Apply Now

Job Description

Mango Global Technologies is seeking a Machine Learning Engineer – Evals to lead end-to-end evaluation for an AI memory and identity system. This role focuses on creating new evaluation methodologies, building production-grade evaluation infrastructure, and using trace-level analysis to drive concrete fixes.

Key Responsibilities

Define how to determine whether evolving entity representations are improving.
Build infrastructure required to measure improvements over time.
Use evaluation outcomes to identify and fix system failures.
Define what “better” means for representations that evolve.
Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
Continuously evolve evaluation definitions as models and product capabilities change.
Build production-quality Python pipelines and evaluation harnesses.
Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
Create infrastructure that enables researchers and engineers to answer new evaluation questions quickly.
Integrate experiment tracking and observability into evaluation workflows.
Analyze model outputs, agent trajectories, and execution traces.
Identify meaningful failure modes rather than optimizing for superficial metrics.
Translate evaluation findings into concrete engineering or research improvements.
Run experiments, compare results, document findings, and drive follow-up fixes.
Develop and operate simulation environments capable of modeling large numbers of entities.
Increase the throughput and reliability of evaluation runs.
Build systems that support rapid iteration across models, prompts, representations, and evaluation strategies.
Own the full evaluation workflow: Research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required Qualifications

3+ years of professional experience designing and owning LLM evaluations.
Demonstrated experience designing, building, and running LLM evaluations end to end.
Production-quality Python experience, including testing, maintainability, debugging, and release practices.
Experience creating evaluation methodologies, not only running existing benchmarks.
Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
Ability to work onsite in New York City five days per week.
Unrestricted U.S. work authorization (no visa sponsorship available).

Technologies

Python
PyTorch
Hugging Face / Transformers
Weights & Biases
Hydra
SQL
Kafka

Preferred Technical Skills

Weights & Biases
Hydra
SQL
Kafka or other streaming/event-driven infrastructure
Evaluation and observability frameworks
Agent simulation and multi-agent systems
Large-scale data pipelines

Nice to Have

Experience with AI memory, identity, personalization, agent evaluation, or agentic optimization.
Research experience in LLMs, NLP, machine learning, evaluation, or related areas.
First-author/co-author research at NeurIPS, ICML, ICLR, ACL, EMNLP, or comparable venues.
Alternatively, a high-quality open-source project, benchmark, framework, or technical publication demonstrating comparable research ability.
Degree in Computer Science, Mathematics, Statistics, Machine Learning, or another quantitative discipline.

Compensation

USD 220,000 - 300,000 base salary per year + competitive equity.
Final compensation depends on experience, technical depth, research contributions, and interview performance.

Benefits

Employee assistance program

Hiring Process (Technical Focus)

Designing an evaluation from scratch.
Diagnosing an LLM or agent failure from traces.
Building a production evaluation pipeline.
Research methodology and experimental design.
Python and ML engineering.
Previous research or open-source work.

Application Questions

Can you work full-time on-site in New York City 5 days per week, and do you currently have unrestricted U.S. work authorization without requiring visa sponsorship?
Have you either (a) published ML/LLM research at a recognized conference/journal or (b) released substantial open-source research/ML work of comparable quality?
Do you have hands-on experience with PyTorch AND Hugging Face/Transformers?
Do you have production-level Python experience, including writing tests, maintaining production code, debugging, and managing releases?
How many years of professional experience do you have designing and owning LLM evaluations?
Have you personally designed, built, and run an LLM evaluation system end to end, including defining the evaluation methodology and metrics rather than only executing an existing benchmark?
Share your LinkedIn URL.

Location and Role Details

New York, NY (onsite)
Salary range: USD 220,000 - 300,000 per year
Minimum experience: 3 years

View full job page →

Get Ml Experimentation Developer Jobs alerts

New Ml Experimentation Developer Jobs, daily.