Machine Learning Engineer – Evals

Mango Global Technologies is seeking a Machine Learning Engineer – Evals to lead end-to-end evaluation for an AI memory and identity system. This role focuses on creating new evaluation methodologies, building production-grade evaluation infrastructure, and using trace-level analysis to drive concrete fixes.

Key Responsibilities

• Define how to determine whether evolving entity representations are improving.
• Build infrastructure required to measure improvements over time.
• Use evaluation outcomes to identify and fix system failures.
• Define what “better” means for representations that evolve.
• Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
• Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
• Continuously evolve evaluation definitions as models and product capabilities change.
• Build production-quality Python pipelines and evaluation harnesses.
• Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
• Create infrastructure that enables researchers and engineers to answer new evaluation questions quickly.
• Integrate experiment tracking and observability into evaluation workflows.
• Analyze model outputs, agent trajectories, and execution traces.
• Identify meaningful failure modes rather than optimizing for superficial metrics.
• Translate evaluation findings into concrete engineering or research improvements.
• Run experiments, compare results, document findings, and drive follow-up fixes.
• Develop and operate simulation environments capable of modeling large numbers of entities.
• Increase the throughput and reliability of evaluation runs.
• Build systems that support rapid iteration across models, prompts, representations, and evaluation strategies.
• Own the full evaluation workflow: Research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required Qualifications

• 3+ years of professional experience designing and owning LLM evaluations.
• Demonstrated experience designing, building, and running LLM evaluations end to end.
• Production-quality Python experience, including testing, maintainability, debugging, and release practices.
• Experience creating evaluation methodologies, not only running existing benchmarks.
• Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
• Ability to work onsite in New York City five days per week.
• Unrestricted U.S. work authorization (no visa sponsorship available).

Technologies

• Python
• PyTorch
• Hugging Face / Transformers
• Weights & Biases
• Hydra
• SQL
• Kafka

Preferred Technical Skills

• Weights & Biases
• Hydra
• SQL
• Kafka or other streaming/event-driven infrastructure
• Evaluation and observability frameworks
• Agent simulation and multi-agent systems
• Large-scale data pipelines

Nice to Have

• Experience with AI memory, identity, personalization, agent evaluation, or agentic optimization.
• Research experience in LLMs, NLP, machine learning, evaluation, or related areas.
• First-author/co-author research at NeurIPS, ICML, ICLR, ACL, EMNLP, or comparable venues.
• Alternatively, a high-quality open-source project, benchmark, framework, or technical publication demonstrating comparable research ability.
• Degree in Computer Science, Mathematics, Statistics, Machine Learning, or another quantitative discipline.

Compensation

• USD 220,000 - 300,000 base salary per year + competitive equity.
• Final compensation depends on experience, technical depth, research contributions, and interview performance.

Benefits

• Employee assistance program

Hiring Process (Technical Focus)

• Designing an evaluation from scratch.
• Diagnosing an LLM or agent failure from traces.
• Building a production evaluation pipeline.
• Research methodology and experimental design.
• Python and ML engineering.
• Previous research or open-source work.

Application Questions

• Can you work full-time on-site in New York City 5 days per week, and do you currently have unrestricted U.S. work authorization without requiring visa sponsorship?
• Have you either (a) published ML/LLM research at a recognized conference/journal or (b) released substantial open-source research/ML work of comparable quality?
• Do you have hands-on experience with PyTorch AND Hugging Face/Transformers?
• Do you have production-level Python experience, including writing tests, maintaining production code, debugging, and managing releases?
• How many years of professional experience do you have designing and owning LLM evaluations?
• Have you personally designed, built, and run an LLM evaluation system end to end, including defining the evaluation methodology and metrics rather than only executing an existing benchmark?
• Share your LinkedIn URL.

Location and Role Details

• New York, NY (onsite)
• Salary range: USD 220,000 - 300,000 per year
• Minimum experience: 3 years