Machine Learning Evaluation

Machine Learning Evaluation Developer Jobs

5 jobs found

Today

Evaluation & Insights Machine Learning Engineer

Apple Cupertino, CA

Cupertino, CA

Today

$185k - $325k/yr Full time Ai Evaluation Ai Testing And Evaluation Artificial Intelligence

Apply Now

Job Description

Work on Apple’s Human-Centered AI team to evaluate and improve AI systems using data science, model behavior analysis, and qualitative insights.

Architect and execute comprehensive evaluation suites for LLMs and multimodal models, surfacing edge cases in multi-step reasoning, factuality, adversarial robustness, safety, and alignment
Build deterministic, heuristic, and LLM-assisted evaluation frameworks (including LLM-as-a-judge and reward modeling) to quantify human-perceived quality metrics such as helpfulness and hallucination rates
Convert qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for both training and inference
Partner with engineering teams to refine model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning
Apply advanced ML techniques (embedding-based clustering, representation learning, perturbation analysis) to map error taxonomies and latent failure manifolds
Develop robust MLOps workflows to codify evaluation metrics, automate regression testing across model checkpoints, and integrate human-centric assessments into ML CI/CD pipelines
Design scalable, distributed inference and processing pipelines (for example, Ray and vLLM) to support high-throughput evaluation, automated annotation, and large-scale output analysis
Define quantitative evaluation frameworks capturing nuanced human factors such as trust calibration, conversational state tracking, and interpretability
Build automated evaluation pipelines that use LLMs to assess outputs at scale, with optimization for high correlation to human baseline annotations
Collaborate across Apple with ML researchers, software developers, and product managers to translate product requirements into reliable and efficient evaluation infrastructure

Requirements

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design
Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field
8+ years of relevant industry experience in ML Engineering or Applied Research
Advanced proficiency in Python and modern deep learning ecosystems (PyTorch, JAX, Hugging Face)
Proven experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems
Strong ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives
Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems
Deep familiarity with AI quality metrics, hallucination detection techniques (for example SelfCheckGPT), model alignment (RLHF/DPO), and LLM-as-a-judge frameworks (for example G-Eval and DeepEval)
Experience building internal tools or automated pipelines for ML workflows using tools like MLflow and Weights & Biases or similar platforms
Strong familiarity with advanced prompt engineering, RAG architectures (vector databases, semantic search), and Fine-Tuning

Technologies

Python
PyTorch
JAX
Hugging Face
LLM-as-a-judge
reward modeling
Retrieval-Augmented Generation (RAG)
embedding-based clustering
representation learning
perturbation analysis
MLOps
CI/CD pipelines
Ray
vLLM
trust calibration
SelfCheckGPT
RLHF
DPO
G-Eval
DeepEval
MLflow
Weights & Biases
vector databases
semantic search
Fine-Tuning
LLM-assisted evaluation frameworks

Benefits

Comprehensive medical and dental coverage
Retirement benefits
Discounted products and free services
Reimbursement for certain educational expenses, including tuition
Discretionary restricted stock unit awards
Opportunity to purchase Apple stock at a discount via the Employee Stock Purchase Plan
Base pay range between $184,700 and $324,800
Comprehensive total compensation package may include discretionary bonuses or commission payments as well as relocation

Pay & Benefits

Base pay range: $184,700 to $324,800 (depends on skills, qualifications, experience, and location)
Employee stock programs: eligibility-dependent discretionary employee stock programs and Employee Stock Purchase Plan participation
Benefits include comprehensive medical and dental coverage, retirement benefits, discounted products and free services, and tuition reimbursement for formal education related to career advancement
This role might be eligible for discretionary bonuses or commission payments and relocation
Note: Apple benefit, compensation, and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program

View full job page →

Today

Machine Learning Engineer

XPO Boston, MA

Boston, MA

Today

$100k - $120k/yr Full time Artificial Intelligence Automation Bigquery

Apply Now

Job Description

In this hybrid role in Boston, MA, XPO is hiring a Machine Learning Engineer to build and maintain the systems that make ML programs reliable in real production settings. You will focus on data preparation and validation tooling, ML infrastructure for training and deployment, and MLOps capabilities such as CI/CD, monitoring, and feedback loops.

This position supports the full path from experimentation to deployment by partnering with applied and data scientists and coordinating with data engineering to ensure dependable, accessible data pipelines.

What you’ll work on

Build and maintain data preparation and validation tooling to ensure high-quality inputs for ML and optimization models
Design and implement ML infrastructure for model training, evaluation, and deployment
Create and maintain CI/CD pipelines for machine learning models, including automated testing and validation
Implement model monitoring, drift detection, and feedback loops to track performance in production
Partner with applied and data scientists to productionize models and streamline the path from experimentation to deployment
Collaborate with data engineering teams to ensure reliable, accessible data pipelines
Contribute to shared MLOps tooling and best practices across the AI/ML organization

Minimum qualifications

Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent related work or military experience)
1 year of experience in software or machine learning engineering, including hands-on experience building data pipelines, ML infrastructure, or MLOps tooling
Experience developing data preparation, validation, or quality-checking tooling for machine learning pipelines
Proficiency in Python and SQL
Experience with cloud data or ML platforms, such as AWS, GCP, or BigQuery
Strong collaboration skills working with data science/applied science teams and data engineering teams
Master’s degree in Computer Science or related field
3+ years of experience building ML infrastructure for training, evaluation, and deployment at scale
Experience building and maintaining CI/CD pipelines for machine learning models
Experience with model serving and inference infrastructure (batch and real-time)
Experience implementing model monitoring, drift detection, and feedback-loop tooling
Experience with containerization and orchestration (Docker, Kubernetes)
Experience partnering with data engineering teams on data pipeline reliability and access

Technologies

Python
SQL
AWS
GCP
BigQuery
Docker
Kubernetes

Compensation and benefits

Annual salary range: $100,000 to $120,000 (actual compensation may vary based on experience and skill set)
Full health insurance benefits available on day one
Life and disability insurance
Earn up to 15 days of PTO over your first year
9 paid company holidays
401(k) option with company match
Education assistance
Opportunity to participate in a company incentive plan
This is an incentive-based position, which may include bonuses, incentive or commission plans

Location

Boston, MA (hybrid)

Education

Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent related work or military experience).

View full job page →

1d ago

Machine Learning Engineer

Johns Hopkins Applied Physics Laboratory (APL) Laurel, MD

Laurel, MD

1d ago

$100k - $245k/yr Full time Ai Engineer Ai Ml Artificial Intelligence

Apply Now

Job Description

The Machine Learning Engineer role at Johns Hopkins Applied Physics Laboratory (APL) supports the full lifecycle of machine learning algorithm development, from design through evaluation and implementation. The work is focused on national defense non-kinetic systems and the development of advanced AI methods and software pipelines for real-world planning and decision-making.

Location

Laurel, MD (onsite)

Salary

USD 100,000 - 245,000 per yearly

Role Summary

As a Machine Learning Engineer, you will design, implement, and evaluate ML algorithms for challenging problems in planning, perception, coordination, and control. The position includes building and integrating intelligent decision-making capabilities, leveraging cutting-edge AI approaches and development pipelines.

Responsibilities

Design, implement, and evaluate advanced machine learning algorithms to address challenging real-world planning, perception, coordination, and control problems for national defense.
Develop software pipelines that connect data streams, simulation environments, and intelligent decision-making algorithms.
Apply technologies and concepts at the forefront of AI, including deep reinforcement learning, foundation models, large language models, convolutional/recurrent/graph neural networks, computer vision, and physics-based modeling and simulation tools.
Collaborate with scientists and engineers within the group and with partner teams across APL.
Engage directly with sponsors to communicate proposed concepts, solutions, and analysis.

Requirements

Have a Bachelor’s degree in Mathematics, Physics, Engineering, Computer Science, or a related field.
Have at least 2+ years of experience in machine learning and data science fields.
Have at least one year of hands-on experience applying or developing machine learning algorithms using common libraries such as PyTorch or TensorFlow.
Have strong foundational knowledge in at least two of the following: classification, clustering, deep learning, reinforcement learning, computer vision (object detection and visual tracking), multi-agent systems, or optimization/control theory.
Demonstrated experience working with version control software such as Git.
Strong communication skills, both verbal and written.
Be able to obtain an Interim Secret security clearance by your start date and ultimately obtain a Secret level clearance. If selected, you will be subject to a government security clearance investigation and must meet requirements for access to classified information, including U.S. citizenship.

Technologies

PyTorch
TensorFlow
Git
Deep reinforcement learning
Foundation models
Large language models
Convolutional neural networks
Recurrent neural networks
Graph neural networks
Computer vision
Physics-based modeling and simulation tools

Preferred Qualifications

You will go above and beyond minimum requirements if you have one or more of the following:

An MS in Mathematics, Physics, Engineering, Computer Science, or a related field.
5+ years of experience designing and implementing AI/ML algorithms for a variety of datasets.
Proven experience applying state-of-the-art deep learning techniques to solve distributed resource allocation problems.
Hands-on experience building computer vision pipelines for detection, tracking, segmentation, or multi-modal sensor fusion.
Experience with modeling and simulation platforms such as AFSIM, Blender, Unity, or Unreal.
Comfort working in high performance computing environments (GPU/CPU clusters).
Proficiency in one or more technology areas including multi-agent reinforcement learning, geometric deep learning, multi-modal sensor fusion, or agentic AI.
A track record of writing deployable, production-level code in Python and/or C/C++ for real-world applications.

View full job page →

4d ago

Senior Machine Learning Engineer

Motion Recruitment Boston, MA

Boston, MA

4d ago

Full time Senior Artificial Intelligence Automation

Apply Now

Job Description

Motion Recruitment is seeking a Senior Machine Learning Engineer to own the end-to-end machine learning lifecycle for medical-device, robotics, and automation applications. This role focuses on taking sensor data from collection through modeling, then delivering deployable solutions that run on constrained hardware, with close collaboration across embedded/firmware, software, validation, and documentation needs.

What you’ll work on

Develop and troubleshoot workflows for collecting, cleaning, and organizing sensor data
Build and refine ML models for real-time device applications and performance improvements
Partner with firmware teams to embed and test AI features on hardware platforms
Set up and oversee tools for tracking experiments, automating evaluations, and managing deployments
Analyze model behavior, maintain reliability, and resolve issues to keep outputs high quality

Requirements

5+ years of experience
Strong proficiency in Python
Hands-on experience with PyTorch or TensorFlow
Experience deploying models to edge using TFLite, ONNX, CoreML, TensorRT, or equivalent
Experience building sensor data pipelines
Proficiency with MLOps
Solid software engineering fundamentals
Proficiency in C or C++

Nice to have

Experience with physiological signal processing for medical or wearable applications
Background in robotics or autonomous systems
Experience in a startup or small team
Degree in a relevant field

Tools you’ll use

Python, PyTorch, TensorFlow
TFLite, ONNX, CoreML, TensorRT
MLOps, C, C++

Benefits

Bonus OR Commission eligible
Medical Insurance
Dental Benefits
Vision Benefits
Paid Time Off (PTO)
401(k) including match (if applicable)

Location: Boston, MA 02116 (onsite)

Daily responsibility: 100% hands on

View full job page →

5d ago

Machine Learning Engineer

UST Portland, OR

Portland, OR

5d ago

$82k - $123k/yr Full time Agent Orchestration Agentic Ai Agentic Ai Orchestration

Apply Now

Job Description

UST is looking for a Machine Learning Engineer (ML Engineer II) to help build and optimize multi-agent workloads end-to-end, from application behavior down to inference runtime and accelerator hardware. This onsite role in Oregon supports engineers who want to translate agentic AI into measurable performance improvements across latency, throughput, and resource utilization.

What you’ll do

Develop a multi-agent workload at the application level, then profile and optimize execution all the way down to inference runtime, OMIX/middleware, compute runtime, Linux GPU driver, and accelerator hardware.
Optimize for latency, throughput, tokens/sec, memory footprint, and accelerator utilization.
Select and tune models and inference engines based on agent workload characteristics, hardware capabilities, and performance requirements.

Core qualifications

Hands-on development of multi-agent workloads.
Experience with agent frameworks including LangGraph/LangChain, AutoGen, CrewAI, or similar.
Strong understanding of LLMs, SLMs, and multimodal models, including Transformer architecture, attention mechanisms, tokenization, context windows, and KV cache.
Hands-on experience with model selection, evaluation, and deployment for different agentic workload requirements.
Understanding of model formats and optimization, including ONNX, Open VINO IR, safe tensors, and quantization such as FP16, BF16, INT8, and INT4.
Experience with inference engines/runtime frameworks such as Open VINO, ONNX Runtime, vLLM, llama.cpp, or TGI.
Understanding of prefill vs. decode, batching, continuous batching, speculative decoding, KV-cache management, and memory optimization.
Working knowledge of CPU/GPU execution, device placement, and heterogeneous inference.
Ability to benchmark and compare models, inference engines, and runtime configurations to identify performance bottlenecks.
Understanding of GPU memory, kernel execution, synchronization, device selection, and host/device data movement.
Familiarity with middleware and accelerator compute runtimes such as OMIX/OneAPI/SYCL, Level Zero, or OpenCL.
Strong Python and working knowledge of C/C++.
Git/GitHub, Linux shell, and debugging tools.
Practical use of GitHub Copilot for development, debugging, and code generation.
Docker/container fundamentals.
Ability to benchmark and profile AI workloads and analyze whether performance issues originate in the agent, model, inference engine, runtime, or driver.

Technologies you may work with

LangGraph, LangChain, AutoGen, CrewAI
Python, C/C++, Linux, Git, GitHub, GitHub Copilot, Docker
ONNX, Open VINO IR, safe tensors, FP16, BF16, INT8, INT4
Open VINO, ONNX Runtime, vLLM, llama.cpp, TGI
OMIX, OneAPI, SYCL, Level Zero, OpenCL

Compensation and benefits

Compensation range: USD 82,000 to 123,000 per year.

Full-time employees accrue minimum 10 days of paid vacation per year.
6 days of paid sick leave each year (pro-rated for new hires throughout the year).
10 paid holidays.
Eligible for paid bereavement leave and jury duty.
Eligible to participate in the 401(k) Retirement Plan with employer matching.
Medical, dental, and vision insurance for employees and their dependents residing in the US.
Company-paid Employee Only benefits: basic life insurance, accidental death and disability insurance, and short- and long-term disability benefits.
Regular employees may purchase additional voluntary short-term disability benefits.
Eligible to participate in a Health Savings Account (HSA).
Eligible to participate in a Flexible Spending Account (FSA) for healthcare, dependent child care, and/or commuting expenses.

View full job page →

Machine Learning Evaluation Developer Jobs

Evaluation & Insights Machine Learning Engineer

Machine Learning Engineer

Machine Learning Engineer

Senior Machine Learning Engineer

Machine Learning Engineer

Get Machine Learning Evaluation Developer Jobs alerts