Gen Ai Evaluation Developer Jobs

1 job found

Today

Machine Learning Engineer, Evaluation

Apple Cupertino, CA

Cupertino, CA

Today

$185k - $325k/yr Full time Artificial Intelligence Data Analysis Data Analytics

Apply Now

Job Description

Build and lead the evaluation work that helps shape global generative AI products. This role focuses on operationalizing safety and risk assessments into measurable criteria, then turning those results into reliable evidence for product ship or no-ship decisions. You will lead a team and stay hands-on with evaluation findings to improve feature safety, fairness, and user trust at scale.

What you’ll do

Provide critical technical direction to translate policy and risk assessments into clear, measurable evaluation criteria for safety and fairness.
Develop metrics to evaluate safety and fairness risks in generative models and Gen-AI features.
Design datasets, identify data needs, and expand coverage using human and synthetic generation methods.
Create sampling strategies and combine human annotation with auto-grading to generate high-quality insights efficiently at large scale.
Use and implement data pipelines and collaborate cross-functionally to run end-to-end safety evaluations.
Drive operational excellence by automating manual workflows, optimizing evaluation velocity, and scaling infrastructure for rapid product release cycles.
Distill evaluation outcomes into actionable insights for product engineering and for safety policy development.
Work with highly sensitive content, with exposure to offensive and controversial material.
Remain hands-on analyzing evaluation results, conducting failure investigations, and leveraging human and automated feedback to continuously improve feature safety.

Required experience and qualifications

Experience in Responsible AI.
Curiosity about fairness and bias in generative AI systems and a strong interest in making technology more equitable.
A Social Science or Humanities background, or other demonstrable interest in socio-technical issues.
Prior scientific research and publication experience.
Proven ability to contribute to diverse, collaborative teams.
Experience with multimodal models, agentic systems, and/or diverse languages and locales.
Experience with compliance and regulatory needs in AI, specifically related to child safety, clinical/health applications, or companionship chatbots.
Experience conducting large-scale benchmarking across models and capabilities.
Education: MS or PhD in Computer Science, Linguistics, Cognitive Science, HCI, Psychology, Mathematics, Physics, or a similar science or technology field, with a strong basis in scientific data collection and analysis, plus at least 4 years of relevant work experience (or BA/BS with 8+ years relevant experience and 2+ years directly managing engineering teams).
Experience collecting and analyzing varied language, image, voice, video, and/or multimodal data, including both human- and LLM-generated data.
Experience with LLM-Judges and human-in-the-loop methods, including designing human annotation projects and writing annotation guidelines for multi-labeled, nuanced, and sometimes conflicting data.
Proficiency in data science, machine learning, and analytics; programming with Python and Pandas; experience with at least one plotting and visualization library.
Strong cross-functional collaboration skills to prioritize competing needs, manage milestones, and communicate with stakeholders.
Rigorous approach to metric development, interpretation of evaluations, and presenting results to executives.
Comfort leading a team that works with sensitive or potentially offensive content.

Tools and technologies

Python
Pandas

Location, salary, and work model

Cupertino, CA (onsite). Salary range: USD 184,700 - 324,800 per year. Base pay is determined within the range based on skills, qualifications, experience, and location.

Benefits

Comprehensive medical and dental coverage
Retirement benefits
Range of discounted products and free services
Reimbursement for certain educational expenses, including tuition
Discretionary bonuses or commission payments as well as relocation (if eligible)
Apple shareholder opportunity through participation in Apple’s discretionary employee stock programs
Eligible for discretionary restricted stock unit awards
Ability to purchase Apple stock at a discount through voluntary participation in the Employee Stock Purchase Plan

Note: Apple benefit, compensation, and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

View full job page →

Get Gen Ai Evaluation Developer Jobs alerts

New Gen Ai Evaluation Developer Jobs, daily.