Llm Evaluation

Llm Evaluation Developer Jobs

3 jobs found

11d ago

Senior Software Engineer

AikiTech, Inc. Scottsdale, AZ

Scottsdale, AZ

11d ago

Full time Senior Ai Guardrails APIs

Apply Now

Job Description

Senior Software Engineer role at AikiTech building and operating LLM-powered applications and production systems for public agencies.

Responsibilities

Build and operate LLM-powered applications on public-cloud platforms, including government-approved environments
Own prompt design, model evaluation, retrieval, guardrails, and production observability
Evaluate additional commercial or open-weight models as appropriate
Design scalable, secure backend services and data pipelines using Python
Develop versioned evaluation and regression-testing frameworks to measure accuracy, latency, and failure modes before releases reach customers
Work on real-time and near-real-time audio pipelines, including telephony integrations, streaming transcription, call metadata, and latency-sensitive flows
Extend the core GenAI platform with configurable capabilities and integrations aligned to individual agencies’ workflows and data environments
Design and operate infrastructure in AWS GovCloud and Microsoft Azure Government, including identity and access management, networking, encryption, secrets management, deployment automation, infrastructure as code, logging, and monitoring
Embed security and privacy across the development lifecycle using least-privilege access, encryption, audit logging, data-retention controls, vulnerability management, testing, and documented change control
Own engineering controls and supporting evidence for SOC 2 work and CJIS-alignment assessments, including remediation of technical findings
Lead design and code reviews, mentor less-experienced engineers, and produce clear technical documentation for engineers, auditors, customers, and agency stakeholders

Requirements

6+ years of professional software engineering experience with significant ownership of production systems or major production components
Strong Python skills and experience building production backend services, APIs, and data-processing systems
Hands-on experience building LLM API-based applications, including retrieval, embeddings, evaluation, structured outputs, and/or model observability
Production experience with AWS, Microsoft Azure, or another major public-cloud platform, including several of: identity and access management, networking, containers or serverless, infrastructure as code, monitoring, and CI/CD
Experience designing for security, reliability, and maintainability, not only getting prototypes working
Strong debugging and problem-solving skills across application, infrastructure, and integration layers
Ability to communicate clearly with engineers, nontechnical stakeholders, customers, and external assessors
Bachelor’s degree in Computer Science (or higher) required

Technologies

Python
LLM APIs
Retrieval
Embeddings
Structured outputs
AWS
Microsoft Azure
AWS GovCloud
Microsoft Azure Government
Identity and access management
Networking
Containers
Serverless computing
Infrastructure as code
Monitoring
CI/CD
Encryption
Secrets management
Audit logging
SOC 2
CJIS
CIJIS-aligned security program

Benefits

Competitive salary based on experience and qualifications
Participation in AikiTech’s stock compensation plan
Comprehensive health benefits
Paid time off
Access to a 401(k) plan
401(k)
Dental insurance
Health insurance
Vision insurance

Preferred Experience

Building production software for, or alongside, government agencies or public-sector vendors
Supporting customer deployments involving security reviews, technical integrations, or regulated data
Public-safety technology experience such as 911 systems, PSAP operations, CAD, NG911, emergency communications, or telephony
Experience in SOC 2, CJIS-aligned, FedRAMP, StateRAMP, or similarly regulated environments, including implementing engineering controls and responding to assessment findings
AWS GovCloud, Microsoft Azure Government, or other government-cloud environments
Real-time audio processing, VoIP or SIP, streaming transcription, or event-driven architectures
FERPA, COPPA, student-data privacy, or education technology experience
Firsthand professional experience in public safety, emergency communications, or military environments

Work Authorization and Personnel Requirements

Applicants must be legally authorized to work in the United States at the time of hire
AikiTech is unable to provide or assume employment-visa sponsorship for this position

Location

On-site in Scottsdale, Arizona, where the engineering team and integration environment are based
Work closely with people building, deploying, and supporting the platform to support critical technology for public agencies outside traditional coastal technology hubs

View full job page →

1mo ago

Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Apple Seattle, WA

Seattle, WA

1mo ago Closed

$142k - $263k/yr Full time Adversarial Ai Evaluation Ai Evaluation Artificial Intelligence

This position is no longer accepting applications (closed Sep 29, 2026).

Job Description

Join Apple’s Human-Centered AI team to build evaluation and insight systems that help foundation models and generative AI deliver reliable, safe, and human-aligned experiences. This role focuses on bridging human perception with model performance through robust evaluation frameworks, scalable MLOps pipelines, and automated analysis that turns findings into actionable guidance for engineering and training.

Location: Seattle, WA (onsite)

Compensation: USD 142,300 - 263,300 per yearly

What you’ll do

Architect and execute comprehensive evaluation suites for LLMs and multimodal models, surfacing edge cases across multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
Build deterministic, heuristic, and LLM-assisted evaluation frameworks (including LLM-as-a-judge and reward modeling) to quantify human-perceived quality metrics such as helpfulness and hallucination rates.
Convert qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for model training and inference.
Partner with engineering teams to improve model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning.
Apply advanced ML methods (including embedding-based clustering, representation learning, and perturbation analysis) to map error taxonomies and latent failure manifolds in model outputs.
Develop robust MLOps workflows that codify evaluation metrics, automate regression testing across model checkpoints, and integrate human-centric assessments into ML CI/CD pipelines.
Design scalable, distributed inference and processing pipelines (such as Ray and vLLM) to run high-throughput evaluations, automated annotation, and output analysis at scale.
Define quantitative evaluation frameworks that capture nuanced human factors, including trust calibration, conversational state tracking, and interpretability.
Build automated evaluation pipelines using LLMs to assess outputs at scale, optimizing for high correlation with human baseline annotations.
Collaborate across Apple teams, including software engineering, product, research, and responsible AI, to translate product needs into scalable and efficient evaluation infrastructure.

Requirements

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
5+ years of relevant industry experience in ML Engineering or Applied Research.
Advanced proficiency in Python and modern deep learning ecosystems (PyTorch, JAX, Hugging Face).
Proven experience building scalable ML inference pipelines, model evaluation workflows, and structured rating frameworks for large-scale AI systems.
Strong ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives.
Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
Deep familiarity with AI quality metrics, hallucination detection techniques (including SelfCheckGPT), model alignment approaches (RLHF/DPO), and LLM-as-a-judge frameworks (such as G-Eval and DeepEval).
Experience building internal tools or automated pipelines for ML workflows using platforms like MLflow and Weights & Biases (or similar).
Strong familiarity with advanced prompt engineering, RAG architectures (vector databases, semantic search), and fine-tuning.
Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.

Technologies

Python, PyTorch, JAX, Hugging Face, Ray, vLLM, MLflow, Weights & Biases, SelfCheckGPT, RLHF, DPO, G-Eval, DeepEval, LLM-as-a-judge, Retrieval-Augmented Generation (RAG), vector databases, semantic search

Benefits

Comprehensive medical and dental coverage.
Retirement benefits.
Range of discounted products and free services.
Reimbursement for certain educational expenses (including tuition) for formal education related to advancing your career.
Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs.
Eligible for discretionary restricted stock unit awards.
Can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan.
May be eligible for discretionary bonuses or commission payments and relocation.

Preferred qualifications

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.

View full job page →

1mo ago

Machine Learning Engineer, GAI Search Platform

ServiceNow Mountain View, CA

Mountain View, CA

1mo ago Closed

$140k - $217k/yr Full time Agentic Ai AI Ai Ml

This position is no longer accepting applications (closed Sep 11, 2026).

Job Description

Join ServiceNow’s GenAI Search Platform team to build machine learning capabilities for LLM-enabled search-based question answering and evaluation.

Responsibilities

Drive development of the search platform with LLM-enabled generative AI features in a fast-paced environment
Enable product teams to rapidly build new applications and features, including shipping prototypes to customers at scale
Evaluate and iterate on ranking improvements using a systematic approach
Enforce guarantees on quality, security, privacy, and scalability for enterprise-ready search and summarization
Design systematic evaluation metrics and methodology for search models
Use LLMs to address enterprise challenges, including security and privacy
Collaborate with ranking, product, design, infrastructure, and data science teams to advance agentic search capabilities with a focus on performance and enterprise readiness
Partner with product, user experience, customer success, and other engineering teams to identify, define, and build solutions

Requirements

3+ years of engineering experience
First-hand experience relevant to search platforms, such as:
- traditional information retrieval
- search ranking and relevance
- recommendation systems
- question answering systems
Background in one or more areas: text processing, information retrieval, natural language understanding, or machine learning
Interest in helping product and ranking teams move faster and scale solutions in an environment focused on customer impact and success

Technologies

LLMs (large language models)
Natural language understanding
Retrieval augmented generation (RAG)

Location and Work Persona

Mountain View, CA (onsite)
Work persona categories may be flexible, remote, or required in office
ServiceNow may confirm eligibility for a work persona by checking the distance between your primary residence and the closest ServiceNow office using a third-party service

Compensation

USD 139,700 - 216,500 per year base pay
Plus equity (when applicable), variable/incentive compensation, and benefits

Benefits

Health plans, including flexible spending accounts
401(k) plan with company match
ESPP
Matching donations
Flexible time away plan
Family leave programs

Nice to Have

Experience with (or desire to learn) large language models, retrieval augmented generation (RAG), and agents
Experience with Python; contributing to mature Python libraries or platform code, including open-source repositories
Experience with Golang is not required

View full job page →

Llm Evaluation Developer Jobs

Senior Software Engineer

Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Machine Learning Engineer, GAI Search Platform

Get Llm Evaluation Developer Jobs alerts