Tech Lead Machine Learning Engineer - LLM Evaluation & Agent Systems

Join TikTok USDS JV in Seattle for an onsite technical leadership role building AI systems that improve safety, integrity, and compliance across the U.S. data ecosystem. You will help shape an AI Risk & Compliance Intelligence Platform and an Intelligent Data Agent Ecosystem, with a strong emphasis on LLM evaluation, synthetic data generation, and modular agent skill optimization for Data, Engineering, and SRE workflows.

This position offers day one medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short-term and long-term disability, and life insurance. You will also have wellbeing benefits, 10 paid holidays per year, 10 paid sick days per year, and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).

Responsibilities

• Design, build, and scale robust LLM evaluation architecture, including automated offline datasets, metric suites, and benchmarking pipelines to measure prompt and model performance.
• Develop LLM-driven synthetic data generation approaches to address data scarcity bottlenecks and evaluate long-tail and edge-case risk scenarios.
• Move beyond criteria-based prompt engineering by extracting deep risk patterns from complex case data to create native, ML-driven decision models and reasoning abstractions.
• Architect, evaluate, and continuously optimize modular agent skills and tools for Data Science, Analytics, Data Engineering, and SRE workflows to improve tool-use accuracy, execution efficiency, and task success rates.
• Apply advanced evaluation methods such as LLM-as-a-judge and multi-agent benchmarking across both risk intelligence and agentic ecosystems.
• Provide technical leadership by defining core ML system design patterns, benchmarking standards, and engineering best practices, translating complex operational requirements into scalable ML solutions with cross-functional partners.

Requirements

• 5+ years of software/ML engineering experience with a strong record designing, building, and deploying ML/LLM systems in production.
• Deep hands-on expertise in LLM architectures, fine-tuning, RAG, prompt tuning, and agentic frameworks.
• Experience building robust LLM evaluation frameworks (including LLM-as-a-judge and automated benchmark pipelines) and addressing data scarcity or imbalance.
• Strong coding skills with production experience in scalable data processing pipelines.
• Domain experience in Risk & Compliance Intelligence and Automated Decision Systems.
• Hands-on experience with synthetic dataset generation or bootstrapping models for cold-start scenarios.
• Experience building AI tools or agents for technical personas such as Data Analysts, Data Scientists, Data Engineers, and Data SRE.
• Demonstrated ability to serve as a founding technical lead on ambiguous, high-impact initiative pods.

About the Team

The TikTok U.S. Data Security (USDS) team is responsible for the security, integrity, and compliance of TikTok’s data within the United States. The Data Platform Team builds scalable, reliable, cost-efficient infrastructure that supports data integrity and enables self-service tools and scientific insights across business units.

About USDS

TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on TikTok apps. The role requires onsite presence across teams to support speed, alignment, and agility, and the organization is shifting from a hybrid work model to a fully in-person schedule of up to 5 days a week.

Accommodation

USDS is committed to providing reasonable accommodations in recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs, or other reasons protected by applicable laws.

Location: Seattle, WA (onsite)
Compensation: USD 198,360 - 416,100 per year
Minimum Experience: 5 years