As a Principal Machine Learning Engineer on the AI Canvas team, you will provide technical ownership of the AI layer behind a next-generation cybersecurity data exploration experience. The role focuses on building production-grade LLM, RAG, and agent systems with reliability and trust.

Location

Santa Clara, CA (onsite)

Salary

USD 163,200 - 264,000 per year

Compensation Disclosure

$163,200.00 - $264,000.00/yr
Compensation depends on qualifications, experience, and work location.
For candidates offered at the posted level (non-sales roles), starting base salary is expected to fall within the annual range listed.
The offered compensation may also include restricted stock units and a bonus.

Role Summary

You will lead architecture and development of the AI capabilities powering AI Canvas, from early design through production deployment and continuous improvement. Responsibilities include building scalable systems that support natural-language exploration of complex security data, including natural-language-to-query generation and follow-up interactions for investigation and troubleshooting.

Responsibilities

Provide technical leadership for architecture and development of AI capabilities for AI Canvas, from early design through production deployment and continuous improvement.
Design and build scalable, production-grade systems using LLMs, RAG, AI agents, assistants, tools, skills, and agent harnesses.
Architect workflows that enable exploration of complex security data through natural language, including natural-language-to-query generation, follow-up interactions, clarification, investigation, and troubleshooting.
Define and evolve architectures for agentic AI systems, including orchestration, context management, tool invocation, memory, reasoning workflows, and multi-step execution.
Build evaluation frameworks and harnesses to measure AI quality, including correctness, relevance, reliability, regression detection, and end-to-end product behavior.
Establish evaluation methodologies using automated metrics, LLM-based evaluators, human evaluation, and representative production datasets.
Design and implement guardrails and safety mechanisms to improve reliability, reduce hallucinations, enforce system constraints, and support responsible behavior.
Drive improvements in model and system quality through prompt engineering, retrieval strategies, model selection, fine-tuning where appropriate, and systematic experimentation.
Build scalable RAG and knowledge-retrieval systems that ground responses in large, complex, and evolving security datasets.
Partner with backend and platform engineers to build reliable APIs, services, and infrastructure for AI workloads at production scale.
Establish engineering best practices for observability, debugging, tracing, versioning, reproducibility, testing, and monitoring of LLM and agent systems.
Evaluate emerging models, frameworks, and AI infrastructure and determine when and how to incorporate them into the product.
Balance rapid experimentation with engineering rigor needed for mission-critical production AI systems.
Mentor engineers, influence technical direction across teams, and raise the engineering bar for applied AI development.

Required Qualifications

Bachelor’s degree in Computer Science, Machine Learning, Engineering, or a related technical field, or equivalent practical experience.
7+ years of software engineering, machine learning engineering, or related industry experience, including significant experience building production systems.
Strong experience designing and building machine learning or AI-powered applications at scale.
Hands-on experience building applications using large language models and generative AI technologies.
Experience with areas such as RAG, AI agents, AI assistants, tool-calling systems, agent orchestration, or LLM-based workflows.
Strong proficiency in Python and experience developing production-grade backend or ML services.
Experience designing evaluation systems for AI/ML applications, including offline evaluation, regression testing, quality measurement, and production monitoring.
Strong understanding of modern ML and AI concepts, including embeddings, retrieval, ranking, prompt engineering, model inference, and experimentation.
Experience building scalable systems on public cloud platforms such as GCP, AWS, or Azure.
Strong software engineering fundamentals, including system design, distributed systems, APIs, testing, CI/CD, and observability.
Ability to lead complex technical initiatives across multiple teams while remaining hands-on.
Strong communication skills and ability to translate ambiguous product problems into clear technical architectures and execution plans.

Technologies

LLMs
Retrieval-augmented generation (RAG)
AI agents
AI assistants
Natural-language-to-query generation
Python
GCP
AWS
Azure
Embeddings
Prompt engineering
CI/CD
Docker
Kubernetes
Vector databases
Natural-language-to-SQL
LLM-based evaluators

Preferred Qualifications

Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
Deep experience building and operating LLM-powered products or agentic AI platforms in production.
Experience with AI development frameworks and tooling for model orchestration, agent systems, evaluation, tracing, or observability.
Experience building natural-language-to-SQL or natural-language-to-query systems, semantic layers, or AI-powered data exploration products.
Experience designing LLM evaluation harnesses, synthetic test generation, LLM-as-a-judge approaches, or human-in-the-loop evaluation workflows.
Experience with vector databases, search and retrieval infrastructure, embeddings, and large-scale knowledge systems.
Experience with containerization and orchestration technologies such as Docker and Kubernetes.
Experience designing highly available, low-latency AI inference and backend services.
Familiarity with AI security, prompt injection defenses, data privacy, access control, and guardrails for enterprise AI applications.
Experience in cybersecurity, security analytics, observability, or large-scale data platforms.
Contributions to open-source AI, ML, agent, or infrastructure projects.

Immigration Sponsorship

Yes

Principal Machine Learning Engineer

Job Description