Palo Alto Networks is seeking a Senior Principal Machine Learning Engineer to shape architectural strategy and deliver advanced AI capabilities within cloud security systems. This onsite role in Santa Clara, CA supports multi-year engineering roadmaps and Security-as-a-Service innovation with an emphasis on production-ready design.
Responsibilities
- Act as the organization’s primary technical compass by assessing frontline AI research, distinguishing production-viable approaches from short-lived hype, and directing engineering investment toward high-leverage architectural paths.
- Strengthen engineering rigor by establishing RFC standards, leading architectural review boards (ARBs), and mentoring Staff and Principal engineers to raise technical quality across teams.
- Translate ambiguous business goals and complex threat landscapes by partnering with Product Management, Security Researchers, Data Science, and Infrastructure to produce concrete technical requirements, milestone architectures, and executable designs.
- Own macro- and micro-level architecture for highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale.
- Design production-grade agentic frameworks with dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails.
- Define ingestion, preprocessing, and feature engineering pipeline architecture using Apache Spark, SQL, and BigQuery across Google Cloud platforms including Dataproc, GKE, and GCP SDKs.
- Maintain hands-on expertise across evolving model approaches, guiding architectural choices across Large Language Models (LLMs), Small Language Models (SLMs), and multimodal systems to optimize cost, latency, privacy, and domain effectiveness.
- Build systematic methods to diagnose underperforming production models, including drift, context dilution, alignment collapse, and hallucination spikes, and lead remediation through fine-tuning approaches such as SFT, DPO, RL, and LoRA/PEFT.
- Troubleshoot low-level compute bottlenecks on distributed GPU clusters, including distributed training and inference issues, CUDA out-of-memory (OOM) states, NCCL communication overhead, and kernel performance stalls.
Requirements
- 10+ years of progressive software engineering and systems design experience, with a record of technical leadership delivering high-availability, enterprise-scale platforms.
- Proven ability to author complex system designs from scratch, define data contracts, manage architectural trade-offs, and guide legacy-to-modern system migrations at scale.
- Demonstrated experience aligning cross-functional teams across Product, Data, Applied Research, SecOps, and Infrastructure to deliver unified technical visions and resolve competing priorities.
- Deep hands-on knowledge of modern AI architectures including Transformers, attention mechanisms, and tokenization strategies, with practical experience deploying and serving LLMs and SLMs.
- Expert capability in identifying causes of model degradation and executing targeted recovery pipelines involving data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning.
- Practical proficiency profiling and debugging distributed GPU workloads, including memory leak analysis, concurrency locks, and cluster communication bottlenecks.
- Advanced proficiency in Python for AI/ML pipelines and scientific computing.
- Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
- Strong foundation in data engineering with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration using Dataproc and GKE.
- BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.
Technology Stack
Python, Go, Java, Apache Spark, SQL, BigQuery, Dataproc, GKE, GCP SDKs, Large Language Models (LLMs), Small Language Models (SLMs), multimodal systems, Transformers, attention mechanisms, tokenization strategies, SFT, DPO, RL, LoRA, PEFT, CUDA, NCCL, agentic frameworks, supervised fine-tuning, preference alignment, parameter-efficient tuning.
Preferred Qualifications
- Deep background in threat modeling, endpoint detection and response (EDR), SIEM architectures, or cloud security posture management.
- Working knowledge of L3–L7 network protocols, packet telemetry, and deep packet inspection systems.
- Contributions to open-source foundation model tooling, published research, or active participation in the broader AI systems community.
Location and Work Model
- Location: Santa Clara, CA, United States of America
- Work mode: Onsite
Compensation
- Compensation offered depends on qualifications, experience, and work location. Starting base salary (non-sales roles) or base salary plus commission target (sales/commissioned roles) expected at $190,400.00 to $308,000.00 per year.
- Offered compensation may also include restricted stock units and a bonus.
Immigration Sponsorship
Yes. This role is eligible for immigration sponsorship.