Principal Machine Learning Engineer (Advanced Threat Protection)
Job Description
Palo Alto Networks is seeking a Principal Machine Learning Engineer to provide technical leadership for next-generation, AI-driven cloud security solutions. This onsite role in Santa Clara, CA focuses on security telemetry and Security-as-a-Service (SECaaS), spanning the full path from prototyping to enterprise-grade production systems.
Responsibilities
- Lead the design and implementation of ML and GenAI-powered agents with dynamic planning, multi-step execution, self-correction, and collaboration across multi-agent setups and human-in-the-loop workflows.
- Build, evaluate, and fine-tune LLMs and SLMs, including multimodal architectures.
- Investigate model regressions and sub-par performance by identifying root causes such as data distribution shift, alignment failures, and loss anomalies, then apply targeted fine-tuning approaches including SFT, DPO/RLHF, and PEFT/LoRA.
- Profile, debug, and optimize model training and inference on GPU clusters, addressing memory allocation issues (OOMs), distributed communication overhead, and compute utilization.
- Architect scalable data preprocessing and feature extraction pipelines to support real-time endpoint telemetry feeding agentic and analytical workflows.
- Own end-to-end engineering from prototype to enterprise production, defining best practices for prompt architecture, agent memory persistence, context window optimization, tool-use execution, and safety guardrails.
- Continuously evaluate state-of-the-art agent frameworks, model serving engines, and accelerator architectures to guide long-term technical roadmaps.
Requirements
- 7+ years of professional software engineering with a history of deploying production-grade ML, GenAI, or agentic architectures at enterprise scale.
- Advanced Python proficiency for AI/ML development, plus production-level experience with Go or Java for building distributed microservices.
- Strong hands-on experience building and scaling data pipelines.
- Practical experience with Apache Spark, core data tools, SQL, BigQuery schema design, and Google Cloud (including Dataproc, GKE, and GCP SDKs).
- Operational experience with both LLMs and SLMs.
- Deep expertise in diagnosing model performance issues and executing remedial fine-tuning strategies, including dataset curation, parameter-efficient fine-tuning, loss monitoring, alignment, and evaluation benchmarks.
- Practical GPU profiling and debugging experience, such as CUDA memory optimization, multi-GPU synchronization, batch throughput tuning, and handling GPU compute crashes or stalls.
- Strong understanding of Retrieval-Augmented Generation (RAG), vector storage, prompt optimization, model evaluation metrics, and applied statistical modeling.
- BS or MS in Computer Science, Data Science, Software Engineering, or a related technical discipline.
Technologies
- Python, Go, Java
- Apache Spark, SQL, BigQuery
- Google Cloud, Dataproc, GKE, GCP SDKs
- LLMs, SLMs, Retrieval-Augmented Generation (RAG), vector storage
- SFT, DPO/RLHF, PEFT/LoRA
- CUDA, GPU clusters, multi-GPU synchronization
Preferred Qualifications
- Background in cybersecurity, threat detection, or cloud security principles.
- Working knowledge of L3–L7 networking protocols, packet inspection, and network security topologies.
- Experience handling high-volume cybersecurity telemetry, endpoint detection data, or real-time threat intelligence feeds.
Compensation Disclosure
The compensation offered depends on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the expected annual starting base salary range is $163,200.00 - $264,000.00/yr (non-sales roles: starting base salary; sales/commissioned roles: base salary plus commission target). The offer may also include restricted stock units and a bonus.
Equal Opportunity and Accommodations
- Palo Alto Networks is committed to providing reasonable accommodations for all qualified individuals with a disability.
- Palo Alto Networks is an equal opportunity employer.
Similar Jobs
F