Apple Cloud AI Platform is seeking a Senior / Staff Machine Learning Engineer to lead large-scale model training efforts and help set technical direction across teams. The role focuses on building training recipes and scalable infrastructure while addressing stability, performance, and post-training needs at scale.
What You’ll Do
- Partner with researchers and ML engineers to develop ML models and translate their requirements into training features and performance improvements that can be adopted across the platform.
- Design model distillation and compression approaches, including teacher-student distillation and quantization-aware training, to improve quality for smaller models.
- Train models that support long-context extension, mixture-of-experts architectures, and multimodal inputs, addressing stability and efficiency challenges introduced by each setup.
- Investigate large-scale training behaviors such as loss spikes, divergence, and throughput drops, identifying root causes across data, numerics, code, and hardware.
- Build and take ownership of scalable training frameworks and services that run efficiently on GPUs and TPUs, using PyTorch or JAX and higher-level frameworks.
- Create reinforcement learning infrastructure for post-training, including rollout and inference services, reward pipelines, and secure sandboxes for tool use and code execution.
- Integrate the training platform with Apple’s data and governance stack to support privacy and security requirements for training runs.
- Set technical direction across teams, mentor engineers, and raise the engineering bar.
Minimum Qualifications
- 8+ years of experience in large-scale model training, including reinforcement learning, from research to production.
- 5+ years of experience building and operating large-scale model training systems in production.
- Experience with fine-tuning and distillation techniques, including supervised fine-tuning, parameter-efficient fine-tuning with LoRA, and teacher-student distillation.
- Hands-on experience with at least one higher-level training framework such as Megatron-LM, DeepSpeed, TorchTitan, NeMo, AXLearn, or MaxText.
- Deep expertise in PyTorch or JAX, including distributed training features.
- Strong programming skills in Python or C++.
- Strong collaboration and written and verbal communication skills, with a track record of leading technical work across teams.
- BS, MS, or PhD degree in Computer Science, Computer Engineering, or equivalent practical experience.
Preferred Qualifications
- Experience with low-precision training and the numerical issues it brings.
- Experience training large models with data, tensor, pipeline, or expert parallelism across hundreds to thousands of GPUs or TPUs.
- Strong understanding of GPU and TPU architecture, including memory hierarchy, matrix units, and high-bandwidth memory, and how they affect training performance.
- Experience with collective communication libraries such as NCCL and high-speed interconnects such as NVLink, InfiniBand, and TPU ICI.
- Experience profiling and optimizing training workloads using tools such as Nsight Systems, PyTorch Profiler, or XProf.
- Experience building reinforcement learning or post-training systems for large models.
- Open source contributions to training frameworks and/or publications at top ML venues such as NeurIPS, ICML, or ICLR.
Core Skills and Technologies
- PyTorch, JAX
- Distributed training features
- Parameter-efficient fine-tuning (LoRA), supervised fine-tuning
- Teacher-student distillation, quantization-aware training
- Megatron-LM, DeepSpeed, TorchTitan, NeMo, AXLearn, MaxText
- NCCL, NVLink, InfiniBand, TPU ICI
- Nsight Systems, PyTorch Profiler, XProf
- Python, C++
Compensation and Benefits
Base pay range for this role is USD 175,000 - 308,500 per year, depending on skills, qualifications, experience, and location. Apple notes that base pay is one part of a total compensation package and provides additional opportunities for growth within the role.
- Comprehensive medical and dental coverage
- Retirement benefits
- Range of discounted products and free services
- Reimbursement for certain educational expenses, including tuition for formal education related to advancing your career
- Participation in Apple’s discretionary employee stock programs, including discretionary restricted stock unit awards
- Employee Stock Purchase Plan to purchase Apple stock at a discount if voluntarily participating
- Discretionary bonuses or commission payments, as applicable
- Relocation, as applicable
Note: Apple benefit, compensation, and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
Location
Seattle, WA (onsite)
Minimum Education
BS, MS, or PhD degree in Computer Science, Computer Engineering, or equivalent practical experience.
