Machine learning engineer II, AWS Inference Optimization

As a Machine Learning Engineer II on an AWS inference optimization team, you will help improve inference performance for large language OSS models such as Qwen, Gemma, gptoss, and others. The work centers on inference optimization techniques and owning price-performance for Amazon LLM inference across SageMaker, Hyperpod, and Bedrock.

What You’ll Do

• Develop model servers to support optimized LLM inference workflows.
• Build CUDA kernels in vLLM and SGLang.
• Create services for GenAI optimization and benchmarking.
• Create and modify a mid-size application or service.
• Optimize large language OSS models by working on inference stacks such as vLLM and SGLang.
• Perform performance tuning for GenAI inference on Amazon SageMaker, Amazon Bedrock, and Hyperpod.
• Implement inference optimization techniques including Speculative decoding, Disaggregated Inference, and CUDA kernel tuning.
• Own the price-performance of Amazon LLM inference across SageMaker, Hyperpod, and Bedrock.
• Deliver solutions that leverage existing academic and industrial research or apply pragmatic problem-solving to improve inference systems.
• Bring solutions into production in customer-facing products.

Required Qualifications

• 3+ years of non-internship professional software development experience.
• 2+ years of non-internship design or architecture experience for new and existing systems, including design patterns, reliability, and scaling.
• 1+ years of software development engineer or related occupational experience.
• 1+ years designing and developing large-scale, multi-tiered, multi-threaded, embedded, or distributed software applications, tools, systems, and services using experience with C#, C++, Java, or Perl.
• Bachelor’s degree (or foreign equivalent) in Computer Science, Engineering, Mathematics, or a related field.
• Knowledge of Machine Learning and LLM fundamentals, including transformer architecture, training/inference lifecycles, and optimization techniques.

Technologies You’ll Work With

• vLLM, SGLang
• Amazon SageMaker, Amazon Bedrock, Hyperpod
• CUDA
• C#, C++, Java, Perl
• Transformer architecture

Preferred Qualifications

• 3+ years of full software development life cycle experience, including coding standards, code reviews, source control management, build processes, testing, and operations.
• Bachelor’s degree in computer science or equivalent.

Compensation and Location

Location: Santa Clara, CA (onsite).
Salary: USD 165,200 - 223,600 per year.

Benefits

• Sign-on payments
• Restricted stock units (RSUs)
• Health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance, option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• Paid time off
• Parental leave
• Workplace accommodation or adjustment during the application and hiring process (including support for the interview or onboarding process)

Los Angeles County Applicants

Job duties include:

• Work safely and cooperatively with other employees, supervisors, and staff.
• Adhere to standards of excellence despite stressful conditions.
• Communicate effectively and respectfully with employees, supervisors, and staff to ensure exceptional customer service.
• Follow all federal, state, and local laws and Company policies.
• Criminal history may have a direct, adverse, and negative relationship with some of the material job duties, including the abilities to adhere to company policies, exercise sound judgment, effectively manage stress, work safely and respectfully with others, exhibit trustworthiness and professionalism, and safeguard business operations and the Company’s reputation.
• Pursuant to the Los Angeles County Fair Chance Ordinance, qualified applicants with arrest and conviction records will be considered.