Gpu Inference

Gpu Inference Developer Jobs

3 jobs found

2d ago

Senior Machine Learning Engineer

Cloudflare New York, NY

New York, NY

2d ago

$194k - $266k/yr Full time Artificial Intelligence Engineer Generative AI

Apply Now

Job Description

Design and productionize machine learning inference across Cloudflare’s global network, from frontier open LLMs and real-time speech models to customer-deployed inference on heterogeneous accelerators.

Responsibilities

Develop, optimize, and productionize machine learning models for Cloudflare’s serverless inference platform, prioritizing performance, reliability, and model quality.
Build benchmarking and evaluation frameworks to measure latency, throughput, and cost efficiency, plus model behavior across LLMs, speech, vision, and other model families.
Improve inference performance using techniques such as quantization, batching, caching, model compilation, and accelerator-aware runtime tuning.
Work with systems engineers to integrate models into Cloudflare’s distributed inference infrastructure across a heterogeneous fleet of GPUs and next-generation accelerators.
Advance model deployment workflows, including validation, rollout safety, observability, regression testing, and operational readiness.
Collaborate with product and engineering teams to translate customer requirements into scalable ML capabilities for Workers AI.
Mentor engineers, contribute to technical direction, and raise the quality bar for production ML practices.

Requirements

Experience building, optimizing, and operating machine learning models in production.
Strong proficiency in Python and modern ML frameworks such as PyTorch, TensorFlow, JAX, or equivalent.
Hands-on inference optimization for large-scale models, including quantization, batching, caching, compilation, and serving runtime tuning.
Experience with large-scale inference serving frameworks or runtimes such as SGLang, vLLM, TensorRT-LLM, ONNX Runtime, Triton, llama.cpp, or similar.
Familiarity with modern deep learning architectures including LLMs, speech models, vision models, embeddings, multimodal models, and retrieval-augmented generation.
Experience optimizing models for GPUs or specialized accelerators.
Understanding of production ML concerns: evaluation, monitoring, model regressions, rollout safety, and reliability.
Ability to work across ML and systems boundaries, with familiarity in distributed systems, networking, or serverless platforms.
Track record leading complex technical projects and mentoring engineers.

Technologies

Python
PyTorch
TensorFlow
JAX
SGLang
vLLM
TensorRT-LLM
ONNX Runtime
Triton
llama.cpp
Workers AI

Compensation

Compensation may be adjusted depending on work location.
Bay Area based hires: estimated annual salary $194,000 - $266,000.
New York City, New Jersey, Washington, Washington DC, and California (excluding Bay Area) based hires: estimated annual salary $185,000 - $254,000.

Equity

Eligible to participate in Cloudflare’s equity plan.

Benefits

Medical/Rx Insurance
Dental Insurance
Vision Insurance
Flexible Spending Accounts
Commuter Spending Accounts
Fertility & Family Forming Benefits
On-demand mental health support and Employee Assistance Program
Global Travel Medical Insurance
Short and Long Term Disability Insurance
Life & Accident Insurance
401(k) Retirement Savings Plan
Employee Stock Participation Plan
Flexible paid time off covering vacation and sick leave
Leave programs, including parental, pregnancy health, medical, and bereavement leave

Bonus Points

Experience contributing to open source ML tooling, model serving frameworks, or inference runtimes.

Location: New York, NY (hybrid)

Salary Range Listed: USD 194,000 - 266,000 per yearly

View full job page →

14d ago

Software Engineer, DL Kernel Libraries - New College Grad 2027

NVIDIA Santa Clara, CA

Santa Clara, CA

14d ago

$108k - $196k/yr Internship Cuda Deep Learning Gpu Inference

Apply Now

Job Description

NVIDIA is building software infrastructure to accelerate deep learning inference and training at scale, and the DL Kernel Libraries team plays a central role in that effort. As a Software Engineer Intern for the 2027 New College Grad program, you will help develop the systems software stack behind efficient AI workloads, including GPU kernel technologies, libraries, and supporting code generation.

This onsite role is based in Santa Clara, CA, with an estimated salary range of USD 108,000 - 195,500 per year.

What you’ll do

Innovate and develop new AI systems technologies focused on efficient inference and training
Design, implement, and optimize kernels for high impact AI workloads
Design and implement extensible abstractions for serving engines and training frameworks
Build efficient just-in-time domain specific compilers and runtimes
Collaborate closely with other NVIDIA engineers across deep learning frameworks, libraries, kernels, and GPU architecture teams
Contribute to open source communities including FlashInfer, cuDNN, vLLM, SGLang, PyTorch, JAX, and XLA

Requirements

BS, MS, or PhD degree in Computer Science, Electrical Engineering, or related field (or equivalent experience)
Strong experience developing or using deep learning frameworks (for example PyTorch, JAX, TensorFlow, ONNX) or inference engines and runtimes such as vLLM, SGLang, and MLC
Strong Python and C/C++ programming skills

Technologies you may work with

PyTorch, JAX, TensorFlow, ONNX
vLLM, SGLang, MLC
Python, C/C++, CUDA C/C++
FlashInfer, cuDNN
CUDA, CUTLASS DSL, cuTile, Triton
Flash Attention, Apache TVM, MLIR
XLA

Benefits

Eligible for equity and benefits

Ways to stand out

Background in domain specific compiler and library solutions for LLM inference and training (for example FlashInfer, Flash Attention)
Expertise in training frameworks and inference engines
Expertise in machine learning compilers (for example Apache TVM, MLIR)
Strong experience in GPU kernel development and performance optimization, including experience with CUDA C/C++, CUTLASS DSL, cuTile, Triton, or similar
Open source project ownership or contributions

View full job page →

1mo ago

Principal Software Engineer, Inference

Hewlett Packard Enterprise Spring, TX

Spring, TX

1mo ago Closed

$160k - $303k/yr Full time Backend Developer Artificial Intelligence Distributed Inference

This position is no longer accepting applications (closed Oct 18, 2026).

Job Description

Hewlett Packard Enterprise is building HPE AI Essentials, an inference platform designed to run large language models on customer-owned infrastructure, including air-gapped and sovereign environments. In this Principal Software Engineer role, you will lead the model runtime and the Kubernetes orchestration architecture that support efficient, low-latency, high-utilization inference.

This is a hybrid position based in Spring, TX, where you will help shape how requests are executed end-to-end, from runtime capabilities through distributed serving and production deployment patterns.

What you’ll do

Define and own the technical direction for LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
Collaborate with inference performance engineering teams, taking accountability for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency.
Design distributed inferencing strategies, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage.
Assess emerging approaches such as new runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and decide whether to adopt, develop in-house, or decline each option.
Define orchestration for the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling.
Mentor and lead engineers through design and architecture reviews, and present technical direction to business unit and executive audiences.

Qualifications

12+ years of relevant experience.
Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including the ability to modify engine internals.
Deep knowledge of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding.
Expertise in tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics.
Strong Kubernetes platform experience, including operators, custom resources, controllers, and scheduling.
Proficiency in Go and Python, plus the ability to read, debug, and profile C++/CUDA using tools such as Nsight.
Experience debugging and profiling multi-tier application workloads such as RAG and Agents.
Excellent analytical and problem-solving skills.

Tech you’ll work with

vLLM, SGLang, TensorRT-LLM, TGI, NVIDIA NIM
Go, Python, C++, CUDA, Nsight
Kubernetes, NCCL, RDMA, GPUDirect Storage
InfiniBand, RoCE, RAG, Agents

Education

A degree in Computer Science or a related field.

Benefits

HPE aims to offer a comprehensive set of benefits that supports physical, financial, and emotional wellbeing for team members and their loved ones.
Programs are available to help you work toward career goals, including pathways for becoming a knowledge expert or applying your skills to another division.
Work practices are described as unconditionally inclusive, with flexibility to manage work and personal needs, and recognition of individual uniqueness.

Preferred

Upstream contributions to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe.
Experience with disaggregated prefill/decode serving, or KV cache offload and reuse at scale.
Experience with RDMA, GPUDirect Storage, InfiniBand, or RoCE.
Experience with MIG, fractional GPU allocation, and multi-tenant GPU isolation.
Experience delivering on-premises, air-gapped, or regulated enterprise software.

Work location and salary

Hybrid setup with an expectation to work on average 2 days per week from an HPE office.
Primary work location: Spring, TX (US remote options are considered; other HPE US sites may apply).
Base salary range (United States):
Annual Salary USD 160,000 - 303,000 in Colorado.
Annual Salary USD 152,000 - 349,000 in North Carolina & Texas.
Variable incentives may be offered in addition to base salary.

View full job page →

Gpu Inference Developer Jobs

Senior Machine Learning Engineer

Software Engineer, DL Kernel Libraries - New College Grad 2027

Principal Software Engineer, Inference

Get Gpu Inference Developer Jobs alerts