gpu mps

GPU Multi-Process Service (MPS) is the NVIDIA runtime service that enables multiple CUDA processes to share a single GPU concurrently with improved efficiency — replacing the default time-slicing behavior (where processes alternate GPU access) with true spatial sharing where multiple processes' kernels execute simultaneously on the same GPU, improving utilization for workloads like multi-rank MPI jobs, inference serving with multiple workers, and Kubernetes GPU sharing.

Why MPS

• Default GPU sharing: Time-slicing via context switching → only one process uses GPU at a time.
• Context switch cost: ~25-50 µs → each process gets exclusive GPU access for a time quantum.
• Problem: Small kernels from one process don't fill the GPU → 30-50% utilization waste.
• MPS: Funnel all processes through a single CUDA context → kernels from different processes run simultaneously.

How MPS Works

Without MPS (time-slicing):
 Process A: [kernel][idle ][kernel][idle ]
 Process B: [idle ][kernel][idle ][kernel]
 GPU:       [  A  ][  B  ][  A  ][  B  ]  ← context switches

 With MPS:
 Process A: [kernel][kernel][kernel]
 Process B: [kernel][kernel][kernel]
 GPU:       [ A+B  ][ A+B ][ A+B ]  ← concurrent execution

Starting MPS

# Start MPS daemon (run as root or GPU owner)
export CUDA_VISIBLE_DEVICES=0
nvidia-cuda-mps-control -d

# All CUDA processes on GPU 0 now go through MPS
# Run multiple processes
mpirun -np 4 ./my_cuda_app  # 4 MPI ranks share GPU via MPS

# Stop MPS
echo quit | nvidia-cuda-mps-control

MPS Benefits

MPS vs. MIG vs. Time-Slicing

Resource Limits (Volta+)

# Limit each MPS client to 25% of GPU threads
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=25

# With Volta MPS: Up to 48 clients per GPU
# Each client gets guaranteed thread allocation

Use Cases

• MPI + GPU: 4-8 MPI ranks per GPU → each rank launches small kernels → MPS packs them together.
• Inference serving: Multiple model workers share one GPU → reduce cost per query.
• Kubernetes: GPU sharing without MIG hardware support → MPS as lightweight alternative.
• Hyperparameter search: Multiple small training runs share GPU resources.

Limitations

• No memory protection between clients → one process can corrupt another's data.
• One client failure can crash all MPS clients on that GPU.
• Unified memory not fully supported with MPS.
• Cannot mix MPS and non-MPS processes on the same GPU.

GPU Multi-Process Service is the lightweight GPU sharing solution for cooperative workloads — by eliminating context switching and enabling true spatial multiplexing of multiple CUDA processes on a single GPU, MPS transforms underutilized GPUs running many small tasks into efficiently packed compute resources, making it essential for MPI-based HPC applications and cost-effective inference serving where workloads are trusted and isolation requirements are relaxed.