Kubernetes and GPUs: orchestration for ML/AI workloads

This article’s angle: how Kubernetes places a GPU workload, the scheduler’s role, the scarcity of the resource, and what observability has to show. It sits one level above technical integration. Sharing between teams is covered in MIG, time-slicing and quotas, and inference engine optimisation in batching and latency.

Introduction

The explosion in AI usage, from training complex models to inference workloads at scale, has put GPUs at the heart of modern architectures. Scarce and expensive, these accelerators have to be used optimally. Kubernetes, now the orchestration standard for cloud-native applications, is progressively establishing itself as the central platform where GPU management, scalability and AI pipeline automation converge.

Increasingly native GPU integration in Kubernetes

GPU support in Kubernetes rests mainly on the Device Plugin Framework, stable for several releases now. This mechanism exposes specialised hardware resources, NVIDIA and AMD GPUs, and other accelerators, to the Kubernetes API without modifying the kubelet. In most environments that integration builds on vendor-supplied stacks such as the NVIDIA GPU Operator, which automatically deploys the drivers, DCGM, the Device Plugin and the components needed to use the GPUs.

The NVIDIA plugin remains the most widespread: it registers GPUs dynamically and exposes them as nvidia.com/gpu resources. That model guarantees exclusive allocation of resources while leaving the scheduler free to place AI pods only on compatible nodes. Kubernetes therefore supports workloads requiring 1, 2 or more GPUs, a common case in fine-tuning and multi-model inference.

Emerging capabilities such as Multi-Instance GPU (MIG), forms of GPU sharing through MPS or third-party operators, and granular allocation profiles all illustrate one trend: giving light inference workloads isolated partitions while reserving whole GPUs for intensive training.

Advanced orchestration of machine learning workloads

Beyond allocation, Kubernetes provides the primitives needed to orchestrate distributed processing. Modern models, LLMs, multimodal architectures, vision pipelines, rarely run on a single GPU. AI environments generally rely on specialised operators:

• Kubeflow Training Operator, DeepSpeed and Megatron-LM for distributed training
• Ray Serve for distributed inference and parallel execution graphs
• MPI Operator for workloads requiring high-performance communication

These operators handle the critical phases of the lifecycle, worker initialisation, failure handling, synchronisation, horizontal scaling, and automatic checkpoint saving to NVMe volumes or distributed storage (CephFS, Longhorn, Lustre and others).

On the network side, the heaviest AI processing benefits from RDMA or SR-IOV capable CNIs to cut synchronisation latency between nodes. Kubernetes can expose those interfaces through Device Plugins dedicated to high-throughput NICs, a critical point for models beyond several tens of billions of parameters. Some multi-AZ or multi-datacenter environments, such as those offered by sovereign cloud providers like Hikube, also optimise that communication through low-latency interconnects.

Managing GPU scarcity: quotas, governance and efficiency

In multi-tenant environments, data labs, internal platforms, R&D clusters, GPUs are an operational bottleneck. Kubernetes brings a set of governance mechanisms for controlling allocation:

• ResourceQuota to cap GPU usage per namespace
• LimitRange to impose minimum or maximum rules per pod
• PriorityClass to arbitrate between critical workloads (live inference) and batch jobs (training)
• Affinities and anti-affinities to optimise the placement of demanding tasks

Compliance controllers such as OPA Gatekeeper and Kyverno go further by enforcing security policies, restricting the permitted images, controlling privileges, requiring labels for GPU scheduling, or verifying the rights needed to access the /dev/nvidia* devices.

Observability, essential for optimising performance and cost

Visibility over real GPU usage is essential. Kubernetes builds on the NVIDIA DCGM ecosystem to expose precise metrics:

• occupancy of the streaming multiprocessors (SM)
• memory bandwidth
• power consumption
• PCIe and NVLink usage
• ECC errors

The DCGM Prometheus exporters allow GPU saturation to be analysed and under-performing workloads or network bottlenecks in distributed training to be identified. That data feeds the optimisation strategies: adjusting batch size, redistributing workers, tuning topological placement, or reconfiguring the GPU pool within the nodes.

In this context, training workloads (batch jobs, often preemptible and very demanding) do not carry the same constraints as inference workloads (continuous services with latency requirements). Kubernetes provides a unified framework for handling both families coherently.

Kubernetes, the cornerstone of modern AI architectures

The Kubernetes ecosystem has consolidated around the needs of AI:

• fast NVMe volumes for datasets,
• networks optimised for model synchronisation,
• specialised inference servers (vLLM, TGI, Triton),
• integrated MLOps tooling (MLflow, Argo, Kubeflow Pipelines).

The developments currently under way, topology-aware scheduling, support for heterogeneous accelerators (TPU, NPU, RDU), GPU auto-scaling mechanisms, dynamic logical partitioning and multi-GPU placement optimisation, reinforce that trend further. At a moment when GPUs are becoming a strategic asset, Kubernetes is establishing itself as one of the key tools for maximising their use while keeping cost, performance and quality of service under control in modern AI environments.