GPUs on Kubernetes: what to know before you start

This article’s angle: what a decision-maker has to settle before opening a GPU cluster, what it is for, what it changes in the organisation, what it costs in skills. It contains no procedure. For the step-by-step of sharing between teams, see MIG, time-slicing and quotas; for cost per job, GPU FinOps.

Introduction

The spread of artificial intelligence workloads, model training, inference at scale, hybrid pipelines, is accelerating demand for GPUs in both cloud and on-premises environments. As companies seek to industrialise their AI platforms, Kubernetes is establishing itself as the reference orchestrator for unifying deployment, scalability and accelerator allocation. In a context where GPUs remain scarce and expensive resources, using them efficiently becomes a strategic question of both performance and financial control.

Faced with these needs, several players in the Kubernetes ecosystem, hardware vendors, AI operator publishers, cloud providers, set out good practices and technical components for optimising GPU usage. Those recommendations run from node configuration to network tuning, by way of resource governance and observability.

GPU support now structured in Kubernetes

Since the Device Plugin Framework stabilised, Kubernetes exposes specialised hardware resources in a standardised way, including NVIDIA and AMD GPUs and other accelerators. NVIDIA states that its GPU Operator, now the preferred method for integrating GPUs into a cluster, automates the installation of the drivers, the CUDA runtime, the device plugin and monitoring tools such as DCGM (Data Center GPU Manager). The operator detects the GPUs present on each node and publishes them as nvidia.com/gpu resources, consumable by pods through a simple resource request.

That native integration lets the scheduler place AI workloads on compatible nodes and guarantee exclusive allocation of one or several GPUs per container. Light inference workloads can build on capabilities such as Multi-Instance GPU (MIG), available on the NVIDIA A100 and H100 architectures, while distributed training can use several GPUs inside one node or across nodes.

The arrival of alternative accelerators (AMD Instinct, Habana Gaudi, dedicated NPUs) is also pushing vendors to publish their own device plugins, widening Kubernetes capabilities in this area.

Essential technical configuration for maximum performance

1. Preparing the GPU nodes

Good practice recommends:

• using a GPU-capable runtime such as containerd plus the NVIDIA Container Runtime,

• a kernel and driver version aligned with the required CUDA stack,

• enabling fast interconnects such as PCIe Gen4/Gen5 or NVLink where available,

• local NVMe volumes to minimise dataset access latency.

In distributed environments, AI nodes should ideally have RDMA or RoCEv2 network interfaces, allowing faster synchronisation between GPUs during parallel training (DeepSpeed, Horovod, Megatron-LM and others).

---

2. Configuring Kubernetes for optimal scheduling

Kubernetes offers several mechanisms for optimising placement:

• NodeFeatureDiscovery (NFD) to detect hardware capabilities and publish labels,

• affinities and anti-affinities to distribute GPU workloads in a controlled way,

• taints and tolerations to isolate GPU nodes from non-AI workloads,

• Topology Manager to align CPU, memory and GPU at NUMA level,

• suitable pod resource requests (neither oversized nor undersized).

NVIDIA also notes that enabling time-slicing or GPU sharing should be reserved for inference workloads or multi-user environments, and is not advisable for latency-sensitive distributed training.

---

3. Orchestrating machine learning workloads

Several operators extend Kubernetes to orchestrate intensive workloads:

• Kubeflow Training Operator to manage TensorFlow, PyTorch and MXNet jobs,

• Ray for distributed inference or parallel compute pipelines,

• MPI Operator for HPC workloads needing low-latency inter-node communication.

These operators natively handle worker creation, synchronisation, recovery after an incident, and management of checkpoints stored on NVMe or distributed storage (CephFS, Longhorn, Lustre and others). They also automate horizontal scaling and multi-GPU scenarios.

---

4. GPU observability and monitoring

To optimise performance, providers recommend putting the following components in place:

• DCGM Exporter to collect precise GPU metrics, SM utilisation, memory, NVLink, temperature, ECC errors,

• Prometheus and Grafana to visualise occupancy rates and identify bottlenecks,

• analysis of NVMe throughput to detect I/O-bound workloads,

• network supervision to guarantee inter-node latency compatible with large models.

These metrics allow batch size to be adjusted, pod placement to be redefined, or resources to be reallocated between training and inference.

Impact and benefits for companies

Putting an optimised GPU architecture on Kubernetes in place brings several concrete gains:

• higher performance for distributed training and high-frequency inference,

• lower cost through fine-grained GPU allocation and the option of partitioning accelerators with MIG,

• better workload density in multi-tenant environments,

• stronger governance through ResourceQuota, LimitRange or Gatekeeper and Kyverno policies,

• reliability thanks to the self-healing and rescheduling mechanisms native to Kubernetes.

Companies operating in sensitive sectors, finance, healthcare, industry, research, cybersecurity, thus gain a more stable platform for developing and deploying their AI solutions.

Conclusion, an essential building block for modern AI platforms

Pairing Kubernetes with GPUs is now a central foundation for artificial intelligence architectures. In a market where models keep growing more complex and demand regularly outstrips accelerator availability, optimising the cluster becomes a key element of performance and operational control. The next steps announced by the vendors, improved topology-aware scheduling, wider support for heterogeneous accelerators, and the rise of MLOps operators, confirm that trajectory.

IT leaders and technical teams now have every component needed to build a robust, scalable and optimised AI platform. In that context, Kubernetes environments equipped with GPUs, whether operated in house or through a sovereign cloud provider, are a major lever for industrialising the next generation of AI workloads.