GPU FinOps on Kubernetes: autoscaling, quotas and cost per job

Intelligent autoscaling, batch scheduling and quotas: maximising GPU ROI

GPU Kubernetes clusters generate substantial costs while frequently showing sub-optimal utilisation. According to Sedai, traditional autoscalers (HPA, VPA, Cluster Autoscaler) work reactively, waiting for utilisation thresholds to be crossed before adjusting resources. That inherent latency produces two expensive pathologies: preventive over-provisioning to absorb peaks (wasting 30 to 50% of the GPU budget according to Kedify), and chronic under-provisioning that degrades SLAs and frustrates end users.

Optimising GPUs in Kubernetes requires a systemic approach combining four technical levers, intelligent autoscaling with Karpenter to provision optimal GPU nodes dynamically, batch scheduling through Volcano or Kueue to maximise bin-packing and remove fragmentation, hierarchical quotas preventing one team from monopolising resources, and granular observability through DCGM to identify inefficiency. This article details those four levers, sets out the operational traps observed in production, and proposes a progressive deployment framework validated on multi-tenant clusters hosting both training and inference workloads.

What comes from Hikube, and what comes from the ecosystem. On Hikube the control plane is operated by Hidora SA, the CNI is Cilium, and workers, GPUs included, are sized as node groups that follow the load. Everything else this article names (Karpenter, Volcano, KEDA, Kyverno, third-party autoscalers) belongs to the Kubernetes ecosystem: components you install and run in your own cluster, not catalogue features, and nothing here turns them into a product promise. One point does not transpose at all: Hikube offers no Spot or preemptible instances, so the savings described on that lever do not apply here. Managed scope: managed Kubernetes, GPU pricing, GPU.

This article’s angle: what a GPU job costs and how to bring it down, autoscaling, quotas, cost per job. This is the cost question: the technical sharing between teams is in MIG, time-slicing and quotas.

The four levers of GPU optimisation in Kubernetes

Effective optimisation of GPU resources in Kubernetes rests on four complementary technical pillars, each addressing a distinct source of waste.

1. Dynamic autoscaling: Karpenter against Cluster Autoscaler

Cluster Autoscaler (CA), the historical system, works through AWS Auto Scaling Groups (ASG) or their cloud equivalents. CA detects unschedulable pods, identifies the node group matching their constraints, then asks the ASG to add nodes. That indirect architecture creates three critical limitations: provisioning time of 3 to 5 minutes (pod detection plus ASG scaling plus instance boot), granularity limited to predefined node groups (no way to request an m5.8xlarge dynamically if only m5.4xlarge is configured), and resource fragmentation (pods requesting 6 vCPU placed on 8 vCPU nodes leave 2 vCPU unusable).

Karpenter, which reached stable v1.0.0 in August 2024, changes Kubernetes autoscaling by interacting directly with the EC2 API. When a pod becomes unschedulable, Karpenter evaluates its constraints (CPU, RAM, GPU, zone, architecture), queries the EC2 API to identify the instance types satisfying them, sorted by cost (price-capacity-optimized), then provisions the optimal instance in 45 to 60 seconds. This approach yields three measurable advantages: three times the speed of CA (45 s against 3 to 5 min per Medium's 2025 benchmarks), near-infinite granularity (Karpenter picks from 800+ EC2 instance types with no prior configuration), and better economics (automatic consolidation replaces under-used nodes with cheaper instances).

Karpenter configuration for GPU:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-pool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-gpu-count
          operator: Gt
          values: ["0"]              # Any instance with a GPU
        - key: karpenter.k8s.aws/instance-gpu-name
          operator: In
          values: ["a100", "h100"]   # A100 or H100 only
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]  # Spot/On-Demand mix
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-class
  limits:
    cpu: "1000"
    memory: 4000Gi
    nvidia.com/gpu: "32"             # Max 32 GPUs in this pool
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30m            # Consolidate after 30 min of under-use

Karpenter introduces automatic consolidation, when a GPU node falls below 50% utilisation for 30 minutes (configurable), Karpenter drains its pods to other nodes and terminates the instance. This capability, absent from the standard Cluster Autoscaler, cuts cost by 20 to 35% according to nOps, which observes under 1% Spot terminations thanks to built-in good practice (instance diversification across several AZs, real-time reconsideration of workloads).

2. Batch scheduling: Volcano and Kueue for training workloads

The standard Kubernetes scheduler works at pod level, each pod is evaluated independently and placed as soon as a node satisfies its constraints. That approach fails for distributed workloads where all pods must start simultaneously. A PyTorchJob with 16 pods (16 GPUs) can end up with 12 pods scheduled and 4 Pending indefinitely, blocking the 12 GPUs already allocated with no training progress.

Volcano, a CNCF project in growing adoption through 2025, implements gang scheduling through PodGroups: either all pods in the group are scheduled simultaneously, or none are. Volcano keeps pods Pending until enough resources are available for the whole group, then schedules them atomically. That guarantee removes deadlocks and GPUs wasted waiting for missing pods. The InfraCloud 2025 benchmarks show Volcano cutting GPU fragmentation by 40% against the standard scheduler on distributed training workloads.

Volcano PodGroup example:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: llama-training
spec:
  minAvailable: 16        # Gang scheduling: 16 pods or nothing
  schedulerName: volcano
  plugins:
    env: []
    svc: []
  policies:
    - event: PodEvicted
      action: RestartJob
  tasks:
    - replicas: 16
      name: worker
      template:
        spec:
          containers:
            - name: pytorch
              image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
              resources:
                limits:
                  nvidia.com/gpu: 1
                  cpu: "8"
                  memory: 32Gi

Kueue, developed by the Kubernetes SIG Scheduling and production-ready since 2024, implements a queuing layer ahead of the scheduler. Workloads (Jobs, PyTorchJobs, RayJobs) are submitted into LocalQueues bound to ClusterQueues with configured quotas. Kueue simulates scheduling and only admits the workload if resources are available and quotas respected. Cohorts let several ClusterQueues share unused quota with weighted fairness logic: if team A uses only 50% of its quota, team B can borrow the surplus temporarily.

Yunikorn, an Apache project originating in the Hadoop ecosystem, offers a universal scheduler with hierarchical queues and fairness policies (DRF, Dominant Resource Fairness). Unlike Volcano, which requires custom resources (VolcanoJob), Yunikorn plugs in as a replacement scheduler and handles standard Kubernetes workloads natively. The PEARC 2024 benchmarks show Yunikorn cutting workflow execution times by 4.6× and improving cluster utilisation threefold against the standard scheduler on multi-tenant scientific workloads.

3. Resource quotas and policies: preventing monopolisation

Kubernetes ResourceQuotas limit total consumption per namespace. A typical GPU quota such as requests.nvidia.com/gpu: "16" stops a namespace consuming more than 16 GPUs at once. LimitRanges define a minimum and maximum per pod: blocking pods that request 0 GPUs (a configuration error) or more than 8 (a monopolisation risk).

These native mechanisms have two limitations. First, they work at namespace level, not team or project level: an organisation with 10 teams needs 10 distinct namespaces, complicating RBAC and networking. Second, they are static: there is no way to redistribute one team's unused quota to another dynamically during a load spike.

Kueue solves both through hierarchical ClusterQueues and cohorts. A ClusterQueue represents a resource pool with a nominal quota. Cohorts enable borrowing: several ClusterQueues form a cohort and share their unused quota according to configured weights. For example: team A with a 20-GPU quota, team B with 10, sharing a cohort. If team A uses only 10 GPUs, team B can borrow up to 10 more (20 in total), which are released automatically when team A needs its nominal capacity back.

Kueue configuration with cohorts:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: a100-80gb
spec:
  nodeLabels:
    nvidia.com/gpu.product: A100-SXM4-80GB
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-research
spec:
  cohort: shared-gpu          # Cohort member, for borrowing
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: a100-80gb
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 20      # Guaranteed quota
              borrowingLimit: 10    # Can borrow up to 10 GPUs

Preemption policies let high-priority workloads reclaim resources by preempting (terminating) low-priority ones. Volcano implements configurable preemption: production workloads (high priority) can preempt dev/test workloads (low priority) when the cluster saturates. Preemption respects a set of laws (the Yunikorn laws): a workload can only preempt if its priority is strictly higher, preemption respects PodDisruptionBudgets, and preempted workloads are automatically requeued.

4. Observability and FinOps: measuring in order to optimise

Optimising without measuring is navigating without instruments. GPU observability in Kubernetes combines three layers: low-level metrics (DCGM), Kubernetes metrics (kube-state-metrics), and business metrics (cost per job, cost per team).

DCGM Exporter, deployed automatically by GPU Operator, exposes Prometheus metrics on port 9400. The critical metrics for optimisation include: DCGM_FI_DEV_GPU_UTIL (utilisation %, target 60 to 80%), DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE (VRAM, to identify oversized GPUs), DCGM_FI_DEV_POWER_USAGE (power draw, to detect energy inefficiency), and DCGM_FI_DEV_XID_ERRORS (hardware errors, to anticipate failures).

Kubernetes metrics (through kube-state-metrics) expose, the number of GPU pods Pending (an indicator of cluster saturation), the ratio of GPU requested against allocatable per node (fragmentation), and GPU pods by priority (load distribution). Correlating DCGM with Kubernetes reveals the pathologies: nodes with allocatable GPU above 0 but GPU utilisation at 100% indicate fragmentation (pods requesting 2 GPUs on a 4-GPU node block the remaining 2, insufficient for later pods).

FinOps metrics turn technical data into business insight. Karpenter exposes karpenter_nodes_total_pod_requests and karpenter_nodes_allocatable, which allow the real bin-packing rate to be calculated. Correlating that with EC2 pricing data gives the cost per GPU-hour actually used, not merely allocated. Kubecost, an open-source tool acquired by AWS in 2024, aggregates these metrics and generates reports: cost per namespace, per workload, per team, with optimisation recommendations based on observed patterns.

Advanced optimisation strategies

Intelligent GPU sharing, when and how

GPU sharing (MIG, time-slicing) improves utilisation but introduces trade-offs. MIG provides hardware isolation guaranteeing predictable performance, ideal for multi-tenant production. Time-slicing offers finer granularity but no isolation, which makes it better suited to dev/test. A hybrid strategy combines the advantages of both: MIG for production inference workloads (strict SLAs), time-slicing for interactive notebooks and experimentation, and exclusive GPUs for intensive training.

The AWS EKS Best Practices 2024 recommend, for ML/AI workloads, using time-slicing for spiky workloads with utilisation below 30%, MIG for steady-state inference needing isolation, and exclusive GPUs through Karpenter for distributed training. The Karpenter configuration allows karpenter.k8s.aws/instance-gpu-name: "a100" to force a specific GPU model, avoiding placement on unsuitable T4s.

Spot instances, cutting training cost by 70%

AWS Spot instances offer up to 70% off On-Demand, in exchange for an interruption risk with two minutes' notice. Karpenter supports Spot natively through karpenter.sh/capacity-type: spot in the NodePool requirements. Karpenter uses a price-capacity-optimized strategy: prioritising the Spot pools offering the best compromise between price and interruption probability.

Spot good practice for ML according to nOps includes, diversifying instances across several AZs (multiplying Spot capacity pools reduces the probability of simultaneous interruption), using SpotToSpotConsolidation (a Karpenter feature gate replacing under-used Spot instances with cheaper Spot without disruption), and checkpointing regularly for training workloads (saving state every N minutes to resume after interruption). nOps reports a Spot interruption rate below 1% on well-configured Karpenter clusters, against the 5 to 15% observed with static ASGs.

For critical workloads tolerating no interruption (production inference, final training of 70B+ models), a Spot/On-Demand mix through NodePool priorities remains relevant: 80% Spot capacity for the savings, 20% On-Demand for availability. Karpenter provisions Spot preferentially, then falls back to On-Demand if Spot is unavailable.

Predictive autoscaling: anticipating instead of reacting

Reactive autoscaling (HPA, Karpenter) waits for metrics to cross thresholds before acting, creating 30 to 90 seconds of latency (metric collection plus evaluation plus scaling). Predictive autoscaling uses time-series models (ARIMA, Prophet) to forecast future load and pre-warm capacity 5 to 10 minutes in advance.

Kedify, a commercial platform built on KEDA, introduces predictive autoscaling driven by error budgets: forecasts estimate the load of the next 15 minutes, the system pre-warms just enough capacity to absorb the predicted peak without exceeding the SLO budget, and reactive scalers take over if the forecast is wrong. Kedify recommends short prediction horizons (10 to 30 minutes, beyond which uncertainty explodes), preferring quantiles to averages (a P95 forecast rather than a mean), and setting strict caps (a maximum pre-warm budget to avoid waste when the forecast is wrong).

Predictive autoscaling delivers, according to Kedify, a 15 to 25% reduction in P95 latency, but requires investment: enough historical data (two weeks minimum), models retrained regularly (weekly), and close monitoring (forecast against actual, adjustments on drift). The ROI is mainly justified for applications demanding ultra-low latency (below 50 ms P99) or handling predictable peaks (batch publications, marked daytime traffic).

The five mistakes that cap GPU utilisation

1. Running Cluster Autoscaler and Karpenter at the same time

Symptom: installing Karpenter without disabling Cluster Autoscaler, hoping the two coexist and complement each other.

Impact: CA and Karpenter both watch unschedulable pods and each try to provision nodes. The result: race conditions with duplicated nodes, thrashing (continuous scale up and down from conflicts between the two systems), and temporarily doubled cost. The AWS Karpenter documentation is explicit: "We recommend not using Kubernetes Cluster Autoscaler at the same time as Karpenter because both systems scale up nodes in response to unschedulable pods." Organisations ignoring that warning report 30 to 50% extra cost during the coexistence period.

Fix: take a progressive migration, identify the node pools managed by CA, create equivalent Karpenter NodePools, test in staging, switch production in waves (20% → 50% → 100%), then disable CA once the cluster is 100% Karpenter. During the transition, use node taints to separate CA and Karpenter nodes strictly, avoiding any overlap. Document the rollback procedure in the runbook (re-enable CA if Karpenter causes problems). Typical timeline: 2 to 4 weeks for a complete migration on a cluster of more than 50 nodes.

2. Ignoring disruption budgets and Karpenter consolidation

Symptom: enabling Karpenter consolidation (consolidationPolicy, WhenUnderutilized) without configuring PodDisruptionBudgets (PDBs) on critical workloads.

Impact: Karpenter consolidates aggressively, as soon as a node falls below 50% utilisation for 30 minutes, it drains the pods and terminates the node. Without PDBs, Karpenter can drain every replica of a service simultaneously, causing downtime. On stateful workloads (databases, queues), consolidation forces a restart that can take several minutes, degrading SLAs. Organisations report production incidents (P1/P2) caused by Karpenter consolidation on services without PDBs during the first weeks after deployment.

Fix: audit every critical deployment and statefulset and create appropriate PDBs, minAvailable, 1 for services with 2 or more replicas (guaranteeing at least one replica always runs), maxUnavailable, 1 for multi-replica services (limiting simultaneous disruption). For stateful workloads, add the annotation karpenter.sh/do-not-disrupt: "true" on the pods to block consolidation. Test consolidation in staging with synthetic traffic, verify the PDBs really prevent downtime, and monitor karpenter_disruption_pods_disrupted_total correlated with production incidents.

3. Not sizing node pools with limits

Symptom: creating Karpenter NodePools or Kueue ClusterQueues without specifying limits, relying on cloud budgets or on teams' good will.

Impact: an application bug generating an infinite pod loop (CrashLoopBackOff continuously recreating pods) can trigger Karpenter, which will provision GPU nodes indefinitely. In two hours a cluster can go from 10 to 200 H100 nodes, and the AWS bill explode to CHF 1,400/h (200 × CHF 7/h). Without limits nothing stops the escalation until AWS billing limits are exhausted (if configured) or someone intervenes in a panic. Real incidents document surprise bills of CHF 50,000 to 100,000 over 48 hours.

Fix: always define limits in NodePools, for example limits, nvidia.com/gpu, "32" (a maximum of 32 GPUs in that pool), combine that with AWS Service Quotas (to cap the number of GPU instances per region), create CloudWatch alarms on estimated cost (alerting if projected monthly cost passes a threshold), and implement admission controllers (OPA, Kyverno) to validate that every pod specifies reasonable resource limits. For Kueue, configure nominalQuota plus borrowingLimit so no team can monopolise the whole cluster, even during a bug.

4. Underestimating the network impact on GPU autoscaling

Symptom: Karpenter provisions GPU nodes quickly (45 to 60 s), but pods stay in ContainerCreating for another 5 to 10 minutes.

Impact: the bottleneck is not Karpenter but pulling large Docker images. ML/AI images (PyTorch, TensorFlow with CUDA) weigh 5 to 15 GB. On a standard network, the image pull takes 3 to 8 minutes depending on bandwidth and registry cache. Fast autoscaling then becomes illusory: pods are only available after 6 to 11 minutes in total (45 s Karpenter plus 5 to 10 min of image pull), against 3 to 5 minutes with CA. Karpenter's speed promise does not materialise, and users are frustrated by the perceived latency.

Fix: put a multi-part strategy in place, pre-cache the images on custom AMIs (customising the AMI with images already pulled cuts cold start below a minute), use optimised image pull policies (imagePullPolicy, IfNotPresent avoids needless repulls), deploy an internal registry cache (Harbor, Dragonfly) in the same VPC as the cluster, and compress the images (multi-stage builds, removing dev dependencies). For critical production, keeping a warm pool of pre-provisioned GPU nodes (Karpenter --reserved-eni or a minimal static node pool) avoids cold start entirely.

5. Neglecting real-time cost monitoring

Symptom: deploying Karpenter, Kueue and various optimisations, then waiting for the monthly cloud bill to measure the savings.

Impact: poorly calibrated optimisations can create invisible extra cost until the end-of-month billing shock. Frequent examples: over-aggressive Karpenter consolidation causing thrashing (continuous scale up and down) and raising cost by 15 to 25%, badly configured Kueue quotas leaving GPUs idle (allocated but unused) for hours, or Spot instances with frequent interruptions (above 10%) wasting training time for lack of suitable checkpointing. Without real-time visibility, these pathologies cannot be identified and corrected before their financial impact.

Fix: implement a complete FinOps pipeline, Kubecost or an equivalent for real-time cost per namespace and workload, Grafana dashboards with metrics such as cost per GPU-hour used (not allocated), utilisation rate per node pool, cost per team with 7-day and 30-day trends, plus alerts on anomalies (cost more than 20% above the weekly average). Weekly engineering and finance reviews then analyse the dashboards and adjust the configuration. Typical monitoring ROI: 1 to 2 days of setup plus 2 hours a week of review, for 10 to 20% of the GPU budget saved through the optimisations it reveals.

Illustrative scenario, optimising an ML cluster on AWS EKS

This scenario is an illustration, not a Hikube customer and not a Hikube deployment. It runs on AWS EKS, the instance families (p3, g4dn, m5), the Spot instances and CloudWatch give it away, and the figures describe that specific case, not a result Hikube measures or reproduces. What transposes: the reasoning about GPU fragmentation, gang scheduling, per-team quotas and cost visibility. What does not: the AWS instance families, and anything resting on Spot instances, which Hikube does not offer.

The case in one sentence, a fintech startup with 40 data scientists and ML engineers, running an EKS cluster with training workloads (fraud detection models) and inference (real-time scoring), on a GPU budget of CHF 35,000 a month that was regularly exceeded (overruns of CHF 45,000 to 50,000). The initial infrastructure used Cluster Autoscaler with three fixed node groups (p3.8xlarge for training, g4dn.xlarge for inference, m5.2xlarge for system), ResourceQuotas per team and basic monitoring through CloudWatch.

Problems identified in the audit:

• GPU fragmentation, fixed node groups (4 GPUs per p3.8xlarge node) with jobs requesting 2 to 3 GPUs left 1 to 2 GPUs idle, for overall utilisation of 42%.
• Training jobs failing 25% of the time (no gang scheduling, partial pods blocking GPUs).
• Defensive over-provisioning, 8 p3.8xlarge nodes (32 GPUs) kept running 24/7 to absorb occasional 2-hour daily peaks.
• No visibility on cost per team or project, making waste impossible to identify.

Optimisations deployed (timeline, six weeks)

Weeks 1-2, migrating from Cluster Autoscaler to Karpenter

• Karpenter v1.0.2 installed through Helm.
• NodePools created, gpu-training (p3, p4d instances, Spot priority), gpu-inference (g4dn, g5, On-Demand), system (m5).
• CA disabled progressively, 25% of traffic to Karpenter (one week of monitoring), then 100%.
• Result, GPU node provisioning time down from 4.5 min to 55 s (-80%), better bin-packing (Karpenter picks p3.2xlarge for 2-GPU jobs instead of p3.8xlarge).

Weeks 3-4, implementing Volcano and gang scheduling

• Volcano v1.9 deployed.
• PyTorchJobs migrated to VolcanoJob with PodGroups.
• Queues configured, production (high priority, 16-GPU quota), research (medium priority, 24-GPU quota), dev (low priority, 8-GPU quota, preemptable).
• Result, training job success rate up from 75% to 96%, average job time down 22% as partial allocations and retries disappeared.

Week 5, deploying Kubecost and FinOps dashboards

• Kubecost v2.1 installed.
• Allocation tags configured per team and project.
• Grafana dashboards created, real-time cost per team with a GPU/CPU/network breakdown, efficiency scores (GPU used against allocated), automatic recommendations.
• Findings, team A was taking 45% of the budget at 28% utilisation (interactive notebooks never closed), while team B showed 82% efficiency but an insufficient quota.

Week 6, final optimisations

• Karpenter consolidation enabled with PDBs on critical services.
• 80% of training jobs moved to Spot (70% saving).
• Quotas redistributed on measured efficiency.

Results measured after three months:

Projected annual saving: (47,000 - 26,500) × 12 = CHF 246,000, an ROI under three months on the engineering investment (6 weeks × 1 senior DevOps FTE ≈ CHF 25,000).

In short, three key points

Karpenter makes GPU autoscaling substantially more effective

Cluster Autoscaler creates 3 to 5 minutes of latency and fragmentation through static node groups. Karpenter v1.0 (stable since August 2024) dynamically provisions the optimal instance from 800+ EC2 types in 45 to 60 seconds, cuts cost by 20 to 35% through automatic consolidation, and supports Spot natively with under 1% interruptions thanks to multi-AZ diversification and the price-capacity-optimized approach. The illustrative AWS EKS scenario shows gains measured in that specific case: provisioning time -80%, GPU utilisation +76%, monthly cost -44%. Karpenter does demand rigour, though: set strict limits, configure PDBs before consolidation, and pre-cache images to avoid the network bottleneck. A progressive CA to Karpenter migration over 2 to 4 weeks reduces the risk.

Gang scheduling through Volcano or Kueue eliminates most distributed training failures

The standard Kubernetes scheduler places pods independently, which creates deadlocks on distributed workloads: a 16-pod PyTorchJob can end up with 12 pods scheduled and 4 Pending, blocking 12 GPUs with no progress. Volcano guarantees atomicity: all pods scheduled together or none, which removes partial allocations. The benchmarks show a sharply higher job success rate and lower average time as retries disappear. Kueue adds hierarchical quotas with borrowing, letting teams share unused quota dynamically while avoiding monopolisation. Yunikorn, for its part, offers a solid alternative for hybrid K8s/Hadoop environments and scientific workloads.

Real-time FinOps turns optimisation from a one-off effort into a continuous practice

Waiting for the monthly bill to judge optimisations leads to recurring billing shocks. The expensive pathologies, over-aggressive consolidation, badly calibrated quotas, mismanaged Spot, stay invisible without real-time monitoring. Kubecost plus Grafana expose cost per namespace, team and workload in real time, with trends and alerts. The scenario showed one team taking 45% of the budget at only 28% utilisation, which allowed resources to be redistributed. Weekly engineering and finance reviews then continuously identify another 10 to 20% of savings. The investment in monitoring pays for itself quickly.