Kubernetes 1.37 gang scheduling goes beta: what changes for AI

Every platform team running distributed training on Kubernetes has written the same workaround: a script, an affinity rule or a third-party scheduler that stops a job from starting with half its workers. On September 8, 2026, SIG Scheduling published the v1.37 update that makes that workaround optional. Kubernetes 1.37 gang scheduling is now beta: the Workload and PodGroup APIs, workload-aware preemption and group-level DRA ResourceClaims all leave alpha, a new CompositePodGroup API brings hierarchy, and the plain batch/v1 Job learns to ask for all-or-nothing placement.

If your workloads are multi-node PyTorch jobs, disaggregated LLM serving or GPU batch pipelines, this release changes which layer of your stack owns the "start together or not at all" rule.

Background

Kubernetes schedules pods one at a time. For a stateless web service that is exactly right. For a distributed job it is a trap: an 8-worker job that gets 5 GPUs starts 5 pods, which sit idle waiting for the other 3 while holding expensive accelerators, and a second job in the same state holds the GPUs the first one needs. That deadlock is why Volcano, Kueue and friends reimplemented gang scheduling (place every pod of a group, or none) and group-level preemption on top of the default scheduler.

Since v1.36 the project has been folding these ideas into the core under the Workload-Aware Scheduling (WAS) effort: a Workload object holds the job template, a PodGroup holds runtime state, and kube-scheduler treats them as a single unit. Version 1.37, described in the official release post, promotes that foundation to beta and adds the missing pieces for composite workloads.

What graduates to beta: Workload, PodGroup and preemption

Workload and PodGroup move to scheduling.k8s.io/v1beta1, "one step away from GA" in the authors' words. The gang scheduling they enable sits behind the GenericWorkload feature gate, which now also covers workload-aware preemption (the separate WorkloadAwarePreemption gate is gone). Three changes matter if you tested the alpha.

The PodGroup becomes the queueing unit

Until now, pods that belonged to a group were still queued individually. In v1.37 only the top-level PodGroup is queued, so every member shares the same queueing behaviour. It sounds like plumbing, but it is what makes per-group queueing strategies (priority, fairness) possible in later releases.

minCount is now mutable

minCount, the minimum number of pods that must be placed together, used to be immutable. A controller can now grow or shrink a gang's minimum size on the fly without disturbing pods that are already running. For elastic training that tolerates losing two workers, or serving that absorbs a spike, that is the difference between degrading gracefully and restarting everything.

Preemption finally respects groups

In v1.36 the default single-pod preemption ignored PodGroup objects and could evict one pod out of a group declared as indivisible. v1.37 closes that gap. The disruptionMode values are renamed so they read the same for flat and composite groups: all (evict the whole group or nothing) and single (each pod may be evicted alone). A preemptionPolicy field appears on the PodGroup behind the PodGroupPreemptionPolicy gate, and the preemption algorithm no longer reruns placement for every candidate victim: one simulation pass, then reprieval checks, which cuts the cost on large clusters.

CompositePodGroup: hierarchy for JobSet and LeaderWorkerSet

A flat group is not enough for real workloads. Training jobs have a driver and workers; disaggregated serving with LeaderWorkerSet has leaders and worker groups; a JobSet chains several jobs. The new CompositePodGroup API (scheduling.k8s.io/v1alpha3, alpha, gate CompositePodGroup) arranges groups as a tree: each composite node carries a gang policy expressed as minGroupCount (how many child groups must be placed), each leaf PodGroup keeps its own minCount.

The scheduler evaluates the whole tree as one unit, root to leaves, and binds pods only if every level is satisfied. The release post's example is a Workload with a composite template workload-root (minGroupCount: 2) and two child templates, workers (minCount: 4) and driver (minCount: 1): either all five pods start, or none do.

Preemption follows the same rule. A CompositePodGroup can preempt to make room and can itself be a victim, with disruptionMode: all evicting the entire hierarchy together or single letting a child group go on its own.

Multi-level topology

The second thing hierarchy buys you is nested placement constraints. You can pin the root to an availability zone (topology.kubernetes.io/zone) and each child group to a rack (topology.example.com/rack) inside that zone. Resolution is top-down: the candidate domains for a child are confined to the domain chosen for its parent. For training where cross-rack bandwidth is the bottleneck, that is precisely the constraint people have been hand-rolling with brittle affinities. Multi-level topology is alpha (gate TopologyAwareWorkloadScheduling), and v1.37 also speeds up placement evaluation for the single-level topology-aware scheduling introduced in v1.36.

The standard Job learns gang scheduling

This is the part you can use soonest. The batch/v1 Job gains an explicit .spec.scheduling field, behind the alpha WorkloadWithJob gate, built from four blocks: schedulingPolicy (basic or gang), schedulingConstraints (the topology domain to co-locate in), disruptionMode (single or all) and resourceClaims (DRA claims shared by every pod). An 8-worker training job reads:

apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training-job
spec:
  parallelism: 8
  completions: 8
  scheduling:
    schedulingPolicy:
      gang: {}          # minCount omitted: defaults to parallelism (8)
    schedulingConstraints:
      topology:
      - key: topology.kubernetes.io/zone
    disruptionMode:
      all: {}
  template:
    spec:
      containers:
      ...

The controller compiles this into a Workload and a PodGroup owned by the Job and sets .spec.schedulingGroup.podGroupName on every pod it creates. Omit .spec.scheduling and you get today's behaviour. The field is immutable after creation, with one exception: schedulingPolicy.gang.minCount, so a running gang can be resized.

Under the hood the Job controller uses the new controller integration APIs and the workloadbuilder Go library (in kubernetes/component-helpers, no feature gate). A third-party controller describes its workload as a tree of WorkloadItem nodes; Validate() reports errors at the exact field path of the controller's own API, BuildWorkload() compiles the tree, NewPodGroup() and NewCompositePodGroup() stamp out the runtime objects. Validation is deny-by-default: a controller explicitly lists the policies and disruption modes it supports. That is the intended path for Kubeflow Trainer, JobSet, LWS and in-house operators.

DRA: one ResourceClaim for the whole group

Sharing ResourceClaims at the PodGroup level, introduced in v1.36, graduates to beta (gate DRAWorkloadResourceClaims). A PodGroup declares its claims through a ResourceClaimTemplate and each member pod references them; the claim is replicated and reserved for the whole group instead of being created per pod. v1.37 also fixes a surprising edge case: with the gate disabled, a pod whose claim matched its group's claim used to get an individual claim, which could flood the cluster with ResourceClaims and exhaust DRA resources. Now no claim is created in that situation.

For GPUs, network accelerators and MIG partitions exposed through DRA, this is what makes gang scheduling genuinely atomic: the group gets its resources together, or not at all. The other DRA changes in this release are covered in the v1.37 DRA updates post.

What this changes for AI teams

Native gang scheduling changes how you architect an ML platform. Concretely:

• A shared cluster no longer needs a parallel scheduler for the basics. Volcano and Kueue still earn their keep for queues, quotas and fair share, but all-or-nothing placement and group preemption now live in kube-scheduler. The v1.38 roadmap explicitly aims for Kueue to use WAS as its underlying engine. A platform starting today should target the scheduling.k8s.io/v1beta1 APIs rather than a third-party CRD.
• A training job is four lines of intent. gang: {}, a zone constraint and disruptionMode: all replace affinities, anti-affinities and readiness scripts. Rack-level constraints via CompositePodGroup are alpha: test them, do not ship them.
• Elasticity becomes a parameter, not a redesign. Mutable minCount lets you build training that degrades gracefully, provided the framework (PyTorch elastic, Ray) knows how to react.
• Mind the API break. v1alpha2 is removed and replaced by v1alpha3, with breaking changes around disruptionMode (PodGroup becomes all, Pod becomes single). Alpha testers must migrate manifests before upgrading.
• Everything is off by default. GenericWorkload (beta) on apiserver, controller-manager and scheduler; DRAWorkloadResourceClaims (beta) on those plus the kubelet; TopologyAwareWorkloadScheduling, CompositePodGroup, WorkloadWithJob and PodGroupPreemptionPolicy (alpha). On a managed cluster that means waiting for your provider to expose the gates, or running your own test cluster.
• Group preemption cuts both ways. disruptionMode: all protects a 200-GPU training run from partial eviction, but makes total eviction a single decision. Priority classes and disruption budgets need to be rethought at group level.

The good news for DevOps teams is that the semantics were designed to be consumed by controllers, not just by humans: workloadbuilder and the WorkloadPodGroup* building blocks are how your internal operators speak the same Kubernetes gang scheduling language as the native Job.

In short

• Kubernetes v1.37 promotes the Workload and PodGroup APIs, gang scheduling, workload-aware preemption and group-level DRA claims to beta.
• CompositePodGroup (alpha) adds group hierarchies with minGroupCount and zone-to-rack multi-level topology.
• The standard Job gains .spec.scheduling (gang, topology, disruption, claims) behind the alpha WorkloadWithJob gate.
• minCount becomes mutable, preemption respects groups, and v1alpha2 gives way to v1alpha3.
• v1.38 targets GA for the core APIs, beta for topology and CompositePodGroup, and Kueue integration.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Domaintechnik on Unsplash.