Fleet-Scale Kubernetes: An Operating Model for Homogeneous Clusters with Decoupled Capacity
Kubernetes was designed for a single cluster. As organisations scale to fleets of tens, hundreds, or thousands of clusters, the operational model hasn't kept up. This paper proposes one that does.
Kubernetes was designed for a single cluster. A team deploys workloads. kube-scheduler places pods. The cluster autoscaler adds nodes. Everything operates within one control plane, one etcd, one scheduling domain.
The industry outgrew this model. Organisations now operate fleets of 10 to 10,000 clusters, but the tooling and operational patterns haven't evolved to match.
Capacity is fragmented. Each cluster manages its own nodes independently. One cluster is GPU-starved while another has idle GPUs. There's no mechanism to rebalance. Datadog's State of Cloud Costs report shows average CPU utilisation of ~18% across enterprise Kubernetes fleets [1][2], with overprovisioning factors of 2–5× and estimates of annual waste ranging from $50,000 to $500,000 per cluster [3]. The capacity exists in the fleet — it's just trapped in per-cluster islands.
Maintenance doesn't scale. Upgrading, patching, and draining are per-cluster operations that require per-cluster knowledge. When clusters are snowflakes — different configurations, node types, workload assumptions — each upgrade requires its own runbook. Airbnb reached 30+ distinct cluster types with 100+ total clusters and found upgrades untenable because each type required individual testing [4]. Large operators routinely spend months on fleet-wide upgrades and years building proprietary lifecycle tooling.
The cluster became an unnecessary unit of concern. Teams think about "which cluster do I deploy to" when they'd rather think about "I need resources." The cluster is infrastructure plumbing that could be invisible, like a rack or an availability zone. But because scheduling is cluster-scoped and capacity is cluster-managed, the cluster is the unit everyone has to reason about.
AI/ML outgrew the single-cluster model. GPU scarcity, gang scheduling, multi-node training with topology constraints, preemption across priority tiers — these all require fleet-level capacity decisions. A training job needs 2,000 GPUs on the same rack. A serving workload needs to be preempted fleet-wide when a higher-priority training run arrives. No single cluster's autoscaler can make these decisions.
There's no fleet-level control plane. The hyperscalers built them — Google has Borg [5], Meta has Twine [6] — but most organisations manage their fleets through GitOps tooling and per-cluster automation. The operational sophistication gap is significant.
Autoscalers duplicate the scheduler. Both Cluster Autoscaler and Karpenter embed full scheduling simulators to decide what to provision. These simulators diverge from kube-scheduler, producing topology bugs, over-provisioning, and 600+ second decision times at scale [8]. The scheduler already knows why a pod can't be placed. The autoscaler re-derives this information independently, and the two frequently disagree.
The Cluster Autoscaler watches for unschedulable pods, simulates scheduling against its configured node groups, and scales up the winning group. It runs inside the cluster, embeds a full bin-packing estimator (BinpackingNodeEstimator) [36], and uses the scheduler framework's filter plugins to evaluate expansion options.
ProvisioningRequest API. Since CA v1.30, the ProvisioningRequest CRD (autoscaling.x-k8s.io/v1beta1) [7] provides a fire-and-forget capacity request with pluggable provisioning classes (modeled after StorageClass). GKE uses this for queued provisioning of large GPU workloads. ProvisioningRequests are created by users or Kueue, not by the scheduler.
Karpenter provisions nodes directly from pod requirements without predefined node groups. Its Scheduler.Solve() [37] runs a full scheduling simulation — processing pods through a priority queue, evaluating topology spread constraints pod-by-pod, and creating NodeClaim CRDs that represent "a node is on the way."
Karpenter's scheduling simulation is a known source of divergence: its internal scheduler and kube-scheduler can reach different conclusions about pod placement. At scale, the pod-by-pod topology simulation takes 600+ seconds for ~2000 pending pods [8].
Kueue makes topology decisions at admission time, before pods reach kube-scheduler. Its Topology CRD [9] defines a hierarchy of node labels (block → subblock → host → hostname). The TASFlavorCache builds a tree of free capacity per topology domain and finds the tightest domain that fits an entire workload. Kueue injects NodeSelector entries to pin pods to specific domains, then removes scheduling gates to release them.
KEP-5732 [10] proposes making topology-aware scheduling native to kube-scheduler via the Placement-based algorithm. PlacementGeneratorPlugin generates candidate topology domains, PlacementStatePlugin manages state across simulations, and PlacementScorerPlugin ranks candidates. Built on the Workload API (KEP-4671 [11], gang scheduling).
Every major fleet management solution distributes workloads to clusters but treats each cluster's capacity as independently managed. Each member cluster runs its own Cluster Autoscaler or Karpenter. The fleet scheduler checks available capacity but doesn't trigger node provisioning across cluster boundaries.
Every existing autoscaler embeds scheduling simulation. Both CA and Karpenter maintain a parallel model of the scheduler to decide what to provision. This simulation diverges from the real scheduler, creates a large class of topology-related bugs, and doesn't scale. The fundamental insight: the scheduler already knows why a pod can't be placed. Re-deriving that information in the autoscaler is redundant and error-prone.
Autoscaling is coupled to a single cluster. CA and Karpenter run inside the cluster and provision nodes directly. There is no interface for a fleet-level system to manage capacity across clusters. Salesforce running 1,000+ clusters with independent autoscalers [12], and Datadog's industry-wide data showing average CPU utilisation of ~18% with overprovisioning factors of 2–5× [1][2], document what happens. If you want cross-cluster preemption, capacity pooling, or fleet-wide capacity planning, you have to build it from scratch.
No standard capacity request API. ProvisioningRequest comes closest, but it's processed by the Cluster Autoscaler binary — not by an arbitrary external system. Karpenter's NodeClaim is internal to Karpenter. There is no "CSI-for-nodes" — a standardised contract that any autoscaler can implement.
Topology is handled at the wrong layer. CA and Karpenter try to solve topology spread inside the autoscaler's scheduling simulation. Kueue TAS and KEP-5732 handle it at the scheduling layer, which is architecturally correct. But there's no mechanism for the scheduling layer to tell the autoscaler "distribute nodes across 3 zones" without the autoscaler re-simulating the topology constraints.
Fleet management doesn't manage capacity. Karmada, GKE Multi-Cluster, Azure Fleet Manager — they place workloads onto clusters, but don't provision nodes. If the fleet scheduler places a workload on a cluster that doesn't have enough nodes, it waits. The fleet scheduler can't tell a capacity provider "give cluster X 64 more GPU nodes" because there's no standard interface for that.
Large Kubernetes operators typically reach for one of these solutions. Each has real merit — and real limitations that production experience has revealed.
The 5,000-node threshold is real. SIG Scalability tests against this limit [13], and every operator that pushed past it had to substantially re-engineer the control plane. Google replaced etcd with Spanner to reach 130,000 nodes [14][15]. AWS replaced the Raft consensus protocol with a proprietary journal system to reach 100,000 [16]. ByteDance built KubeBrain [17], a custom high-performance etcd replacement, to run 20,000+ node clusters with 1,000,000+ pods. In each case, the result is a Kubernetes-compatible system, not vanilla Kubernetes.
PayPal documented the most candid scaling post-mortem: at ~4,000 nodes, API server throttling delays grew from 1 second to 31 seconds during ramp-ups [18]. Alibaba needed years of custom engineering including a modified page allocation algorithm that extended etcd storage from 2 GB to 100 GB [19].
Your cluster size limit only needs to accommodate your single biggest workload, not your entire fleet. A training job that needs 2,000 GPUs with NVLink co-location — that's your actual hard constraint. Everything else can be scattered across as many clusters as you want. A 100,000-node fleet runs as 20 clusters of 5,000 nodes each, all on stock Kubernetes.
Tellingly, the hyperscalers converge on this architecture for their own infrastructure: Google, AWS, Azure, and Alibaba all give each customer an isolated control plane and scale by running many independent control planes. AWS described their approach at re:Invent 2025 as operating "tens of millions of clusters" via a cell-based architecture [20]. Alibaba runs a Kube-on-Kube meta-cluster where customer control planes are managed as pods [21].
| Approach | Clusters | Max cluster size | Control plane complexity | Cross-cluster preemption |
|---|---|---|---|---|
| Monolith | 1 | 100,000 nodes | Extreme (custom storage, forked K8s) | N/A |
| Specialised | 10-20 | 5,000-10,000 nodes | Moderate | Not supported |
| Homogeneous + capacity contract | 20-200 | ≤ largest single workload | Stock Kubernetes | Via autoscaler implementation |
KubeFed was archived in April 2023, never having reached GA [22]. ByteDance adopted KubeFed v2 in 2019 and abandoned it by 2021 [23], citing uneven resource utilisation, service disruption during rescheduling, and high onboarding costs from the non-native API. They built KubeAdmiral [23] as a replacement, now running 10 million+ pods across dozens of federated clusters.
Karmada (CNCF Incubating [24], 40+ production adopters [25] including Bloomberg, ICBC, and Trip.com) fixed KubeFed's core failures: native Kubernetes APIs, separated propagation and override policies, and both push and pull modes. It has been tested managing 100 clusters with 5,000 nodes each — 500,000 nodes total [26].
Karmada's success is instructive: it works because it treats clusters as independent scheduling domains with workload placement policies on top — not because it unified the Kubernetes API across clusters. Cross-cluster component distribution is still not supported. Federation works well when workloads are self-contained. It struggles when they have cross-cluster dependencies.
The practical reality: most organisations chose not to federate at all. GitOps tools deploying to multiple clusters from a single Git repo became the dominant pattern.
Airbnb progressed through three stages: homogeneous clusters, then 30+ distinct cluster types, then back toward consolidation[4]. Salesforce runs 1,000+ EKS clusters with 1,180+ node pools [12]. Lyft organises by workload type: a main cluster for 600+ microservices (~300,000 containers), a separate ML cluster, and a Flyte workflow cluster [27].
Specialisation creates fragmentation. The GPU cluster is full but the CPU cluster has spare capacity. You can't rebalance. Maintenance becomes per-cluster. At 20 specialised clusters, you need 20 upgrade runbooks. Configuration drift is endemic.
Salesforce documented the result across their 1,000+ EKS clusters [12]: thousands of node groups creating operational bottlenecks, multi-minute scaling delays, inefficient bin-packing, and stranded resources. Their migration from Cluster Autoscaler to Karpenter achieved 5% cost savings and 80% reduction in manual operational overhead — but still couldn't solve cross-cluster fragmentation. No fleet-level capacity view. No cross-cluster preemption.
The alternative is not a bigger cluster, a smarter federator, or a better autoscaler. It's a different way of thinking about what clusters are — and a shift in what operators need to care about.
A cluster is not a capacity pool. It's a logical boundary in a wider fleet — a scheduling domain where kube-scheduler can see nodes and place pods. The only reason two workloads share a cluster is if they reference each other at the Kubernetes API level: shared Services, ConfigMaps, RBAC, network policy, or gang-scheduled PodGroups. Everything else can run on any cluster.
This means your cluster size only needs to accommodate your single biggest workload, not your entire fleet. A 100,000-node fleet runs as 20 clusters of 5,000 nodes each. Each cluster stays within well-tested scaling limits. Stock Kubernetes.
There's no "GPU cluster" or "batch cluster." There are just clusters. The autoscaler serves capacity to whichever cluster needs it, and that capacity can run whatever the scheduler puts on it. You can still separate workloads with taints and tolerations — but that's a policy choice, not a structural requirement.
The autoscaler owns the nodes. It provisions them, tracks them, and reclaims them. Clusters request capacity through a standard contract. The autoscaler decides where to source it — cloud APIs, bare-metal racks, other clusters, a shared fleet pool. Clusters don't know and don't care.
This is how the hyperscalers already operate internally. The missing piece was a standard interface for the rest of us.
The autoscaler does not simulate the scheduler. It receives a declaration of what a cluster needs and provisions nodes. The scheduler handles pod placement when nodes arrive. Topology spread constraints, DRA device allocation, gang scheduling, Kueue admission — all unchanged. We operate below the scheduling layer. We make nodes appear. Everything after is kube-scheduler's job.
This eliminates the entire class of bugs caused by simulation divergence — the same class that produces Karpenter's 600-second topology simulations and the Cluster Autoscaler's 4x over-provisioning with topology spread.
Three CRDs and a protobuf message. CapacityRequest ("I need resources"), UpcomingNode ("a node is on the way"), AvailableCapacity ("here's what I could provision"). An operator bridges CRDs to your autoscaler. That's the entire contract. Its complexity is fixed — it doesn't grow with new Kubernetes features because it doesn't embed scheduling logic.
A namespaced CRD that declares a resource need. Lives in the same namespace as the owning pod, so ownerRef GC works. Created by whatever you want — an optional controller, Kueue, CI/CD, kubectl apply. Deleted via ownerRef when the pod is deleted.
Lifecycle: Two phases, one transition. Pending → Acknowledged. The operator writes Acknowledged once, the first time it includes a CR in a roll-up. Never touched again.
apiVersion: fleet.lucy.sh/v1alpha1
kind: CapacityRequest
metadata:
name: cr-trainer-worker-42
namespace: training
ownerReferences:
- apiVersion: v1
kind: Pod
name: trainer-worker-42
uid: <uid>
spec:
requirements:
- key: node.kubernetes.io/instance-type
operator: In
values: ["a3-highgpu-8g", "p5.48xlarge"]
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a", "us-east-1b"]
resources:
requests:
cpu: "96"
memory: "768Gi"
nvidia.com/gpu: "8"
priority: 1000000 # from pod's PriorityClass
topologySpread:
- topologyKey: topology.kubernetes.io/zone
maxSkew: 1
whenUnsatisfiable: DoNotSchedule
status:
phase: Acknowledged # Pending | AcknowledgedOne CR per pod. No count field — the roll-up aggregates. Minimal status — one write, never touched again.
How CapacityRequests get created is not part of the contract. We ship an optional controller that watches for unschedulable pods and creates one CR per pod. But you could use Kueue, a CI/CD pipeline, a custom controller, or kubectl apply. The operator rolls up whatever exists.
CRs represent total desired state, not incremental asks. The sum of all CRs is "what this cluster needs in total." The autoscaler diffs against its own inventory (it provisioned every node, so it knows what each cluster has) and provisions the difference. Withdrawal is implicit: pod deleted → ownerRef GC deletes the CR → absent from next roll-up.
An eventually consistent hint from the autoscaler about what capacity could be provisioned. Not a source of truth, not a reservation, not required.
Real capacity is inherently racy. Between writing availableCount: 200 and reading it, someone else may have claimed 50. AvailableCapacity is a hint with a confidence signal (availability: High / Medium / Low / None).
apiVersion: fleet.lucy.sh/v1alpha1
kind: AvailableCapacity
metadata:
name: gpu-h100-us-east
namespace: fleet-system
spec:
requirements:
- key: node.kubernetes.io/instance-type
operator: In
values: ["a3-highgpu-8g"]
resources:
cpu: "208"
memory: "1872Gi"
nvidia.com/gpu: "8"
availableCount: 200
availability: High
cost: 31.22
supportsAtomicProvisioning: true
estimatedProvisioningTime: 3m
nodeTemplate:
labels:
accelerator-type: nvidia-h100-80gb
taints:
- key: nvidia.com/gpu
effect: NoScheduleKueue reads it for admission decisions. Operators read it for capacity planning. If nothing reads it, the system works without it.
A node matching this spec is being provisioned. Written by the operator. Deleted when the real Node is Ready.
apiVersion: fleet.lucy.sh/v1alpha1
kind: UpcomingNode
metadata:
name: un-gpu-east-0142
namespace: fleet-system
spec:
labels:
node.kubernetes.io/instance-type: a3-highgpu-8g
topology.kubernetes.io/zone: us-east-1a
topology.kubernetes.io/rack: rack-07-14
accelerator-type: nvidia-h100-80gb
resources:
cpu: "208"
memory: "1872Gi"
nvidia.com/gpu: "8"
taints:
- key: nvidia.com/gpu
effect: NoSchedule
status:
phase: Provisioning # Provisioning | Launched | Registered | Ready | Failed
nodeRef:
name: node-gpu-east-0142
providerID: "aws:///us-east-1a/i-0abc123def456"
provisioningStartTime: "2026-04-14T10:30:00Z"
estimatedReadyTime: "2026-04-14T10:33:00Z"Useful for observability (kubectl get upcomingnodes) and duplicate suppression. Nothing in the contract requires anything to read them.
The operator runs anywhere that can reach the cluster's API server and the autoscaler. It does three things:
1. Roll up and send. The operator uses an informer to cache all CapacityRequests. Periodically, it aggregates CRs by (requirements, resources, priority, topologySpread) into CapacityNeed entries with counts, and sends the aggregate with a cluster identifier. Each send is a full replacement. Pods with identical profiles become one entry.
func (o *Operator) rollUpAndSend(ctx context.Context) error {
// Aggregate CRs by profile. Compresses 10,000+ CRs into ~5-20 entries.
groups := groupByProfile(o.crInformer.List())
msg := &pb.ClusterCapacityNeeds{
ClusterId: o.clusterID,
Timestamp: timestamppb.Now(),
Needs: make([]*pb.CapacityNeed, 0, len(groups)),
}
for _, g := range groups {
msg.Needs = append(msg.Needs, &pb.CapacityNeed{
Requirements: toProto(g.Requirements),
Resources: toProto(g.Resources),
Priority: g.Priority,
Count: int32(g.Count),
Spread: toProtoSpread(g.TopologySpread),
})
}
_, err := o.autoscaler.UpdateClusterNeeds(ctx, msg)
if err != nil {
return err
}
// Mark Pending → Acknowledged (one write per CR, once, ever).
for _, cr := range o.crInformer.GetPending() {
cr.Status.Phase = "Acknowledged"
o.Status().Update(ctx, cr)
}
return nil
}2. Write UpcomingNode CRDs when the autoscaler provisions nodes.
3. (Optional) Write AvailableCapacity CRDs from the autoscaler's capacity data.
syntax = "proto3";
package fleet.lucy.sh.v1alpha1;
import "google/protobuf/timestamp.proto";
service InfrastructureAutoscaler {
rpc UpdateClusterNeeds(ClusterCapacityNeeds) returns (Acknowledgement);
}
message ClusterCapacityNeeds {
string cluster_id = 1;
google.protobuf.Timestamp timestamp = 2;
// Aggregated by (requirements, resources, priority, topology).
// Typically 5-20 entries for an entire cluster.
repeated CapacityNeed needs = 3;
}
message CapacityNeed {
// Standard operators: In, NotIn, Exists, DoesNotExist.
// One additional operator: Same.
// Same (no values) → all `count` nodes must share a value for this key.
// Same (with values) → must share one of these values.
repeated NodeSelectorRequirement requirements = 1;
map<string, string> resources = 2;
int32 priority = 3;
int32 count = 4;
repeated TopologySpread spread = 5;
}
message TopologySpread {
string topology_key = 1;
int32 max_skew = 2;
string when_unsatisfiable = 3; // DoNotSchedule | ScheduleAnyway
}
message NodeSelectorRequirement {
string key = 1;
string operator = 2; // In | NotIn | Exists | DoesNotExist | Same
repeated string values = 3;
}
message Acknowledgement { bool acknowledged = 1; }The autoscaler provisioned every node. It knows what each cluster has because it put the nodes there. The roll-up only contains desired state. The message stays tiny.
Two kinds of topology constraint, both passed through to the autoscaler. The scheduler handles final pod placement in both cases.
Same operator)Same is the only new concept in the API. During roll-up, for CRs needing co-location, the operator adds {key: "topology.kubernetes.io/rack", op: "Same"}. The autoscaler provisions all nodes in one topology domain.
{key: "topology.kubernetes.io/rack", op: "Same"}
→ all nodes must share a rack. Autoscaler picks which.
{key: "topology.kubernetes.io/zone", op: "Same", values: ["us-east-1a", "us-east-1b"]}
→ must share a zone, but only these two are candidates.
{key: "topology.kubernetes.io/zone", op: "In", values: ["us-east-1a", "us-east-1b"]}
→ each node independently in one of these zones. Can be mixed.Same only exists in the protobuf. The CRD uses standard operators. The operator translates during roll-up.
Copied from the pod's topologySpreadConstraints. Three fields pass through: topologyKey, maxSkew, whenUnsatisfiable.
DoNotSchedule: Don't provision nodes if spread can't be satisfied — the scheduler will refuse to place pods on them.
ScheduleAnyway: Try to spread, but provision regardless.
Without this information, the autoscaler might provision all nodes in one zone. With one topology domain, maxSkew is trivially zero and DoNotSchedule is vacuously satisfied — incorrect with no unschedulable pods to trigger correction.
Priority is the pod's PriorityClass value. Whatever creates the CR copies pod.Spec.Priority. If consistent PriorityClasses are used across the fleet, the autoscaler gets a meaningful global ordering for free.
Preemption is the autoscaler's decision. The CR carries the priority number. The autoscaler decides what's preemptible based on relative values across all clusters.
Reclamation is standard node shutdown. The autoscaler signals the kubelet to shut down. The kubelet handles pod eviction, PDB respect, and graceful termination. No new API.
t=0 User kubectl apply -f training-job.yaml (64 pods, 8 GPUs each)
t=1 Pod controller Creates 64 CRs, one per unschedulable pod
t=2 Operator Roll-up: {gpu: 8, count: 64, rack: Same, spread: [{rack, maxSkew: 1}]}
Autoscaler provisions 64 nodes across racks.
t=3 Nodes arrive. kube-scheduler schedules 64 pods.
t=4 Job finishes. 64 pods deleted. 64 CRs GC'd. Next roll-up: absent. Autoscaler reclaims.t=0 Pod controller 64 pods fail. Creates 64 CRs.
t=1 Operator Roll-up: 64 needs. Autoscaler has 32 available. Provisions 32.
t=2 32 nodes arrive. 32 pods schedule. 32 still pending. CRs still exist.
t=3 Operator Roll-up: still 64 CRs. Autoscaler sees 32 nodes exist. Diff = 32.t=0 74 CRs: 64 GPU training + 10 CPU batch.
Roll-up: [{gpu: 8, count: 64, rack: Same}, {cpu: 96, count: 10, zone spread}]
Autoscaler provisions 74 nodes.
t=1 All scheduled. 74 CRs exist. Diff = 0. ✓
t=2 Training finishes. 64 CRs GC'd.
Roll-up: [{cpu: 96, count: 10, zone spread}]. 64 GPU nodes excess.Fleet shape: ~20 clusters × 5,000 nodes. ~250K pods per cluster. 5M pods total.
Everything works comfortably. Each cluster has ~5K CRs (assuming one per unschedulable pod at any time, not all pods). Roll-up compresses to ~15 entries per cluster. 20 clusters × 2KB = 40KB per roll-up cycle. The autoscaler handles 20 clusters as a single process. Operator uses a standard informer. etcd is well within limits. Stock Kubernetes, no special tuning.
Fleet shape: ~200 clusters × 5,000 nodes. ~50M pods total.
Still comfortable. Per-cluster behaviour is identical to the 100K case — the system scales horizontally by adding clusters, not by making them bigger. 200 clusters × 2KB = 400KB per roll-up cycle. The autoscaler processes 200 roll-ups every 10 seconds (20/sec) — trivial. Single autoscaler instance is sufficient.
First consideration: fleet-wide preemption decisions across 200 clusters require the autoscaler to maintain a global priority view. This is an autoscaler implementation concern, not a protocol concern — the roll-up message carries priority values, and the autoscaler decides.
Fleet shape: ~2,000 clusters × 5,000 nodes. ~500M pods total.
The roll-up protocol is still tiny. 2,000 clusters × 2KB = 4MB per cycle. 200 messages per second to the autoscaler. Each message has ~15 entries. The autoscaler needs to process each in <5ms — straightforward.
First real concern: CRs for total desired state. If CRs exist for ALL pods (not just unschedulable ones), each cluster has ~250K CRs. The informer cache for 250K objects consumes ~500MB in the operator. etcd handles it but the watch stream generates continuous traffic. At this scale, operators should profile whether total-desired-state CRs or unschedulable-only CRs are the right model for their fleet.
The operator must use informers, not periodic List. This was always true but becomes non-negotiable at this scale. Listing 250K CRs every 10 seconds would saturate the API server.
Fleet shape: ~20,000 clusters × 5,000 nodes. ~5 billion pods total.
The roll-up message is still tiny. 20,000 clusters × 2KB = 40MB per cycle. The protobuf compression (10,000+ CRs per cluster → ~15 entries) means the autoscaler receives kilobytes per cluster regardless of fleet size.
The autoscaler needs sharding. 20,000 clusters sending roll-ups every 10 seconds = 2,000 messages/sec. Each requires a diff against inventory and a provisioning decision. The autoscaler should shard by cluster range — each shard handles ~2,000 clusters. Sharding is an implementation detail; the protocol doesn't change.
etcd object count is the primary per-cluster concern. At 250K CRs per cluster (total desired state), etcd is stressed. The mitigation: switch to unschedulable-only CRs. This reduces CR count from ~250K to ~25K per cluster (assuming 10% unschedulable at any time) but changes the withdrawal model — the autoscaler tracks its own provisioned inventory rather than diffing against total demand.
ownerRef GC on large Job completion. A 10,000-pod Job finishing triggers 10,000 CR deletions. At etcd's ~10K writes/sec, this takes ~1 second — fine in isolation, but concurrent large completions across clusters stack.
| Concern | Threshold | Mitigation |
|---|---|---|
| Autoscaler throughput | ~2,000+ clusters | Shard by cluster range |
| etcd object count (total-state CRs) | ~250K CRs/cluster | Switch to unschedulable-only CRs |
| Operator memory (informer cache) | ~250K CRs/cluster | Profile and right-size |
| GC write storms | ~10K+ simultaneous pod deletions | Accept async cleanup, size etcd |
| Fleet-wide preemption decisions | ~200+ clusters | Autoscaler implementation concern |
Ships separately from the contract. A standard Kubernetes controller — no scheduler modification needed. Watches for pods with PodScheduled=False, reason=Unschedulable and creates one CR per pod.
func (c *Controller) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
pod := &corev1.Pod{}
if err := c.Get(ctx, req.NamespacedName, pod); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
if !isUnschedulable(pod) || c.capacityRequestExistsForPod(ctx, pod) {
return ctrl.Result{}, nil
}
if c.anyUpcomingNodeSatisfies(ctx, pod) {
return ctrl.Result{}, nil
}
cr := &v1alpha1.CapacityRequest{
ObjectMeta: metav1.ObjectMeta{
GenerateName: "cr-" + pod.Name + "-",
OwnerReferences: []metav1.OwnerReference{{
APIVersion: "v1", Kind: "Pod",
Name: pod.Name, UID: pod.UID,
}},
},
Spec: v1alpha1.CapacityRequestSpec{
Requirements: extractNodeRequirements(pod),
Resources: podResources(pod),
Priority: pod.Spec.Priority,
TopologySpread: convertTopologySpread(pod.Spec.TopologySpreadConstraints),
},
}
return ctrl.Result{}, c.Create(ctx, cr)
}You don't need this controller if: Kueue manages your capacity requests, you have a custom controller, you pre-provision via CI/CD, or you kubectl apply CRs manually.
How nodes are provisioned. Cloud API? Bare metal? Borrowed from another cluster? The contract doesn't care.
How nodes are deprovisioned. The autoscaler signals the kubelet to shut down. Standard Kubernetes graceful node shutdown. No new API.
How clusters are managed. Fleet membership, cluster stamping, control plane scaling — fleet orchestration concerns.
Where anything runs. The autoscaler and operator can run anywhere reachable.
$ kubectl get availablecapacity -n fleet-system
NAME INSTANCE TYPE AVAILABLE COST AVAILABILITY
gpu-h100-us-east a3-highgpu-8g 200 31.22 High
gpu-h100-spot a3-highgpu-8g 450 9.50 Medium
cpu-general m6i.8xlarge 1000 2.44 High
$ kubectl get capacityrequests -A
NAMESPACE NAME PHASE GPU PRIORITY AGE
training cr-trainer-worker-0-7f3a Acknowledged 8 1000000 5m
training cr-trainer-worker-1-b2c1 Acknowledged 8 1000000 5m
batch cr-batch-etl-0-x9d2 Acknowledged 500000 2h
research cr-research-0-k4f1 Pending 8 1000000 3s
$ kubectl get upcomingnodes -n fleet-system
NAME PHASE NODE INSTANCE TYPE AGE
un-research-046 Provisioning a3-highgpu-8g 30s
un-research-047 Launched a3-highgpu-8g 30s
un-research-048 Registered node-gpu-east-0291 a3-highgpu-8g 45s
un-research-049 Ready node-gpu-east-0292 a3-highgpu-8g 2m[1] Datadog, "State of Cloud Costs," datadoghq.com, 2024. https://www.datadoghq.com/state-of-cloud-costs/
[2] Datadog, "Kubernetes autoscaling guide," datadoghq.com, 2024. https://www.datadoghq.com/blog/kubernetes-autoscaling-datadog/
[3] DevZero, "How to Reduce Your Kubernetes Spend: The Complete Guide," devzero.io. https://www.devzero.io/guides/how-to-reduce-your-kubernetes-spend-the-complete-guide
[4] Sheng, E. & Morrison, D., "Dynamic Kubernetes cluster scaling at Airbnb," Airbnb Tech Blog, May 2022. https://medium.com/airbnb-engineering/dynamic-kubernetes-cluster-scaling-at-airbnb-d79ae3afa132
[5] Verma, A. et al., "Large-scale cluster management at Google with Borg," EuroSys '15, ACM, 2015. https://research.google/pubs/large-scale-cluster-management-at-google-with-borg/
[6] Tang, C. et al., "Twine: A Unified Cluster Management System for Shared Infrastructure," OSDI '20, USENIX, 2020. https://www.usenix.org/conference/osdi20/presentation/tang
[7] kubernetes/autoscaler, "Provisioning Request CRD proposal." https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/proposals/provisioning-request.md
[8] GitHub Issue, aws/karpenter-provider-aws#5505, "Scheduling 2000 Pods takes more than an hour," Jan 2024. https://github.com/aws/karpenter-provider-aws/issues/5505
[9] Kueue Documentation, "Topology Aware Scheduling." https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
[10] kubernetes/enhancements, "KEP-5732: Topology Aware Workload Scheduling." https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5732-topology-aware-workload-scheduling
[11] kubernetes/enhancements, "KEP-4671: Gang Scheduling using Workload Object." https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4671-gang-scheduling/README.md
[12] Jawad, S. et al., "How Salesforce migrated from Cluster Autoscaler to Karpenter across their fleet of 1,000 EKS clusters," AWS Architecture Blog, Jan 2026. https://aws.amazon.com/blogs/architecture/how-salesforce-migrated-from-cluster-autoscaler-to-karpenter-across-their-fleet-of-1000-eks-clusters/
[13] kubernetes/community, SIG Scalability, "Kubernetes Scalability thresholds." https://github.com/kubernetes/community/blob/main/sig-scalability/configs-and-limits/thresholds.md
[14] Massri, B. & Róźacki, M., "How Google Does It: Building the largest known Kubernetes cluster, with 130,000 nodes," Google Cloud Blog, Nov 2025. https://cloud.google.com/blog/products/containers-kubernetes/how-we-built-a-130000-node-gke-cluster
[15] Bradstock, D. & Róźacki, M., "65,000 nodes and counting," Google Cloud Blog, Nov 2024. https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting
[16] Ramakrishnan, A. et al., "Under the hood: Amazon EKS ultra scale clusters," AWS Containers Blog, Jul 2025. https://aws.amazon.com/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/
[17] KubeWharf/ByteDance, "KubeBrain: A High Performance Metadata System for Kubernetes." https://github.com/kubewharf/kubebrain
[18] Qadeer, A., "Scaling Kubernetes to Over 4k Nodes and 200k Pods," PayPal Technology Blog, Jan 2022. https://medium.com/paypal-tech/scaling-kubernetes-to-over-4k-nodes-and-200k-pods-29988fad6ed
[19] Chen, X., "Performance Optimization of etcd in Web Scale Data Scenario," CNCF Blog, May 2019. https://www.cncf.io/blog/2019/05/09/performance-optimization-of-etcd-in-web-scale-data-scenario/
[20] Joshi, S. & Tripathi, R., "Under the hood: Architecting Amazon EKS for scale and performance," AWS re:Invent 2025, Session CNS429, Dec 2025. https://repost.aws/articles/ARZ3qVRVY8SUKZ_6mj32xAiQ/re-invent-2025-under-the-hood-architecting-amazon-eks-for-scale-and-performance
[21] Tang, Z. et al., "Demystifying Kubernetes as a Service — How Alibaba Cloud Manages 10,000s of Kubernetes Clusters," CNCF Blog, Dec 2019. https://www.alibabacloud.com/blog/demystifying-kubernetes-as-a-service---how-alibaba-cloud-manages-tens-of-thousands-of-kubernetes-clusters_595623
[22] GitHub, kubernetes-retired/kubefed, archived April 25, 2023. https://github.com/kubernetes-retired/kubefed
[23] Liu, G., "KubeAdmiral: next-generation multi-cluster orchestration engine," CNCF Blog, Nov 2023. https://www.cncf.io/blog/2023/11/24/kubeadmiral-next-generation-multi-cluster-orchestration-engine-based-on-kubernetes/
[24] CNCF, "Karmada brings Kubernetes multi-cloud capabilities to CNCF Incubator," Dec 2023. https://www.cncf.io/blog/2023/12/12/karmada-brings-kubernetes-multi-cloud-capabilities-to-cncf-incubator/
[25] Karmada Project, "Adopters." https://karmada.io/adopters/
[26] Karmada Project, "Test Report on Karmada's Support for 100 Large-Scale Clusters," Oct 2022. https://karmada.io/blog/2022/10/26/test-report/
[27] Altoros, "Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment." https://www.altoros.com/blog/lyft-runs-300000-containers-in-a-multicluster-kubernetes-environment/
[28] kubernetes/community, SIG Scalability, "Scalability SLOs." https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md
[29] KubeCon EU 2026 Scheduling Summit, "[Public] Kubecon Scheduling Summit 03'2026." https://docs.google.com/document/d/1HDj4od6qml71T4lq1ELjfNwKO0xNkqIeHo112z3AEGk/
[30] "[Public] API Design for WAS Controller Integration." https://docs.google.com/document/d/1VG7Zto9JYuPG4Anb01WMRryJlfV6met0jgob3T2NjZ4/
[31] "[Public] Workload Aware-Scheduler Cluster Autoscaling." https://docs.google.com/document/d/1ergKWH28EpGyYVISZbqPqIBS1wN7VNhjNa_Sl1pAFI8/
[32] "[Public] WAS and Kueue integration strategy." https://docs.google.com/document/d/13I3bMY-abBtsWmDhCl4RdDHKchHeB3G8zMSTpplmxHA/
[33] kubernetes/enhancements, "KEP-5832: Decouple PodGroup API." https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5832-decouple-podgroup-api
[34] kubernetes/enhancements, "KEP-5710: Workload-Aware Preemption." https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5710-workload-aware-preemption
[35] kubernetes/enhancements, "KEP-5729: ResourceClaim Support for Workloads." https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5729-resourceclaim-support-for-workloads
[36] kubernetes/autoscaler, cluster-autoscaler/estimator/binpacking_estimator.go. https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/estimator/binpacking_estimator.go
[37] kubernetes-sigs/karpenter, pkg/controllers/provisioning/scheduling/scheduler.go. https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/controllers/provisioning/scheduling/scheduler.go