Happy Dussehra! May good always win. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Kube-burner: Performance and Scale Testing for Kubernetes Clusters

CNCF Sandbox. A Go tool that creates, patches, reads and deletes Kubernetes objects at a controlled rate, measures latencies such as pod startup, collects and indexes Prometheus metrics and evaluates alerts, so you can prove a cluster meets its scale targets before real load arrives.

18 min read

CNCF Sandbox

  • Kube-burner
  • Kubernetes
  • Prometheus
  • OpenSearch
  • Go
  • Load Testing
  • Performance Testing
  • AWS EKS
CNCF project page

Kube-burner is a Kubernetes performance and scale test orchestration tool. You write a short YAML file that says what to create (namespaces, pods, services, anything you can template), how many iterations, and how fast. Kube-burner creates the objects through the API, waits for them, measures how long things took, queries Prometheus for cluster metrics over the test window, evaluates your alert expressions, and sends the results to a store such as OpenSearch or local files.

The result is a repeatable answer to “does this cluster still behave when it holds ten thousand pods?”, with numbers you can compare between runs, versions or node sizes.

CNCF daily · Sprint 5, post 13 of 14 · Continuous Integration & Delivery

At a glance

Category App Definition and Development, subcategory Continuous Integration & Delivery (CNCF Landscape)
CNCF status Sandbox. The CNCF project page says it was accepted on 18 December 2023 at the Sandbox maturity level
Origin Created by Red Hat performance and scale engineers for Kubernetes and OpenShift testing; the repository has since moved to a dedicated kube-burner GitHub organization
Written in Go, using the official client-go library. A static binary
License Apache-2.0
Distribution Pre-compiled binaries on the GitHub releases page and a container image on Quay
Latest release Not confirmed in this pass. Check the releases page and pin a version
Job types create, delete, read, patch, kubevirt
Measurements podLatency, jobLatency, nodeLatency, pvcLatency, serviceLatency, netpolLatency, volumeSnapshotLatency, dataVolumeLatency, vmiLatency, vmimLatency, pprof
Indexers opensearch, elastic (Elasticsearch 7), local files, tsdb
Wrapper kube-burner-ocp, an OpenShift-focused wrapper with ready-made workloads. The core tool works on any Kubernetes cluster
State None kept by the tool. Each run has a UUID that tags its results

The problem Kube-burner solves

“It works with 50 pods” tells you little. Control-plane and network problems appear at scale: API server latency, scheduler throughput, etcd size, controller backlogs, DNS and service programming delays.

Ad-hoc scripts are not benchmarks. A shell loop that applies manifests has no rate limit, no readiness accounting, no latency numbers and no comparison across runs.

Performance regressions need evidence. After a Kubernetes, CNI or node-size change you want to know whether pod startup got slower, not guess.

Kube-burner standardizes the load shape, rate and collection so results from different runs can be compared.

Key concepts

Term Meaning
Job One unit of load. It has a type, iteration count, rate limits and a list of objects
Iteration One repetition of a job. By default each iteration gets its own namespace
Object A template (a file path or URL) for a Kubernetes resource and how many replicas of it to create per iteration
Churn Deleting and recreating a share of objects for a period, to test cluster behaviour under change, not just growth
Measurement A latency or profiling collector that runs alongside the jobs, such as pod startup latency
Metrics profile A list of PromQL queries run after the jobs; each entry has a query and a metricName
Alert profile A list of PromQL expressions with a severity and a description; error and critical alerts fail the run
Indexer Where documents go: OpenSearch, Elasticsearch, local files or Prometheus TSDB blocks
UUID Identifier for one run, added to every indexed document
Wrapper A binary such as kube-burner-ocp that embeds ready-made workloads and flags

Architecture

Kube-burner architecture: a job configuration and object templates are read by the kube-burner binary, which uses client-go to create, patch, read or delete objects through the Kubernetes API at a limited rate. It measures latencies, queries Prometheus with a metrics profile and an alert profile, and sends the results to an indexer such as OpenSearch or local files.

Kube-burner creates load through the API, measures it, and indexes Prometheus metrics. Blue is input, purple is kube-burner, grey is a cluster service and green is load or results. Open the zoomable diagram, or download it as SVG or PNG.

Component Responsibility
Configuration and templates Describe jobs, measurements and object templates; the config file itself can be templated with Go templates and environment variables
kube-burner binary Runs jobs, rate-limits requests, waits for readiness, collects measurements and metrics
Kubernetes API server Receives the load. Kube-burner holds no cluster-side component for the core run
Prometheus Source of cluster and node metrics and the engine for alert expressions
Indexer Stores per-run documents for later comparison and dashboards
Helper pods Some features create pods in the cluster: image preloading with a DaemonSet, and the pod used for service latency checks in a kube-burner-service-latency namespace

Design principle. Keep the load generator outside the cluster’s control path, use only the public API, and treat measurements and Prometheus data as first-class outputs, so a run is a benchmark and not just a burst of requests.

One benchmark run

Kube-burner run flow: a run starts with the configuration and an optional cluster health check. Jobs create objects in iterations, limited by queries per second and burst. The run waits for objects to be ready and can churn them. Measurements such as pod latency are collected, then Prometheus metrics and alerts are evaluated and indexed. Error or critical alerts and latency threshold breaches make the exit code non-zero, and created namespaces can be garbage collected.

The exit code and indexed documents tell you whether the cluster met your targets. Open the zoomable diagram, or download it as SVG or PNG.

  1. kube-burner init loads the config. With clusterHealth enabled it first checks that all nodes are Ready.
  2. Each job runs its iterations, creating objects at the configured qps and burst.
  3. The run waits for objects to become ready (depending on the wait settings) and, if configured, churns objects.
  4. Measurements such as podLatency collect timings while this happens.
  5. After the jobs finish, the metrics profile queries run against Prometheus and the alert profile is evaluated. Documents are indexed, with a job summary document at the end.
  6. A latency threshold breach, a failed object verification, or an error or critical alert makes kube-burner exit with code 1. Created namespaces can be garbage collected.

Kube-burner documentation

Install Kube-burner

Commands follow the project’s documentation and guides. I did not run them here. Pin versions.

Binary

Download a release for your platform from the project’s GitHub releases page, make it executable and put it on your PATH. The project README also describes an install script that installs to ~/.local/bin by default (set INSTALL_DIR to change it); review any script before running it.

kube-burner version
kube-burner --help

Container image

A container image is published on Quay (quay.io/kube-burner/kube-burner). Use it to run kube-burner as a Job or pod inside or next to the cluster. Prefer a pinned tag over latest.

Prerequisites

  • A kubeconfig for the cluster, with permission to create and delete the objects in your jobs (cluster-admin is common for lab clusters; scope it down for shared ones).
  • Optional: a Prometheus endpoint (and token) for metrics and alerts, and an OpenSearch or Elasticsearch endpoint if you want to index results.

Using Kube-burner

Snippets are adapted from the documentation and project guides and are untested here. Template variable names follow the project’s examples; check the reference for your version.

A first job

global:
  measurements:
    - name: podLatency

jobs:
  - name: pause-pods
    jobType: create
    jobIterations: 10
    qps: 20
    burst: 40
    namespace: kb-demo
    namespacedIterations: true
    cleanup: true
    podWait: false
    waitWhenFinished: true
    objects:
      - objectTemplate: pod.yml
        replicas: 5

pod.yml is an ordinary Pod manifest, rendered as a Go template for each iteration and replica (the project examples use variables such as the iteration number and replica number in names). A pause container keeps the test about the platform, not the workload.

Run it

kube-burner init -c config.yml

With metrics and indexing, pass a metrics endpoints file:

kube-burner init -c config.yml -e metrics-endpoints.yml
# metrics-endpoints.yml
- endpoint: http://localhost:9090
  token: <token>
  metrics: [metrics.yml]
  alerts: [alerts.yml]
  indexer:
    type: local
    metricsDirectory: collected-metrics

Newer configuration files can also embed a metricsEndpoints list with prometheusURL, metrics, alerts and indexer fields; the reference page shows both styles.

A metrics profile

# metrics.yml
- query: sum(irate(node_cpu_seconds_total[2m])) by (mode,instance)
  metricName: nodeCPU

- query: kube_node_role
  metricName: nodeRoles
  instant: true
  captureStart: true

Range queries use the benchmark (or job) start and end time; instant: true runs an instant query at the last timestamp of each job; captureStart adds a document from the first timestamp with a -start suffix on the metric name.

An alert profile

# alerts.yml
- expr: avg_over_time(histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[2m]))[{{ .elapsed }}:]) > 1
  description: 99th percentile API request latency above 1 second
  severity: error

{{ .elapsed }} is the job duration (or the range passed to check-alerts). Descriptions can include {{$labels.<name>}} and {{$value}}.

Fail a run on slow pod startup

global:
  measurements:
    - name: podLatency
      thresholds:
        - conditionType: Ready
          metric: P99
          threshold: 30s

A threshold breach makes kube-burner exit with code 1, which is what lets a CI job fail on a performance regression.

Re-check alerts later

kube-burner check-alerts -u https://prometheus.example -t "$TOKEN" -a alerts.yml
kube-burner index ...   # collect and index metrics for a past time range
kube-burner measure ... # collect podLatency for resources that still exist

The check-alerts, index and measure subcommands let you evaluate alerts, index metrics or collect pod latency for a past run. Check --help for the exact flags in your version.

Churn

jobs:
  - name: churn-demo
    jobIterations: 20
    churnConfig:
      percent: 10
      cycles: 3
      duration: 10m
      delay: 30s
      mode: namespaces

Churn applies to create jobs and stops when either cycles or duration is reached first. Set cycles: 0 to use only the duration.

Features

Area What you get
Job types Create, delete, read and patch objects, plus a kubevirt job type for virtual machine operations
Rate control qps and burst limit request rates; requestTimeout bounds each client call
Templating Go templates in object files and in the configuration; function templates can be shared; inputVars inject extra values per object
Waiting Per-object wait with custom waitOptions, per-iteration podWait, per-job and global waitWhenFinished (global overrides job, which overrides podWait)
Verification verifyObjects and errorOnVerify check object counts after each job
Churn Namespace- or object-level churn with percent, cycles, duration and delays
Image preload preLoadImages pre-pulls job images with a DaemonSet so pull time does not distort results
Measurements Latency collectors for pods, jobs, nodes, PVCs, services, network policies, VolumeSnapshots, DataVolumes, VMIs and VM migrations, plus pprof profile capture
Metrics PromQL profiles, range or instant queries, multiple Prometheus endpoints
Alerting PromQL alert expressions with info, warning, error, critical severities
Indexing OpenSearch, Elasticsearch 7, local files (optionally a tarball) and Prometheus TSDB blocks; a job summary document per job
Hooks Global and job hooks run before or after jobs
Cluster health and cleanup clusterHealth check before the run; gc to remove created namespaces
Extensibility Custom measurements by implementing the Measurement interface; the core code can be used as a library
OpenShift wrapper kube-burner-ocp bundles workloads such as cluster-density and node-density, with health checks and OpenShift metric profiles

Configuration reference

Setting Level Purpose and default
measurements global or job List of measurements; job-level overrides global
requestTimeout global Client-go request timeout, default 60s
gc, gcMetrics global Garbage collect namespaces; collect metrics during collection. Both default false. Global waitWhenFinished and job gc cannot both be on
clusterHealth global Check all nodes are Ready; default false
timeout global Whole-benchmark timeout, default 4hr
deletionStrategy global default or gvr
jobType job create (default), delete, read, patch, kubevirt
jobIterations job Number of executions, default 1
namespace, namespacedIterations job Base namespace name; one namespace per iteration, default true
cleanup job Remove old namespaces first, default true
qps, burst job Request rate limits, default 0 (unset)
waitWhenFinished, podWait, maxWaitTimeout job Wait behaviour; default max wait 4h
preLoadImages job Pre-pull images, default true
jobPause job Pause after the job, default 0s
verifyObjects, errorOnVerify job Count verification, both default true
churnConfig job percent, cycles, duration, delay, deleteDelay, mode
objects[] job objectTemplate, replicas, kind, apiVersion, inputVars, wait, waitOptions, runOnce, labelSelector, patchType
kube-burner init flags CLI -c config, -e metrics endpoints, plus log level and others; see --help
Indexer fields metrics endpoint type, esServers, defaultIndex, insecureSkipVerify, metricsDirectory, createTarball, tarballName

Production reference design

  • Test a copy of production, not production. A scale test is a deliberate stress. Use a dedicated cluster built like the real one (same instance types, add-ons and CNI) and discard it afterwards.
  • Run from outside the cluster under test (a CI runner or separate node) so the load generator is not competing for the resources you measure.
  • Always collect Prometheus metrics. Latency numbers without API server, etcd and node metrics tell you something is slow but not why.
  • Fix the workload shape (iterations, replicas, qps, burst) and version the config in Git, so runs are comparable.
  • Preload images so image pulls do not dominate pod latency.
  • Set thresholds and alerts for your SLOs so CI fails on regression.
  • Index every run with its UUID and compare against a baseline run.
  • Clean up. Enable gc or delete the test namespaces, and destroy dedicated clusters.
  • Start small, then ramp. Run at a tenth of target scale first to validate the config.

Security

Area What to know
Cluster permissions Kube-burner creates and deletes objects with your kubeconfig. Use a dedicated test cluster and a scoped service account where possible
Destructive by design cleanup and gc delete namespaces. Check names and labels so they never match real workloads
Prometheus access A bearer token or basic auth is used to query metrics. Keep tokens out of the config file and Git
Indexer credentials OpenSearch or Elasticsearch credentials can be part of the server URL. Use secrets management and avoid committing them
TLS insecureSkipVerify exists for lab endpoints; keep it false elsewhere
Templates from URLs Object templates can be loaded from a URL; only use sources you trust
Images Job pods pull images you choose. Pin and scan them, and consider a private mirror for large tests
Cost and quota Large tests consume real cloud resources and can hit API rate limits or service quotas

Monitoring and day-2 operations

Task How
Follow a run Kube-burner logs progress; with indexing on, query documents by UUID
Compare runs Index into OpenSearch and compare latency quantiles and metrics between UUIDs
Re-evaluate alerts kube-burner check-alerts with a Prometheus URL and an alert profile
Index a past window The index subcommand collects and indexes metrics for a time range
Clean up leftovers kube-burner destroy with the run’s UUID, or delete the labelled namespaces
Upgrade Download a newer binary or image; re-run a baseline to separate tool changes from cluster changes
Add a metric Add an entry to the metrics profile
CI integration Run kube-burner init in a pipeline and rely on its exit code
Symptom First checks
Run exits with code 1 Look for a threshold breach, an error or critical alert, or failed object verification
Objects never become ready Image pulls, scheduling capacity, quotas; check maxWaitTimeout
Throttling or timeouts qps, burst and requestTimeout; API server saturation
No metrics indexed Prometheus endpoint and token, metrics profile path, indexer settings
Service latency measurements missing The measurement creates a pod in kube-burner-service-latency; check it can start
Leftover namespaces cleanup and gc settings; run destroy for the UUID
pprof files missing Targets need curl in the pods, plus correct URL and credentials

Usage cases

  • Control-plane scale tests. Create thousands of pods, services and ConfigMaps and watch API server, etcd and scheduler behaviour.
  • Regression gates. Fail a pipeline when pod startup latency or API latency exceeds a threshold after a version change.
  • Comparing configurations. Same workload on different node sizes, CNIs or Kubernetes versions.
  • Network and storage checks. Network policy, service and PVC or snapshot latency measurements.
  • Virtualization workloads. kubevirt jobs and VM latency measurements.
  • Profiling. Capture heap and CPU profiles from control-plane components during load.
  • Cold-chain sprint project. A fleet-scale load test of the Sprint 5 final project’s platform, simulating a large number of devices and workloads.

Operational considerations

  • It generates real load. Never point it at a cluster that serves users.
  • Results depend on the whole setup. Node types, networking, add-ons and Prometheus scrape settings change the numbers, so compare like with like.
  • Cluster-level, not application-level. It tests the platform’s ability to create and run objects; it does not model user traffic to your services. Use an application load tool for that.
  • Docs and flags change. The configuration format evolved (for example metrics endpoint handling), so follow the documentation for the version you pin.
  • OpenShift focus in places. The wrapper and some metric profiles target OpenShift; the core tool is generic.
  • Sandbox maturity. Smaller governance and adoption evidence than graduated tools.

Adoption

Kube-burner was built by Red Hat performance and scale engineers and is used for Kubernetes and OpenShift scale testing. Red Hat’s announcement of the Sandbox acceptance describes it as the first and only performance and scale testing tool to reach that status; that is the vendor’s claim. A 2024 ICPE companion paper presents a benchmarking case study using it, and a DevConf.US 2025 session covers it alongside ClusterLoader2. I did not find a public adopters list in the pages I read, so I am not naming adopters.

Alternatives

Solution Model Best suited to
ClusterLoader2 The official Kubernetes scalability and performance test framework; YAML tests in a semi-declarative style Upstream-style scalability testing and comparisons with Kubernetes project results
KWOK Simulates nodes and pods without running kubelets Testing control-plane and client behaviour at huge scale cheaply; it does not run real workloads, and it can be combined with a load driver
k6, JMeter, Locust Application and HTTP load Testing your service’s performance, not the cluster’s object-handling
Shell scripts and kubectl loops DIY Quick smoke tests; no rate control, measurements or comparison
Cloud-provider scale testing guidance Provider documentation Understanding managed-service limits such as EKS quotas

Strengths and limitations

Strengths Limitations
Repeatable, rate-controlled load with built-in measurements You must design a realistic workload shape
Prometheus metrics and alerts tied to each run Needs Prometheus (and an indexer) for full value
Exit code works as a CI gate Can damage a cluster if pointed at the wrong one
Many job types and latency measurements Configuration format changes between versions
Single binary, any Kubernetes cluster Not a user-traffic or application load tool
OpenShift wrapper with ready workloads Sandbox project with a smaller community

Project status and roadmap

Kube-burner started in 2020 as a Red Hat performance tool, was accepted into the CNCF Sandbox on 18 December 2023, and has continued to add measurements and job types (the docs now list kubevirt, VolumeSnapshot, DataVolume and network policy measurements). I did not find a published roadmap page in this pass, so I am not stating one; follow the releases page and the CNCF project page.

Recommendation

Trial Kube-burner if you run Kubernetes clusters that must hold a known number of pods, services or nodes and you want repeatable evidence: start with one create job and podLatency, add Prometheus metrics and one alert, then wire the exit code into CI. Adopt it as a regression gate once the workload shape and thresholds match your SLOs and a baseline is indexed. Assess ClusterLoader2 if you want the upstream Kubernetes scalability methodology, KWOK if you need to simulate very large node counts cheaply, and an application load tool for service-level performance. Before a big run: use a dedicated test cluster, fix the config in Git, preload images, collect Prometheus metrics, set thresholds, and plan cleanup.

References: Kube-burner documentation · Configuration reference · Measurements · Indexing · Metrics profiles · Alerting · Command line · kube-burner-ocp · CNCF project page · Red Hat: kube-burner joins the CNCF Sandbox · Red Hat: Test Kubernetes performance and scale with kube-burner · Red Hat: Introducing kube-burner · InfoQ: new CNCF Sandbox projects · ICPE 2024 companion paper · ClusterLoader2

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.