OpenKruise: In-Place Updates and Advanced Workloads for Kubernetes

OpenKruise is a set of controllers and custom resources that extend the workload features of Kubernetes. Its CloneSet works like a Deployment but can update a container’s image in place, without recreating the pod, and its Advanced StatefulSet, Advanced DaemonSet, SidecarSet and image pre-pull resources give operators control that the built-in workloads do not provide. It addresses the cost of rolling updates at scale: recreating a pod reschedules it, repulls images and restarts every container, which is slow and disruptive for large fleets and heavy images. OpenKruise grew out of Alibaba Cloud’s internal practice of running very large Kubernetes estates. It joined the CNCF Sandbox on November 10, 2020, and moved to Incubating on March 2, 2023.

CNCF daily · Sprint 3, post 2 of 21 · Continuous Integration & Delivery

At a glance


Category	App Definition and Development › Continuous Integration & Delivery
CNCF status	Incubating. Accepted November 10, 2020; incubating March 2, 2023
Written in	Go
License	Apache 2.0
Core workloads	CloneSet, Advanced StatefulSet, Advanced DaemonSet, BroadcastJob, AdvancedCronJob
Sidecar and image tools	SidecarSet, ImagePullJob, ContainerRecreateRequest
Placement and safety	WorkloadSpread, PodUnavailableBudget, ResourceDistribution
Progressive delivery	Kruise Rollouts, a companion project for canary and blue-green releases

Architecture

flowchart TB
  USER[Platform or GitOps tool<br/>applies manifests] ==>|CloneSet SidecarSet| API[Kubernetes API server]
  API ==>|watch custom resources| MGR[kruise-manager<br/>controllers and webhooks]
  MGR ==>|create or patch pods| POD[Application pods]
  MGR -.->|image update in place| KLT[kubelet on node]
  KLT ==>|restart one container| POD
  MGR -.->|ImagePullJob| DMN[kruise-daemon<br/>on every node]
  DMN ==>|pre pull images| CRT[Container runtime]
  API -.->|admission webhook| MGR
  MGR -.->|inject sidecar| POD
  RO[Kruise Rollouts<br/>canary and blue green] -.->|partition and traffic steps| MGR
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  class MGR,DMN,RO core
  class API,POD,KLT,CRT data
  class USER actor

Key: teal blocks are OpenKruise components, blue are Kubernetes parts, grey is the delivery tool. Thick arrows show the main control path, dashed arrows webhooks and helpers.

Component	Responsibility
kruise-manager	Runs the controllers for all OpenKruise resources and the admission webhooks that inject sidecars and validate settings.
kruise-daemon	Runs on each node. Pulls images ahead of time and performs container restarts needed for in-place updates.
CloneSet	Manages stateless pods with in-place update, partition-based canaries, `maxUnavailable` and `maxSurge`, and pod deletion priority.
SidecarSet	Injects and upgrades sidecar containers independently of application releases.
Kruise Rollouts	Adds canary, blue-green and A/B release steps with traffic routing and analysis hooks.

Design principle. Change as little as possible when you update. If only an image changes, the pod’s identity, network address, volumes and scheduling placement can stay the same, so the update restarts one container instead of replacing the pod.

Production reference design

A common deployment installs OpenKruise on a cluster that runs many replicas of heavy services, then moves selected workloads from Deployment to CloneSet.

Install the controllers. Use the project Helm chart and pin the version:

helm repo add openkruise https://openkruise.github.io/charts/
helm install kruise openkruise/kruise \
  --version <version> --namespace kruise-system --create-namespace

Create a CloneSet. A web service with in-place updates and a canary partition:

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata: { name: web, namespace: shop }
spec:
  replicas: 10
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: registry.example.com/shop/web:1.4.0
  updateStrategy:
    type: InPlaceIfPossible
    partition: 8
    maxUnavailable: 20%

With partition: 8, a new image reaches 2 of 10 pods first. Lowering the partition releases the rest in steps.

Pre-pull images. An ImagePullJob loads the next image onto selected nodes before the rollout, removing pull time from the update window.
Manage sidecars centrally. A SidecarSet injects and upgrades a logging or proxy sidecar for matching pods without editing every application chart.
Add progressive delivery. Install Kruise Rollouts, or call the partition from a GitOps controller such as Argo CD, and gate each step on metrics.

Typical uses include large stateless fleets, game servers and other stateful workloads that should keep their identity, image-heavy applications, and organisations that need sidecars upgraded on their own schedule. Verify API versions and field names against the documentation for your release.

Operational considerations

In-place limits. Only changes to container images and some metadata can be applied in place. Changes to resources, volumes or the pod spec still recreate the pod, so test which path an update takes.
Webhooks are on the critical path. Pod creation passes through the OpenKruise webhooks. Run the manager with replicas, and set failure policies deliberately.
Adoption path. Existing Deployments can be converted to CloneSets with care. Plan the migration so scale and selectors are preserved.
GitOps interplay. A GitOps tool that reverts the partition will fight a manual canary. Decide whether Git or Kruise Rollouts owns the rollout steps.
Observability. Track update progress, ready replicas and in-place update counts, and alert on stalled partitions.

Adoption

Alibaba Cloud created OpenKruise from its internal workload management and runs it at very large scale.
Ant Group is a long-standing contributor and user.
The project maintains an ADOPTERS file in its repository listing further organisations that run OpenKruise in production; consult it for current names.

Alternatives

Solution	Model	Best suited to
Kubernetes Deployment and StatefulSet	Built-in workloads	Most applications; recreate pods on every change
Argo Rollouts	Progressive delivery controller	Canary and blue-green releases with analysis; see the Argo brief
Flux with Flagger	GitOps with progressive delivery	Git-driven releases with automated canaries; see the Flux brief
Tekton or CI pipelines	Build and release automation	Building and promoting images rather than running workloads; see the Tekton brief
Custom operators	In-house controllers	Highly specific workload behaviour

Strengths and limitations

Strengths	Limitations
In-place updates keep pod identity and cut rollout time	Adds a second layer of workload controllers to learn and run
Image pre-pulling removes pull latency from releases	Only some changes can be applied in place
Central sidecar management with SidecarSet	Tools expecting Deployment objects may need adapting
Fine-grained partitions, surge and unavailability control	Incubating project with a community concentrated in a few regions
Used at very large scale in production	Webhook and controller availability become critical

Recommendation

Assess. OpenKruise earns its place when rollout speed, image size or sidecar management is a measured problem, typically in large fleets. Start with one stateless service on a CloneSet and an ImagePullJob, compare update time and disruption with a Deployment, and adopt it further if the gain is real. Smaller clusters usually do well with standard Deployments and a progressive delivery tool.

References: CNCF project page · OpenKruise website · OpenKruise repository · CNCF Landscape

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.