Dragonfly is a peer-to-peer (P2P) distribution system for container images, AI model weights and other large files. Instead of every node downloading the same layer from a registry, nodes download pieces from each other, so the registry and network serve each piece only a few times however many nodes need it. It addresses a problem that appears at scale: when hundreds or thousands of nodes pull the same large image at once, during a rollout, an autoscaling burst or the start of an AI training job, the registry and its bandwidth become the bottleneck. Alibaba Group open-sourced Dragonfly in November 2017. It joined the CNCF sandbox in 2018, reached incubating status in 2020, and graduated in October 2025, with the graduation announced in January 2026.
CNCF daily · Sprint 2, day 6 · Container Registry
At a glance
| Category | Provisioning › Container Registry |
| CNCF status | Graduated. Accepted November 13, 2018; incubating April 9, 2020; graduated October 28, 2025 (announced January 14, 2026) |
| Written in | Go for the manager and scheduler; Rust for the client (dfdaemon) |
| License | Apache 2.0 |
| Distributes | OCI container images, AI model weights and artifacts, and general files over HTTP |
| Integrations | containerd, CRI-O, Docker, Harbor preheating, Nydus lazy-loading images |
| Subproject | Nydus, an image format and filesystem that lets containers start before the whole image is downloaded |
Architecture
flowchart TB
RT[containerd or CRI-O<br/>on each node] ==>|image pull via proxy| P1[Peer dfdaemon<br/>node A]
P1 <==>|pieces| P2[Peer dfdaemon<br/>node B]
P1 <==>|pieces| P3[Peer dfdaemon<br/>node C]
P1 -.->|which peers have it| SCH[Scheduler<br/>builds the P2P tree]
P2 -.-> SCH
SCH -.->|cache miss| SEED[Seed peer]
SEED ==>|back to source once| REG[Registry or object storage<br/>Harbor, ECR, S3]
SEED ==>|pieces| P1
MGR[Manager<br/>clusters, preheat, console] -.-> SCH
MGR -.-> SEED
classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
classDef ctrl fill:#ede7f6,stroke:#5e35b1,stroke-width:1.5px,color:#1d1c1a
classDef store fill:#fff3e0,stroke:#e65100,stroke-width:1.5px,color:#1d1c1a
class RT actor
class P1,P2,P3,SEED core
class SCH,MGR ctrl
class REG store
Key: teal blocks are the project’s data path, purple are its control components, amber is storage, grey is the container runtime. Thick arrows show data transfer, dashed arrows scheduling and control.
| Component | Responsibility |
|---|---|
| Peer (dfdaemon) | Runs on every node, usually as a DaemonSet. Acts as an HTTP proxy for the container runtime, splits files into pieces, downloads pieces from other peers and serves pieces it already has. |
| Scheduler | Tracks which peers hold which pieces and tells each downloading peer which parents to fetch from, forming an efficient P2P tree. |
| Seed peer | Downloads content from the origin (registry or object storage) when no peer has it yet, then serves it to the cluster. |
| Manager | Manages clusters of schedulers and seed peers, provides the console and API, and runs preheating: pulling content into seed peers before it is needed. |
| dfget | Command-line client for downloading files through the P2P network. |
Design principle. The origin should serve each piece once per cluster, not once per node. Every node that downloads a piece becomes a source for the next, so total capacity grows with the number of nodes instead of being capped by the registry’s bandwidth.
Production reference design
A typical deployment is a large Kubernetes cluster (EKS, GKE, AKS or on-premises) pulling images from Harbor or a cloud registry.
- Install a pinned chart version with Helm. The chart deploys the manager, scheduler, seed peers and the dfdaemon DaemonSet:
helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/ && helm repo update helm install dragonfly dragonfly/dragonfly -n dragonfly-system --create-namespace -f values.yaml kubectl -n dragonfly-system get pods - Route image pulls through the local dfdaemon. For containerd, a registry host configuration sends pulls to the dfdaemon proxy on the node, which falls back to the registry if Dragonfly is unavailable:
The Helm chart can write this configuration on each node automatically.# /etc/containerd/certs.d/docker.io/hosts.toml server = "https://index.docker.io" [host."http://127.0.0.1:4001"] capabilities = ["pull", "resolve"] [host."http://127.0.0.1:4001".header] X-Dragonfly-registry = ["https://index.docker.io"] - Preheat before rollouts. Use the manager API, or Harbor’s built-in P2P preheat policies with Dragonfly as the provider, to load new images into seed peers before a large deployment.
- Consider Nydus images for very large images. Containers start while the image is fetched on demand, and Dragonfly distributes the chunks.
- Monitor the Prometheus metrics for P2P traffic ratio, back-to-source downloads and piece download latency.
Typical uses include clusters that roll out new image versions to hundreds of nodes at once, batch and autoscaling workloads that start many identical pods together, and AI platforms that must distribute model weights of tens or hundreds of gigabytes to many GPU nodes quickly.
Operational considerations
- It only pays off at scale. On a cluster of ten nodes pulling small images, Dragonfly adds components with little benefit. The gains grow with node count, image size and how many nodes pull the same content at once.
- Fallback path. Configure the runtime so that pulls fall back to the registry if dfdaemon is down; otherwise a Dragonfly outage can block every image pull.
- Node disk and network. Peers cache pieces on local disk and serve them to other nodes. Size the cache, set eviction limits, and account for east-west traffic between nodes.
- Security. Peers exchange content with each other, so restrict network access to the peer ports, keep content integrity checks enabled, and handle registry credentials carefully for private images.
- Scheduler and seed peer availability. Run several replicas of the scheduler and seed peers. If they are unavailable, peers fall back to downloading from the origin, which is safe but loses the P2P benefit.
Adoption
- The CNCF graduation announcement names Ant Group, Alibaba, Datadog, DiDi and Kuaishou as adopters, and reports production use supporting tens of millions of container launches a day.
- The same announcement reports storage bandwidth savings of up to 90 percent and image launch times reduced from minutes to seconds, and describes distributing AI model weights at hundred-terabyte scale to hundreds of nodes in minutes.
- Contributors grew from 45 people at 5 companies at incubation to 271 people at more than 130 companies by graduation.
Alternatives
| Solution | Model | Best suited to |
|---|---|---|
| Kraken (Uber) | P2P Docker registry | Teams wanting a P2P registry rather than a distribution layer in front of one |
| Spegel | Stateless node-to-node image sharing using containerd’s local store | Simpler clusters that want P2P pulls without extra control-plane components |
| Registry pull-through caches (Harbor proxy cache, a local mirror) | A cache close to the cluster | Moderate scale, where one nearby cache removes most external traffic |
| Lazy-loading formats (Nydus, eStargz, SOCI) | Start containers before the full image arrives | Large images where start time matters more than total transfer; combinable with P2P |
Strengths and limitations
| Strengths | Limitations |
|---|---|
| Registry load and egress stay nearly flat as node count grows | More components to run: manager, schedulers, seed peers, DaemonSet |
| Large speed-ups for mass rollouts and AI model distribution | Little benefit in small clusters or with small images |
| Works with containerd, CRI-O, Docker and Harbor preheating | Adds east-west network traffic and local disk use on every node |
| Nydus subproject adds lazy loading for faster starts | Pull path depends on correct runtime proxy configuration and fallback |
| Graduated project, proven at very large scale | Most large public adopters are in China; fewer case studies elsewhere |
Recommendation
Assess, and Adopt at scale. For clusters with hundreds of nodes, frequent mass rollouts or large AI models, Dragonfly can cut image distribution time and registry egress dramatically, and it is a proven, graduated project. For small and medium clusters, a pull-through cache or Spegel is usually enough. Introduce Dragonfly with registry fallback enabled, measure the P2P ratio and pull times, and expand once the gains are clear.
References: CNCF project page · Dragonfly documentation · Dragonfly repository · CNCF graduation announcement · Nydus · CNCF Landscape