Happy Dussehra! May good always win. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Kuberhealthy: Synthetic Monitoring as Kubernetes Pods

CNCF Sandbox. A Kubernetes operator that runs your own synthetic checks as short-lived pods on a schedule, records pass or fail on a HealthCheck resource, and serves the results as a status page, a JSON endpoint and Prometheus metrics. Covers the v3 rewrite, install and a custom check.

16 min read

CNCF Sandbox

  • Kuberhealthy
  • Kubernetes
  • Synthetic Monitoring
  • Prometheus
  • Helm
  • Go
  • AWS EKS
CNCF project page

Most monitoring tells you what components report about themselves. Kuberhealthy checks what a user or another service would actually experience. You describe a check as a Kubernetes resource (a HealthCheck): which container to run, how often, and how long to wait. The operator starts that container as a short-lived pod on schedule, the container tries the real thing (resolve a name, call an API, create a Deployment, read a queue) and reports pass or fail, and the verdict is shown on a status page and exported to Prometheus.

That makes the monitoring itself a set of Kubernetes manifests you can keep in Git, review and deploy like any other workload.

CNCF daily · Sprint 5, post 14 of 14 · Observability

At a glance

Category Observability and Analysis, subcategory Observability (CNCF Landscape)
CNCF status Sandbox. The CNCF project page says it was accepted on 30 March 2021 at the Sandbox maturity level
Origin Started at Comcast and now hosted in the kuberhealthy GitHub organization
Written in Go. Check clients exist for Go, Python, TypeScript, JavaScript, Rust, Ruby, Java and Bash
Current line v3, a complete rewrite. The repository page shows v3.0.0 dated 2 June 2026; the newest patch I found is v3.0.20 (scheduler wake-up fix). Confirm the latest on the releases page
Main resource HealthCheck (one custom resource that is either a single-use check or a scheduled one, and holds its own status)
Outputs Status UI, a /json endpoint and a /metrics endpoint for Prometheus
Install routes Helm chart, Kustomize and an Argo CD manifest, all documented in the repository
State Check status is stored on the HealthCheck resource itself
v2 users v3 breaks compatibility with earlier versions; the repository has a migration guide (MIGRATING_TO_V3.md)

The problem Kuberhealthy solves

Healthy components, broken experience. Pods can be Ready while DNS fails, a network policy blocks a path, an image registry is unreachable or storage cannot be provisioned. Resource metrics do not show that.

Failures found by users. If the only test of “can I deploy a workload and resolve a service name?” is a real deployment, you discover problems at the worst time.

Checks scattered across scripts. Cron jobs and ad-hoc probes have no common scheduling, timeout, status or metrics.

Kuberhealthy gives synthetic checks one shape: a container, a schedule, a timeout, a verdict.

Key concepts

Term Meaning
HealthCheck The custom resource that defines a check: runInterval, timeout and a podSpec for the checker pod. Its status section holds results
Checker pod A short-lived pod created for each run. It runs your validation logic and exits
Run interval How often the check runs, such as 10m
Timeout How long Kuberhealthy waits before it considers a run failed. The default checker pod run timeout is 30 seconds
Check client A small library that reports ReportSuccess or ReportFailure(errors) to the controller
Reporting URL and run UUID Injected into the pod so the controller can match a report to the run that produced it
Reaper A controller component that removes old checker pods
Status UI A live web page for all checks, with a “run now” button and running pod details
Check registry A list of ready-made checks, now kept in separate repositories under the kuberhealthy organization

Architecture

Kuberhealthy architecture: a HealthCheck custom resource is watched by the Kuberhealthy controller, which starts a short-lived checker pod on a schedule. The pod makes real requests to what you test, such as DNS, an API, a queue or storage, and reports the result back. Results appear in a status page, a JSON endpoint and Prometheus metrics.

A HealthCheck starts a short-lived pod on a schedule and records the verdict. Blue is the definition, purple is Kuberhealthy, grey is the checker pod and green is the target or output. Open the zoomable diagram, or download it as SVG or PNG.

Component Responsibility
Controller Watches HealthChecks, schedules runs, creates checker pods, validates reports, writes status
Scheduler Wakes when a run is due; since v3.0.20 it also wakes when a HealthCheck is added, changed or deleted
Checker pods Do the real test and report back
Reaper Cleans up checker pods
Web server Serves the status UI, /json and /metrics
HealthCheck CRD Definition and status in one place

Design principle. Treat a synthetic check as a workload: it is a container, so it can use any language, any image and normal Kubernetes scheduling, resources and service accounts.

One check run

Kuberhealthy run flow: the scheduler fires when the run interval elapses, the controller creates a checker pod with a deadline derived from the timeout, the pod runs your check logic, the pod reports success or errors back with its run UUID, the controller records the status on the HealthCheck, serves it on the UI, JSON and metrics, and a reaper removes old and finished pods.

No report before the deadline counts as a failure. Open the zoomable diagram, or download it as SVG or PNG.

  1. The run interval elapses and the scheduler starts a run.
  2. The controller creates the checker pod from spec.podSpec, with a run UUID and a deadline derived from spec.timeout.
  3. The pod runs your logic.
  4. The pod reports back to the controller (the README shows a POST /check). The check clients handle the reporting URL, run UUID and deadline for you.
  5. The controller records the result in the HealthCheck status. The reaper cleans up old pods.

Kuberhealthy repository and docs

Install Kuberhealthy

Commands follow the repository README. I did not run them here. Pin versions.

git clone https://github.com/kuberhealthy/kuberhealthy
cd kuberhealthy
helm install kuberhealthy deploy/helm/kuberhealthy -n kuberhealthy --create-namespace

The README installs from the chart directory in the repository. Older guides used a published chart repository (helm repo add kuberhealthy ...); that is v2-era, so use the v3 instructions.

Kustomize

kubectl apply -k github.com/kuberhealthy/kuberhealthy/deploy/kustomize/base?ref=main

Replace main with a release tag you have reviewed.

Argo CD

The repository includes deploy/argocd/kuberhealthy.yaml for an Argo CD application.

Look at it

kubectl -n kuberhealthy port-forward svc/kuberhealthy 8080:80
# open http://localhost:8080  (status UI)
# http://localhost:8080/json  (machine-readable status)
# http://localhost:8080/metrics (Prometheus)
kubectl -n kuberhealthy get healthcheck

Using Kuberhealthy

Snippets are adapted from the README and are untested here. The README’s example uses apiVersion: kuberhealthy.github.io/v2 for the v3 HealthCheck; confirm against your installed CRD with kubectl get crd and kubectl explain healthcheck.spec.

A built-in style check

apiVersion: kuberhealthy.github.io/v2
kind: HealthCheck
metadata:
  name: deployment
  namespace: kuberhealthy
spec:
  runInterval: 10m
  timeout: 5m
  podSpec:
    serviceAccountName: deployment-sa
    containers:
      - name: main
        image: <deployment-check image and tag>
        resources:
          requests:
            cpu: 10m
            memory: 50Mi

The deployment check creates a Deployment, waits for it to be ready and cleans up, which tests scheduling, image pulls and the API end to end. Use the image and service account from the check’s own repository.

A custom check

A custom check is any container that does a test and reports. The README’s Go example calls checkclient.ReportFailure([]string{...}) or checkclient.ReportSuccess():

package main

import (
	"time"

	// import path of the Go check client, per the Kuberhealthy docs
	"github.com/kuberhealthy/kuberhealthy/v3/pkg/checkclient"
)

func main() {
	// Example: fail if the newest record is more than 10 minutes old.
	age, err := newestRecordAge() // your own function
	if err != nil {
		_ = checkclient.ReportFailure([]string{err.Error()})
		return
	}
	if age > 10*time.Minute {
		_ = checkclient.ReportFailure([]string{"newest record is older than 10 minutes"})
		return
	}
	_ = checkclient.ReportSuccess()
}

Package it as an image and reference it:

apiVersion: kuberhealthy.github.io/v2
kind: HealthCheck
metadata:
  name: data-freshness
  namespace: kuberhealthy
spec:
  runInterval: 5m
  timeout: 1m
  podSpec:
    containers:
      - name: main
        image: <registry>/data-freshness-check:<tag>
        env:
          - name: STATUS_URL
            value: https://status.example.internal/latest

I could not read the current check creation guide in this pass (the fetch was rate limited), so the import path above and the exact reporting details are unverified. Older v2 documentation describes a non-Go route: POST JSON with an OK flag and an Errors list to the reporting URL, with the run UUID in a header and an environment variable carrying the deadline. Check the v3 guide before relying on it.

Run once

The v3 release notes describe a HealthCheck that can run once as a single-use check or repeatedly on a schedule. I did not verify the field name that selects single-run mode; check kubectl explain healthcheck.spec.

Run now

The status UI has a “run now” button that triggers a check and shows details of the running pod.

Alert on failures

kuberhealthy_check{check="kuberhealthy/deployment", status="0"} == 1

kuberhealthy_check carries check, namespace and status labels. In the older Kubernetes blog’s availability query a status of 0 marks a failing run; confirm the label values on your version before alerting.

Features

Area What you get
One resource HealthCheck for scheduled and single-use checks, with status stored on the resource
Any language Checks are containers; client libraries for Go, Python, TypeScript, JavaScript, Rust, Ruby, Java and Bash
Scheduling and timeouts runInterval and timeout per check; run-now from the UI
Status UI A live-updating web page, served on port 80 of the service
JSON API /json returns an overall ok and a checks map keyed by namespace/name with ok, errors, lastRun and runDuration
Metrics kuberhealthy_check (check, namespace, status), kuberhealthy_check_duration_seconds and kuberhealthy_check_success_total, plus controller metrics
Events and reaping Lifecycle events for checks and a reaper for checker pods
Check registry A registry of community checks, such as AWS IAM role, AMI exists, HTTP, HTTP content and image pull checks listed in the older registry. In v3 checks live in separate repositories under the kuberhealthy organization
Deployment options Helm, Kustomize, Argo CD and a local Podman-based setup for development
Possible built-in probes A proposal (issue 1557) for HTTP and TCP probes run by the controller without a pod was closed as completed on 15 November 2026; I could not confirm it has shipped in a release

Configuration reference

Setting Purpose
spec.runInterval How often a scheduled check runs
spec.timeout When a run counts as failed; the default checker pod run timeout is 30s
spec.podSpec The checker pod template: image, env, resources, serviceAccountName, tolerations and so on
status Written by the controller: outcome, errors, last run, duration and consecutive failure counts (v3.0.19 fixed consecutive failure tracking)
Injected environment A reporting URL, a run UUID and a run deadline, handled by the check clients
Helm values Controller image, service, ServiceMonitor and reaper settings; read deploy/helm/kuberhealthy for your version
Service The controller is reached at svc/kuberhealthy on port 80 in its namespace
Metrics endpoint /metrics on the controller service
Status endpoints / (UI) and /json
v2 settings Older docs describe maxCheckPodAge, maxCompletedPodCount and Influx forwarding in a ConfigMap; these were v2 and I did not confirm which still exist in v3

Production reference design

  • Pick checks that match user journeys. Deploy-and-resolve, DNS, image pull, storage provisioning, and one or two checks against your own critical services.
  • Keep checks cheap and idempotent. A check runs repeatedly and must clean up after itself.
  • Set realistic intervals and timeouts. Short intervals create many pods; set the timeout comfortably above normal run time.
  • Use least-privilege service accounts. A deployment check needs rights to create and delete Deployments in a test namespace, not cluster-admin.
  • Scrape and alert. Add a ServiceMonitor or scrape annotation, and alert on consecutive failures rather than a single blip.
  • Keep checks in Git next to your other manifests and deploy them with Argo CD or Flux.
  • Run the controller with resource requests and a PodDisruptionBudget suited to your cluster.
  • Test failure. Break a dependency in a lab and confirm the alert and status page show it.

Security

Area What to know
Check pods run your code Treat check images like any workload image: pin tags, scan, and use a trusted registry
Service accounts Each check can use its own service account. Grant only the verbs the check needs
Cross-namespace behaviour Checks that create resources should use a dedicated namespace with quotas
Status UI exposure The service is ClusterIP by default in older docs. The UI has a run-now button, so do not expose it publicly without authentication in front
Secrets in checks Pass credentials through Kubernetes Secrets, not environment values in plain YAML
Egress A check that calls external endpoints needs network policy and egress rules, which are also what it tests
Supply chain v3 moved checks into separate repositories; review each check’s source and image before running it
Project health Sandbox maturity and a large v3 rewrite mean you should review recent activity before adopting

Monitoring and day-2 operations

Task How
See all checks Status UI, /json or kubectl get healthcheck -A
Inspect one kubectl describe healthcheck <name> -n <ns> and the checker pod logs while it runs
Trigger now The “run now” button in the UI
Change a schedule Edit the HealthCheck; v3.0.20 makes the scheduler pick up the change immediately
Upgrade Controller and UI images and CRDs together; patch releases state when no migration is needed
Move from v2 Follow the migration guide; the CRD and API changed
Add a check Apply a new HealthCheck manifest
Clean up The reaper removes old pods; delete a HealthCheck to stop its checks
Symptom First checks
Check always fails after 30s Default timeout; raise spec.timeout
Check never runs Controller logs, HealthCheck namespace and service account, scheduler after a recent change
Pod fails to start Image pull, resources, service account and quotas
Metrics missing Prometheus scrape of /metrics and the service monitor
Wrong status after timeout Check pod finished late; compare run duration with the deadline
Too many finished pods Reaper behaviour and check frequency
Upgrade from v2 broke checks v3 changed the CRD; migrate checks

Usage cases

  • Cluster basics. Deploy a workload, resolve DNS, pull an image, provision a volume, on every cluster you run.
  • Platform guardrails. Check that a network policy still allows the paths it should.
  • Dependency checks. HTTP, queue or database reachability from inside the cluster.
  • AWS-specific checks. The older registry lists an IAM role check (can pods call AWS APIs) and an AMI exists check.
  • Release verification. Run a check after an upgrade as a gate.
  • Cold-chain sprint project. Synthetic checks for the Sprint 5 final project, such as confirming that no vehicle has been silent for 10 minutes.

Operational considerations

  • It is synthetic. It tells you the path works at the moment of the run, not what every user saw in between.
  • Each run is a pod. Frequent checks create churn on the scheduler and API server.
  • v3 is a rewrite. Documentation, check repositories and examples from the v2 era may not match.
  • Documentation is moving. Some pages and guides still describe v2 resources and settings.
  • A single controller. Plan its availability like any monitoring component.
  • Check quality is on you. A weak check gives false confidence.
  • Sandbox maturity. Smaller community and governance than graduated projects.

Adoption

Kuberhealthy began at Comcast, and the Kubernetes blog post that introduced its KPI approach was written by Comcast engineers. The repository has an ADOPTERS.md, which I did not read in this pass, so I am not naming adopters here.

Alternatives

Solution Model Best suited to
Prometheus Blackbox Exporter Probes HTTP, TCP, DNS and ICMP endpoints from an exporter Endpoint probing when you do not need checks that create Kubernetes resources
Grafana Cloud synthetic monitoring Hosted probes and agents Teams already on Grafana Cloud who want external probe locations
Commercial synthetic monitoring (Datadog, New Relic and others) SaaS Teams that prefer a managed service
Kubernetes CronJobs with scripts DIY Simple one-off checks; no common status or metrics
Argo Workflows or Tekton on a schedule Workflow engines Multi-step test pipelines
End-to-end test suites CI-driven Pre-release validation rather than continuous production checks

Strengths and limitations

Strengths Limitations
Checks are Kubernetes manifests in Git v3 is new; docs and tooling still catching up
Any language and any container Each run creates a pod
One resource holds definition and status Single controller to run and protect
Status page, JSON and Prometheus metrics Synthetic, so it samples rather than observes every request
Strong fit for cluster-level validation You write and maintain the checks
CNCF Sandbox project Smaller community than graduated tools

Project status and roadmap

Kuberhealthy began at Comcast, was accepted into the CNCF Sandbox on 30 March 2021, and was completely rewritten for v3, which has one HealthCheck resource, a live status UI, a reaper, Prometheus metrics and new deployment options. Patch releases since then fix scheduler and failure-tracking behaviour. I did not find a published roadmap page in this pass. The repository has a TODO.md that, according to an issue, listed ideas such as port and HTTP content checks, but I did not read it.

Recommendation

Trial Kuberhealthy if you run Kubernetes clusters and want continuous end-to-end validation (deploy, DNS, image pull, storage, key service paths) expressed as manifests with Prometheus alerts. Adopt it for a defined set of cluster-level checks once the failure alerts are tested and the check images and service accounts are reviewed. Assess Blackbox Exporter if you only need endpoint probing, a hosted synthetic service if you want external vantage points, and a workflow engine if checks are long multi-step pipelines. Before production: confirm the v3 install and CRD version, set realistic timeouts, scope service accounts, scrape the metrics, alert on consecutive failures, and test one failure end to end.

References: Kuberhealthy repository · v3.0.0 release notes · v3.0.5 release notes · v3.0.20 release · v3.0.19 release · HTTP and TCP check proposal · Scheduler wake issue · CNCF project page · Check creation guide (v2 era) · Checks registry · Kubernetes blog: K8s KPIs with Kuberhealthy · InfraCloud: synthetic monitoring with Kuberhealthy

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.