Vector: A Rust Observability Data Pipeline to Collect, Normalise, Filter and Route Telemetry

Vector is an open-source observability data pipeline. It reads events (logs, metrics and, in newer versions, traces) from sources such as files, Kubernetes pods, Kafka topics or HTTP endpoints, passes them through transforms that parse, rename, enrich, filter or sample them, and sends them to sinks such as Kafka, Prometheus, Loki, object storage or a time-series database. Everything is one static binary written in Rust, configured in YAML, TOML or JSON, and you run it either as an agent on every node or as an aggregator that many agents and services send to. It was started by Timber and has been maintained by Datadog since the 2021 acquisition; the code is licensed under MPL-2.0. NON CNCF: Vector is not a CNCF project. It appears as a card in the CNCF Landscape, and this brief covers it as a Landscape item. Why it is here: this slot was planned for Tremor, an event-processing tool in the CNCF Sandbox. Its last stable release is 0.12 and the 0.13 line stopped at release candidates, its repository was last updated in July 2025, and the CNCF page shows falling contributor numbers, so I swapped it for a tool that is actively released. This brief explains Vector from the problem it solves to installing, configuring, securing and operating it.

CNCF daily · Sprint 5, post 3 of 14 · Observability · NON CNCF

At a glance

Category	Observability and Analysis (Landscape listing as “Vector”, by Timber; confirm the exact subcategory on the Landscape page)
CNCF status	NON CNCF. Listed in the CNCF Landscape. Maintained by Datadog’s Community Open Source Engineering team
Written in	Rust
License	MPL-2.0
Model	A directed graph of sources, transforms and sinks, each with an ID, wired by inputs
Transform language	Vector Remap Language (VRL), checked when the config loads
Roles	Agent (collect at the edge) and aggregator (process centrally), or both in stages
Delivery	At-least-once where sources and sinks support end-to-end acknowledgements. No exactly-once
Config formats	YAML, TOML, JSON; one file, several files or a directory
Command line	vector validate, vector test, vector top, vector tap, vector vrl
Release cadence	A minor release every two to six weeks. 0.61.0 shipped on October 29, 2026; check the releases page for the current one
Install	Install script, apt/yum packages, Docker, Helm chart from https://helm.vector.dev

The problem Vector solves

Telemetry usually leaves a system through whatever agent the destination vendor provides. Each agent has its own format, its own filters and its own failure behaviour, and changing the destination means redeploying agents everywhere. The same raw events also carry fields you do not want stored (tokens, noisy debug lines, health checks), or arrive in shapes that no two teams agree on.

Vector puts one pipeline between producers and destinations. Producers send raw events to it, you decide in one place how to normalise, drop, sample or enrich them, and you choose the destinations by config, not by agent. That makes it a natural fit for a processing hop in a streaming design: read from a message bus, clean the data once, and write it to the stores that need it.

Key concepts

Term	Meaning
Event	The unit Vector moves: a log (a structured object), a metric or a trace.
Source	Where events come from: file, kubernetes_logs, kafka, http_server, host_metrics, vector and many more.
Transform	A step that changes the stream: remap (VRL), filter, route, reduce, sample, dedupe, lua and others.
Sink	Where events go: kafka, influxdb_logs, prometheus_exporter, loki, aws_s3, console and more.
Component ID	The unique name of each component; a downstream component lists the IDs it reads in inputs.
VRL	A small expression language for events. It is not Turing complete and is type-checked at compile time, so many mistakes show up at startup, not at 3 a.m.
Buffer	A queue in front of each sink: in memory (default) or on disk, with when_full set to block or drop_newest.
Acknowledgements	End-to-end tracking, enabled per sink, that lets a source commit its position only after the sink delivered.
Agent and aggregator	Roles, not different programs: the same binary on every node, or as a central service.
Enrichment table	A lookup (file, CSV, geoip) that a transform can query to add fields.

Architecture

The diagram shows one aggregator pipeline; agents on nodes can feed it through the vector sink and source. Each sink has its own buffer. Purple is a Vector component, blue is a producer, dark grey is Kafka and green is the destination. Open the zoomable diagram, or download it as SVG or PNG.

Component	Responsibility
Source	Reads from the outside world and turns input into events. A kafka source joins a consumer group and tracks offsets.
remap transform	Runs a VRL program on each event: parse JSON, rename fields, convert units, add tags, drop on error or abort.
filter transform	Keeps an event only if a VRL condition is true. Everything else is dropped.
route transform	Sends events to named outputs by condition, so one stream can feed different sinks.
Sink	Batches and delivers events, retries on failure, and reports to its source when acknowledgements are on.
Buffer	Absorbs bursts and slow destinations. Memory is fast; disk survives a restart for events already written to it.
Top-level options	data_dir (where disk buffers and state live), api (for vector top and vector tap), secret backends, enrichment tables and global acknowledgements.

Design principle. Treat the pipeline as a graph of small, named steps. Every step has one job, every step can be tested without the others, and delivery guarantees come from how a source and a sink are paired, not from a separate system.

From a raw reading to a delivered point

The second diagram follows one telemetry message through an aggregator.

1. Publish. A producer writes a raw message to a Kafka topic.
2. Read. The kafka source, part of a consumer group, receives it as an event.
3. Normalise. A remap transform parses the payload and sets consistent field names, units and tags.
4. Filter. A filter transform drops events that fail a condition, such as a missing device ID or a value outside a plausible range.
5. Buffer. The sink puts the surviving events in its buffer and batches them.
6. Write. The sink sends the batch to the database.
7. Acknowledge. When the sink reports delivery, the source commits the Kafka offset. If delivery fails, the offset is not committed and the message can be read again.

Open the zoomable flow diagram, or download it as SVG or PNG.

Install Vector

On a host

The quick start uses an install script; review it before you pipe it to a shell:

curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash
vector --version

Packages are also published for apt and yum (set up the repository with bash -c "$(curl -L https://setup.vector.dev)", then sudo apt install vector). The package installs the config at /etc/vector/vector.yaml and a vector systemd service.

On Kubernetes

The official Helm chart supports three roles:

helm repo add vector https://helm.vector.dev
helm repo update
helm install vector vector/vector \
  --namespace vector --create-namespace \
  --values values.yaml

role	Workload	Use it for
Agent	DaemonSet	Collecting logs and host metrics on every node
Aggregator	StatefulSet (the chart default)	Central processing that needs persistent storage for disk buffers
Stateless-Aggregator	Deployment	Central processing without disk buffers

Put the pipeline in the chart’s customConfig value. Check the chart’s values.yaml for your version before you rely on a key; chart options change between releases.

Using Vector

A first pipeline you can validate

Save this as vector.yaml. It generates sample events, so nothing outside the machine is needed:

data_dir: /var/lib/vector

sources:
  demo:
    type: demo_logs
    format: json
    interval: 1

transforms:
  shape:
    type: remap
    inputs: [demo]
    source: |
      . = parse_json!(.message)
      .ingested_by = "vector"

sinks:
  out:
    type: console
    inputs: [shape]
    encoding:
      codec: json

vector validate vector.yaml
vector --config vector.yaml

vector validate checks the config, including the VRL, without starting the pipeline. I have not run these commands live for this brief; the field names come from the official component reference, so confirm them against the version you install.

Normalise and filter with VRL

A pattern for sensor-style JSON messages: parse, normalise, then drop what you do not trust.

transforms:
  normalise:
    type: remap
    inputs: [from_kafka]
    drop_on_error: true
    reroute_dropped: true
    source: |
      . = parse_json!(.message)
      .device_id = to_string!(.device_id)
      .temp_c = to_float!(.temp_c)
      .ts = parse_timestamp!(.ts, format: "%+")

  keep_valid:
    type: filter
    inputs: [normalise]
    condition: |
      exists(.device_id) && .temp_c > -60.0 && .temp_c < 60.0

With reroute_dropped: true, events that fail a remap go to a separate normalise.dropped output, so you can send them to a dead-letter sink and not lose the evidence. The ! after a function means “abort the event if this fails”; without it VRL makes you handle the error, which is why many mistakes surface at compile time.

Read from Kafka and write to a time-series database

sources:
  from_kafka:
    type: kafka
    bootstrap_servers: broker-1:9093,broker-2:9093
    group_id: vector-normaliser
    topics: [telemetry.raw]
    auto_offset_reset: earliest
    tls:
      enabled: true
      ca_file: /etc/vector/tls/ca.pem

sinks:
  tsdb:
    type: influxdb_logs
    inputs: [keep_valid]
    endpoint: http://tsdb.example.internal:8086
    database: telemetry
    measurement: readings
    tags: [device_id]
    acknowledgements:
      enabled: true
    buffer:
      type: disk
      max_size: 1073741824
      when_full: block

The influxdb_logs sink speaks the InfluxDB line protocol. A database that accepts that protocol on a /write endpoint, such as openGemini, which documents HTTP writes of line protocol to /write?db=..., is a plausible target. I have not tested Vector’s sink against openGemini; the sink’s exact behaviour (field typing, tag handling, authentication options) should be checked against your database with a throwaway pipeline before you build on it. Part 2 of the Sprint 5 project is where this wiring gets built and checked.

Test the config in CI

tests:
  - name: drops out-of-range readings
    inputs:
      - insert_at: normalise
        type: log
        log_fields:
          message: '{"device_id":"v1","temp_c":99,"ts":"2026-11-12T10:00:00Z"}'
    outputs:
      - extract_from: keep_valid
        conditions:
          - type: vrl
            source: "false"

Run vector test vector.yaml in the pipeline. Unit tests inject events at a named component and assert on the output of another, so routing and filtering logic gets regression-checked like code. Treat the exact test syntax above as a sketch and check it against the unit-testing reference for your version.

Reload without a restart

Send SIGHUP to the process, or start Vector with --watch-config and it reloads when the file changes (--watch-config-method poll for network filesystems). vector validate --deny-warnings in your deploy step catches mistakes before they reach a node.

Features

• One graph for logs, metrics and traces, with sources, transforms and sinks for the common cloud, message bus and storage systems.
• VRL, a purpose-built, type-checked language for reshaping events, plus Lua for the cases VRL does not cover.
• Agent and aggregator roles, with vector source and sink to chain instances.
• Delivery controls. Per-sink buffers (memory or disk), block or drop_newest when full, retries and end-to-end acknowledgements.
• Built-in testing and validation. vector validate, unit tests with vector test, and vector vrl to try expressions.
• Live inspection. vector top shows events in and out per component, and vector tap samples events in flight (both need the api block enabled).
• Secrets backends. SECRET[backend.key] placeholders resolved from a file, a directory, an exec command or AWS Secrets Manager.
• Enrichment tables for lookups inside transforms.
• Hot reload by signal or --watch-config.

Production reference design

This design runs a two-stage pipeline on Kubernetes: agents on nodes, a small aggregator tier that does the real work, and a message bus in between when the data matters.

1. Pick roles. Agents (DaemonSet) only collect, add host metadata and forward. Do heavy parsing, filtering and fan-out in the aggregator so you change the logic in one place.
2. Decouple with a bus when loss matters. For data that must not be dropped, have producers or agents write to Kafka (or a similar log) and let the aggregator consume it. A consumer that has not committed its offset can read again after a crash.
3. Pair sources and sinks for delivery. Turn on acknowledgements on each sink and use a source that supports them. Vector checks this at startup and warns when a sink has acknowledgements but its source cannot take part.
4. Choose buffers on purpose. Use disk buffers with when_full: block for sinks whose events must survive restarts; use memory buffers where freshness matters more and a short loss is acceptable. Back-pressure then flows to the source instead of data being discarded.
5. Run the aggregator as a StatefulSet (or a Deployment if you use no disk buffers), with at least two replicas and requests and limits set from a load test.
6. Keep configs in Git. Validate and unit-test on every change, then roll out with your GitOps tool.
7. Send the dropped. Route events that fail remap to a dead-letter sink so bad inputs are visible.
8. Watch Vector itself. Enable its internal_metrics source and export them with a Prometheus sink; alert on buffer size, discarded events and sink errors.

Security

• TLS on every network edge. Kafka, HTTP and Vector-to-Vector connections all take tls options (CA, certificate and key files); Kafka also supports SASL.
• Keep secrets out of the config. Use SECRET[backend.key] with the file, directory, exec or AWS Secrets Manager backend, or environment variables. Mount the secrets read-only and restrict who can read them.
• Disk buffers can hold plaintext. Events buffered to disk are stored unencrypted, and so can any secrets inside them; protect the data_dir volume and encrypt it at the storage layer.
• Log levels. Verbose logging can expose secret values; keep production logging at normal levels.
• Filter before you store. A remap that deletes tokens, passwords or personal fields is a cheap privacy control, if applied before the first sink.
• Least privilege on Kubernetes. The agent role needs read access to pod metadata through a service account; give it nothing more.
• Supply chain. Pin image tags or digests and review the install script before you run it on hosts.

Monitoring and day-2 operations

Task	How
Is the config valid?	vector validate --deny-warnings --config <file>, in CI and before each reload
What is flowing?	vector top for events in and out per component; vector tap <component> for sample events. Both need the api block enabled
Vector’s own health	The internal_metrics source plus a Prometheus-compatible sink; also internal_logs
Change config	Edit and SIGHUP, or run with --watch-config; restart if a change needs it
Upgrade	Read the release notes (they list breaking changes) and test with vector validate on the new version first. The observability API moved from GraphQL to gRPC in 0.55, which affects vector top and vector tap
Replay	Reset the consumer group offset on Kafka and let the aggregator read again; sinks must tolerate duplicates

Symptom	Check
Service will not start	journalctl -u vector, then vector validate; check file permissions and data_dir
Source reads but sink is silent	Filter conditions that drop everything, a sink health check failing, or a route with no matching output
Memory grows	Sink is slower than the source with a memory buffer; use a disk buffer, add replicas, or tune batch settings
Events missing after a restart	A memory buffer was in use, or acknowledgements were not enabled on the sink, or the source cannot acknowledge
Duplicates downstream	Expected with at-least-once delivery after a retry or a replay; make the destination idempotent (for example by writing the same key and timestamp)

Usage cases

• Log and metric collection on every node. An agent per host or node, with a few transforms before the data leaves.
• A central aggregator. One tier that parses, enriches, samples and fans out to several destinations.
• Cost control. Drop or sample low-value logs, and turn logs into metrics, before paying to store them.
• Vendor-neutral routing. Change a destination in config without touching producers.
• Stream processing on a bus. Read from Kafka or Pulsar, normalise, and write to storage.
• Redaction and compliance. Remove or hash fields before data reaches a store.
• In the Sprint 5 project. The final project of this sprint, a cold-chain logistics monitoring platform on AWS, uses Vector to normalise and filter telemetry between Kafka and the time-series store.

Operational considerations

• At-least-once, not exactly-once. Design destinations to accept a repeated event.
• Acknowledgements need both ends. Enabling them on a sink helps only if its source supports them. Enabling them on the source alone has no effect.
• Disk buffers are not a guarantee by themselves. They protect events already written to them; end-to-end delivery still depends on acknowledgements and on the destination.
• Agents can lose data. An agent with a memory buffer loses what is in memory if it crashes; use a bus or a disk buffer for data that matters.
• VRL is deliberately small. It has no loops or unbounded recursion; use Lua or preprocess elsewhere for anything complex.
• Fast releases. A minor release every few weeks needs a regular upgrade routine and a CI validate step.
• Not a store or a query tool. Vector moves data; it does not keep it or let you search it.
• Governance. Vector is maintained by a single company’s team under MPL-2.0. That is a different governance model from a CNCF project, and you should weigh it.

Adoption

• Datadog maintains Vector, and the project is the open source base of Datadog’s Observability Pipelines product, according to Datadog’s open source site.
• Datadog’s open source site also names contributors from other companies, for example a Kubernetes platform provider that uses Vector for log shipping in its platform.
• I did not find an official public adopters list for Vector, so ask the community for current references and review production reports before relying on it at large scale.

Alternatives

Solution	Model	Best suited to
OpenTelemetry Collector	Receivers, processors and exporters for OTLP-first telemetry; see the OpenTelemetry brief	Teams standardising on OTLP and the CNCF ecosystem
Fluent Bit	Lightweight log and metric forwarder with filters	Small footprint log collection at the edge
Fluentd	Ruby log collector with a large plugin set; see the Fluentd brief	Existing Fluentd estates and plugin coverage
Logstash	JVM pipeline in the Elastic stack	Teams already running the Elastic stack
Kafka Connect and Kafka Streams	Connector and stream processing inside Kafka; see the Strimzi brief for running Kafka	Processing that belongs inside the Kafka platform
Tremor	CNCF Sandbox event-processing engine with its own query language	Projects willing to adopt a language and a slower-moving codebase; I did not choose it because its recent activity is low
Cloud pipelines	Provider-managed ingest and routing	Teams that want no pipeline to operate

Strengths and limitations

Strengths	Limitations
One binary for agent and aggregator roles	Not a CNCF project; governance sits with one vendor’s team
VRL is type-checked, and config is validated before it runs	At-least-once only; duplicates are your problem
Built-in unit tests, validate, top and tap	Delivery guarantees depend on the source and sink pair and need care to set up
Broad set of sources and sinks	Frequent releases and some breaking changes
Small, fast Rust runtime	Traces support is newer than logs and metrics
Disk buffers and back-pressure	Disk buffers need protecting and sizing

Project status and roadmap

Vector is actively released: 0.61.0 shipped on October 29, 2026, following 0.60.0 on October 19 and 0.59.0 on October 6. The repository sits at roughly 22,800 GitHub stars and was updated on November 5, 2026. Datadog’s Community Open Source Engineering team maintains it. I did not find a single published roadmap page; use the release notes and the issue tracker to see what is planned, and check the releases page for the version current on the day you read this.

Recommendation

Adopt for a processing hop between producers and stores, Trial for anything that depends on strict delivery. Vector is a strong choice when you want one place to normalise, filter, redact and route telemetry, with a testable config and a small runtime. Start with vector validate and the demo pipeline, then build a Kafka-to-database pipeline with acknowledgements and a disk buffer on a staging cluster. Kill the sink on purpose, restart the pod, and check what you lose and what you read twice. If you are already committed to the OpenTelemetry Collector and OTLP, the benefit of adding Vector is smaller.

References: Vector documentation · Vector repository · Acknowledgement guarantees · End-to-end acknowledgements · Kafka source · Kafka sink · Install on Kubernetes · Install with Helm · Secrets configuration · Command line reference · Datadog open source: Vector · openGemini line protocol writes · CNCF Landscape