Skip to main content

Data Pipeline

ServiceRadar uses two primary data paths:

  • Agent ingestion (edge status, discovery, sync) via mTLS gRPC to agent-gateway
  • Bulk telemetry ingestion (logs/flows/etc.) via NATS JetStream streams into CNPG

This page is intentionally high-level. It focuses on the mental model and the moving parts you will see when debugging.

NATS JetStream (Bulk Ingestion)​

Bulk collectors publish to NATS JetStream. The most common stream is events.

Zen normalizes logs (syslog, SNMP traps, OTEL logs) and raw OTEL metrics. NetFlow data publishes to flows.raw.> and does not pass through Zen—it is written directly by the writer path. log-promotion is an in-process JetStream pull consumer running inside serviceradar_core (not a separate deployment); it subscribes to processed log subjects and promotes matching logs into OCSF events. db-event-writer persists records into CNPG.

Data Service (datasvc)​

datasvc is a gRPC service (port 50057) that fronts the platform's NATS-backed key-value and object stores. Other components use it for shared configuration and state—for example, the KV bucket serviceradar-datasvc holds rule definitions and runtime settings, and the object store carries larger payloads. Routing this state through one service keeps NATS KV/object access consistent and avoids components manipulating JetStream buckets directly.

CNPG (System Of Record)​

CNPG is the system of record for inventory, telemetry, and analytics (Timescale hypertables and AGE graph features are enabled in the cluster).

Querying happens through the web UI and SRQL, which is embedded in web-ng.