Skip to main content

NetFlow Ingest Guide

ServiceRadar ingests flow telemetry to expose traffic matrices, top talkers, and application reachability trends. The flow collector is a high-performance Rust daemon that receives NetFlow v5/v9/IPFIX and sFlow exports from network devices and processes them through the ServiceRadar pipeline.

Template-based NetFlow v9/IPFIX over UDP is rejected by default because exporter source addresses are spoofable; template-free v5 and sFlow over UDP remain available. Use mutually authenticated native IPFIX (v10) over TLS/TCP for template traffic. See Flow Collector Scaling and Exporter Authentication.

Host process and workload attribution joins collected NetFlow/sFlow rows to netprobe observations in the warehouse; see Host Network Visibility for the observation path, the correlator, and the in:attributed_flows evidence.

Architecture Overview​

ServiceRadar uses a single canonical NetFlow ingest path:

Network Devices → Flow Collector → NATS JetStream `flows` → EventWriter → StarRocks ocsf_network_activity
 v5 UDP :2055       (Rust)          flows.raw.netflow      (Broadway)     (+ CNPG bgp_routing_info)
 IPFIX TLS :4740

Raw flows use a dedicated JetStream stream (flows by default), not the shared multi-signal events bus used for logs/OTEL/Falco. EventWriter consumes flows on a separate Broadway pipeline so GenStage demand is not fair-shared with other telemetry.

Key Components:

  • Flow Collector: Rust daemon for NetFlow, IPFIX, and sFlow. Ports and the UDP template opt-in are under Collector Layout.
  • AutoScopedParser: RFC-compliant per-source template isolation
  • NATS JetStream (flows): Dedicated stream for protobuf FlowMessage bytes on flows.raw.netflow / flows.raw.sflow (owned max_bytes/max_age, R=3 in HA)
  • EventWriter flow pipeline: Dedicated Elixir/Broadway demand domain that long-polls JetStream, decodes protobuf, Stream Loads OCSF flow rows into StarRocks, and derives BGP observations
  • StarRocks (required for NetFlow): The only flow store. EventWriter Stream Loads flow rows into the warehouse ocsf_network_activity table. See Flow cutover and delivery for routing, acknowledgement and upgrade behavior.
  • CNPG: Derived bgp_routing_info and the enrichment and attribution current state the warehouse joins. CNPG stores no flow rows.
  • SRQL: Query flows via in:flows from the warehouse's ocsf_network_activity.
  • Web UI: NetFlow dashboard with BGP topology visualization

Flow cutover and delivery​

With the warehouse enabled, an unset or blank cutover list selects flows by default. Helm uses analytics.starrocks.cutoverDatasets; Compose uses STARROCKS_CUTOVER_DATASETS. A non-blank list replaces the default.

The NetFlow dashboard and in:flows require warehouse reads. The dashboard throughput sparkline and device Flows-tab presence probes also route through Readers: without flow cutover, the sparkline renders empty and the probes report no flows. They do not fall back to CNPG on a warehouse failure. DeviceRiskIocExposure reads warehouse flows after cutover and otherwise finds no flow evidence; it never reads CNPG flows. Its IOC cache and device-identity lookups remain in CNPG.

NetFlow, sFlow and IPFIX flows are stored only in the warehouse. EventWriter never writes them to CNPG: the legacy CNPG platform.ocsf_network_activity hypertable and its continuous aggregates are dropped by migration 20261019160000_drop_cnpg_flow_hypertable_and_caggs. Flow Stream Load success or durable quarantine is required before JetStream ACK, independently of the read-cutover list. Warehouse failures backpressure ingestion; JetStream retention bounds how long unacknowledged data remains available, and redelivery reuses stable identities, so the warehouse upserts rather than duplicates.

Without the warehouse, flows are not stored at all. The Helm chart refuses to render flowCollector.enabled unless analytics.starrocks.enabled is set. Flows that still arrive from another publisher are acknowledged without being decoded or stored, and EventWriter logs once per node that NetFlow ingestion requires StarRocks. No BGP observation is derived from them.

Full-row ingestion and correlation updates carry agent and process attribution. The risk reader resolves device identity through ocsf_agents first, then the destination IP in device_identifiers. Migration 0023_ocsf_network_activity_agent_id.sql adds the agent column without a backfill. Older warehouse rows lack agent identity, and older best-effort shadow loads may have left gaps. Immediate cutover can therefore miss agent-only hostile-IOC exposure until these rows leave the effective risk lookback (window_seconds, default one hour). There is no automatic hold or backfill. The default cutover is intended for the accepted dev/test rollout.

BGP Routing Support​

NetFlow v9 and IPFIX exports can carry BGP information elements (AS numbers, communities). ServiceRadar derives BGP analytics from these flows into the bgp_routing_info table. To keep this guide focused on the ingest path, all BGP field details, query examples, indexing, and device configuration for BGP elements live in BGP Routing.

Collector Layout​

Components​

  • Listener: Receives UDP packets on port 2055 (default); template-based v9/IPFIX over UDP is rejected unless explicitly opted in, and parses NetFlow v5, sFlow, and opted-in template traffic (authenticated IPFIX (v10) arrives over TLS/TCP 4740)
  • Parser: AutoScopedParser with per-source template caching (prevents collisions)
  • Publisher: Batches flows and publishes to NATS JetStream (default: 100 flows/batch)
  • Metrics Reporter: Logs template cache statistics every 30 seconds

Deployment Options​

Kubernetes:

# Service definition exposes UDP 2055
apiVersion: v1
kind: Service
metadata:
  name: serviceradar-flow-collector
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  sessionAffinity: ClientIP
  ports:
    - port: 2055
      protocol: UDP
      name: netflow
    - port: 6343
      protocol: UDP
      name: sflow

Send template-free NetFlow to <FLOW_COLLECTOR_ADDRESS>:2055/UDP and sFlow to <FLOW_COLLECTOR_ADDRESS>:6343/UDP. Mutually authenticated IPFIX uses TLS/TCP 4740; see Flow Collector Scaling and Exporter Authentication. Keep the actual collector address in private operations material. Syslog can use a shared Gateway address instead; see Kubernetes External Ingestion.

Docker Compose:

The shipped Compose stack leaves the flow collector off unless you pass --profile flows or --profile network-ingest. --profile starrocks starts the StarRocks warehouse (starrocks/allin1-ubuntu:3.5.21) independently; set STARROCKS_ENABLED=true alongside it so EventWriter shadows telemetry, and set STARROCKS_ROOT_PASSWORD in .env (the container refuses to start without it; see README-Docker.md). NetFlow requires the warehouse: start the flows profile together with --profile starrocks and STARROCKS_ENABLED=true. Without it EventWriter acknowledges flow batches without storing them, and the dashboard NetFlow panel and in:flows are refused.

The trap, flow, and BMP collectors together start with STARROCKS_ENABLED=true docker compose --profile starrocks --profile network-ingest up -d. Its bounded JetStream reservations are KV 2 GiB + objects 2 GiB + events 2 GiB + flows 1 GiB + BMP 128 MiB = 7.125 GiB. That leaves 896 MiB of account headroom under the generated 8 GiB platform-account quota, which itself stays below the NATS server's 10G decimal per-server file-store limit.

services:
  flow-collector:
    image: ghcr.io/carverauto/serviceradar-flow-collector:latest
    ports:
      - "2055:2055/udp"
      - "6343:6343/udp"
    environment:
      - NATS_URL=nats://nats:4222
    networks:
      - serviceradar-net

Standalone:

cd rust/flow-collector
cargo build --release
./target/release/serviceradar-flow-collector --config flow-collector.json

Device Configuration​

Network devices (routers, switches, firewalls) must be configured to export NetFlow data to the ServiceRadar collector.

Configuration Requirements​

  1. Destination IP: ServiceRadar collector IP address
  2. Port: 2055/udp (default, configurable)
  3. Protocol: NetFlow v5 or sFlow over UDP; IPFIX (v10) over mutually authenticated TLS/TCP for template traffic (convert NetFlow v9 to IPFIX, or use the explicit insecure UDP opt-in)
  4. Timeouts: Active 60s, Inactive 15s (recommended)
  5. Interfaces: Which interfaces to monitor

Per-vendor flow-export snippets (Cisco IOS-XE/NXOS, Juniper, MikroTik, Fortinet, Palo Alto, VyOS) are maintained in the device quick reference at rust/flow-collector/DEVICE-CONFIG.md. BGP-specific flow-record configuration is covered in BGP Routing.

Multi-Source Deployments​

ServiceRadar's flow collector uses AutoScopedParser, which provides RFC-compliant template scoping for multi-source environments.

Why AutoScopedParser Matters​

The Problem:

  • Router A sends template ID 256 with fields: [SRC_IP, DST_IP, BYTES]
  • Router B sends template ID 256 with fields: [SRC_IP, DST_IP, PACKETS, PROTOCOL]
  • Without scoping, Router B's template overwrites Router A's → data corruption

The Solution:

  • AutoScopedParser isolates templates per source IP address
  • RFC 3954 (NetFlow v9) and RFC 7011 (IPFIX) compliant
  • Each source maintains independent template cache
  • Template ID 256 from 192.168.1.1 is different from 256 from 192.168.1.2

Template Cache Isolation​

Source: 192.168.1.1:2055 (Router A)
  V9 Template Cache: 15 templates
  V9 Data Cache: 8 active flows

Source: 192.168.1.2:2055 (Router B)
  V9 Template Cache: 12 templates
  V9 Data Cache: 5 active flows

Source: 192.168.1.3:2055 (Firewall C)
  IPFIX Template Cache: 20 templates
  IPFIX Data Cache: 15 active flows

Each source has completely isolated caches, preventing template ID collisions.

Monitoring and Observability​

Template Cache Metrics​

The collector logs cache statistics every 30 seconds:

V9 Template Cache [192.168.1.1:2055] - Templates: 15/2000, Data: 8/2000,
  Template Hits/Misses: 1250/15, Data Hits/Misses: 8420/8

IPFIX Template Cache [192.168.1.2:2055] - Templates: 20/2000, Data: 12/2000,
  Template Hits/Misses: 3200/20, Data Hits/Misses: 12500/12

Metrics:

  • Size (current/max): Number of templates cached / maximum cache size
  • Hits: Cache lookups that found the template (good)
  • Misses: Cache lookups that didn't find template (requires fetch)
  • Evictions: Templates removed due to cache size limits (logged separately)

Healthy Cache:

  • Hit ratio > 95% (Hits / (Hits + Misses))
  • Size well below max (< 50% utilization)
  • Few or no evictions

Unhealthy Cache:

  • High miss ratio (< 90%) → increase max_templates
  • Size near max → increase max_templates
  • Frequent evictions → increase max_templates
  • Many "Missing template" warnings → network issues or router reboots

Template Event Hooks​

The collector logs important template lifecycle events:

[INFO] Template learned - ID: 256, Protocol: V9
[WARN] Template collision - ID: 256, Protocol: V9
[DEBUG] Template evicted - ID: 512, Protocol: V9
[DEBUG] Template expired - ID: 1024, Protocol: IPFIX
[WARN] Missing template - ID: 300, Protocol: V9. Flow data received before template definition.

Event Types:

  • Learned: New template successfully cached
  • Collision: Template ID reused with different definition (shouldn't happen with AutoScopedParser)
  • Evicted: Template removed from cache due to size limits
  • Expired: Template TTL expired
  • MissingTemplate: Flow data arrived before template definition (normal during startup)

Performance Metrics​

Monitor these in logs and system metrics:

  • Flow ingestion rate: Flows/second processed
  • Channel utilization: Publisher channel usage (warn if >80%)
  • NATS publish latency: Time to publish batches
  • Drop rate: Flows dropped due to backpressure

Configuration Reference​

Collector Configuration​

The flow collector reads a single JSON file (/etc/serviceradar/flow-collector.json). Listener tuning fields (buffer_size, max_templates, max_template_fields, pending_flows) belong inside each listener entry, not at the top level.

{
  "nats_url": "nats://nats:4222",
  "nats_creds_file": "/etc/serviceradar/creds/platform.creds",
  "stream_name": "flows",
  "stream_subjects": ["flows.raw.netflow", "flows.raw.sflow"],
  "stream_max_bytes": 10737418240,
  "stream_max_age_secs": 21600,
  "stream_replicas": 3,
  "partition": "default",
  "listeners": [
    {
      "protocol": "netflow",
      "listen_addr": "0.0.0.0:2055",
      "subject": "flows.raw.netflow",
      "buffer_size": 65536,
      "max_templates": 2000,
      "max_template_fields": 10000,
      "pending_flows": {
        "max_pending_flows": 256,
        "max_entries_per_template": 1024,
        "max_entry_size_bytes": 65531,
        "ttl_secs": 300
      }
    },
    {
      "protocol": "sflow",
      "listen_addr": "0.0.0.0:6343",
      "subject": "flows.raw.sflow",
      "buffer_size": 65536
    }
  ],
  "channel_size": 10000,
  "batch_size": 100,
  "publish_timeout_ms": 5000,
  "security": {
    "mode": "mtls",
    "cert_dir": "/etc/serviceradar/certs",
    "tls": {
      "cert_file": "flow-collector.pem",
      "key_file": "flow-collector-key.pem",
      "ca_file": "root.pem"
    }
  },
  "metrics_addr": "0.0.0.0:50046"
}

Top-level parameters:

  • nats_url: NATS endpoint for JetStream publishing
  • nats_creds_file: Optional path to NATS credentials file
  • stream_name: Dedicated JetStream stream for flow subjects (default/production: flows; do not share with the multi-signal events stream)
  • stream_subjects: Stream subjects to ensure exist for canonical raw flow ingest (each listener's subject is merged in automatically)
  • stream_max_bytes: Stream size cap in bytes. Binary default for stream_name=flows is 10 GiB (recovery headroom). Docker Compose and the OCI-baked config override to 1 GiB as part of the bounded 7.125 GiB aggregate described above. Tenant overlays use ≤256 MiB under 2G file stores. Helm sizes it from nats.jetstream.profile (8 GiB / R=3 in small; see Helm configuration). Never apply these retention values when stream_name is still events (legacy mode is subject-merge only). Must not change an existing StatefulSet PVC size via Helm (volumeClaimTemplates are immutable); moving to a larger profile follows docs/nats-jetstream-profile-runbook.md in the repository. The SERVICERADAR_JS_FLOWS_MAX_BYTES and SERVICERADAR_JS_FLOWS_REPLICAS environment variables override stream_max_bytes and stream_replicas (environment, then JSON, then default); an empty value is treated as unset and any other non-positive or non-integer value fails startup. On startup the collector reconciles an existing flows stream to the resolved size and replicas, evicting the oldest messages if the stream is over the new cap.
  • stream_max_age_secs: Stream MaxAge in seconds (default: 21600 / 6 hours). The flow-collector is the retention owner for the flows stream: it marks the stream with serviceradar.owner: flow-collector metadata, and EventWriter must not shrink those limits on reconcile.
  • stream_replicas: JetStream replica count (default: 1 in the binary; Helm HA sets 3). Prefer R=3 in multi-node NATS so demo matches production HA. The chart checks the resulting JetStream reservations against max_file_store at render time.
  • partition: Partition tag applied to ingested flows (default: default)
  • listeners: One entry per UDP socket (netflow or sflow)
  • channel_size: Bounded channel depth (default: 10,000)
  • batch_size: Flows per NATS publish (default: 100)
  • publish_timeout_ms: NATS publish timeout (default: 5,000)
  • Backpressure is fixed to drop-newest: each listener owns a bounded mpsc channel of depth channel_size, and when it is full the listener drops the incoming datagram and increments a per-subject drop counter (no operator-tunable policy).
  • metrics_addr: Optional address for the collector metrics endpoint

Per-listener parameters:

  • buffer_size: UDP socket receive buffer (default: 65,536) — applies to both netflow and sflow listeners
  • max_templates (netflow only): Template cache size per source (1–10,000, default 2,000)
  • max_template_fields (netflow only): Max fields per template for security (1–100,000, default 10,000)
  • pending_flows (netflow only): Optional cache for flow data that arrives before its template. Fields: max_pending_flows (1–10,000, default 256), max_entries_per_template (1–100,000, default 1,024), max_entry_size_bytes (1–65,531, default 65,531), ttl_secs (1–3,600, default 300)
  • max_samples_per_datagram (sflow only): Optional cap on samples parsed per datagram

Tuning for High Volume​

For high flow rates, raise the per-netflow-listener cache sizes and the top-level channel/batch settings. The listener tuning fields stay inside the netflow listener entry:

{
  "channel_size": 50000,
  "batch_size": 500,
  "publish_timeout_ms": 10000,
  "listeners": [
    {
      "protocol": "netflow",
      "listen_addr": "0.0.0.0:2055",
      "subject": "flows.raw.netflow",
      "buffer_size": 131072,
      "max_templates": 5000
    }
  ]
}

For deployments with many routers, increase max_templates on the netflow listener so each source has enough template cache headroom.

Registry and Metadata​

  • Use the embedded sync runtime (agent) to register flow exporters with site, account, and device tags.
  • Populate interface maps in the registry so flows can be joined with SNMP interface stats.
  • Capture application dictionaries (port to service mapping) in the control plane so SRQL and the UI can present friendly names.

Verification​

1. Check Collector is Running​

# Docker
docker ps | grep flow-collector
docker logs serviceradar-flow-collector-mtls

# Kubernetes
kubectl get pods -l app=serviceradar-flow-collector
kubectl logs -l app=serviceradar-flow-collector --tail=100

# Standalone
ps aux | grep flow-collector
journalctl -u flow-collector -f

2. Verify Packets Arriving​

# Capture on collector host
sudo tcpdump -i any -n port 2055

# Should see:
# 15:30:45.123456 IP 192.168.1.1.12345 > 10.0.0.50.2055: UDP, length 1480
# 15:30:46.234567 IP 192.168.1.2.54321 > 10.0.0.50.2055: UDP, length 1200

3. Check Collector Logs​

# Look for template learning
grep "Template learned" /var/log/flow-collector.log

# Check cache stats
grep "Template Cache" /var/log/flow-collector.log

# Look for errors
grep -i "error\|warn" /var/log/flow-collector.log

4. Query NATS Stream​

# Raw NetFlow lives on the dedicated `flows` stream (not shared `events`).
nats stream info flows

# Should list flows.raw.netflow / flows.raw.sflow in the subjects list and show
# recent last_seq growth while exporters are active.
#
# Do NOT delete the `flows` stream — it is the canonical raw-flow bus.
#
# Cutover (coordinated):
# 1. Deploy core EventWriter with flow pipeline dual-consume enabled
#    (EVENT_WRITER_FLOW_DRAIN_EVENTS=true, default). Drain durables reuse the
#    pre-cutover durable names so ACK cursors continue (no full-history replay).
# 2. Helm uses Deployment strategy Recreate for flow-collector so the legacy
#    publisher is terminated before the new pod starts. Helm readiness is an
#    exec probe of the publisher marker file, which exists only after the
#    publisher is ready (written only after JetStream rehome/ensure succeeds). Rehome marker lives on the data PVC at
#    rehome_state_path (/var/lib/serviceradar/flow-collector-rehome.json).
# 3. New collector detaches subjects from events, attaches them on flows, then
#    publishes with Nats-Msg-Id retries (preferred dedup window 120s, capped by stream max_age and the NATS server limit).
# 4. When events drain consumers report num_pending=0, set
#    EVENT_WRITER_FLOW_DRAIN_EVENTS=false and restart core.

Downgrading across the stream-ownership cutover​

Do not run a plain helm rollback to a revision whose flow collector still uses stream_name: events. Helm restores that old image and old ConfigMap together, but the old image cannot detach flows.raw.* from the dedicated flows stream. NATS then rejects its attempt to attach the same subjects to events. The old chart's process-only readiness probe can still report Ready while publishing is stuck.

Use the migration-capable current image to transfer ownership back first, then restore the old revision:

scripts/prepare-flow-collector-rollback.sh \
  --release serviceradar \
  --namespace demo \
  --revision <legacy-helm-revision>

The helper fails closed unless the target revision contains the legacy events configuration and the current Deployment uses Recreate plus Helm readiness as an exec probe of the publisher marker file, which exists only after the publisher is ready. It patches only the current flow ConfigMap to events, restarts the current image, waits until its reverse-transfer path is ready, checks the legacy events stream confirmation log, and only then invokes helm rollback.

For ArgoCD or another GitOps controller, pause automatic reconciliation and export the target revision's rendered flow-collector.json ConfigMap value to a standalone file. ArgoCD does not create Helm release history, so validate that artifact directly:

scripts/prepare-flow-collector-rollback.sh \
  --namespace demo \
  --prepare-only \
  --target-config /path/to/legacy-flow-collector.json

After it reports success, immediately point GitOps at the revision that produced that file. If you abandon the downgrade, re-apply the current chart/values so the current image rehomes the subjects forward to flows; do not leave the live ConfigMap different from the desired release.

5. Query the Warehouse​

Run these queries on the StarRocks Frontend (MySQL protocol, port 9030), using your configured warehouse database (serviceradar by default).

-- Check recent flows
SELECT
  time,
  src_endpoint_ip,
  dst_endpoint_ip,
  dst_endpoint_port,
  protocol_name,
  bytes_total,
  packets_total
FROM ocsf_network_activity
WHERE time > NOW() - INTERVAL 5 MINUTE
ORDER BY time DESC
LIMIT 20;

-- Count flows per source
SELECT
  src_endpoint_ip,
  COUNT(*) as flow_count,
  SUM(bytes_total) as total_bytes
FROM ocsf_network_activity
WHERE time > NOW() - INTERVAL 1 HOUR
GROUP BY src_endpoint_ip
ORDER BY total_bytes DESC
LIMIT 10;

6. Query via SRQL​

# Top talkers last hour
srql "in:flows time:last_1h groupby:src_endpoint_ip limit:10"

# Specific destination
srql "in:flows dst_endpoint_ip:8.8.8.8 time:last_24h"

# High bandwidth flows
srql "in:flows bytes_total:>10000000 time:last_1h"

7. Check Web UI​

Navigate to http://localhost/netflows to view:

  • Flow summary statistics
  • Top talkers (source IPs)
  • Top destinations
  • Protocol distribution
  • Bandwidth over time
  • Prefix tag chips and filters (when prefix-tag enrichment is enabled)

Common Issues​

No Flows Appearing​

Check:

  1. Device is configured and exporting (check device logs)
  2. Network path allows UDP 2055 (firewall rules)
  3. Collector is listening: netstat -ulnp | grep 2055
  4. Packets arriving: tcpdump -i any port 2055
  5. Collector logs show "Received X bytes from..."

Template Warnings​

"Missing template - ID: 256"

  • Normal during startup: Router sends data before template
  • Wait 60 seconds: Router will re-send template (per timeout)
  • Persistent: Router may have lost template, reboot router or wait for TTL

"Template collision" (should not happen)

  • AutoScopedParser prevents this
  • If seen, report as bug

High CPU Usage​

Causes:

  • Very high flow rate
  • Complex templates with many fields
  • Insufficient batching

Solutions:

  • Increase batch_size to 500-1000
  • Enable sampling on routers (1:100 or 1:1000)
  • Scale horizontally (multiple collectors)

Dropped Flows​

Log message: "Publisher channel full, dropping flow message"

Causes:

  • NATS JetStream slow or unavailable
  • Channel too small for burst traffic
  • Batch publish taking too long

Solutions:

  • Increase channel_size to 50,000+
  • Check NATS JetStream health
  • Increase batch_size for better throughput
  • Check network latency to NATS

Performance Characteristics​

The Rust flow collector is designed for high-throughput ingest with bounded memory. Actual sustainable rate depends on CPU allocation, template complexity, batch settings, and NATS/JetStream performance, so benchmark in your own environment rather than relying on fixed numbers. As general guidance:

  • Throughput scales with CPU cores and batch_size; enable router-side sampling for very high flow rates.
  • Memory stays bounded by channel_size, the per-listener template caches, and the optional pending_flows cache.
  • End-to-end latency from UDP receipt to NATS publish is dominated by batching (batch_size) and publish_timeout_ms.

Security Considerations​

Network Security:

  • Restrict UDP 2055 to known exporter IPs via firewall
  • Use VPN or private network for exporter-to-collector communication
  • Monitor for unusual sources in logs
  • v9/IPFIX template traffic over UDP is spoofable even behind allowlists; keep the secure default or restrict any allow_unauthenticated_templates listener to a separately protected network. See Flow Collector Scaling and Exporter Authentication.

Template Validation:

  • Max template fields enforced (default: 10,000)
  • Prevents memory exhaustion attacks
  • Malformed templates rejected

mTLS Support:

  • NATS connection can use mTLS
  • Authenticates collector to NATS
  • Encrypts flow data in transit

Further Reading​