Network Performance Testing (netperf)
ServiceRadar actively measures network throughput, jitter, and packet loss
between agents using embedded netperf inside serviceradar-agent. Unlike
passive monitoring (which observes existing traffic), netperf generates
synthetic TCP or UDP traffic and measures how the network handles it.
For each test, netperf reports:
- Availability and duration - test success indicator (
rperf.available) and measured duration in seconds (rperf.duration_seconds). - Throughput - bits per second (
rperf.bits_per_second) achieved for TCP or UDP traffic. - Jitter - variation in packet delay (
rperf.jitter_ms), in milliseconds (UDP). - Packet loss - packets sent, received, and lost (
rperf.packets_sent,rperf.packets_received,rperf.packets_lost) and loss percentage (rperf.loss_percent) (UDP). - Bytes transferred - total bytes sent and received (
rperf.bytes_sent,rperf.bytes_received) over the test duration.
Agents running with Hub Responder enabled also emit responder session gauges:
rperf.responder_active_sessions, rperf.responder_sessions_total,
rperf.responder_busy_rejections_total, and rperf.responder_bytes_total.
Architecture
Active network performance testing is embedded directly into
serviceradar-agent (go/pkg/netperf).
| Role | Binary | Transport / Ports | Auth / Authorization |
|---|---|---|---|
| Spoke Sender | serviceradar-agent | Outbound TCP 5199 (control) + UDP 5199 (optional UDP data) | mTLS + Ed25519 NetperfPermitV1 + HMAC-SHA256 |
| Hub Responder (opt-in) | serviceradar-agent | Inbound TCP 5199 (mTLS) + single UDP socket 5199 | mTLS (RequireAndVerifyClientCert) + NetperfPermitV1 verification |
Key architectural properties:
- No dynamic port pools - TCP data streams ride over the mTLS-authenticated
5199/TCPgRPC stream (netperf.v1.NetworkPerfPeerService), and UDP tests multiplex over the single5199/UDPsocket using HMAC-SHA256 authenticatedsession_idheaders. Ports5200-5210are no longer required. - Zero gateway synthetic payload load - Spoke agents talk directly to Hub
Responders over
5199, never routing synthetic test traffic throughagent-gateway. - Permit-authorized execution - Control-plane
NetperfCompilercompiles policies (:hub_and_spokeor:hub_mesh) intoAgentConfigResponseand mints Ed25519-signedNetperfPermitV1tokens verified against the agent'sRecordTrustkeyset. - Durable delivery - Completed test summaries commit directly to the
agent's
RecordSpoolunderserviceradar.metrics.batch(source = "rperf-metrics",metric_type = "rperf"), then flow throughEdgeRecordIngestService-> NATS JetStream (metrics.rperf.>) ->EventWriter-> CNPG/StarRocks.
Configuring Policies, Responders, and Profiles
Manage active network performance testing in the web UI at
Settings -> Networks -> Netperf (/settings/networks/netperf):
- Profiles - Define protocol (
tcporudp), direction (upload,download, orbidirectional), duration, parallel streams, omit warm-up interval, target bandwidth, and poll interval. - Hub Responders - Opt in specific agents to listen on
5199/TCP(+5199/UDPwhen UDP is enabled) with explicit concurrency and bandwidth caps. - Policies - Bind spoke agents to hub responders (
hub_and_spoke) or link responders together (hub_mesh) using a selected profile. - On-demand execution - Trigger immediate runs from
/settings/networks/netperf("Run Test Now"), which dispatches anetperf.runcommand overControlStreamwith a single-shotNetperfPermitV1.
Revoking Issued Permits
Scheduled permits are valid for up to eight days and carry the Hub Responder's authority epoch. Editing or disabling a policy rotates its permit IDs and drops the target from the next config push, but it does not invalidate a permit a spoke already holds. To revoke outstanding permits for a responder (for example after tightening a profile or disabling a policy), click Bump Epoch on the Hub Responder. Disabling a Hub Responder bumps its epoch as well. The responder rejects any permit whose epoch is below its current epoch once it receives the new config.
Firewall Rules (Hub Responders only)
Only agents explicitly enabled as Hub Responders accept inbound traffic:
- TCP
5199- Required for all Hub Responders (mTLS control and TCP data). - UDP
5199- Required only if the responder serves UDP tests.
Spoke-only agents require no inbound netperf ports.
Querying Results with SRQL
Netperf measurements are stored in timeseries_metrics and queried in SRQL
with in:netperf or in:netperf_metrics.
Recent netperf measurements:
in:netperf sort:timestamp:desc limit:50
Throughput for a target over the last day:
in:netperf metric_name:"rperf.bits_per_second" tags.target:"hub-01" time:last_24h sort:timestamp:desc
UDP packet loss:
in:netperf metric_name:"rperf.loss_percent" tags.protocol:udp sort:timestamp:desc
See the SRQL Language Reference for the full
timeseries_metrics field list (metric_name, metric_type, value,
agent_id, timestamp, and point tags such as tags.target,
tags.responder_agent_id, tags.policy_id, tags.profile_id,
tags.direction, tags.protocol, tags.execution_source, and
tags.parallel).
Network Performance Dashboard
The built-in Network Performance dashboard at /dashboards/network-performance
shows netperf throughput, loss, jitter and availability per agent and target for
the last 24 hours.
Alerting on Degradation
Because results land in timeseries_metrics, you
can build alerts the same way as any other metric:
- Throughput drop - alert when
rperf.bits_per_secondfalls below an expected baseline. - Packet loss - alert when
rperf.loss_percentrises above a threshold for UDP targets. - Jitter - alert when
rperf.jitter_msexceeds tolerance for latency-sensitive workloads. - Missing results - alert when no netperf metrics arrive within a multiple of the policy poll interval.
Use the Rule Builder to turn these into notifications.