Network Topology
The Network Topology view is the high-density topology experience for large graphs with causal blast-radius overlays. The overview renders server-authored world tiles fetched by zoom level; drill-down opens bounded ELK detail scenes.
Feature Flag
The Network Topology view is controlled by:
SERVICERADAR_GOD_VIEW_ENABLED=true|false
Runtime behavior:
false(default):/topologyis hidden/disabled.true:/topology, the topology channel stream, and the world-tile and latest-snapshot endpoints are available.
Serving model
The overview is a quadtree of world tiles over a persisted, versioned radial
ELK layout. The background worker composes bounded ELK batches before publishing
coordinates; individual tiles never run their own layout. Additions that leave
the trusted parent tree unchanged preserve existing positions. A change to that
tree, including deletion of a connected endpoint, stages a fresh layout under a
new layout version and keeps the accepted world until that replacement
activates. Timestamp-only observation refreshes do not swap an equally trusted
forest or move unchanged geometry. An operator relayout or a staged layout
upgrade can also replace the coordinate space. Arrow tiles stay bounded; the
binary path audit records the remaining
producer and browser copies and the scale evidence still missing.
The optional bitmap membership protocol uses
server-side set queries and bounded binary row overlays without changing tile geometry.
The client fetches only the tiles in the viewport, keyed by
layout_version/z/x/y, and renders schema-3 Arrow batches; telemetry
(health and interface rates) rides a separate per-tile overlay so geometry
tiles stay cached. Zoom reveals aggregates first, then backbone, infrastructure
and endpoints, subject to density limits. Large populations remain navigable
as groups rather than requiring every device or label on screen at once.
Reset view fits the active devices. Searching for a device resolves its world
coordinates and flies to them; opening a device or attachment group enters a bounded ELK detail scene
and returns to the tile map on exit. These bounded map pages reuse the radial
overview projection, including real attachment fans; older detail payloads
without that profile retain their layered layout.
The tiled world reads Dgraph's admitted topology view, including fresh and last-known stale attachment, hosted and inferred evidence alongside the canonical backbone. Stale links remain visible and marked stale, but never supply current traffic or packet animation. The separate canonical traversal API remains backbone-only. Overview routes follow the preferred physical-first forest; retained cross-links remain available in bounded details. Zoom changes grouping and label admission, not device coordinates.
Rollout Guidance
Recommended rollout order:
- Enable in a non-production environment first.
- Validate stability and performance.
- Enable broader environments only after SLO validation.
For Helm-based deployments:
- Set
webNg.extraEnv.SERVICERADAR_GOD_VIEW_ENABLED: "true"in the target values file.
Dgraph topology store
Mapper evidence stays in CNPG (mapper_topology_links). Dgraph is the only
topology graph: every topology write and read (mapper links, interfaces,
management relations, risk scalars, MTR paths, canonical rebuild, telemetry
refresh, ServiceRadar Atlas, SRQL in:graph / in:device_graph, and the causal
hydrator) uses it. There is no alternative backend and no fallback. A Helm
release without Dgraph (dgraph.enabled=false and no dgraph.external.host)
fails to render. If core starts without a resolvable Dgraph endpoint, it logs
an error once and drops topology writes until the endpoint resolves.
Canonical reconciliation
Core's TopologyGraph.rebuild_canonical_links_from_current/0 reconciles the
policy-approved mapper evidence and existing canonical edges in Dgraph through
the typed replacement API. It preserves directional interface attribution,
existing canonical telemetry, and observation timestamps. Retained edges
participate in same-port conflict resolution with their stored support rank.
Reconciliation retains the starvation and mass-deletion guards for stale
pruning. Its input fingerprint and heartbeat persist in CNPG
(runtime_topology_projection_meta), so an unchanged topology is not rewritten.
Reconciliation reads its inputs in pages of at most 256 edges from one Dgraph snapshot, and resolves the interfaces those edges name by key in chunks of the same size, so no response grows with the inventory. A page or chunk that is still too large for the gRPC receive limit is retried at half the size. If any page fails, the read fails as a whole and that run writes and prunes nothing: the pruning decision is only ever made from a complete read.
Stale pruning and in:device_graph follow the inventory too and are bounded the
same way. Stale pruning ages edges a few thousand at a time, one transaction per
page, so a large backlog of aged edges is cleared across several transactions
and a repeat finishes anything a failure left behind. in:device_graph pages
the device's own ports and edges, and reads from each neighbour only the port
an edge names instead of the neighbour's whole port list. A query passed to
in:graph is sent as written: give a broad one first: and after:.
Every outcome is recorded as the topology-canonical-rebuild core health
check. A failed reconciliation turns it unhealthy with the failure reason,
which opens an alert through the seeded core_health_check_unhealthy rule,
and the next completed (or unchanged) reconciliation returns it to healthy.
While it is unhealthy, ServiceRadar Atlas and every other reader of canonical
edges keep showing the last reconciled topology. Core also logs
Canonical topology rebuild failed: <reason> for each failed run.
Projection lag
Mapper ingestion commits interface and topology evidence to CNPG first, inside
the result-ingestion lane's worker budget. The Dgraph write runs afterwards as
a TopologyProjectionWorker job on the topology_projection Oban queue
(OBAN_QUEUE_TOPOLOGY_PROJECTION, default 2 per core node). A slow or
unavailable Dgraph therefore delays the graph instead of timing the mapper
result out.
Topology views read the graph, so they trail the CNPG interface and evidence tables by the job's queue wait plus its run time. One device's interfaces are written in one Dgraph request, and that device's links in two (endpoints, then edges), however many interfaces or links it carries.
Each projection job is one device and carries that device id only. When it
runs it reads the current committed discovered interfaces (including aliases)
and mapper topology links for that device, and rewrites that device's graph
state through the same batched writes. The newest of that device's link rows
that sets a field owns its IP and interface fields. For each link the job also
creates the neighbor vertex, the neighbor interface and the edge, and writes
the neighbor device's IP and name and the neighbor interface name the reporting
device observed when it has them; a missing value never erases a known one. A
retry of an older job therefore leaves the graph equal
to that device's latest CNPG rows, including when it runs after a newer job has
already completed. A newer result for the same device replaces a job that has
not started, so a pending backlog collapses to one run per device. Oban keeps at
most one pending job per device; an executing job is not pending, so a result
that lands during a run still schedules the next one. Each graph statement
commits on its own. When a job reads the device's rows it records a watermark
(an md5, computed in SQL, of the full content of the device's interface rows
and of its link rows, so any change to, addition of or removal of a row
changes it), and after a successful projection it reads the watermark again. If it
changed, because a write landed during the run or another run for the device
read newer rows and finished first, the job schedules one follow-up projection
for the device. Every run writes what it read, including a report identical to
an earlier one, so the graph converges to the newest committed rows whichever
core node runs each job.
A job reads only links observed at or after the stale-link cutoff that pruning
uses, so retained rows older than that are neither re-projected nor a reason
for a follow-up. The graph-wide passes (global stale-evidence pruning and the
canonical rebuild) run in a separate TopologyGlobalPassWorker job on the same
queue. Each projection schedules it 30 seconds ahead, and it is unique while
pending, so a burst of device projections runs the passes once. A failed projection is logged,
reported as an Oban job exception, and retried with backoff. The ingestion
runtime metrics
(metrics.ingestion_lanes) carry a topology_projection lane (scheduled,
coalesced, accepted, not accepted and failed projections, with queue_wait_ms
and duration_ms) and a dgraph lane (every Dgraph call's outcome, timeouts
and retries, with duration_ms and queue_wait_ms).
Dgraph superuser credentials
With the in-chart cluster (dgraph.enabled=true), the groot password is
generated into the Dgraph ACL Secret alongside the ACL HMAC key, and the
post-install Job serviceradar-dgraph-acl-bootstrap rotates the cluster off
Dgraph's well-known default before the schema Job runs. Application pods and
both Jobs read it through DGRAPH_PASSWORD; nothing embeds a literal password.
An external cluster (dgraph.enabled=false with dgraph.external.host) is not
provisioned by this chart, so an ACL credential for it must already exist as a
Secret in the namespace: point dgraph.external.credentialsSecret (and
credentialsKey, default password) at it, with dgraph.external.username
naming the ACL user. Leave credentialsSecret empty to dial an external
cluster that has ACL disabled. A password is never a chart value, because
graph.env renders into the Deployment spec.
Dgraph reset
The canonical topology is derived from evidence. To rebuild it, reset mapper
evidence with the existing topology-evidence cleanup (CNPG
mapper_topology_links stays the source of truth; never drop_all on Dgraph)
and let the next mapper cycle and canonical reconciliation repopulate Dgraph.
in:graph / in:graph_dql and in:device_graph query Dgraph;
in:graph_cypher was retired with AGE. The entities are described in the
SRQL Language Reference.
Operator Controls
Primary controls in the Network Topology view:
- Zoom mode (
auto,world,region, ordetail) - Health toggles (
unavailable,healthy,unknown) - Layer toggles (links, traffic, and opt-in inferred relations)
Interpretation:
healthy: last observed availableunavailable: last observed unavailableunknown: not yet observed
Health is availability only, independent of geometry and causality; it does not
imply a causal root_cause/affected status.
Known Limitations
- Rendering requires WebGPU; unsupported clients receive an error rather than a fallback renderer.
- Performance depends on browser/GPU capability. Authenticated million-device product acceptance remains pending.
- Large revisions may be dropped under budget pressure to preserve interaction responsiveness.
- Causal confidence is bounded by telemetry quality/completeness.
Telemetry and Signals
Link traffic
World-tile traffic uses the shared telemetry reader: StarRocks while enabled, CNPG otherwise. It does not read the frozen CNPG history when warehouse writes are enabled. Only fresh, unambiguous physical interface measurements can drive packets. Attachment, inferred and hosted evidence does not supply traffic.
A fully selected bundle can animate its measured contribution even when some selected bindings have no telemetry. Complete totals remain unknown and observed coverage is reported separately. Bundles with unselected membership do not animate; rates are never extrapolated across telemetry pages. Bounded detail scenes currently have no live traffic overlay.
Canonical graph telemetry refresh
Link traffic shows packets per second (pps) and bits per second (bps), derived from cumulative interface counters. Each interval uses two samples from the same collector and metric series within the last 30 minutes; octet rates are converted to bits per second. For each device/IP, interface and metric, the freshest producer is selected before validating its interval. A reset, negative sample, invalid interval or single sample yields no rate; it does not fall back to an older collector's traffic.
Both telemetry backends use the shared SRQL-compatible wrap and plausibility
rule in ServiceRadar.Analytics.StarRocks.MetricConsumers.counter_rate_sql/1.
CNPG supplies the producer's max_counter_rate_per_second metadata when present.
The StarRocks reader has no producer ceiling, so it uses the rule's default
32-bit wrap bound and rejects 64-bit decreases. A decrease without enough
metadata to distinguish a plausible 32-bit wrap from a reset can still be
interpreted as a wrap.
Telemetry refresh updates only existing canonical Dgraph edges' directional
traffic, capacity and telemetry eligibility. It preserves discovery evidence,
last_seen and endpoints, and cannot recreate a pruned edge.
Operational metrics
The Network Topology view emits operational telemetry for:
- Snapshot build latency
- Snapshot payload size
- Snapshot dropped count
- Snapshot error count
Use these metrics to validate rollout health and SLO readiness.
Troubleshooting
/topology is not visible
- Confirm
SERVICERADAR_GOD_VIEW_ENABLED=trueforweb-ng. - Confirm the pod has restarted with updated env.
Stream fails to join (god_view_disabled)
- Feature flag is disabled at runtime.
- Verify
web-ngruntime env and effective config.
Snapshot decode errors
- Check schema/version compatibility between server and client.
- Verify latest deployed frontend bundle matches backend snapshot contract.
Frequent dropped snapshots
- Check snapshot budget configuration and host load.
- Reduce update pressure and verify telemetry around build time and drop counts.