Camera Recording & NVR
ServiceRadar records camera video streams through serviceradar-nvr StatefulSet
members orchestrated by core-elx, tracks sealed blocks and retention state in
CNPG, optionally archives sealed blocks to S3-compatible object storage, and
serves unified wall-clock playback and evidence holds from the Web UI.
Operator configuration lives under Settings -> System -> Camera Recording ->
Recording & Storage (/settings/recording), while historical playback and
evidence holds live on each camera's Recordings page
(/cameras/:camera_source_id/recordings).
Architecture and prerequisites
Camera recording separates deployment-owned workload topology from operator-managed runtime settings:
| Layer | Component | Responsibility |
|---|---|---|
| Media spool & object I/O | serviceradar-nvr StatefulSet | Owns one replicated PVC and one immutable storage-lifetime UUID (storage_id, for example 00000000-0000-4000-8000-000000000011) per ordinal (serviceradar-nvr-0, serviceradar-nvr-1). Seals fragmented MP4 (init.mp4, media.m4s, descriptor.json) blocks on disk and uploads or verifies archive objects directly against S3-compatible storage. |
| Relay & orchestration | core-elx | Hosts the shared camera relay pipeline, acquires recorder ownership fences in CNPG, supervises capture and archive operations, and queries NVR member endpoints over pinned mTLS and bearer tokens. |
| Control plane & catalog | CNPG (serviceradar_core) | Stores block reservations (RecordingBlock), sealed and quarantined catalog entries (RecordingCatalogEntry), verified archive receipts (NVRArchiveReceipt), gaps (RecordingGap), holds (RecordingHold), append-only deletion/loss dispositions (RecordingDisposition), access audits (RecordingAccessAudit), and runtime settings. |
| Operator UI & playback API | web-ng | Renders /settings/recording and /cameras/:camera_source_id/recordings, enforces RBAC and camera scope on every request, assembles multi-block HLS v7 VOD playlists, and enforces bounded in-flight read admission. |
Deployment prerequisites
- Enable the NVR workload in Helm (
nvr.enabled: true) on replicated storage. NVR does not implement application-level media replication; each StatefulSet ordinal relies on a qualified replicated CSI storage class (such as Longhorn or Ceph RBD) with a retaining PVC lifecycle so local spools and unarchived backlogs survive single-node loss. - Verify
core-elxreachability and NVR client configuration.web-ngqueriescore-elxover bounded ERTS RPC to inspectCameraRecording.Supervisorstatus and mounted NVR member identities. Whencore-elxis unreachable or NVR is not installed,/settings/recordingreports explicitunknownorNVR is not installedbanners rather than rendering empty lists. - Satisfy the Deployment Recovery Gate. Every recording, reconciliation, and
destructive retention path checks the read-only deployment recovery gate
(
:nvr_recovery_gateon Core and the per-member gate on each NVR pod). See NVR runtime and replicated storage: Deployment recovery gate for the bootstrap and post-restore gate procedure.
The Settings UI and HTTP API report Core and per-member recovery gate states
(ready, closed, not_configured, or unknown with the active generation
prefix), and never open, close, or modify the gate. Gate authority stays in
deployment control outside CNPG backups and GitOps rollbacks so a database
restore cannot silently re-open recording or retention against unverified media
volumes.
Install topology vs CNPG runtime settings
No runtime recording or retention choice uses a Helm value. Install topology
stays in Helm/deployment configuration (nvr.enabled, nvr.replicas, storage
UUIDs, workload mTLS, nvr.archive.providerEgress network policies, and
install capacity bounds nvr.maxSpoolBytes / nvr.reserveBytes). Operational
choices -- archive destinations, selected s3_compatible credentials,
per-camera recording policies, local and object retention rules, and per-member
desired byte budgets -- persist in CNPG with deterministic code defaults when no
row has been saved yet.
Recording & Storage settings (/settings/recording)
Open Settings -> System -> Camera Recording -> Recording & Storage
(/settings/recording), or search the Settings command palette (Ctrl+K /
Cmd+K) for camera, nvr, vms, recording, archive, s3, bucket,
retention, or playback. Opening the view requires at least one of
settings.recording.view, settings.recording.manage, or
settings.recording.operate.
The top banner displays four live status indicators: NVR client installation,
Core recovery gate, core-elx reachability, and Capture supervisor,
along with a Refresh status action. Below the banner, six tabs organize
day-to-day operations.
1. Recorders (?tab=recorders)
The Recorders tab shows NVR runtime readiness, active storage mode (Local + archive or Local only: no object storage configured), and per-member local
byte budgets (NVRStorageSettings, platform.camera_nvr_storage_settings).
- Honest unknown states. If
core-elxor an NVR member'sGET /v1/statusendpoint does not respond within the RPC deadline, member readiness, recovery gate state, and live disk measurements render asunknown, never as an empty or absent recorder list. - Install defaults from PVC capacity. By default (
install_default), Helm derives each member's reserve floor asmax(64 MiB, 1% of PVC)(nvr.reserveBytes) and its local spool ceiling/default asfloor(85% of PVC) - reserve(nvr.maxSpoolBytes). On a100 GiBPVC, that yields a1 GiBreserve and an84.0 GiBspool budget, leaving 15% headroom for filesystem metadata, journals, and publication records. - Per-member desired budget and reserve (
NVRStorageSettings). Keyed by the member's storage UUID (storage_id, for example00000000-0000-4000-8000-000000000011), administrators withsettings.recording.managecan setbudget_modeandreserve_modetoinstall_defaultorcustom(AshPaperTrail-audited, with monotonicrevision):- Custom budget bounds: must be at least
pinned_bytes + 33,554,432(pinned_bytes + 32 MiB, reserving two16 MiBblock reservations above the member's non-evictableP1..P5bytes) and at most1 PiB, the install ceiling (nvr.maxSpoolBytes), andvolume_capacity_bytes - effective_reserve_bytes. Lowering a budget below current disk usage (while staying at or above the pinned floor) is permitted: it pauses new capture admission until retention eviction brings usage below the new high-water mark without triggering immediate NVR-initiated deletes. - Custom reserve bounds: must be at least
67,108,864bytes (64 MiB) and the install reserve floor, and at most10%of volume capacity.
- Custom budget bounds: must be at least
- Desired vs applied tracking. Each row displays the desired
revision,sync_state(pending,applied, orrefusedwithlast_refusal_class),applied_revision,applied_at, currentpinned_bytesfloor, and projected effective local window.
2. Archive (?tab=archive)
The Archive tab manages S3-compatible object storage destinations and binds
reusable credentials on the RecordingArchiveSettings singleton
(platform.camera_recording_archive_settings).
Local-only mode vs S3-compatible archive mode
- Local-only mode (
Local only: no object storage configured) is a complete, first-class operating mode whenarchive_enabledisfalseor no active destination generation is selected. No S3 bucket or archive credential is required, and the UI presents local-only mode as normal state rather than a warning. - S3-compatible archive mode (
Local + archive) copies sealed NVR blocks to the activeArchiveBindingdestination and records readback-verified receipts (NVRArchiveReceipt). Enablingarchive_enabledrequires an active destination generation. - Ending or clearing the active destination while
archive_enabledremains enabled displays an advisory warning: blocks registered while no destination is active (and unassigned quarantine copies) wait inawaiting_binding(pinned locally inP1), whereas blocks that already recorded a binding in retained history for a non-compromised reason continue archiving under their recorded binding.
Registering immutable operator:<name> destination generations
Archive destinations are immutable ArchiveBinding records. Registering a
destination creates provisioning_ref = "operator:<name>" at the next monotonic
generation using ArchiveDestination schema v2:
| Field | Validation rule | Synthetic example |
|---|---|---|
| Destination name | [a-z0-9][a-z0-9-]{0,62}, persisted as operator:<name> | site-a-archive (operator:site-a-archive) |
| Endpoint | https://<dns-name> on port 443 (no IP literals) | https://objects.example.com |
| Region | [a-z0-9-]{1,40} | example-1 |
| Bucket | 3..63 character DNS-compatible bucket name | example-archive |
Key prefix (namespace) | 1..128 bytes of [a-z0-9/_-] with no leading, trailing, or empty slash segments (included in destination_sha256; v1 bindings retain recordings/v1) | recordings/site-a |
Existing ArchiveBinding generations cannot be edited or deleted
(on_delete: :restrict from blocks and receipts). Changing an endpoint, bucket,
region, or prefix registers a new generation while older blocks retain the
binding they were sealed under. Every operator destination displays capability
state unqualified -- NVR connectivity probes write and verify a deterministic
_probe/ object to confirm network reachability, TLS trust, and bucket access,
which does not constitute storage provider qualification.
Binding reusable s3_compatible credentials
Secret key material is never entered or rendered on /settings/recording. Create
credentials first under Settings -> System -> Security -> Credentials
(/settings/networks/credentials) using S3-Compatible Object Storage - Access key (see Credential Management), then
bind them by reference across three slots:
| Slot | Field | Broker purpose & HTTP verbs | Behavior when unset |
|---|---|---|---|
| Writer credential | writer_credential_secret_id | camera_archive_write (PUT, HEAD, GET on /<bucket>/<prefix>/*) | Archive uploads and destination probes cannot run. |
| Reader credential (optional) | reader_credential_secret_id | camera_archive_read (HEAD, GET) | Defaults to reusing the writer credential scoped down to the read-only :reader broker role. |
| Retention delete credential (optional) | retention_credential_secret_id | camera_archive_retention (DELETE, HEAD) | Displays Object retention not enforced: no delete credential; expired and capacity-reaped archive objects are never deleted until a retention credential is bound. |
Selecting or changing any of the three credential references requires both
settings.recording.manage and settings.credentials.manage. Administrators
with settings.recording.manage alone see the bound credential's redacted name
and a disabled selector explaining the missing permission. Bound credentials are
protected against deletion by GuardCredentialDestroy while referenced.
3. Cameras (?tab=cameras)
The Cameras tab inspects per-camera NVR block reservations and cataloged
copy locations in keyset-paginated pages of 25 rows rendered via LiveView
streams (listing camera sources and relay-eligible stream profiles additionally
requires devices.view):
- Core Record (
RecordingBlock&RecordingCatalogEntry): Distinguishes sealed cataloged blocks from uncatalogedopen_localCore reservations and surfacesRecordingBlock.reconciliation_state(unresolved,adopted,block_absent,unrecoverable_journal). Core's reservation state is never presented as local NVR seal state. - Local Copy (NVR): Shows cataloged
local_state(present,quarantined,reclaiming,reclaimed,missing), media byte size, and fragment count.provisionalbadge: Local copies carry aprovisionalindicator untilLocalStorageBindingregistration and current storage qualification exist for thatstorage_id.unknownvsabsent: When an uncataloged reservation's NVR lookup is unreachable or unavailable, the local copy is reported asunknown, never as missing or absent.
- Archive Copy (Catalog & Receipt): Shows
archive_state(not_required,awaiting_binding,pending,failed_retryable,failed_terminal,verified,deleting,deleted,missing) alongside verifiedNVRArchiveReceiptobject details (init.mp4,media.m4s,descriptor.jsonkeys, byte lengths, SHA-256 digests, ETags, optional receipt v2version_id, andverified_at). - Hold Pin: Displays active
RecordingHoldpin badges (archiveorlocal) orNo active hold. Holds are placed and released on the camera Timeline page, not in Settings. - Timeline Window (Time Basis): Shows the derived UTC wall-clock window
tagged with
receivedtime basis (uncertainty: unmeasured) and atime re-anchoredbadge when receipt clock drift caused a re-anchor without a media gap. Quarantined entries displayNot playable (quarantined).
4. Policy (?tab=policy)
The Policy tab configures per-camera, per-profile continuous recording
intent (CameraRecordingPolicy, platform.camera_recording_policies,
AshPaperTrail-audited). Keeping policy in its own resource ensures camera
discovery syncs never overwrite operator recording intent.
Every policy save atomically increments policy_revision and acts as a
recording-plane assignment transition: existing blocks retain the storage_id
and archive_binding_id they were recorded under.
Policy fields
- Camera source & relay-eligible stream profile: Locked to one policy per
(camera_source_id, stream_profile_id)pair. The form displays each profile's codec hint, observed bitrate, estimated daily storage bytes, and estimated daily block count (unknown until recordedwhen a profile has never streamed). - Recording mode (
mode):off(default) orcontinuous. - Assigned NVR member (
nvr_member): Target StatefulSet member name (for exampleserviceradar-nvr-0), required whenmodeiscontinuous. - Storage tier (
storage):local_and_archive-- seals locally on the NVR spool and queues archive upload after sealing (default when deployment archiving is enabled).local_only-- seals locally witharchive_state = not_required(PD8). Allowed even when deployment archiving is enabled for other cameras; its blocks follow the deployment's local-only retention rules.
- Archive retention override (
archive_retention_override_mode):inherit(uses deploymentarchive_retention_days),limited(1..3,650days viaarchive_retention_days_override), orunlimited(no time expiry). For example, a camera can record a high-bitrate main stream with a short override and a low-bitrate substream with long archive retention.
Pre-save What-If capacity and throughput preview
Changing any form input updates the Pre-Save What-If Capacity & Throughput
Projection (C1a) panel live before saving:
- Projected local window &
time_to_fullon the target NVR member, naming which bound binds first (budget,max_local_age, orblock_cap). - Archive throughput (
~2xcapture rate) needed for upload plus full readback verification, compared against trailing measured provider throughput and archive goodput. - Projected
blocks/dayagainst the member's volume block cap (max_blocks). - Active continuous sessions on the target member against the session limit
(
max_sessions, default1..16viaSERVICERADAR_NVR_MAX_SESSIONS). - Object effective retention &
days_to_capacityacross policies. - Simulated nominal bitrate (
nominal_bitrate_kbps, preview-only): When a stream profile has not been measured yet (unknown until recorded), entering a nominal bitrate inkbit/smodels local window, block rate, and archive throughput impact before saving without persisting the simulated value.
Non-blocking advisory warnings highlight if the projected low-water window drops
below the window alert threshold, sessions exceed max_sessions, required
provider throughput exceeds measured throughput, the volume block cap binds
before bytes, or local_only is chosen without a confirmed
local_only_full_policy.
5. Retention (?tab=retention)
The Retention tab manages deployment-wide local and object-tier retention on
RecordingArchiveSettings, displays live CNPG usage and RetentionCalculator
projections, provides an interactive What-If rate/spool simulator (camera_count,
nominal_bitrate_kbps with the 1.02 CMAF packaging factor, and
local_budget_gib), and previews eviction consequences before saving. Every
optional limit is stored as an explicit mode + value pair enforced by SQL
CHECK constraints -- NULL never stands in for a hidden default or sentinel
number. The tab remains fully usable when object storage is disabled (Not used: local only).
Local tier watermarks and maximum local age ceiling
Local NVR storage acts as a capacity-driven cache rather than a minimum-duration guarantee:
- Watermarks: Space eviction begins when usage reaches
local_high_water_percent(default90%, bounds50..95%of the member's applied budget) and evicts oldest space-evictable blocks first until usage reacheslocal_low_water_percent(default80%, bounds40..(high_water - 5)%). - Maximum local age ceiling (
max_local_age_mode+max_local_age_seconds): Defaults tolimitedat86,400seconds (24hours), configurable across3,600..315,360,000seconds (1hour to10years) orunlimited(No limit). This is an upper ceiling ("keep at most this long locally"), never a minimum floor: high-water space eviction can reclaim archive-verified local blocks earlier when disk usage requires it. - Effective local window alert (
local_window_alert_mode+local_window_alert_seconds): Defaults toenabledat3,600seconds (1hour, bounds60..31,536,000s) ordisabled; warns when the low-water effective window drops below the threshold. - Never evicted automatically: Unarchived, deleting, or missing-archive
blocks (
kind (c)), unreconciled disk blocks (P2), local-pinned held blocks (P4), and quarantined blocks (row (q)/P5) are never selected by space or age eviction.
Local-only mode full policy (local_only_full_policy)
Blocks recorded with archive_state = not_required (kind (b) local-only
footage) never reuse max_local_age_seconds. Instead, they follow dedicated
local-only settings:
stop_when_full(default behavior when unset): Local-only blocks are pinned inP3and never evicted for space; when the member's spool budget fills, new capture admission stops and records a gap.overwrite_oldest(confirmed ring buffer): Evicts the oldest unheld local-only blocks from high water down to low water alongsidekind (a)blocks. Because no archive copy exists, savingoverwrite_oldestrequires ticking an explicit confirmation checkbox, recordinglocal_only_full_policy_confirmed_by_idandlocal_only_full_policy_confirmed_atin CNPG.- Local-only target alert & age ceiling:
local_only_target_mode(targetdefault604,800s /7days, bounds3,600..315,360,000s, ornone) alerts when the ring-buffer window shrinks below target;local_only_max_age_modedefaults tounlimited(Keep until the volume is full) orlimited(3,600..315,360,000s).
Application-managed object retention and object-tier byte budget
ServiceRadar enforces archive expiry and capacity reaping in the application
using the :retention (DELETE, HEAD) credential role so every object
deletion serializes with CNPG catalog locks and evidence holds. Configure bucket
lifecycle rules on the storage provider only for aborting incomplete
multipart uploads and expiring _probe/ connectivity-test objects -- never on
the recorded footage prefix.
- Object archive retention (
archive_retention_mode+archive_retention_days): Defaults tolimitedat30days (bounds1..3,650days) orunlimited(No limit). Setting a limited archive retention shorter than a limited maximum local age displays a non-blocking Single-copy local tail warning (archive objects will expire before local copies age out). - Optional object-tier byte budget (
C3a): Setobject_budget_modetounlimited(default) orlimited(object_budget_bytes >= 1 GiB/1,073,741,824bytes), withobject_high_water_percent(default90%,50..95%),object_low_water_percent(default80%,40..(high - 5)%), andobject_capacity_policy:stop_archiving(default behavior when unset): holds archive upload jobs once usage reaches capacity while local NVR recording continues.reap_oldest: deletes only the archive copies of the oldest unheld blocks from high water down to low water, keeping any local copy on disk. Requires explicit actor confirmation (object_capacity_policy_confirmed_by_idandobject_capacity_policy_confirmed_at) acknowledging that later local eviction of reaped blocks removes the last copy. (On self-hosted Phase 1 installs, the object budget computes effective retention and raises capacity alerts while the background reaper flag remains disabled.)
- Same-bucket usage accounting: Active archive usage (
archive_used_bytes), in-flight uploads, andnon_reapable_bytes(held archive copies, in-flight uploads,blocked_by_providerentries, and_quarantine/objects) are summed across all retainedArchiveBindinggenerations sharing the active{endpoint, bucket}, regardless of key prefix. Bytes retained in other buckets are displayed separately and excluded from the active budget.
Hold pin rules (PD5) and local_only_after_archive semantics
PD5hold pin rule: An activeRecordingHoldguarantees at least one verified copy without letting multi-week investigations exhaust hot NVR disks:- When a covered block has
archive_state = verified, the hold pins the archive copy (counted innon_reapable_bytes), leaving its local copy evictable by local watermarks andmax_local_age. - In local-only mode, while a block is still unarchived, if the archive copy
was already deleted before the hold was placed, or if a held archive copy
becomes
missing, the hold pins the local copy (P4). When an unarchived held block later reachesverified, the pin moves atomically to the archive copy. - If
non_reapable_bytesreach object high water, new archive uploads pause and raise an alert while local capture continues.
- When a covered block has
local_only_after_archive: When time-based object retention orreap_oldestdeletes an archive copy (archive_state = deleted, witharchive_deleted_reasonset toobject_retention,object_capacity, oroperator), any remaining local NVR copy becomes local only after archive expiry/reaping (kind (a)). It is never counted as unarchived, is never re-archived automatically (preventing upload loops), and remains evictable under local watermarks andmax_local_age. The Retention tab reports totalLocal only after archivebytes and the timestamp of the oldest such block.
6. Health (?tab=health)
The Health tab aggregates live CNPG backlog, capacity, reconciliation, disposition, gap, and alert state across NVR storage members:
- Archive Upload Backlog: Counts and bytes of unarchived catalog entries
(
awaiting_binding,pending,failed_retryable, andfailed_terminal), pinned inP1, plus the oldest unarchived block age (1hour alarm threshold). - Per-Storage Local Capacity & Effective Window: Per
storage_id(00000000-0000-4000-8000-000000000011), shows the calculated effective local window (local storage holds about ...) and binding bound (budget,max_local_age, orblock_cap), present local bytes/blocks, disjointC1apinned bytes (P1unarchived/deleting/missing +P2unreconciled +P3pinned local-only +P4local-pinned holds +P5quarantined), pinned-growthtime_to_full, assigned continuous camera count, and live disk/ledger status (unknown until NVR /v1/status reportsuntil the member status endpoint answers). - Core Reservation Resolution Breakdown: Counts
RecordingBlockrows byreconciliation_state(unresolved,adopted,block_absent, andunrecoverable_journal), plus 24-hourRecordingDispositionstep and failure totals. - Unattributable / Quarantined Blocks (
local_state = quarantined): Blocks whose journal could not be recovered (unrecoverable_journal), that lacked Core reservation/PD1 continuation authority (unattributable), or whose unverified archive binding lacked history or was ended as compromised (binding_without_history,binding_compromised) are isolated inrow (q)(P5) and excluded from playback and automatic eviction. Operators withsettings.recording.operatehave two audited actions:- Export / Preserve Intent -- records an audited preservation/export operation for inspection.
- Audited Delete Intent -- requires ticking an explicit confirmation
checkbox (with an optional audit note) to record a permanent
operator_quarantineRecordingDispositiondelete intent.
- Missing Archive Copies (
archive_state = missing): Lists catalog entries whose archive object was found absent or altered (raisingarchive_missing, plus criticalheld_archive_missingif covered by an accepted hold). Operators withsettings.recording.operatecan:- Re-archive (offered when
local_stateispresentorquarantined) -- queues a transition-8 re-archive under the entry's recorded binding. A re-archive never trusts a retained local receipt; it succeeds only when local digests match the frozen descriptor (or quarantine receipt),HEADconfirms the bucket objects are absent, the recorded binding remains in non-compromised history, and freshPUT+ readback verification pass. - Accept Archive Loss -- records an audited
archive_loss_accepteddisposition (with optional operator note) and transitionsarchive_statetonot_required(leavinglocal_stateunchanged, so a quarantined entry remainsquarantined). When an active hold covers the entry, accepting archive loss additionally requiresdevices.cameras.recordings.hold.
- Re-archive (offered when
- 24-Hour Recording Gaps & Active Health Alerts: Summarizes trailing
24-hour and open
RecordingGaprows by reason, alongside deduplicated, auto-clearingTripwireHealth/HealthTrackerchecks (storage.recordings.*andnvr.*).
Unified timeline playback and holds (/cameras/:camera_source_id/recordings)
Opening /cameras/:camera_source_id/recordings requires both devices.view and
devices.cameras.recordings.view plus fresh camera Source scope (devices.cameras.view
for live video never grants recorded playback). No database queries run during
disconnected LiveView mount, and every LiveView event and HTTP media request
re-verifies active user permissions and camera scope before releasing data.
Timeline mode vs Published (ArchiveUploads) mode
The page header provides a mode toggle between two distinct recording catalogs:
- Timeline (
?mode=timeline): Selected automatically whenever the camera has anyRecordingCatalogEntry,RecordingGap,RecordingHold, orCameraRecordingPolicyrows (RecordingTimeline.has_timeline_data?/2). Plays seamlessly across consecutive NVR blocks from local NVR storage and verified object-archive copies. - Published (
?mode=published): Lists single-block recordings published through the legacyArchiveUploadsworkflow (/api/cameras/:camera_source_id/recordings/:id/playlist.m3u8), labelled explicitly so operators never confuseArchiveUploadspublication times with NVR catalog receipts.
Coverage bar, time basis, and playback controls
In Timeline mode, operators can filter by relay-eligible stream profile,
switch the coverage span (1h, 6h, 24h, 7d, with Previous, Next,
Latest, and Refresh navigation), and choose playback source mode (Auto,
Force local, or Force archive).
The coverage bar aggregates non-quarantined catalog blocks and gaps into bounded
buckets (120 in the UI; up to 1,440 buckets across at most 31 days via the
API) using eight states:
| Coverage state | Meaning |
|---|---|
Local + archived (local_and_archived) | Present on the owning NVR spool and verified in object archive storage. |
Archive only (archive_only) | Local copy reclaimed or absent; playable from the verified object archive copy. |
Pending archive (pending_archive) | Present locally (awaiting_binding, pending, failed_retryable, failed_terminal, deleting, or missing); playable from the local copy while pinned in P1. |
Local-only footage (local_only) | Recorded under a local_only policy or with archiving disabled (archive_state = not_required with no archive binding). |
Local only (archive expired/reaped) (local_only_after_archive) | Archive copy was deleted by object retention or capacity reaping while the local NVR copy remains present and evictable under local retention. |
Gap (gap) | Recorded RecordingGap interval displaying its reason (for example nvr_refused, incomplete_tail, block_absent, keyframe_wait, or quarantined) and open/closed timestamps. Quarantined entries are never playable and appear as a quarantined gap only when both camera and media interval are known. |
Not scheduled (not_scheduled) | No continuous recording policy covered the interval. |
Deleted by retention (deleted_by_retention) | Local copy reclaimed and archive copy deleted (or not_required). |
Above the bar, banners and markers surface timing and durability provenance:
- Explicit
receivedtime basis: DisplaysTime basis: received (uncertainty: unmeasured). Wall-clock positions are derived fromcore-elx's UTC receipt timestamp of the first access unit plus 90 kHz media PTS spacing -- never unverified camera clocks. - Clock re-anchor markers (
Clock re-anchored): Marks points where receipt clock drift triggered a timeline re-anchor without a media gap. provisionalbadge: Indicates local copies on NVR storage lifetimes that do not yet haveLocalStorageBindingregistration and current qualification.- Adjacent-window loading: Each HLS VOD playlist covers up to
60minutes (2,048segments). As playback nears the end of the active window, theCameraTimelinePlayerhook automatically advances into the adjacent window.
Timeline HTTP API, copy resolution, and read bounds
All timeline endpoints enforce Cache-Control: private, no-store and
X-Content-Type-Options: nosniff, reject client-supplied provider URLs or
object keys, and forbid HTTP Range headers on application endpoints (400 invalid_request):
| Endpoint | Purpose & limits |
|---|---|
GET /api/cameras/:camera_source_id/timeline?from=&to=&profile= | Returns coverage buckets (1..1,440), non-quarantined blocks, gaps, holds, and clock re-anchor markers for spans up to 31 days (416 recording_window_not_satisfiable if exceeded). |
GET /api/cameras/:camera_source_id/timeline/playlist.m3u8?from=&to=&profile=&source= | Emits an HLS v7 VOD playlist for windows up to 60 minutes and 2,048 segments (EXT-X-MAP deduplicated per init_sha256, EXT-X-DISCONTINUITY across init/epoch/gap boundaries, EXT-X-PROGRAM-DATE-TIME), and writes one RecordingAccessAudit row. |
GET /api/cameras/:camera_source_id/timeline/blocks/:entry_id/init?source= | Serves verified CMAF initialization bytes (<= 1 MiB) with Content-Type: video/mp4 and X-Recording-Copy. |
GET /api/cameras/:camera_source_id/timeline/blocks/:entry_id/fragments/:index?source= | Serves verified fragment 0..4095 (<= 4 MiB) with Content-Type: video/mp4 and X-Recording-Copy. |
Source modes, SHA-256 verification, and X-Recording-Copy
source=auto(default): Attempts apresentlocal NVR copy first (2 stimeout). On timeout, unavailability, or not-found, falls back automatically to averifiedarchive copy.source=local/source=archive(forced source): Reads only the requested copy and never falls back (409 recording_unavailablewhen absent).- Cryptographic digest verification: Before releasing any init or fragment
bytes,
CameraTimelineReaderverifies exact byte length and lowercase SHA-256 against the block's frozen descriptor, andCameraTimelineControllerre-checks user permissions and camera scope. If a local read fails SHA-256 or length verification, the corrupt bytes are discarded, the catalog entry's local copy is markedmissing(local_integrity), andautomode falls back only if averifiedarchive copy exists (otherwise returning409 recording_integrity_failed). X-Recording-Copyresponse header: Every200init/fragment response includesX-Recording-Copy: local,X-Recording-Copy: local; provisional, orX-Recording-Copy: archive, updating the player's liveServing:indicator.- Bounded in-flight read admission (
CameraTimelineAdmission): Enforces non-queuing ETS counters of at most4concurrent init/fragment reads per playback session ({user_id, camera_source_id}) and64concurrent reads perweb-ngnode. Excess requests return503 Service UnavailablewithRetry-After: 1and{"error": "playback_capacity_exhausted"}.
Append-only access audit (RecordingAccessAudit)
Every issued playlist window and every hold placement or release appends an
immutable row to platform.camera_recording_access_audits (event_kind in
playlist_window, hold_placed, hold_released), recording the actor ID,
camera, stream profile, [window_start_us, window_end_us), source_mode,
time_basis, segment count, and metadata. Database triggers refuse UPDATE and
DELETE on audit rows. Individual fragment fetches are covered by their issued
playlist window rather than writing per-fragment rows.
Placing and releasing investigation and export holds
Users with devices.cameras.recordings.hold (along with devices.view and
devices.cameras.recordings.view) can place and release holds in the
Recording holds panel on /cameras/:camera_source_id/recordings:
- Select a coverage bucket (which pre-fills
Start (us UTC)andEnd (us UTC)for the active playback window) or enter explicit UTC microsecond bounds. - Choose Hold kind (
InvestigationorExport), enter a required ticket or incident Reason, and click Place hold. RecordingHolds.place_hold/2locks intersecting non-quarantinedRecordingCatalogEntryrows (SELECT ... FOR UPDATEin primary-key order) in one transaction withRecordingHoldandRecordingAccessAudit, applying thePD5pin rule (Pinned: archivewhen verified, otherwisePinned: local). Any block that already had an in-progress deletion with no surviving pinnable copy is recorded inunavailable_blocks(unavailable, deletion in progress) and flagged on the hold card.- Holds never expire automatically on disconnect, policy edits, or disk pressure. To release a hold, enter an optional release note on the accepted hold card and click Release.
RBAC permissions summary
All camera recording and settings permissions default to admin (except
devices.view, which defaults to all built-in roles) and can be delegated
through custom role profiles (see Roles & Permissions):
| Permission | Default roles | What it authorizes |
|---|---|---|
devices.view | viewer, helpdesk, operator, admin | Required alongside camera permissions to resolve camera inventory, list camera sources and stream profiles in Settings, and open /cameras/:camera_source_id/recordings. |
devices.cameras.recordings.view | admin | Open /cameras/:camera_source_id/recordings (Timeline and Published modes), query timeline coverage, issue HLS VOD playlists, and stream verified init/fragment bytes (auto, local, or archive). |
devices.cameras.recordings.hold | admin | Place and release investigation or export holds on the camera Timeline, and authorize Accept Archive Loss on /settings/recording?tab=health when an entry is covered by an active hold. |
settings.recording.view | admin | Open /settings/recording and inspect recorder status, desired/applied storage budgets, archive bindings, redacted credential names, block locations, policies, retention projections, and health summaries. |
settings.recording.manage | admin | Register immutable operator:<name> archive destinations, choose the active destination, toggle archive_enabled, save per-camera CameraRecordingPolicy rows, save deployment retention settings, and save per-member NVRStorageSettings budgets and reserves. |
settings.credentials.manage | admin | Required in addition to settings.recording.manage to list and change the writer, reader, or retention s3_compatible credentials on /settings/recording?tab=archive. |
settings.recording.operate | admin | Run supervised recorder operations (bounded captures, manual archive uploads, connectivity probes, retrieval checks) and execute audited Health tab actions (quarantine export/preserve intent, confirmed quarantine delete intent, missing-archive re-archive, and unheld missing-archive loss acceptance). |
Operational note: immediate control-plane enforcement vs runtime workers
When operating /settings/recording and /cameras/:camera_source_id/recordings,
distinguish between control-plane state enforced immediately in CNPG/web-ng
and background runtime workers in core-elx and serviceradar-nvr:
- Enforced immediately in CNPG and
web-ngtoday:- All RBAC checks, split-duty credential authorization,
GuardCredentialDestroyprotection across writer/reader/retention slots, and immutableArchiveBindingv2 destination registration. - Auditing and SQL constraints on
CameraRecordingPolicy,NVRStorageSettings(pinned_bytes + 32 MiBfloor), andRecordingArchiveSettings(watermarks, age ceilings, confirmedlocal_only_full_policy, and confirmedobject_capacity_policy), plus liveRetentionCalculatorusage and What-If projections. - Historical block freeze (
freeze_historical/3),PD1rotated-block adoption, reservation resolution tracking, transactionalRecordingHoldplacement/release (PD5), append-onlyRecordingDispositionandRecordingAccessAudittriggers, and audited quarantine / missing-archive operator intents. - Unified Timeline UI, coverage bucketing, multi-block HLS v7 VOD playlist
generation, forced-source routing (
auto,local,archive), SHA-256 digest verification,X-Recording-Copyheaders, and 4-per-session / 64-per-node read admission.
- All RBAC checks, split-duty credential authorization,
- Requires active
core-elxandserviceradar-nvrruntime workers:- Continuous capture & auto-archive: Saving a
continuouspolicy persists the assignment in CNPG immediately; unattended continuous capture and background upload retries execute throughContinuousRecorderandArchiveBlockJoboncore-elx. - Volume reconciliation & live member sync: Automatic 5-minute inventory
scans, journal recovery, and desired budget pushes execute through
NVRReconcileJob,NVRStorageSettingsSyncJob, and the NVR/v1/statusandPUT /v1/storage/settingsendpoints. - Background eviction & byte-range media readers: Automatic local disk
reclamation (
LocalRetentionJob+ NVRPOST /v1/blocks/<id>/reclaim), expired object deletion (ArchiveRetentionJob), and self-hosted capacity reaping (gated behind the disabled Phase 1 reaper flag) run as background jobs; until local reclamation is active, NVR spools retain sealed blocks and refuse new capture admission once spool budget is reached. Similarly, timeline init/fragment streaming requires the NVR local read listener and per-request isolated archive reader (CameraTimelineReaderreturns503 recording_read_unavailablewhen neither reader is configured).
- Continuous capture & auto-archive: Saving a