Deployment

Local development

# 1. PostgreSQL
createdb mydlp
psql -c "CREATE ROLE mydlp LOGIN PASSWORD 'dev'"
export DATABASE_URL="postgres://mydlp:dev@127.0.0.1/mydlp"

# 2. Migrate and bootstrap
cargo run -p dlp-management-api -- --migrate-only
DLP_BOOTSTRAP_PASSWORD='DevPassword123!' \
  cargo run -p dlp-management-api -- --bootstrap \
    --bootstrap-tenant acme --bootstrap-email admin@acme.test

# 3. API
cargo run -p dlp-management-api          # 127.0.0.1:8080

# 4. Dashboard
npm --prefix apps/dashboard ci
npm --prefix apps/dashboard run dev      # 127.0.0.1:5173, proxies /api

# 5. Agent
cargo build -p dlp-agent
cat > /tmp/agent.toml <<'TOML'
server_url = "http://127.0.0.1:8080"
state_dir = "/tmp/dlp-agent"
TOML
DLP_AGENT_CONFIG=/tmp/agent.toml ./target/debug/dlp-agent enroll --token <token>
DLP_AGENT_CONFIG=/tmp/agent.toml ./target/debug/dlp-agent scan

Plain http:// works only in debug builds. A release agent refuses it, because the agent transmits enforcement decisions - verified in production during bring-up when the release binary rejected an http:// URL.

Docker Compose

The development path. PostgreSQL, the management API and the dashboard, all as non-root containers with read-only root filesystems.

cp deploy/docker/.env.example deploy/docker/.env
# set POSTGRES_PASSWORD in that file, then:
docker compose -f deploy/docker/docker-compose.yml up --build

The password step is not optional and compose will not start without it. An unset POSTGRES_PASSWORD makes the official PostgreSQL image fall back to trust authentication - a database anyone who can reach the port may write to - so the compose file fails on the missing variable instead of defaulting.

Dashboard	http://localhost:5173
API	http://localhost:8080
PostgreSQL	not published; use docker compose exec postgres psql -U mydlp

Both published ports bind to 127.0.0.1, not to every interface. Postgres is not published at all: a development database on a laptop's wifi interface is a database on the café's network.

Migrations run as their own one-shot service rather than on API start-up, and the API waits for it to succeed. Two API replicas starting together would otherwise race each other through the same migration, and that failure looks like a corrupt schema rather than like a race.

Data lives in a named volume, mydlp-pgdata. docker compose down keeps it; docker compose down -v is the deliberate way to discard it.

VITE_API_BASE_URL is baked into the dashboard bundle at build time, so it must be the address the browser uses - not the compose service name, which a browser cannot resolve. Changing it needs --build again.

Production does not use any of this; it runs deploy/deploy.sh onto a host with systemd and nginx, described below.

Production (the deployment running at dlp.erpona.app)

Production is released by a job, not by whoever pushed (DLP-632). deploy/systemd/mydlp-release.timer runs deploy/cadence.sh at quarter past every second hour, from a checkout of its own at /var/lib/mydlp/release. If anything reached main since the last release, the job cuts release: vX.Y.Z (tools/release.py) and runs deploy/deploy.sh --ref=<that commit>. The outcome, success or failure, is written to /var/lib/mydlp/release-status.json. The log is in /var/log/mydlp-release/. A failure also leaves the unit failed. The job's fetches and pushes are tried up to three times when the remote refuses in the way GitHub's SSH endpoint intermittently does (publickey refusal, dropped connection); any other git error is not retried, and after the third refusal the release fails and the status file says why (DLP-754). If another deploy holds /var/lock/mydlp-deploy.lock when the job starts, it cuts nothing: a timer run skips its slot and exits 0, and cadence.sh --now says a deploy is running and exits 3 (DLP-755).

A release that ends without recording its end (DLP-1496). While a release runs, the status says deploying and names its owners: the job's pid and its deploy's, each with the kernel's start time, so a reused pid does not pass for the run. A job that exits or is signalled records failed with the cause ("interrupted": "SIGTERM") and the log's last line, unless the deploy it started is still running. Nothing here stops a running deploy. A run killed with nothing left to record its end (SIGKILL, the whole group gone) is recorded by the next reconcile, once neither owner exists. That happens at the start of every slot, when the release unit stops (ExecStopPost, from the unit file in deploy/systemd/ once that copy is installed), or on demand with deploy/cadence.sh --reconcile. An owner that cannot be read counts as alive. Only the job writes deployed, from the deploy's exit status and its verification block, and only while the status still names it, so an old run ending late never overwrites a newer release. A deploying status with no owners, written before this, is left as it is.

The release is run by origin/main's cadence.sh (DLP-1899). After it resets the release checkout, a copy that differs from that checkout's deploy/cadence.sh prints "handing over" and re-runs as it, with the same arguments. A run that has already handed over and still differs stops with exit 5 and a failed status, rather than looping. A copy from before this cannot hand over, which is how a pre-DLP-1496 copy in a stale checkout left v1.68.0's status deploying with no owner.

Whether CI actually ran (DLP-1220). Every slot, including the ones that release nothing, runs tools/ci_state.py record. It reads the latest completed CI run on main from the Actions API and writes /var/lib/mydlp/ci-status.json with one of these states:

• did-not-run: every job executed zero steps. This is billing, runner availability, or a workflow that cannot be parsed, not a test.
• failed
• passed
• pending
• unknown: the API did not answer, or no token is configured.

The first slot that moves into did-not-run or failed files one tracker ticket; later slots of the same outage file nothing. It never holds a release. GitHub being down, or CI not running, is reported, not gated.

It needs a fine-grained GitHub token with actions:read on this repository only, in /root/.config/mydlp/github-actions-token, mode 0600, created by the owner. Without it the state is unknown, and the file says which path was looked for. This check exists because GitHub-hosted runners started no job for sixteen days, from 2026-09-15 to 2026-10-01, and a run that never started showed the same red "failure" as a failing test (DLP-1215).

deploy/cadence.sh --now            # a release that cannot wait
deploy/deploy.sh --ref=<sha>       # redeploy a commit as it is, without a release

Classification overview rollup (DLP-1602). The release migration adds transactional event triggers but does not backfill historical paths. Until a tenant is backfilled, the overview continues using the original event query. On the deployed host, run the released dlp-management-api binary with the same protected service environment as mydlp-api and first use --backfill-classification-rollups (read-only counts, no paths). Review the counts and database capacity. Only then run with both --backfill-classification-rollups --apply-classification-rollups; this is idempotent and commits one tenant at a time. Do not place DATABASE_URL in shell history, command arguments or logs. A failed tenant transaction leaves that tenant on the original overview query and can be retried.

The agents are rebuilt only when something they are built from has changed since their last build: agent/, packages/, the toolchain, the Windows packaging, the root Cargo.toml, or a dependency the agents build. A change to a version line alone does not count. A Cargo.lock change is judged by what cargo resolves for the agent packages on each agent target (tools/agent_build_inputs.py, DLP-1459): packages, versions, sources, features and checksums. So a dependency only the server gained keeps the agents. The lock alone could not say that, because it records every workspace member's features at once. If the comparison cannot be made (cargo fails, or the earlier tree cannot be opened), the agents are rebuilt, and the reason line says why. Otherwise the published packages stay as they are, and the verification block checks them against the version they were built at. The server and console are built every time. The server therefore measures an endpoint as current or outdated against the newest published agent package, and falls back to its own version only when no package says what it is.

deploy/deploy.sh does:

1. Checks out the commit being deployed into a detached worktree, and builds from there rather than from your working tree.
2. Builds release binaries and the dashboard. Then, before installing anything, times the detector engine against a calibration loop on each 16 KB piece of a fixed corpus, one straight after the other. It stops the release if the median of those paired ratios is more than 1.5 times its baseline (DLP-1785, DLP-2043, packages/detector-engine/tests/scan_throughput.rs). The ratio, not the time, is compared, because the host is shared. Pairing keeps both halves under the same load: idle it read 0.52-0.55; during a parallel cargo build (load 12-18) 0.57-0.63; with 16 busy loops 0.57-0.59. An engine made to scan twice read 0.96 idle and 1.14-1.20 during the build, and failed. The line is repeated in the verification block.
3. Creates the mydlp service user and /var/lib/mydlp.
4. Writes /etc/mydlp/api.env (mode 0600) if absent.
5. Installs binaries to /opt/mydlp/bin.
6. Runs migrations.
7. Copies the dashboard to /var/www/dlp.erpona.app - nginx cannot read /root, so a copy is required.
8. Installs and starts mydlp-api.service.

What is deployed, and why it is not your working tree (DLP-483)

By default the deploy builds origin/main, not whatever is in your checkout. --ref=<commit-ish> chooses something else.

This replaced a clean-working-tree check that was correct and measured the wrong property. The old script tested that the tree was clean once, at the start, and then built for ten to fifteen minutes. The tree was clean at 14:02 is not these bytes are that commit, and no amount of strictness in that gate closes the window between them - CLAUDE.md §4 records v0.146.0 doing exactly this, where edits landed mid-build and the Windows stage failed on a tree that compiles.

So the fix is a different source, not a better gate. Consequences worth knowing:

• A dirty working tree no longer blocks a deploy. Somebody else editing the checkout cannot stop a release, and - the point - cannot alter one either.
• An unpushed commit is refused. Code nobody else can see must not run on a server. --allow-unpushed overrides it, and you should have a reason.
• The build cache is shared with target/, deliberately: it is 192 GB and a second copy has twice filled this disk. The sources differ, the artefacts do not. The consequence is that a deploy holds the cargo build lock, so a concurrent cargo test blocks until it finishes. That is a better failure than a mislabelled artefact, and it is stated here rather than left to be discovered.
• --dry-run resolves and verifies the commit without building or touching the server. tests/deploy_source.sh uses it.

One deploy at a time

deploy/deploy.sh takes an exclusive lock on /var/lock/mydlp-deploy.lock and refuses rather than waits if another deploy holds it.

Both refusals come from the same incident. On 2026-09-17 two sessions deployed to this host within minutes of each other; each built from its own checkout state, each installed its binaries over the other's, and the service ended up running v0.51.2 while the deploy that had just finished believed it had installed v0.51.1. The version check at the end caught it - after the service had already been restarted twice.

Waiting for the lock would be worse than refusing it. The second deploy's tree is not newer than the first's; it is simply different, and usually older. A queued deploy would therefore roll the first one back, quietly, as its final act. Refusing makes somebody look.

The dirty-tree check exists because /healthz reports the build commit (#440). Code deployed from an uncommitted tree corresponds to no commit, so that field becomes a number nobody can resolve - which is worse than having no field, because it reads as authoritative.

Service hardening

deploy/mydlp-api.service runs as a dedicated non-root user under a systemd sandbox: ProtectSystem=strict, ProtectHome, PrivateTmp, PrivateDevices, NoNewPrivileges, MemoryDenyWriteExecute, an empty CapabilityBoundingSet, a @system-service syscall filter, and ReadWritePaths=/var/lib/mydlp as the only writable location.

The API binds 127.0.0.1 only. The sole route in is nginx.

The agent's hardening and the isolated parser are in tension, on purpose

deploy/mydlp-agent.service sets RestrictNamespaces=yes and a CapabilityBoundingSet of CAP_DAC_READ_SEARCH alone. That is right for an agent that reads files and talks to one server, and it is exactly what stops the isolated parser helper (DLP-230) from starting: creating a network namespace needs CAP_SYS_ADMIN, and dropping to an unprivileged uid needs CAP_SETUID.

So on a deployed endpoint dlp-agent doctor reports the sandbox as unavailable with the reason, and anything that depends on it - OCR, when it lands - does not run. That is the intended state today and not a fault: the mechanism exists, nothing needs it yet, and the agent stays as narrow as it can be.

Widening the unit to permit it is a real trade and belongs to the ticket that gives it a tenant, not to this paragraph. When it is made, it is RestrictNamespaces=~ for the namespaces actually used and CAP_SYS_ADMIN CAP_SETUID CAP_SETGID added to the bounding set - which is a larger agent in exchange for a parser that cannot reach the agent at all. The helper binary also has to be installed somewhere the unprivileged uid can read and execute, because it runs as nobody.

nginx

/etc/nginx/sites-available/dlp.erpona.app:

• TLS 1.2/1.3, HSTS with includeSubDomains, OCSP stapling
• A strict CSP: no unsafe-inline for scripts, frame-ancestors 'none'
• Separate rate limits: 5/minute for /auth/login, 120/minute for agent traffic
• The ACME webroot is matched before the HTTPS redirect, so renewal keeps working
• SPA fallback to index.html; hashed assets cached for a year
• A 16 MiB request body site-wide, and 128 MiB on /api/v1/edm-datasets/import-file only, to match DLP_EDM_MAX_UPLOAD_BYTES. That path also has proxy_request_buffering off, so an EDM source file is not written to a temporary file on the proxy host (DLP-1097, PRIVACY_MODEL.md §5). Raise both together: nginx refuses first, and a limit above the API's only moves the refusal.
• An access log without query strings: log_format mydlp_noquery in /etc/nginx/conf.d/mydlp-log-format.conf logs $uri in place of the request line, and a Referer cut at ?. The default combined format records the whole URI, and a query string can carry free text that names a person (DLP-952). Set per server block, so other sites on the host keep their own format.

Certificates

Let's Encrypt via webroot, ECDSA:

certbot certonly --webroot -w /var/www/certbot -d dlp.erpona.app --key-type ecdsa

certbot.timer handles renewal. Verify with certbot renew --cert-name dlp.erpona.app --dry-run.

For the Settings certificate inventory, publish a public-only copy and keep it fresh with Certbot's successful-renewal deploy hook. On this host, install deploy/publish-console-tls-certificate.sh as executable at /etc/letsencrypt/renewal-hooks/deploy/mydlp-public-certificate.sh, then run it once as root for the existing lineage. Set a systemd drop-in for mydlp-api.service containing Environment=DLP_TLS_CERTIFICATE_FILE=/var/lib/mydlp/console-fullchain.pem, reload systemd, and restart only mydlp-api. The hook copies fullchain.pem atomically after renewal; it never opens privkey.pem. The API reports a parsed copy, not proof of the certificate currently served by nginx or of chain trust. A Certbot dry-run can exercise deploy hooks with --run-deploy-hooks; it uses the active certificate, not the temporary test certificate.

Cloudflare in DNS-only mode is what the deployment uses; HTTP-01 reaches the origin directly. Behind a proxying CDN, set DLP_PUBLIC_URL, and have nginx take the client address from the CDN (set_real_ip_from its ranges with real_ip_header), so the address it forwards is the client's and not the CDN's edge.

Whose address is recorded (DLP-1237). The API believes X-Forwarded-For only when the connection comes from a trusted proxy (DLP_TRUSTED_PROXIES, loopback by default, where nginx connects from), and reads it from the right. nginx's $proxy_add_x_forwarded_for appends the address it saw to whatever the client sent, so the first hop is the client's own claim and the last untrusted hop is the one nginx wrote. From any other connection the header is ignored. Until 2026-09-28 the API took the first hop, so a client could put any address into the audit log with one header.

Configuration

Variable	Default	Purpose
DATABASE_URL	-	Required
DLP_BIND	0.0.0.0:8080	Bind 127.0.0.1 behind a proxy
DLP_CORS_ORIGINS	http://localhost:5173	Comma-separated. No wildcard
DLP_PUBLIC_URL	unset	The URL this server is reached on from outside. Used in generated agent configs, and to build the single sign-on redirect URI. Set it if you use single sign-on: unset, the redirect URI cannot be built, the settings page says so, and no sign-in can start. It is never taken from a request header - Host and X-Forwarded-Host are supplied by the caller, and a redirect URI built from one is a redirect URI an attacker chose
DLP_LICENSE_FILE	unset	Absolute path to an operator-installed product licence JSON envelope. Unset means the Licensing & Updates page reports “Not configured”; the console cannot install or edit this file
DLP_LICENSE_PUBLIC_KEY	unset	Base64-encoded 32-byte Ed25519 issuer public key. Required with DLP_LICENSE_FILE to show verified claims. Keep the issuer private key outside this server and repository
DLP_DATA_VOLUME	unset	Absolute path on the volume whose disk use the resource sampler reports (DLP-2182). Unset, disk use is reported as not configured, never guessed
DLP_SYSTEM_SAMPLE_INTERVAL_SECS	60	Seconds between resource samples; the history is kept for seven days (DLP-2182)
DLP_BACKUP_STATUS_FILE	unset	Absolute path to a read-only JSON snapshot produced by pgbackrest --stanza=<name> --output=json info. The API reads only the snapshot; it never runs backup/restore commands or receives the repository key
DLP_HA_EXPECTED_REPLICAS	0	Number of physical PostgreSQL standbys the operator expects (0–16). Set this to detect a disconnected standby; the API never assumes an absent sender is healthy.
DLP_HA_REPLICA_APPLICATION_NAMES	unset	Comma-separated application_name values of slotless physical standbys to include in HA observation. Standbys using a physical replication slot are discovered without names. Other WAL senders may be logical consumers and are excluded. The expected count is required when names are set, so disappearance remains visible. The probe is read-only and bounded to 2 seconds.
DLP_TLS_CERTIFICATE_FILE	unset	Absolute path to an API-readable copy of the ingress public PEM certificate chain (at most 1 MiB, eight certificates). Used only for Settings inventory. Never point it at a private-key file. The API does not replace certificates or verify that the ingress currently serves this copy

| DLP_AUDIT_ARCHIVE_DIR | unset | Where verified audit archive segments are written (DLP-55). Unset means audit entries are never removed: nothing is pruned without a verified copy existing elsewhere, so with nowhere to write one the archiver refuses to run. Point it at whatever append-only storage the organisation has - a WORM volume, an object store with a retention lock, a tape gateway. The append-only property is the storage's; what the product guarantees is that the copy is self-verifying, so storage that turns out not to be append-only is detectable rather than merely trusted | | DLP_AUDIT_TSA_URL | unset | An RFC 3161 timestamp authority that stamps each tenant's audit checkpoints (DLP-1185). Unset means no anchoring: a rewrite of the whole audit log is then caught only by a checkpoint somebody kept elsewhere. http is accepted, because many authorities only offer it, and the answer is trusted for its signature and not for its channel. What is sent is one SHA-256 digest and a nonce per stamp. Set it together with the CA file, or the server refuses to start | | DLP_AUDIT_TSA_CA_FILE | unset | PEM: the CA certificate(s) the authority's tokens must chain to, or its own certificate. Required with DLP_AUDIT_TSA_URL; checked at start-up | | DLP_AUDIT_ANCHOR_INTERVAL_MINUTES | 1440 | How often each tenant's head is stamped when it has moved. Clamped to 5 minutes – 7 days | | DLP_AUDIT_CHAIN_CHECK_SECONDS | 3600 | How often each tenant's audit chain is checked, for the Health page (DLP-2199). Each check continues from where the last one stopped, and a sweep from the start of the chain runs again 24 intervals after one reaches the end (DLP-2279). Clamped to 1 second to 1 day. A check that cannot run shows on the Health page once the last completed check is older than two intervals. A result whose write is refused is held in the server's memory and shown as not yet in the audit log until a write succeeds; a restart drops it (DLP-2280) | | DLP_AUDIT_CHAIN_CHECK_LIMIT | 100000 | Most audit entries one chain check reads (DLP-2279). A chain longer than this is covered over several checks. Clamped to 1 to 10,000,000. Each check reads in pages of at most 5,000 | | DLP_CLASSIFICATION_QUEUE_WARN_SECONDS | 900 | The age of the oldest classification choice waiting on an endpoint before the Health page calls the queue lagging (DLP-2200). Clamped to 1 second to 1 day | | DLP_CLASSIFICATION_QUEUE_CRITICAL_SECONDS | 3600 | The age past which the queue is called stalled. Clamped to 1 second to 1 day. Keep it at or above the warning age | | DLP_FEEDBACK_ADDRESS | unset | The address the console's "Give feedback" link writes to (DLP-729). Unset means no feedback link is shown. A malformed address, or one on a domain RFC 2606 reserves (.invalid, .example, .test, localhost), is treated as unset, because a link that reaches nobody is worse than none. The console has no Help link: no product help is served by the deployment | | DLP_DASHBOARD_URL | first DLP_CORS_ORIGINS entry | Where a browser is sent back to after signing in with an identity provider | | DLP_AGENT_PACKAGE_DIR | /opt/mydlp/packages | Read-only release artefacts served to endpoints. Each file has a <file>.version sidecar. A Windows update is published from dlp-agent-windows-x86_64.msi placed here with its sidecar: the MSI the Windows CI job builds, not the bare .exe, which msiexec cannot install (DLP-1166). deploy.sh does not publish an MSI, so until one is placed here a Windows update is refused with that reason | | DLP_AGENT_UPDATE_STORE_DIR | /var/lib/mydlp/agent-packages | Writable runtime store for immutable, digest-addressed copies referenced by update manifests. It is deliberately separate from DLP_AGENT_PACKAGE_DIR: the API's ProtectSystem=strict sandbox keeps release artefacts under /opt read-only while allowing state only under /var/lib/mydlp | | DLP_SESSION_TTL_HOURS | 12 | | | DLP_REQUEST_FRESHNESS_SECS | 300 | Agent signature window | | DLP_MTLS_MODE | off | off, observe or required (DLP-31). Whether a client certificate forwarded by the TLS terminator is checked against the device that signed the request. See below. An unrecognised value fails the start rather than defaulting to off, because a deployment that meant required and typed requird would otherwise run with no check at all | | DLP_TRUSTED_PROXIES | 127.0.0.0/8,::1/128 | Comma-separated addresses or CIDRs whose X-Forwarded-For is believed (DLP-1237). Set, it replaces the default; empty trusts no proxy. A value that does not parse stops the start. Behind an ingress in Kubernetes, name the ingress controller's pod range | | DLP_BUNDLE_TTL_HOURS | 168 | Policy bundle lifetime. A bundle is re-signed once less than half of it remains (DLP-1232) | | DLP_BUNDLE_RENEWAL_TICK_SECS | 3600 | How often the renewal pass looks; it also runs once at start | | DLP_SMTP_URL | unset | The mail relay the server sends notices through (DLP-1240): smtps://user@host:465 (TLS from the start) or smtp+starttls://user@host:587. smtp:// without TLS is accepted only to a loopback address, for a test sink. Unset means no mail is sent, and the server logs that once at start. There is no password in the URL | | DLP_SMTP_PASSWORD_FILE | unset | A file holding the relay password for the URL's user. Required when the URL names a user; read at start-up, never stored in the database | | DLP_MAIL_FROM | unset | The sender, e.g. MyDLP <dlp@example.com>: an address on the organisation's own domain, with SPF and DKIM set for the relay, or receivers will file it as spam. Required with DLP_SMTP_URL | | DLP_TOKEN_EXPIRY_TICK_SECS | 3600 | How often the server checks for API tokens that expire within 14 or 3 days, and mails each token's creator once per threshold (DLP-1240). The message names the token and its public id and never contains the token | | DLP_DEVICE_CERT_DAYS | 90 | | | DLP_MAX_BATCH_EVENTS | 500 | | | DLP_MAX_BODY_BYTES | 8388608 (8 MiB) | The request body limit for every route but the EDM file import | | DLP_EDM_MAX_UPLOAD_BYTES | 134217728 (128 MiB) | The body limit for POST /api/v1/edm-datasets/import-file alone (DLP-1040). A reverse proxy in front needs a limit at least this large on that path, or it refuses first | | DLP_DISCOVERY_TESTS_PER_MINUTE | 10 | Discovery-target connection tests one organisation may start in a minute (DLP-637). A test makes this server connect to an address an administrator chose and reports whether it was refused, dropped or answered, so an unlimited one is a port scanner over the server's own network. Never below 1 | | DLP_REPORT_RUNS_PER_MINUTE | 60 | Saved-report runs one user may start in a sliding minute (DLP-2088). The console re-runs a report for every sort, filter and page, so this is sized for a person clicking and refuses a loop. An API token spends its owner's budget. Never below 1 | | DLP_REPORT_EXPORTS_PER_HOUR | 30 | Saved-report exports, CSV or print, one user may take in a sliding hour (DLP-2088). Scheduled deliveries spend none. Never below 1 | | DLP_REPORT_EXPORT_ALERT_ROWS | 50000 | Rows one user may export in 24 hours before the export that crosses it is flagged: one report.export_volume_unusual audit entry and a count on system health (DLP-2088). It refuses nothing. Never below 1 | | DLP_M365_GRAPH_URL | https://graph.microsoft.com | Microsoft Graph, for SharePoint discovery (DLP-1193). Set it for a national cloud, e.g. https://graph.microsoft.us. https is required unless the host is this machine | | DLP_M365_FULL_SCAN_EVERY_HOURS | 168 | How long a SharePoint or OneDrive drive may go on incremental scans before it is read whole again (DLP-1201). 0 reads every run whole | | DLP_M365_SUBSCRIPTION_TICK_SECS | 600 | How often SharePoint change-notification subscriptions are made and renewed (DLP-1196). They are made only while DLP_PUBLIC_URL is an https address Microsoft can reach, since Graph posts notifications to <DLP_PUBLIC_URL>/api/v1/m365/notifications | | DLP_M365_LOGIN_URL | https://login.microsoftonline.com | The Microsoft sign-in service a SharePoint target's app registration authenticates against. Set with DLP_M365_GRAPH_URL for a national cloud. The client secret is sent here, so it is the operator's setting and never a tenant's | | DLP_TENANT_DISCOVERY_SCOPE | unset | Which share roots and credential variables each organisation's discovery may use (DLP-675), as JSON keyed by tenant slug: {"acme":{"roots":["/srv/shares/acme"],"credentials":["DLP_DISCOVERY_ACME_"]}}. Roots are matched by path component and after resolving symbolic links; credential entries are name prefixes and must begin DLP_DISCOVERY_. Unset with one organisation on the server, discovery is unrestricted and the log says so. Unset with two or more, every discovery location is refused until they are assigned, and queued scans outside an assignment are paused at start-up with a reason. A malformed value refuses everything. Set it before adding a second organisation | | DLP_REDACT_FILE_PATHS | false | Hash file names, keep directories, for every tenant whose Owner has not chosen (DLP-1302) | | DLP_LOG_JSON | false | Structured logs for shipping | | RUST_LOG | info,dlp_management_api=debug,sqlx=warn | Log filter. Set it to info,dlp_management_api=info,sqlx=warn in production, as the host deploy and the Kubernetes ConfigMap do (DLP-878). The built-in default logs this server at debug, which is where a refused request's message is written |

The licence file is a JSON object with payload_b64 (base64 of the exact UTF-8 JSON payload bytes) and signature_b64 (base64 of the 64-byte Ed25519 signature over those exact bytes). The payload schema is {"version":1,"tenant_id":"<customer tenant UUID>","organization":"…","license_type":"…","expires_at":"2026-12-31T00:00:00Z","limits":{"endpoints":250,"data_sources":50,"users":250}}. The API re-reads and verifies the file on each request, rejects malformed or unknown fields, and reports invalid, expired, unavailable, or not_configured instead of claiming the licence is active. Seat counts shown in Settings are observational; this endpoint does not yet enforce limits on enrollment, user creation, or connector creation.

Read-only Active Directory lookup (DLP-1435)

The API build requires Rust 1.85+ because the pinned LDAP dependency uses Edition 2024; this does not change the agent's compiler requirement. No AD connector is enabled automatically. The operator must provision a read-only account, verified domain identity, limited search base and trusted CA first.

Assign the connection to one tenant using a server environment variable DLP_DIRECTORY_<tenant UUID without hyphens>_<UPPERCASE_SUFFIX>. Its value is base64-encoded JSON containing url (only ldaps://), domain_base, search_base, domain_sid, bind_dn and password. Base64 is encoding, not encryption: keep this only in the protected operator service-secret environment, never in source, a ticket, browser input, logging or an export. The endpoint and credential live together so a tenant cannot redirect a credential to an arbitrary server. Names from another tenant, inline secrets, discovery credentials and arbitrary server variables are refused before resolution. The search base must be the configured domain base or its child.

Install the directory CA in the API host trust store (or a protected SSL_CERT_FILE bundle); TLS hostname/certificate verification cannot be disabled. No cleartext LDAP, NTLM/GSSAPI, directory writes or referral chasing is supported. Assign only read access to permitted user/security-group attributes; do not grant directory-write rights. The connector never requests group memberships. Actual AD ACL provisioning and customer-domain connectivity must be validated by that domain's operator, not inferred from a local fixture.

An Owner registers the reference name only, with a business reason, through POST /api/v1/directory-sources; it starts disabled and unverified. Enable explicitly, then use the bounded lookup endpoint. First successful domain-verified lookup updates connector health. A missing/expired credential, wrong domain, TLS/timeout or partial response remains unknown/stale, not "synchronised". No central sync schedule or membership catalogue exists. An endpoint cannot treat an AD user's metadata as enforceable until its stable user-SID contract is supported. Production logging suppresses LDAP library raw response events even if RUST_LOG enables trace; source health remains visible through sanitized audit entries and API failure messages.

Policy signing-key rotation

A self-generated software policy key has a 365-day validity. The console warns when an active or rotating policy key has 30 days or less remaining. Rotate while the old key can still sign the overlap set. For a database-generated key, use the Signing keys panel's Rotate action or POST /api/v1/system/signing-keys/rotate-software with a reason of at least 10 characters. The route requires tenant:manage and a fresh signing-key step-up, keeps the old key as rotating, and audits the reason and old key ids atomically. It refuses to downgrade an HSM or externally managed key to a database-held key. An active endpoint that has never accepted a signed trust set must check in or be decommissioned before either rotation path proceeds. Before changing the active key, the server also signs and verifies a probe with every still-trusted old key; an unavailable or mismatched signer stops the transition instead of publishing an overlap nobody can authenticate.

The first key can be created for a new tenant. If an existing tenant has no valid policy-bundle key, the server refuses to create an unrelated new root automatically. Policy compilation and agent-update signing then fail with a conflict instead of issuing artefacts enrolled endpoints cannot authenticate. Recovery after expiry needs a verified re-enrolment/reprovisioning plan; changing not_after directly in SQL is not a supported rotation procedure.

Hardware-backed signing keys

Policy bundles are signed with Ed25519 only, by decision (ADR-0016): the bundle verifier runs on every endpoint, offline, and a second algorithm there is a second verification path in the code that decides what a fleet enforces.

That has a consequence worth knowing before buying anything. A token used for bundle signing must implement CKM_EDDSA:

Key store	Can sign bundles
Thales Luna 7.x, Entrust nShield (recent firmware), YubiHSM 2, SoftHSM ≥ 2.6	yes
AWS KMS, Azure Key Vault / Managed HSM, Google Cloud KMS	no - none of the three offers Ed25519 signing
Windows Platform Crypto Provider (TPM 2.0)	no

Configuring one. The key is generated in the token, never here:

softhsm2-util --init-token --slot 0 --label mydlp --so-pin ... --pin ...
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin ... \
    --keypairgen --key-type EC:edwards25519 --label bundle-key --id 01

Then point the product at it. The PIN is an env: reference and a literal is refused - a PIN beside the reference gives back most of what a token is for:

curl -X POST https://dlp.example/api/v1/system/signing-keys/hardware \
  -H 'content-type: application/json' -d '{
    "purpose": "policy_bundle",
    "reference": "pkcs11:module=/usr/lib/softhsm/libsofthsm2.so;token=mydlp;object=bundle-key;pin=env:DLP_HSM_PIN"
  }'

The public half is read from the token, never taken from that request: a typo in a pasted public key produces a fleet that rejects every bundle, and the cause is nowhere near the symptom. The token is made to sign a probe before the configuration is stored, and that signature is verified against its public key. A token that cannot sign or names the wrong key is discovered then rather than at the next policy publish.

The key it replaces moves to rotating, not revoked: agents cache a trust set, and a bundle signed last week must still verify on an endpoint that has not checked in since. On check-in the old key signs the overlap set containing both keys; the endpoint persists that set before it fetches a bundle signed by the new key. Retire the old row only after fleet convergence: the resulting new-key-only set is signed by the new key, and an endpoint that never observed the overlap cannot verify it by design. Use POST /api/v1/system/signing-keys/{old_key_id}/retire; it returns 409 and the number of lagging active endpoints until every one has reported the overlap revision. Do not update signing_keys.status directly, because that bypasses this convergence gate and its audit record.

Policy signing-key compromise

Suspected or confirmed policy-bundle signing-key compromise is not ordinary rotation. The policy-bundle key also signs agent update artefacts; stop policy publishing and agent-package releases, restrict access to the signer and its hosting system, and preserve the signing-key inventory, audit trail, endpoint check-in state and incident timeline before changing keys. If a hardware token is involved, involve its operator; disable signing access when compromise is confirmed. Do not delete a key row, edit its status in SQL, or present a successful key adoption as proof that the fleet no longer trusts the old key.

If the old signer is still trusted and under exclusive control, the staged adoption and retirement above can be used for planned or precautionary rotation. A confirmed compromised private key cannot safely authenticate the {old,new} overlap to an endpoint: an attacker with that key can also sign an apparent trust-set transition. There is currently no immediate, fleet-wide emergency revocation path for endpoints that are offline or still trust only the old key. Treat those endpoints as exposed. Isolate them through external controls, provision a clean signing root independently, and arrange verified re-enrolment/reprovisioning before restoring policy delivery or agent updates. Record which endpoints accepted the clean root and which remain unreachable; do not count an offline endpoint as recovered.

POST /api/v1/agents/{id}/revoke can stop a particular endpoint from using the server and removes it from the active-endpoint retirement gate, but it does not erase that endpoint's locally cached old public key. Use it only with a recorded reason and an explicit plan for that device. The normal /system/signing-keys/{old_key_id}/retire endpoint still refuses to remove a rotating key while active endpoints lag; a 409 is a safety result, not a reason to bypass the gate. Escalate a confirmed compromise as an incident and do not claim complete recovery until every in-scope endpoint is either verified under a clean trust root or explicitly decommissioned.

There is no fallback. If the token is absent, locked, or refuses the PIN, compiling a bundle fails and says which of those it was. Falling back to a software key would have the fleet accept bundles signed by the key the HSM was bought to protect, and nobody would see it.

Device keys are a separate question with a separate answer, because the server is their only verifier: those may be Ed25519 or ECDSA P-256, pinned per device at enrollment, which is what makes a TPM-backed device key possible. The algorithm is looked up by device id and never taken from the request.

Secrets belong in /etc/mydlp/api.env (0600) or a secret manager. Signing keys support env:VAR_NAME so the key material never touches the database.

A pkcs11: key reference signs through the token (DLP-239), as the section above describes, and a token that cannot sign fails the bundle rather than falling back to a software key. This paragraph used to say this build recognised such a reference and refused it (DLP-32), which was true before ADR-0015 and ADR-0016 settled the algorithm and DLP-239 built the signer.

What an agent trusts for the server's certificate (DLP-59)

The agent trusts the machine's own certificate store - the Windows certificate store, the system bundle on Linux - as well as the Mozilla roots compiled into the binary, and any CA given in server_ca_pem.

That is the correct default on a managed endpoint: the machine owner decides what the machine trusts. It is also what makes the two commonest enterprise deployments work without configuration - a management server certificate issued by an internal CA, and a TLS-inspecting proxy re-signing with a private root. With the bundled roots alone, every agent on such a fleet fails to connect with an error that looks like a server fault.

It means the agent's traffic can be inspected by the enterprise, if the enterprise has installed an inspection root on its own machines. A deployment that does not want that configures server_ca_pem and nothing else - but note that the machine store is still trusted alongside it, so pinning exclusively is not available today and is its own decision.

dlp-agent doctor prints which sources are in use, before the connectivity check, because when a connection fails this is the first thing worth reading.

Client certificates for agents (DLP-31)

Request signing already proves which device sent a request, and it stays mandatory. DLP_MTLS_MODE adds a transport-layer check underneath it: the device certificate issued at enrolment carries mydlp://<tenant>/device/<device> as a URI SAN, and the application compares that against the device id that signed the request.

TLS terminates at a proxy in every deployment this product ships, so the proxy does the handshake and forwards two headers - X-Client-Verify and X-Client-San. deploy/nginx-agent-mtls.conf.example has the directives, including the two that are easy to get wrong: the SAN is not $ssl_client_s_dn, and the proxy must clear anything a client sends under those header names or the check is decorative.

Migrate in that order:

1. observe - mismatches are refused, absences are logged. Watch the count of endpoints with no certificate fall to zero.
2. required - an absent or unverifiable certificate is refused.
3. Then, and only then, ssl_verify_client on at the proxy, so an unenrolled client never reaches the application at all.

A header is only as trustworthy as whatever set it. This is defence in depth, not the defence: a client that forges X-Client-Verify: SUCCESS still cannot produce a valid signature, and that is the check doing the work. Which is also why signing is not relaxed when mTLS is on.

Storage, and what actually limits ingestion

Read this before choosing a volume class. Ingestion is the only path in this product with a capacity ceiling worth planning for, and the thing that limits it is not what most people assume.

What was measured, not what is usually assumed

SPIKE-006 sampled pg_stat_activity throughout a sustained load run. The database spent its time in the write-ahead log:

   117  LWLock:WALWrite
    13  IO:WALSync
     2  Client:ClientRead

The obvious reading is "put the WAL on faster disks". On the host that was measured, that reading was wrong, and the measurement that settles it is worth repeating because it is cheap:

Setting synchronous_commit = off - a measurement, never a recommendation - moved throughput from 3,506 to 3,788 events/s, about 8%. If the bottleneck had been waiting for the disk, removing the wait entirely would have transformed it.

LWLock:WALWrite is contention on inserting into the WAL buffer, which is CPU and a lock. IO:WALSync is the disk wait, and it was a tenth of the samples. The real cause was one commit per event in our own code, fixed under DLP-110: 965 transactions down to 271 for the same 500 events, with nothing now scaling per event.

Which case your deployment is in

The distinction matters because the two have different fixes, and a deployment on a network-attached volume genuinely can be fsync-bound where this host was not. Fifteen seconds of sampling tells you which:

SELECT wait_event_type, wait_event, count(*)
  FROM pg_stat_activity
 WHERE state = 'active' AND backend_type = 'client backend'
 GROUP BY 1, 2 ORDER BY 3 DESC;

Run it a few times during your own peak.

• IO:WALSync dominating - you are waiting for the disk. Put pg_wal on the fastest local storage you have, separate from the data directory. On Kubernetes that means a local-NVMe storage class rather than the default network-attached one; the difference between those two is often an order of magnitude of fsync latency, and it is invisible until you look.
• LWLock:WALWrite dominating - faster disks will not help much. Look at transaction count per request first.
• Client:ClientRead dominating - the database is waiting for the application. The limit is elsewhere.

What this product will not decide for you

synchronous_commit = off trades durability for throughput: on a crash, the last fraction of a second of committed transactions can be lost, even though the client was told they committed. For a DLP product that is a real choice - those transactions are audit entries and incidents - and it is the customer's to make, so nothing here sets it. It is mentioned because an operator measuring their own deployment will find it, and should find it explained rather than recommended.

Figures to compare against

From docs/CAPACITY.md §1, one API process and one PostgreSQL on a host that was running tens of cores' worth of other work on 16 cores - so every figure is a floor, not a rating:

	ingestion only	every event matches a policy
Sustained	3,385 events/s at p99 630 ms, concurrency 64	2,622 events/s at p99 283 ms, concurrency 16
Knee	concurrency 128: 2,588 events/s, p99 4,001 ms	concurrency 32: 2,281 events/s, p99 1,147 ms

Size from the right-hand column: it includes correlation, incidents and risk scoring, which a real fleet pays.

If your deployment is materially below this on quieter hardware, the wait-event query above is where to start.

Backup and restore

For a production on-prem installation, keep the backup repository on a separate host or storage failure domain, not in the PostgreSQL data volume. Use an operator-managed pgBackRest stanza with encrypted POSIX storage, scheduled full/incremental backups and WAL archiving for point-in-time recovery. An on-prem S3-compatible repository is optional, not required. The database host's backup account owns the repository credential and encryption passphrase; neither belongs in the web console or its database. Check the repository's retention and run a restore drill on an isolated server before relying on it.

To show actual backup history in Settings without granting the API database-host privileges, schedule a separate status job as the PostgreSQL backup account: run pgbackrest --stanza=<name> --output=json info, atomically replace a group-readable JSON file, and set DLP_BACKUP_STATUS_FILE to its absolute path. Share only this status file, not pgBackRest's configuration or encryption passphrase, with the API service. The endpoint GET /api/v1/system/backup exposes at most 50 backup records and marks a snapshot older than one hour stale; a connected snapshot is not a guarantee that a restore drill or current backup schedule succeeded. A failed status job must leave the previous file in place so staleness becomes visible.

For the documented systemd installation, the release installs /opt/mydlp/bin/dlp-pgbackrest-status. After an operator has installed pgBackRest, created and checked the stanza, and configured an encrypted repository outside the PostgreSQL data failure domain, opt in to its status timer:

install -d -o postgres -g mydlp -m 2750 /var/lib/mydlp/backup-status
test -e /etc/mydlp/backup-status.env || install -m 0640 -o root -g postgres /dev/null /etc/mydlp/backup-status.env
# Edit backup-status.env: DLP_BACKUP_STANZA=mydlp
#                         DLP_BACKUP_STATUS_FILE=/var/lib/mydlp/backup-status/info.json
# Set the same DLP_BACKUP_STATUS_FILE in /etc/mydlp/api.env and restart mydlp-api.
install -m 0644 deploy/systemd/mydlp-backup-status.service /etc/systemd/system/
install -m 0644 deploy/systemd/mydlp-backup-status.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now mydlp-backup-status.timer
systemctl start mydlp-backup-status.service

The timer refreshes status every 15 minutes as postgres, never as mydlp. The worker accepts only a bounded JSON response for the configured stanza; failed commands, timeouts and malformed output retain the old snapshot. It does not start backups. After a successful manual pgBackRest backup and an isolated restore drill, the following optional timers run a weekly full backup and daily incremental backups as postgres:

install -m 0644 deploy/systemd/mydlp-backup@.service /etc/systemd/system/
install -m 0644 deploy/systemd/mydlp-backup-full.timer /etc/systemd/system/
install -m 0644 deploy/systemd/mydlp-backup-incr.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now mydlp-backup-full.timer mydlp-backup-incr.timer

The release installs /opt/mydlp/bin/dlp-pgbackrest-backup, which accepts only full and incr; it never accepts restore. The pgBackRest stanza, encrypted repository, WAL archive_command, retention and any off-host copy are operator configuration, not web API inputs. Check systemctl list-timers and the pgBackRest logs after enabling the units. Disabling the timers stops future scheduled backups, not WAL archiving; verify both before a recovery claim. The Settings form cannot edit these systemd schedules yet.

An operator may separately enable manual backup requests from Settings. This is opt-in and bound to one tenant; it does not grant the API the postgres account, repository key, backup files, or a restore command. First verify the status job above is fresh, the pgBackRest stanza is healthy, and scheduled backups and an isolated restore drill have succeeded. Then install two separate directories and configure the same exact paths for the API and worker:

install -d -o mydlp -g postgres -m 2770 /var/lib/mydlp/backup-requests
# Reuse /var/lib/mydlp/backup-status, owned postgres:mydlp, mode 2750.
# In /etc/mydlp/api.env:
# DLP_BACKUP_REQUEST_FILE=/var/lib/mydlp/backup-requests/pending.json
# DLP_BACKUP_MANUAL_STATUS_FILE=/var/lib/mydlp/backup-status/manual.json
# DLP_BACKUP_MANUAL_TENANT_ID=<the tenant UUID explicitly authorized by the operator>
# DLP_BACKUP_STATUS_FILE=/var/lib/mydlp/backup-status/info.json
# In /etc/mydlp/backup-status.env:
# DLP_BACKUP_STANZA=mydlp
# DLP_BACKUP_REQUEST_FILE=/var/lib/mydlp/backup-requests/pending.json
# DLP_BACKUP_MANUAL_STATUS_FILE=/var/lib/mydlp/backup-status/manual.json
install -m 0644 deploy/systemd/mydlp-backup-request.service /etc/systemd/system/
install -m 0644 deploy/systemd/mydlp-backup-request.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now mydlp-backup-request.timer
systemctl start mydlp-backup-request.service
systemctl restart mydlp-api

The worker polls every minute as postgres; its fresh heartbeat is required before the API accepts a request. A human with tenant:write permission must step up. The API places one short-lived full or incr request atomically; the worker accepts no path, stanza, command, or restore option from it. A second request is refused while one is pending or running, and another request is refused for one hour after a result. The Settings screen reports the worker's sanitized outcome; inspect pgBackRest's operator-only logs on failure. The backup history still comes from the independent info snapshot, not from the request response. A queued response is never proof of a completed or restorable backup. If the timer is disabled or its heartbeat stales, the UI refuses new requests. A stuck pending file must be investigated by the operator rather than deleted through the console.

The commands below are a manual logical-backup fallback, not the scheduled physical-backup/PITR facility in the Settings concept. The current console does not install pgBackRest, configure its stanza or WAL archiving, execute a restore, or claim an operator-managed backup succeeded. A restore remains an offline operator procedure; the manual backup worker deliberately has no restore operation.

pg_dump --format=custom "$DATABASE_URL" > mydlp-$(date +%F).dump
pg_restore --clean --if-exists -d "$DATABASE_URL" mydlp-2026-09-15.dump

Also back up /etc/mydlp/api.env: it holds the database credential, and the signing keys in signing_keys.private_key_ref are unrecoverable without it. Losing them means every enrolled agent must re-enrol.

Verify a restore with GET /api/v1/audit/verify. A restore that silently lost rows shows as a broken chain.

Air-gapped deployment

Everything runs locally. The agent needs only the management server; the optional Tika, Presidio and Tesseract services are all on-premises. No customer document reaches a third-party service unless an operator explicitly enables one.

For an air-gapped install: mirror the crate and npm registries or build elsewhere and ship binaries, use an internal CA (server_ca_pem in the agent config), and disable the certbot timer in favour of internally issued certificates.

Metrics

The management API opens a second listener for Prometheus, separate from the public one, because metrics carry tenant names and volumes. DLP_METRICS_BIND defaults to 127.0.0.1:9090; set it empty to disable the listener entirely.

nginx does not proxy it - the site configuration forwards /api, /healthz and /readyz and nothing else - so on a single-host install it is reachable only from the machine itself. That is the intended arrangement: a scraper runs locally, or an operator tunnels to it.

If the bind is widened, set DLP_METRICS_TOKEN in /etc/mydlp/api.env and give the scraper a bearer_token_file. The port being unroutable is a network control, and network controls are the ones somebody else changes.

In Kubernetes the port is declared on the Deployment and the Service, admitted by allow-metrics-scrape in deploy/kubernetes/60-network-policy.yaml from the monitoring namespace alone, and deliberately absent from the Ingress. On a cluster whose CNI does not enforce NetworkPolicy that rule is silently ignored - see the note in KNOWN_LIMITATIONS - and the token should be set there too.

Kubernetes

deploy/kubernetes/ - applied to a real cluster and verified by tests/kubernetes_manifests.sh (41 assertions). There is no Helm chart.

kubectl apply -f deploy/kubernetes/00-namespace.yaml
kubectl -n mydlp create secret generic mydlp-db \
  --from-literal=POSTGRES_PASSWORD="$(openssl rand -base64 24 | tr -d /=+)"
kubectl apply -f deploy/kubernetes/

File	What it holds
00-namespace.yaml	The namespace, labelled so policies can select it
10-config.yaml	Non-secret configuration
20-secret.example.yaml	The shape of the secret. A placeholder, never applied
30-postgres.yaml	StatefulSet with a volume claim, plus a headless Service
40-api.yaml	Migration Job, Deployment, Service, PodDisruptionBudget
50-dashboard.yaml	Deployment and Service
60-network-policy.yaml	Default deny, then five narrow allowances

One password, in one place. The Secret holds POSTGRES_PASSWORD and nothing else; the Deployment composes DATABASE_URL from it with $(VAR) expansion. An earlier draft kept the password in the Secret and a whole DATABASE_URL containing it in a second Secret, and the two drifted within minutes - the symptom was password authentication failed, which sends an operator to look at credentials rather than at the fact that there were two of them. Note that $(VAR) only expands from the same container's env list, not from envFrom, which is why the user and database names are declared there rather than in the ConfigMap.

Liveness never touches the database. /healthz answers while the process runs; /readyz is the one that checks PostgreSQL. A database-dependent liveness probe restarts every healthy pod at once when the database blips, turning a blip into an outage. Verified by scaling PostgreSQL to zero: the pods went not-ready, took no restarts, and served again when it came back.

Migrations are a Job with an init container that waits for the database. Kubernetes has no equivalent of Compose's depends_on: healthy for a Job. Without the wait the migration simply raced PostgreSQL - observed taking three attempts out of a backoffLimit of three, which a slightly slower database would have exhausted.

The network policy denies by default and then allows exactly: DNS to kube-system, the ingress namespace and the dashboard to the API, the API and the migration job to PostgreSQL, and the ingress namespace to the dashboard. There is deliberately no egress to the internet: a DLP system that can post to an arbitrary host is one that can be used to exfiltrate what it collected. The ingress controller's namespace must carry the label mydlp.io/ingress: "true" - an unlabelled namespace cannot reach anything by accident.

A rollout does not drop requests, and that took a fix rather than a declaration. Two replicas, a PodDisruptionBudget and a readiness probe are not enough on their own: a pod entering Terminating stops being an endpoint and receives SIGTERM at the same moment, while kube-proxy updates every node's rules asynchronously - so for a short window traffic still arrives at a process that has begun shutting down. Measured on this deployment: 5 of 870 requests failed during a rollout. With a preStop wait of five seconds and a thirty-second grace period: 845 requests, none failed.

Verified on kind with kindnet, which does enforce NetworkPolicy. A cluster whose CNI does not enforce it will apply these objects and ignore them, so confirm that before relying on them.

Resource limits are enforced by the runtime, not merely declared: a container given 64 MiB and asked for 256 MiB is OOM-killed, which the test asserts by doing exactly that.

Upgrading

The hardware-id scheme changed (#376)

Linux and macOS agents derive their hardware id without the host name from this release on, so renaming a machine no longer changes its identity. It used to: the server resolves a device by (tenant_id, hardware_id), so a renamed machine enrolled as a new device and its old record was orphaned - still listed, never checking in, indistinguishable from a machine someone switched off.

Running agents are unaffected. The device id is persisted in the identity file; the hardware id only matters at enrolment.

Re-enrolling agents migrate themselves. An agent sends the id it had under the previous scheme alongside the new one, and the server moves the existing record forward - same device id, same history, same incidents - rather than creating a second record. The migration writes an agent.hardware_id_migrated audit entry, which is what explains a device's id changing without anyone re-enrolling it.

Nothing needs to be run by hand. Two things are worth knowing:

• An agent old enough not to send the legacy id will enrol as a new device. Upgrade the agent before re-enrolling it, not after.
• The legacy id is recomputed on the endpoint, and it can only be reproduced where the host name is still what it was. A machine renamed before the upgrade has already been orphaned, and this does not recover it - the old record has to be revoked by hand.

Operating it

Check	How
Service health	systemctl status mydlp-api; curl https://…/healthz
Readiness	/readyz fails when PostgreSQL is unreachable
Fleet health	Console → Administration → System health
Silent endpoints	Console → Endpoints; "stale" means no check-in for an hour
Rejected events	poison_events, surfaced in System health
Audit integrity	GET /api/v1/audit/verify
Agent-side	dlp-agent doctor on the endpoint
What an endpoint can enforce	dlp-agent capabilities - needs no enrollment and no server

Upgrades

1. pg_dump first.
2. Deploy the new binary; migrations run at startup and are idempotent.
3. Agents are backwards compatible within a schema major; a mismatch is rejected explicitly rather than silently misinterpreted.

Rolling the server back (DLP-628)

Redeploy the previous commit: deploy/deploy.sh --ref=<sha>. Any commit already on main is accepted. The deploy builds that commit, migrates with that commit's binary, and verifies that the running server reports that commit's version.

This works because, from v0.327.0, a server starts against a schema newer than itself: a migration it does not know is left alone, not refused (db::migrate, set_ignore_missing). tests/rollback_schema.sh records a future migration and checks that the build still migrates, starts, and signs somebody in. tools/migration_lint.py, in CI, keeps new migrations additive, so that the older build can actually run against what the newer one added.

Two limits, stated rather than discovered:

• Not to a build before v0.327.0, past a migration. Those builds carry sqlx's default and refuse to start ("migration N was previously applied but is missing"). Going back to one of them means restoring the pg_dump from step 1.
• Not past a migration marked -- breaks rollback: DLP-nnn. The lint lets a migration drop, rename, retype or require a column only with that line, which names the ticket and the reason. Rolling back past one means restoring the database. Read the marker before deploying the release that carries it.

Data written by the newer release stays in the database, in tables and columns the older build does not read. Rolling forward again picks it up.

API tokens (DLP-600)

What an operator does with the credentials Administration → API tokens issues. The rules they follow are in docs/API.md (API tokens).

Issuing one. Give it the fewest scopes that do the job; a SIEM export is incidents:read. It acts as you, so it stops working if your account is disabled -- issue integration tokens from an account that will outlive the person, not from a personal one. It expires in 90 days unless you chose otherwise; put the date in whatever tracks renewals.

Rotating one. The old secret stops the moment the new one exists, so rotate when the integration can take the new value straight away. For a change with no interruption: create a second token, move the integration to it, confirm it is used (its Last used moves), then revoke the first.

When one has leaked -- pasted into a ticket, committed, seen on a screen:

1. Revoke it, with the reason. It fails on its next request.
2. Watch mydlp_api_token_refusals_total{reason="revoked"}. If it rises, somebody is still presenting it; the API server's log names the token's public id and the source address on each refusal.
3. Read the audit log for api_token:<id> over the period it was exposed. Every request it made is there under that actor, with what it did.
4. Issue a replacement with the same scopes, and record where it is stored.

What the other refusal classes mean. wrong_secret is somebody holding a token's public half (from a log or a screenshot) and guessing the rest; the secret is 256 random bits, so the guess does not succeed, but the attempt says the public half is out. expired is an integration nobody renewed. owner_disabled is an integration still running as a person who has left. unknown is noise: tokens nobody issued, or ones rotated away.

What cannot be done. A secret is not stored, so it cannot be recovered or shown again; rotate instead. A token cannot create, rotate or revoke tokens, whatever scopes it has.

Confirming it is you (DLP-613)

Some actions ask the person to confirm who they are when their sign-in is more than 15 minutes old: releasing a policy, giving somebody a role, changing where incidents are delivered, changing single sign-on, and API tokens. The full list, including what is deliberately not asked, is in docs/API.md. There is nothing to configure. This section is for when it does something unexpected.

What people see. A Confirm it's you dialog over the page they were on. A password account types its password. A single-sign-on account is sent to the provider in a pop-up window, which closes by itself. The action they were doing then completes, once. Cancelling leaves the page as it was, with the refusal shown where they clicked.

"The sign-in window was blocked." The browser blocked the pop-up. Allow pop-ups for the console's address. Nothing was changed.

"Your sign-in provider did not ask you to sign in again." The provider answered from its own existing session instead of authenticating again: it ignored prompt=login and max_age=0. The person can sign out of the console, sign in again, and retry within 15 minutes. The lasting fix is on the provider: allow the application to request re-authentication.

"Your sign-in provider does not say when you last signed in." The ID token has no auth_time claim, although OpenID Connect Core requires one whenever max_age is requested, and this request sends it. Configure the provider to include it. Until then, single-sign-on accounts use the same sign out, sign in, retry route.

"The provider confirmed a different account." Somebody else is signed in at the provider in that browser. Sign out there and retry.

Automation gets auth.step_up_required. An API token can never confirm, by design: these actions need a person. Do them from the console. If a pipeline has to publish policies, that is a decision to bring to whoever owns the policy process, not something a scope can grant.

What to watch (the API server's metrics port):

• mydlp_step_up_total{outcome="failed"} rising for one action: wrong passwords on live sessions. Somebody guessing on a session they did not sign into looks exactly like this. Read the audit log for step_up.failed, which names the session and source address, and revoke the account's sessions by disabling it if in doubt.
• outcome="token_refused": an integration is being pointed at something only a person may do.
• mydlp_step_up_grants_24h{state="expired_unused"}: people who confirmed and then did not finish, or a console whose retry never arrived. A handful is normal; most of them is a defect worth a ticket.
• mydlp_step_up_grant_to_use_seconds_max_24h: seconds when the console's retry works.

Audit. Every refusal (step_up.required), confirmation (step_up.granted, with method password or oidc) and failure (step_up.failed) is in the tenant's audit chain with the action it was for. No password, token or ID token is ever recorded.

The agent as a service

The agent is supervised by the operating system on both platforms. It does not supervise itself: a process that restarts itself from somewhere unexpected is indistinguishable from malware (see Known limitations §5.0.1).

Linux - systemd

deploy/mydlp-agent.service runs dlp-agent run with Restart=always. systemctl stop sends SIGTERM, which the agent handles: it drains its event queue and releases its run marker, so a legitimate stop is not reported as interference.

Windows - the Service Control Manager

The service binary is dlp-agent.exe service, not dlp-agent.exe run (#375). They are not interchangeable:

dlp-agent.exe service	Registers a ServiceMain and a control handler, reports START_PENDING then RUNNING, and handles stop, shutdown and pre-shutdown by draining the queue before reporting STOPPED. This is the only mode the SCM can start
dlp-agent.exe run	The same loop as an ordinary console process, for diagnostics. It never reports SERVICE_RUNNING, so as a service binary it is killed at the start timeout

From an elevated prompt, install and enrol through Windows Installer:

msiexec.exe /i MyDLPAgent.msi /qn /norestart `
  SERVERURL=https://dlp.example.com ENROLLTOKEN=<one-time-token> `
  /l*v C:\Windows\Temp\mydlp-install.log

Get-Service MyDLPAgent
Get-EventLog -LogName Application -Source MyDLPAgent -Newest 5

A higher-version MSI stops and replaces the service transactionally, preserves the endpoint identity and restarts it. A lower version is refused.

To remove it, quoting an authorisation so the removal is recorded rather than appearing as unexplained silence:

sc.exe stop MyDLPAgent
dlp-agent.exe uninstall --confirm --authorization CHG-12345
sc.exe delete MyDLPAgent

None of the Windows half has been observed running on a Windows host; what is and is not evidenced is in Known limitations §2.3.