Architecture

1. What this system is

MyDLP is an endpoint data loss prevention platform: a lightweight agent on each endpoint detects sensitive data locally, evaluates it against signed policy, acts within what the platform actually supports, and reports a redacted decision to a central server that turns those decisions into incidents an analyst can work.

Why there is no per-platform agent crate

Platform differences live behind cfg attributes inside agent/core, not in per-platform crates. agent/core/src/platform.rs holds the capability matrix, identity.rs the machine identifiers, device_signer.rs the TPM/CNG path, keystore.rs the DPAPI fallback, state_acl.rs the Windows ACLs and removable.rs the device detection - each with a #[cfg(...)] block per platform beside the others it is a variant of.

Splitting those into crates is worth doing when the platform surface is large enough that the build complexity pays for itself. Today it is a few hundred lines, and separating them would mean a reader comparing the Windows and Linux behaviour of one thing has to open two crates to do it.

There were empty directories named agent/windows, agent/linux, agent/installer, agent/updater, packages/ui and packages/shared-config implying otherwise. They are written without backticks here on purpose: tools/check_doc_paths.py treats a backticked path as something a reader can follow, and these are the opposite - a note about what is not there. Git never tracked them - it does not track empty directories - so they existed only in one working copy and never in a clone. They are gone, and this section is the decision they were standing in for.

tools/check_doc_paths.py now refuses documentation that names a file which is not there, because the same defect had been found by hand three times in docs/ alone.

2. Five decisions that shape everything else

2.1 The endpoint decides, not the server

Detection and policy evaluation run on the endpoint, against a cached, signature-verified policy bundle. The server is where policy is authored and where incidents are investigated; it is not in the enforcement path.

The alternative - ship content to a server and wait for a verdict - fails on three counts. It stops working when the network does, which is when data exfiltration is most likely. It sends the very content the product exists to protect across the network. And it puts a round trip in front of every file write.

The cost is that the endpoint holds the detection logic, so a determined local attacker can learn what is detected. That is accepted and recorded in THREAT_MODEL.md §5.2: detection rules are not secrets, and the anti-rollback and signature checks stop an attacker changing them.

2.2 The simulator is the enforcement code

packages/detector-engine and packages/policy-engine compile into both the agent and the server. When an operator simulates a policy in the dashboard, the server runs the identical engine the endpoint runs.

A separate "preview" implementation would drift, and an operator who cannot trust the preview will not publish a blocking policy.

2.3 The product never claims enforcement it does not perform

Every agent reports a per-channel capability matrix. Policy evaluation clamps the decided action to what the endpoint can actually do (ChannelCapability::clamp), and the event records both what the policy asked for and what was done.

On Linux, one channel can be enforced: file_open, over directories the bundle marks, using fanotify permission events (#387). Every other channel is monitor-only, so a policy that says block on one produces an incident whose action is audit with the explanation "Endpoint capability limits this channel to audit".

The clamp reads the endpoint's reported capability, and the endpoint reports what it is actually running - an agent without CAP_SYS_ADMIN, or under a bundle that marks no path, says it cannot enforce. The dashboard shows that matrix per endpoint. Nothing in the interface implies a block that did not happen.

2.4 Matched values never leave the endpoint

Redaction happens at the point of detection, inside the agent, before an event is serialised. The server has no code path that could store a raw match because it never receives one. What travels is: detector id, category, counts, confidence, validation status, context signals, and a masked preview.

2.5 No production Windows kernel component yet

The shipped Windows agent remains user-mode and cannot block file I/O. The tree now contains the development-only, monitor-only B1 minifilter from ADR-0069, plus its service protocol and native WDK build gate. It is not in the production installer, has no pre-operation deny path and cannot advertise block capability. Production use remains gated on Windows lab, Driver Verifier, performance, compatibility, altitude, signing and staged-release evidence. Linux's userspace fanotify open enforcer remains separate; there is no Linux kernel module or LSM hook.

3. Component map

┌──────────────────────── endpoint ─────────────────────────┐
│  dlp-agent                                                │
│   ├── identity        device key, scoped X.509 cert       │
│   ├── policy-runtime  verify signature/tenant/scope/      │
│   │                   expiry/anti-rollback, cache         │
│   ├── detector-runtime  extract → normalise → scan        │
│   ├── monitor         inotify / ReadDirectoryChangesW     │
│   ├── secure-buffer   bounded, severity-aware SQLite queue│
│   └── delivery        batch, sign, backoff with jitter    │
└───────────────────────────┬───────────────────────────────┘
                            │ HTTPS, device-key-signed requests
                            │ (mTLS where TLS terminates in-app)
┌───────────────────────────▼───────────────────────────────┐
│  dlp-management-api                                       │
│   /api/v1/agent/*   enroll, checkin, policy, events       │
│   /api/v1/*         dashboard and integrations            │
│                                                           │
│   domain/  enrollment · policies · ingest · incidents     │
│            identity   · detectors · keys   · housekeeping │
└───────────────────────────┬───────────────────────────────┘
                            │
                   ┌────────▼────────┐
                   │   PostgreSQL    │  32 tables
                   │   append-only   │  composite tenant FKs
                   │   audit chain   │
                   └─────────────────┘
                            ▲
                            │ same-origin, session cookie
┌───────────────────────────┴───────────────────────────────┐
│  dashboard (React, TypeScript strict, Vite)               │
└───────────────────────────────────────────────────────────┘

Optional services, contract-defined, not yet integrated:
  extraction-service  (declined - ADR-0009)
  detection-service   (declined - ADR-0011)
  discovery-service   (declined as a service; a worker in the API - ADR-0013)
  integration-service (not built; webhook, syslog, Slack and ticketing
                       deliver from the API itself)

There is no message broker, and that is measured rather than assumed

SPIKE-006 asked at what rate direct ingestion into PostgreSQL stops being adequate. Two runs differing only in how many events each request carried:

events per batch	requests/s	events/s
20	79.7	1,595
100	21.3	2,127

Requests per second fell fourfold and events per second rose by a third. The cost is per request, not per event, and the database spent its time in LWLock:WALWrite and IO:WALSync - write-ahead log flushing.

A broker does not remove the signature verification, the nonce insert or the WAL sync. It defers them, and they still happen at the same rate. What it adds is a component every customer has to run, monitor and upgrade.

So: no broker. Put pg_wal on fast storage, keep batches large - the agent's default of 100 is already right - and revisit when a real deployment reports a p99 above a second at a rate below these figures. That is the trigger, and it is a number rather than a feeling.

Both figures are floors: the host was running another workload at a load average of 20 throughout, which the harness detects and says.

4. Shared packages

Package	Runs in	Responsibility
dlp-contracts	agent, server	Wire types, schema versioning, redaction primitives
dlp-crypto	agent, server	Ed25519 bundle signing, Ed25519/P-256 request auth, Argon2id, tokens, EDM fingerprints, document shingling
dlp-detector-engine	agent, server	Safe regex gate, normalisation, validators, confidence, overlap suppression, EDM runtime, endpoint extraction
dlp-policy-engine	agent, server	Scope, conditions, exceptions, conflict resolution, explainable risk
dlp-secure-buffer	agent	Durable bounded queue with lease semantics and integrity checks

Sharing these is what makes agent and server behaviour identical by construction rather than by discipline.

5. Trust boundaries

Boundary	Crossing	Control
Endpoint user → agent	Local filesystem, config	Agent runs as a service; identity file is 0600; policy cache is re-verified on every load
Agent → server	HTTPS	Device request signature (Ed25519 or ECDSA P-256, pinned at enrollment) over method, path, body hash, device, timestamp and nonce. Freshness window plus a nonce table make replay useless
Server → agent	Policy bundle	Ed25519 signature; agent checks tenant, device scope, group scope, expiry, clock skew and a monotonic version floor
Tenant → tenant	Every query	tenant_id in the lookup, never a check afterwards. Composite foreign keys make cross-tenant references impossible in the schema
Analyst → evidence	Dashboard	Role permission plus a required justification; every access appended to the audit chain
Untrusted document → parser	Extraction	Endpoint parser is pure Rust with byte, time and recursion limits. Complex formats go to a sandboxed service with no network

6. Data flow: one file, end to end

1. inotify reports a write. The path is checked against exclusions (including the agent's own state directory) and coalesced with other events for the same file.
2. A write event is a report that the content changed, so the file is read (DLP-1528): a same-size rewrite with its old modified time restored was skipped before. Only the periodic walk consults the fingerprint cache, and only for a file nothing reported as changed: unchanged size, unchanged mtime and the same policy bundle version means skip. Including the bundle version is what makes a newly added detector apply to files that already exist.
3. The file is read under a size cap, hashed with BLAKE3, and type-detected from magic bytes rather than the extension. A mismatch is itself a signal.
4. Text is extracted natively, or the file is reported as requiring the central service - never silently treated as empty.
5. Text is normalised: NFKC, invisible characters stripped, whitespace collapsed, with an offset map back to the original.
6. Every compiled detector runs under byte, match and time budgets. Candidates are structurally validated; failures narrow the span and retry before being dropped. When the whole scan's budget runs out, the detectors after that point do not run, and the file is reported partially scanned with how many did not run, never clean (DLP-1530). The detectors a policy reads, by id, through a classification, a data category or a total, run first and are never the ones that budget drops; each is still held to its own budget (DLP-1629).
7. Overlapping matches are reconciled - the more specific validated match wins the bytes, so IBAN digits are not also reported as a payment card. A document-fingerprint match has no location and takes no part: it is reported beside whatever the document also contains, and nothing inside it is suppressed because of it (DLP-966).
8. Confidence is computed from base confidence plus validation, keyword proximity, negative context and match volume. Every adjustment is emitted as a signal.
9. Policy evaluates: tenant, status, effective period, schedule, scope, condition tree, exceptions, conflict strategy.
10. The decided action is clamped to the endpoint's capability for that channel.
11. An event is built with a deterministic id derived from (device, channel, content hash, path, action, bundle version) - so a crash and replay produce the same id. A departure adds how the file left (moved_from, deleted, ...) to the action, so it never shares an id with the arrival at the same path (DLP-1705). Once a sensitive file has left a path, the number of times it has goes in too, so a file put back unchanged is a new arrival and its next departure a new departure (DLP-1804); before that, ids are as they were.
12. The event is queued. If the queue is full, the oldest lowest-severity events are dropped and counted. An event whose id is already queued is not queued twice, and gives its sequence number back, so a sequence gap on the server is an event that never arrived (DLP-1771).
13. Delivery batches, signs and posts. Accepted and duplicate both mean "the server has it"; the agent drops them either way.
14. The server validates, stores under a unique (tenant_id, event_id), and correlates into an incident by a deterministic key that folds repeated activity on the same content into one incident. Activity after that incident was resolved or closed raises a new one, related to it (DLP-1905).
15. Risk is scored from twelve weighted, individually explained factors. An approved exception and an enforced block then reduce it, and a file gone from where it was found soon after is added as points (DLP-1831); each of those is reported beside the factors.

7. What is contract-only

Honest statement of scope. These have a defined contract and a database schema and nothing running behind them:

• Extraction service (Tika) - OOXML, ODF and RTF are read in process by dlp-detector-engine::extract. What needs a service is the pre-2007 binary OLE formats - .doc, .xls, .ppt - which the extractor reports rather than reads. Blocked on DLP-SPIKE-003.
• Detection service (Presidio) - DetectorKind::External exists and compiles to Matcher::Unsupported, so a pack may carry one and no scan acts on it. Blocked on model licensing, DLP-SPIKE-002.
• File-property and file-hash detectors - in the contract, and Matcher::Unsupported in the engine for the same reason as above: a pack can describe one and nothing runs it.
• Browser extension and signed updates - contracts only.

What is not here any more, because it ships: OCR, through a Tesseract helper the agent runs only when it can isolate it (ADR-0012, DLP-231); document fingerprinting, compiled in the engine with corpora built and probed over /api/v1/fingerprint-corpora; EDM, with datasets imported and keys rotated over /api/v1/edm-datasets and a screen in the console; webhook, syslog (RFC 5424 or CEF), Splunk HEC, Slack and Teams delivery with Jira and ServiceNow escalation, where SIEM, e-mail and s3 were removed rather than built (ADR-0040); and discovery of a mounted share and of an S3 bucket, which runs as a worker inside the API rather than as a service, the way ADR-0013 decided.

Each of those has edges, and the edges are in Known limitations rather than here. The distinction this section draws is between nothing behind it and built with limits, and it is worth keeping sharp: every one of the five just listed was still in the list above after it shipped, which is how a register of what is missing becomes the evidence somebody uses to decide the work cannot be done.

Every one is a ticket in the tracker (task.erpona.app, project MyDLP) with acceptance criteria.

8. Related documents

Threat model · Privacy model · Policy model · Detector catalogue · Agent capability matrix · API · Testing · Deployment · Known limitations · The reference console, and what it is not