Appearance
Performance, claim by claim
WIN-DLP-EPIC-026 asks for budgets, benchmarks and the measurements that let the product stop guessing - because "lightweight" without a number is a marketing claim, and an endpoint agent that is not measured will eventually be uninstalled by an angry administrator.
states the discipline in its first line - a number is either measured or marked as a target - and keeps to it. The audit's finding is not a false number. It is that the budget quoted the fastest thing the engine does and omitted the slowest, which had been measured and recorded somewhere else.
1. What is measured
| Figure | Measured by |
|---|---|
| Detector throughput, 21 detectors, text | 13.9 MiB/s - cargo bench -p dlp-detector-engine |
| EDM scan cost, per field | 0.17–0.43 MiB/s - examples/edm_latency.rs (DLP-242) |
| EDM dataset size and collision rate | examples/edm_scale.rs, SPIKE-005 (DLP-34) |
| Ingestion capacity | 3,385 events/s at p99 630 ms - tests/load_test.sh, Capacity |
| Ingestion with correlation | 2,622 events/s at p99 283 ms - as above, DLP_LOAD_MATCH_POLICY=1 |
| Ingestion commit count | tests/ingest_commits.sh - counted, not timed, so a contended host cannot skew it |
| Detector false positives on ordinary prose | examples/precision_probe.rs (DLP-123) |
| What a scan costs the endpoint | agent/core/examples/endpoint_bench.rs (DLP-210) - the figures below |
What a scan costs the endpoint
Measured with cargo run --release -p dlp-agent-core --example endpoint_bench, three runs, median, against the 25 built-in detectors. On Linux, on a host under load - these are a floor, and the Windows figures this ticket actually asks for are not here because no Windows host has run it yet.
| Case | Median | Rate |
|---|---|---|
| 100,000 small files, first pass | 10.40 s | 9,619 files/s · 2.07 MiB/s |
| 100,000 small files, repeated | 9.03 s | 11,071 files/s · 2.38 MiB/s |
| One 23 MiB text file | 1.21 s | 19.5 MiB/s |
| 200 DOCX | 2.76 s | 72.5 files/s · 0.13 MiB/s |
| 200 PDF | 1.40 s | 142.8 files/s · 7.00 MiB/s |
The single-file figure agrees with the 13.9 MiB/s bench above, which is the sanity check that matters: two different harnesses over different content landing in the same order of magnitude.
Office is the expensive one, by an order of magnitude - 0.13 MiB/s against 2.07 for plain text. OOXML is a ZIP, so every document pays decompression before a detector sees a byte. An estate of documents is the estate this product is sold into, so this is the number to quote, not the text one.
The first version of this harness reported 355 MiB/s for a large file, which was the speed of refusing to read it: the file was 43 MiB and the extractor's limit is 32 MiB, so nothing was scanned. The harness now prints what came back - not fully scanned: … or the hit count - and the file is sized under the limit. A throughput figure computed from a file the agent never read is the most flattering possible way to get one wrong.
Idle CPU and working set are not measured by it. They belong to the running service rather than to a scan, and the agent reports both in dlp-agent status --json; the measurement is an idle window on the same host, which belongs with the rest of the Windows host qualification.
That is a genuinely good list for a product this young, and two entries show the discipline working rather than merely being stated:
ingest_commits.sh counts transactions instead of timing them, because a count is unaffected by whatever else the host is doing - which is the objection that makes every throughput figure in CAPACITY.md a floor.
edm_latency.rs measures its own baseline in the same run on the same host, "because a throughput figure compared against one taken on another machine is not a comparison".
2. The finding: the budget quoted the fast path
PERFORMANCE_BUDGET.md's active-scan table led with 13.9 MiB/s. A reader takes that as what a scan costs.
With an EDM set - which is the feature a customer buys this product for - it is 0.17 to 0.43 MiB/s. Measured, by DLP-242, and recorded in TESTING.md where a performance budget's reader has no reason to look:
| EDM set | 1 MiB document | vs no EDM |
|---|---|---|
| none | 116 ms | 1.0x |
| present, 0 fields | 1,837 ms | 15.8x |
| 3 fields - an ordinary customer record | 3,193 ms | 27.5x |
| 8 fields | 5,921 ms | 51.0x |
The zero-field row is the one that changes what somebody would do about it: most of the cost is not the fields. A set with no fields still tokenises the document and normalises every run of up to eight adjacent tokens, and hashes nothing - 1.7 s per megabyte before a single field exists. DLP-256 is the remedy and it attacks the right half.
And the consequence an operator meets first: the endpoint's per-detector budget is 300 ms, against 3.2 s for a 1 MiB document with three fields. The scan stops at the deadline and the file is reported partially scanned - correct behaviour, honestly reported, and not what anybody expects from a product quoting 13.9 MiB/s.
The table now sits in the budget as well, because a performance budget that omits the most expensive component is not a budget.
Also added: a row saying document-fingerprint scan cost is not measured. There is no harness at all, which FINGERPRINTING.md §6 records - and DLP-242 is the proof that "probably fine" is not a safe assumption about a matcher.
3. What is not measured, and is the epic's actual subject
Every endpoint benchmark is unrun:
| Benchmark | Ticket |
|---|---|
| 100,000 small files, cold and warm cache | DLP-210 |
| A single large file at the size cap | DLP-210 |
| Office documents, PDFs, mixed corpus, on the endpoint loop | DLP-210 |
| Thousands of detector rules: compile time and memory | DLP-211 |
| Policy evaluation latency distribution | DLP-211 |
| Rapid change burst and watcher queue overflow | DLP-448 |
| A USB copy of a directory tree | DLP-1176 |
| Offline burst: 24 hours queued, then delivery | endpoint side unrun |
The two idle budgets - < 0.5 % CPU and < 80 MB working set - are now measurable, which they were not: until DLP-274 the agent reported cpu_percent: 0.0 and memory_mb: 0.0 because nothing measured them, so the product could not check itself against its own two headline budgets even in principle and the field that would have shown a violation reported a flattering constant. Every check-in now carries what the agent actually costs, or says nothing where it has not measured yet.
What is still missing is the check: nothing compares the reported figures against these budgets or alerts on a fleet drifting past them. DLP-204 is where that belongs.
4. What is structural rather than measured
Some rows are answered by design rather than by a number, and the budget says so rather than inventing a figure:
- Unchanged file - the fingerprint cache keys on size, mtime and bundle version, so a rescan costs a
stat. - Threads - bounded, with no unbounded spawn anywhere in
agent/core. - Disk - the discovery walk runs on a schedule, not a loop.
- Backpressure - queue full evicts the oldest lowest-severity event and counts the drop.
These are legitimate answers. "Structural" is only an honest word when the structure is actually asserted somewhere, and each of these is.
5. What is not covered
- Nothing has been measured on Windows at all. Every figure above is Linux, on a shared host, and
CAPACITY.mdsays so in its own words.DLP-210,DLP-211,DLP-448, andDLP-1180for a quiet host. - No fingerprint-corpus cost measurement, §2.
- No measurement of the extraction path separately from detection.
DLP-263gave it a time budget - ten seconds, cooperative, reported as an incomplete extraction when it fires - so a hostile input is now bounded. How long extraction actually takes on ordinary documents is still unmeasured, and the budget is a backstop rather than a substitute for that figure.