Appearance
The detector engine, claim by claim
WIN-DLP-EPIC-009 asks for a modular detector interface with validated, explainable, redacted results, because false positives destroy a DLP programme faster than false negatives; validation and confidence are what keep the incident queue workable.
All four properties are built, and the validation and redaction halves are the most heavily property-tested code in the repository. What was missing is a measurement: the engine's defence against false positives was structural and argued, and nothing had ever counted one. This audit counted, and found two.
Implementation: packages/detector-engine/src/engine.rs, with validators.rs, normalize.rs, safe_regex.rs, redaction in dlp-contracts, and the shipped packs in builtin.rs.
1. Modular
| Claim | Proved by |
|---|---|
| A detector kind this build cannot run is reported, not skipped | engine::tests::unsupported_detector_kinds_are_reported_not_silently_skipped |
| A detector that will not compile is dropped and named, and the rest still run | detector_runtime::tests::a_detector_that_will_not_compile_is_dropped_and_reported, …::the_detectors_that_did_compile_still_run (in dlp-agent-core) |
| Every built-in compiles | builtin::tests::every_builtin_detector_compiles |
| Detector ids are unique across packs | builtin::tests::detector_ids_are_unique_across_packs |
| Packs version independently | builtin::tests::every_country_detector_names_its_pack_version; az and tr are separate packs, and the management API accepts a new tenant pack without an agent release |
| A signed new-country pack runs on an already-built agent and reports the pack version | policy_runtime::tests::a_signed_new_country_pack_activates_without_an_agent_upgrade_and_reports_its_pack_version |
| Unsigned pack changes never become runnable | the same acceptance test tampers with the detector payload after signing, observes rejection, and proves the previously verified definition remains active |
2. Validated
Twenty-seven validator tests, sixteen of them property tests. The properties are the ones worth having, because they hold over generated input rather than over examples somebody thought of:
| Claim | Proved by |
|---|---|
| A correct check digit always passes; any other always fails | validators::properties::a_correct_check_digit_always_passes_luhn, …::any_other_check_digit_always_fails_luhn |
| A repeated digit never passes Luhn | validators::properties::a_repeated_digit_never_passes_luhn |
| Separators never change the verdict | validators::properties::separators_never_change_the_luhn_verdict |
| A correctly constructed IBAN always validates; changing one digit always breaks it | validators::properties::a_correctly_constructed_iban_always_validates, …::changing_one_digit_always_breaks_an_iban |
| The same, for a Turkish TCKN | validators::properties::a_correctly_constructed_tckn_always_passes, …::changing_one_digit_always_breaks_a_tckn |
| An unknown country is inconclusive, never validated | validators::properties::an_unknown_country_is_inconclusive_never_validated |
| No validator panics on arbitrary input | validators::properties::no_validator_panics_on_arbitrary_input |
| A validator never invents a verdict | validators::properties::validate_never_invents_a_verdict |
| A Luhn failure suppresses the card match | engine::tests::luhn_failure_suppresses_the_card_match |
| IBAN digits are not also reported as a payment card | engine::tests::iban_digits_are_not_also_reported_as_a_payment_card |
| …and a real card beside an IBAN still is | engine::tests::a_real_card_next_to_an_iban_is_still_reported |
The last pair is the shape every suppression rule should have. Suppressing the wrong match is half the job; not suppressing the right one beside it is the half that gets forgotten, and it has its own test.
What the ranking gets wrong is worse than what it gets right (DLP-276). Until it was measured, specificity ranked a validated pattern above an exact data match, so a record whose account field is also a valid IBAN was found and then discarded in favour of core.iban. A check digit says a number is well formed; an exact data match says the value is in the authorised dataset. Evidence now outranks validation, with length still breaking the tie among equals - and the control that a pattern in no dataset is reported exactly as before has its own test, because that is the half that gets forgotten.
3. Explainable
| Claim | Proved by |
|---|---|
| Overlap suppression is reported, not silent | engine::tests::overlap_suppression_is_reported_not_silent |
| A location cap is flagged and the count stays true | engine::tests::location_cap_is_flagged_but_count_stays_true |
| Negative context lowers confidence | engine::tests::negative_context_lowers_confidence |
| A truncated scan says so | engine::tests::scan_respects_the_byte_budget_and_flags_truncation |
| A low-precision detector cannot fire without context | builtin::tests::low_precision_detectors_require_context_to_fire, …::az_fin_fires_only_with_context |
The count-stays-true row is a small thing that matters: capping the locations recorded must not cap the number found, or a policy thresholding on volume silently stops seeing bulk.
4. Redacted
Twenty tests, eleven of them properties, in dlp-contracts.
| Claim | Proved by |
|---|---|
| Full redaction reveals nothing but the length | redaction::properties::full_redaction_reveals_nothing_but_the_length |
A partial preview keeps no more than keep characters, and they are the ones promised | redaction::properties::a_partial_preview_keeps_no_more_than_keep_characters, …::the_revealed_characters_are_the_ones_promised |
| A short value reveals nothing at all | redaction::properties::a_short_value_reveals_nothing_at_all |
| No input can make a preview unbounded | redaction::properties::no_input_can_make_a_preview_unbounded |
| A silhouette never reproduces the value, and hides the length of long runs | redaction::properties::a_silhouette_never_reproduces_the_value, …::a_silhouette_hides_the_length_of_long_runs |
A blob that merely contains an @ is fully masked | redaction::tests::a_blob_that_merely_contains_an_at_sign_is_fully_masked |
| Credential detectors redact fully | builtin::tests::credential_detectors_redact_fully |
| A preview never contains the matched value | detector_runtime::tests::previews_never_contain_the_matched_value |
5. Safety of the matcher itself
| Claim | Proved by |
|---|---|
| A classic ReDoS pattern is linear here | safe_regex::tests::classic_redos_pattern_is_linear_here |
| Backreferences and lookaround are rejected, with advice | safe_regex::tests::backreferences_are_rejected, …::lookaround_is_rejected_with_advice |
| The gate is stricter than the compiler, and never passes what the compiler would reject | safe_regex::tests::the_gate_is_still_stricter_than_the_compiler, …::the_gate_does_not_pass_something_the_compiler_rejects |
| Zero-width evasion does not work | engine::tests::zero_width_evasion_does_not_work |
| No invisible character survives normalisation | normalize::properties::no_invisible_character_survives |
| Every offset lands on a character boundary of the original | normalize::properties::every_offset_lands_on_a_character_boundary_of_the_original |
| Malformed UTF-8 boundaries do not panic | engine::tests::malformed_utf8_boundaries_do_not_panic |
| Every matcher honours its time budget | packages/detector-engine/tests/detector_budget.rs (DLP-257) |
6. The measurement that was missing
The engine's defence against false positives was structural - a low-precision detector's base confidence sits below its own threshold, so it cannot fire without context - and documented, in Detector catalogue's per-detector precision statements. It was never counted.
cargo run --release -p dlp-detector-engine --example precision_probe counts it, on the only realistic corpus that can live in a repository whose purpose is to stop documents travelling: this repository's own prose. Files carrying deliberate test data are excluded by path, listed in the source, because a detector firing on a published test card number is the detector working. This document is excluded too: it quotes the strings the probe found, so leaving it in would make the corpus measure the measurement.
The first run, over 61 files and 0.57 MiB:
13 finding(s) across 4 detector(s)
core.confidential_marking: 7
core.email: 3
core.ipv4: 2
core.db_connection_string: 1Adjudicated by hand, because the verdict on a finding is a person's:
core.confidential_marking- wrong, seven times out of seven. The term list contains the bare wordrestrictedwithwhole_word: false, so "access is restricted to administrators" is a confidentiality marking at confidence 0.70. So isunrestricted- a document that says in as many words that it is not restricted is flagged as though it were.DLP-265. Fixed. The terms are whole-word now and the barerestrictedis gone, keeping the phrase forms. A second-order finding fell out of it that nobody had looked for:gizliis insidegizlilik,məxfiinsideməxfilikandконфиденциальноinsideконфиденциальность- the words for privacy and confidentiality - so every privacy policy in three of the four languages was a confidentiality marking. A re-run of the probe went from 17 findings to 8 across the corpus, and this detector from 11 to 2. WithDLP-266fixed as well the corpus stands at 6, from 17: the two detectors this audit adjudicated as noise account for every one of the eleven that went.core.ipv4- noise.127.0.0.1and0.0.0.0are never anybody's data and are in every deployment guide. SeverityInfo, so the cost is queue volume rather than a wrong verdict.DLP-266. Fixed. A validator parses the address and drops loopback, the unspecified address and broadcast - parsed rather than pattern-matched, so127.0.0.53goes with127.0.0.1without a second pattern to keep in step. RFC1918 is kept on purpose.core.email- correct.@example.comis a reserved domain and a real address shape. Finding it is the detector working.core.db_connection_string- correct, and the most useful hit of the thirteen: an example connection string in a deployment guide is exactly the thing an estate should know it has written down.
Two thirds of the findings on a corpus of technical writing were noise, from two detectors, and neither would have been visible without counting. That is the argument for keeping this harness.
What the number is not. A rate. Technical documentation is not a finance team's shared drive, 0.57 MiB is small, and the corpus contains no payroll, no contracts and no scanned identity documents. It is a floor and a smoke alarm, not a precision measurement.
7. What is not covered
- No realistic false-positive corpus. §6 is the repository's own prose, which is the best that can live here. A real measurement needs documents this project should not hold, which makes it a customer-side exercise, and the honest place for it is a pilot rather than CI.
precision_probeis not run in CI. It prints findings for a person to read and has no pass or fail, so a gate would have to invent a threshold. Run it after changing a detector.- Shipped-corpus recall is measured, production recall is not. The 58 synthetic labelled documents exercise every built-in detector and gate its current revision in CI. They cannot estimate recall on a customer's real documents; that still needs a representative, locally governed corpus.