Document fingerprinting, claim by claim

WIN-DLP-EPIC-011 asks for exact, normalised and partial matching against protected reference documents, because whole-document controls are what legal and M&A teams actually ask for.

All three kinds of matching are built, tested, and - unlike Exact Data Match - connected end to end: a corpus built on the server reaches an endpoint through a signed bundle and is matched there. The console's Document fingerprints section (Classification) builds a corpus from files, tests what it recognises, shows each version's lifecycle and deployment, and withdraws one behind a confirmation that names what depends on it (DLP-268, DLP-587, DLP-588, DLP-589).

Implementation: packages/crypto/src/fingerprint.rs for the shingling, packages/detector-engine/src/fingerprint_runtime.rs for matching, apps/management-api/src/domain/fingerprint_corpus.rs for the server.

---

1. The three kinds of matching

Kind	Claim	Proved by
Exact	A whole copy matches at any sensible threshold	fingerprint_runtime::tests::a_whole_copy_is_a_match_at_any_sensible_threshold
Normalised	Reformatting does not defeat it	fingerprint_runtime::tests::reformatting_does_not_defeat_it
	Punctuation and case do not change a shingle	domain::fingerprint_corpus::tests::punctuation_and_case_do_not_change_a_shingle
	The same paragraph produces the same shingles	domain::fingerprint_corpus::tests::the_same_paragraph_produces_the_same_shingles
Partial	A pasted paragraph is found inside an unrelated document	fingerprint_runtime::tests::a_pasted_paragraph_is_found_inside_an_unrelated_document
	…and through the whole engine, not just the index	engine::tests::a_pasted_paragraph_is_found_through_the_engine

Matching is on protected-corpus coverage, not on Jaccard similarity. A paragraph pasted into a long unrelated document can cover 40% of the protected reference while the candidate's containment is small; thresholding on the candidate would mean the more a thief surrounded the stolen text, the less it matched. The runtime keeps candidate containment as a diagnostic only.

2. Saying "I don't know" instead of "clean"

The property this epic needs most, because the alternative is a whole-document control that quietly passes short extracts.

Claim	Proved by
A two-line message is too short to judge, not clean	fingerprint_runtime::tests::a_two_line_message_is_too_short_to_judge_rather_than_clean
40% of a reference inside a larger file reaches the configured threshold	fingerprint_runtime::tests::forty_percent_of_a_reference_inside_a_larger_file_matches_the_configured_threshold
"Too short" is not the same answer as "no match"	fingerprint_runtime::tests::too_short_is_not_the_same_as_no_match
An empty corpus matches nothing rather than everything	fingerprint_runtime::tests::an_empty_corpus_matches_nothing_rather_than_everything
A threshold of zero does not match everything	fingerprint_runtime::tests::a_threshold_of_zero_does_not_match_everything
An unrelated document is not a match	fingerprint_runtime::tests::an_unrelated_document_is_not_a_match
A nonsense window does not panic	fingerprint_runtime::tests::a_nonsense_window_does_not_panic
The hash length comes from the corpus rather than being assumed	fingerprint_runtime::tests::the_hash_length_is_taken_from_the_corpus_rather_than_assumed
The corpus version travels with the index	fingerprint_runtime::tests::the_corpus_version_travels_with_the_index

Three of those are the same instinct written three times: an empty corpus, a zero threshold and a nonsense window are the configurations under which a naive implementation matches everything, and each has its own test.

3. The source documents do not travel

Claim	Proved by
A document becomes shingles and the text does not come back	domain::fingerprint_corpus::tests::a_document_becomes_shingles_and_the_text_does_not_come_back
A document over the size limit is refused by size	domain::fingerprint_corpus::tests::a_document_over_the_size_limit_is_refused_by_size
A document shorter than the window is refused, with the reason	domain::fingerprint_corpus::tests::a_document_shorter_than_the_window_is_refused_with_the_reason
A format this build cannot read is named, not treated as empty	domain::fingerprint_corpus::tests::a_format_this_build_cannot_read_is_named_rather_than_treated_as_empty
A wider window produces fewer shingles	domain::fingerprint_corpus::tests::a_wider_window_produces_fewer_shingles

tests/fingerprint_corpus.sh - 85 assertions, wired into CI (whose Windows jobs have not yet passed: Windows CI status) - runs the same ground against a real server and asserts the negative that matters: not a word of the source documents reaches the database, the bundle or the audit log.

4. It is connected, and that is worth stating

apps/management-api/src/domain/policies.rs fills fingerprint_corpora: fingerprint_corpus::for_bundle(...), and the agent passes self.policy.fingerprint_corpora() into DetectorRuntime::compile_with, which reaches the engine as with_fingerprint_indices.

That sentence is here because the line immediately above it in the same function reads edm_sets: vec![]. Exact Data Match is built at both ends and joined at neither (DLP-267); document fingerprinting is the same shape, solved. DLP-244 is the worked example that ticket points at.

build recompiles the bundle immediately, with the reason in the source: "a corpus only reaches an endpoint through a bundle, and only where a policy references it - so compiling here is what makes an upload take effect rather than sit in a table."

5. What used to be missing around it

This section listed three gaps, and all three are closed:

Gap	State
A console page	Built. Classification → Document fingerprints: the list at /classification/fingerprints, building at …/new, the recognition test at …/test and a version's detail at …/{id}. DLP-268, DLP-589
Deleting a corpus	Built as a withdrawal: DELETE /api/v1/fingerprint-corpora/{id} removes every version, recompiles the bundle and is audited with a reason. While a detector or published policy uses the corpus it is refused until the caller acknowledges them. DLP-268, DLP-588
A UI for probe	Built. "Test recognition" pastes a passage and names the corpus that claims it, and it tells "nothing matched" apart from "nothing was checked". DLP-268

---|---| | A console page | Absent. apps/dashboard/src/pages/ has EdmDatasets.tsx and nothing for corpora; grep -rn corpora apps/dashboard/src/ finds nothing. DLP-268 | | Deleting a corpus | Absent. build, list and probe exist; nothing removes one. EDM already has DELETE /api/v1/edm-datasets/{id} | DLP-268 | | A UI for probe | Absent, and it is the most valuable control here: pasting a paragraph and being told which corpus it belongs to is how somebody checks a corpus does what they meant before a policy depends on it | DLP-268 |

The delete is the one with teeth. "We fingerprinted the wrong document" has no answer today, and neither does "this document must stop being referenced" - which, for a corpus built from an M&A document, is a question that arrives with a deadline attached.

---

6. What is not covered

• No endpoint cost measurement. EDM has SPIKE-005, edm_scale.rs and edm_latency.rs; fingerprinting has none of that. What a large corpus costs an endpoint per scanned megabyte is unmeasured, and DLP-257 showed that an EDM set could reach 5.9 s per megabyte - which is the kind of thing that is only discovered by measuring.
• No test of a corpus reaching a real endpoint. tests/fingerprint_corpus.sh proves the corpus reaches a bundle; the engine tests prove matching works. Nothing places a protected paragraph on a monitored path of a running agent and waits for the incident. That is a tests/e2e_vertical_slice.sh extension and is not filed, because it belongs with whoever next touches that script.
• No measurement of false positives for fingerprinting specifically. The probe in DLP-123 covers the pattern detectors only.