Filesystem discovery, claim by claim

WIN-DLP-EPIC-006 asks for scheduled and on-demand scanning of data at rest, with filters, throttling, checkpointing and resume - because most sensitive data on an endpoint is already there, and discovery is what finds it.

On-demand scanning and filters are built and tested. Checkpointing was missing in a way that mattered and is fixed here. The walk repeats on its own (DLP-300) and pauses on battery and while the user is active (DLP-168); a general I/O throttle is still absent.

Implementation: Agent::run_discovery in agent/core/src/runner.rs, the walk in agent/core/src/monitor.rs, the skip-what-has-not-changed cache in agent/core/src/fingerprint_cache.rs.

---

1. On-demand scanning

dlp-agent scan checks in, refuses to run without a verified bundle - there is nothing to scan for without one - and walks every monitored path the bundle marks discovery_scan.

Claim	Proved by
Discovery finds files recursively	monitor::tests::discovery_finds_files_recursively
A non-recursive path stays at the top level	monitor::tests::non_recursive_discovery_stays_at_the_top_level
The file cap is honoured	monitor::tests::discovery_honours_the_file_cap
A path that does not exist is reported absent, not an error	run_discovery - one path list covers a mixed fleet, so a Windows path simply will not exist on a Linux host
A scan with no verified bundle refuses rather than reporting a clean machine	dlp-agent scan bails with the reason and what to do

dlp-agent scan <file> --explain (DLP-1670) reads one file the way a discovery scan reads it, with the bundle already held, and says why it is or is not detected: what was read and what was not, every detector's verdict with the signals behind it (a supporting term found or missing, a candidate set aside for want of one, a budget that ran out), and every policy's decision. Policies are evaluated as when the endpoint reaches its server, and the offline action is named beside it when it differs (DLP-1700). It is for administrators, with the Administrators group enabled in the token (not merely present), and is refused before the agent loads otherwise. It reports and queues nothing, and shows no value.

Claim	Proved by
A card is explained as reported, by the policy that decides, and its digits are not shown	explain_one_file::a_card_is_explained_as_reported_by_the_policy_that_decides_and_its_digits_are_not_shown
Explaining queues nothing	the same test, which reads the queue afterwards
A candidate set aside for want of a supporting term is said	explain_one_file::a_card_set_aside_for_want_of_a_word_is_explained_as_that
REPORTED is exactly what the agent reports for the same file, and a match below its detector's floor is "not reported, below its floor" and is not handed to a policy (DLP-1701)	explain_one_file::explain_reports_exactly_what_the_agent_reports_and_a_match_below_its_floor_is_not_one

2. Filters

Claim	Proved by
Exclusions are honoured during discovery	monitor::tests::exclusions_are_honoured_during_discovery
An exclusion applies to its own path, not to every path	monitor::tests - the per-path ExclusionRules (DLP-148)
A cloud placeholder is not hydrated to scan it	monitor::placeholder_reason - reading the file would download it, which is a data-transfer event caused by the scanner

3. Not scanning what has not changed

The cache keys on size, mtime and the policy bundle version, so a rescan costs a stat per unchanged file rather than a read.

Claim	Proved by
An unseen file is scanned	fingerprint_cache::tests::an_unseen_file_is_scanned
A changed size or mtime triggers a rescan	fingerprint_cache::tests::a_changed_size_or_mtime_triggers_a_rescan
A newer policy bundle forces a rescan	fingerprint_cache::tests::a_newer_policy_bundle_forces_a_rescan
A corrupt cache degrades to a full rescan rather than failing	fingerprint_cache::tests::a_corrupt_cache_file_degrades_to_a_full_rescan
Forgetting a file makes it scannable again	fingerprint_cache::tests::forgetting_a_file_makes_it_scannable_again

The bundle-version row is the one that keeps the cache honest. A new detector must see files a previous scan called clean, or the cache would quietly pin the estate to whatever the rules were the first time anybody looked.

4. Checkpointing - was missing, and it mattered

The cache used to be written once, after every path had been walked.

A scan interrupted before that - a laptop closed, a service stopped, a machine shut down at the end of the day - threw away every file it had scanned, and the next run started from nothing. On a volume that takes longer to walk than a working day, that is a scan which can never finish: each day's run redoes what the last one did and is stopped in the same place.

Fixed (DLP-120): the cache is written every DISCOVERY_CHECKPOINT_FILES (500) files, and at the end of each monitored path, as well as at the end of the run. Writing after every file would put a whole-file save between each one, which costs more than the work it protects; 500 files is a second or two of scanning at the throughput in Capacity - a tolerable amount to lose and an unnoticeable amount to spend.

What this is not is a resume cursor. An interrupted scan still re-walks the tree from the start; what it no longer does is re-scan the files it already finished. The walk is a stat per file against an open-and-read per file, so the expensive half is now protected and the cheap half is not. A true cursor would be worth having on a very large volume and is not worth its complexity before somebody has one - see §6.

5. Scheduling and throttling

Capability	State
Repeating the walk	Built. A running agent walks its monitored paths again every discovery_interval_secs, an hour by default; zero switches it off, and the health report says when the last walk ran (DLP-300)
Pause on battery, and while the user is active	Built (DLP-168). Below discovery_pause_below_battery_percent (30 by default) on battery, and, when discovery_require_idle_secs is set, while the user is active; has the detail
Any other I/O throttle	Absent. run_discovery walks as fast as the disk allows; max_files is a cap, not a rate
Acting on a finding	An endpoint finding is an event and, where a policy matched, an incident, handled in the incident workflow. Recording a remediation on a finding is built for server-side share scans (DLP-234), not for the endpoint walk
Scheduling a scan at a time of day	Absent for the endpoint walk, which repeats on an interval. Server-side share scans repeat on an interval in hours (DLP-232)

A note on the record rather than on the code: DLP-162 is titled "Data-at-rest discovery with filters, throttling and resume" and is closed, with no comment saying what it delivered. Filters were delivered. Throttling was not, and resume was not - which is why they are listed above against their own tickets. A title is not a receipt.

---

6. What is not covered

• The checkpoint trigger is not directly tested. The cache's own behaviour is (§3), and so is discovery's walk (§1); what has no test is whenpersist_cache is called during a scan. Asserting it needs a constructed Agent - identity, keystore, buffer, a verified bundle - which is heavier than the assertion is worth, and a shell test that interrupts a real scan part way would be timing-dependent. Stated here rather than implied.
• No resume cursor, by choice - see §4.
• Discovery on a network share is agent/core/src/network_share.rs and DLP-227, not this document.
• Server-side discovery - the scan targets a console operator sets, and the worker that runs them - is apps/management-api/src/domain/discovery.rs, with its own ten tests, and is a different epic.