Appearance
Content extraction, claim by claim
WIN-DLP-EPIC-008 asks for text to be extracted safely from the formats an endpoint actually meets, under hard resource limits - because a detector that never sees the text detects nothing, and an extractor without limits is a denial-of-service surface reachable by any file.
Both halves are built. The formats are the ones an endpoint meets, minus a documented tail. The limits are thorough in bytes, entries and depth - and, contrary to what claimed until this audit, not in time.
Implementation: packages/detector-engine/src/extract.rs and its per-format modules; packages/detector-engine/src/archive.rs for containers.
1. The formats
| Format | Proved by |
|---|---|
| Plain text, CSV, JSON | extract::tests::plain_text_is_extracted, …::csv_and_json_are_recognised_as_text |
Calendar and contact files (.ics, .ifb, .vcs, .vcf): the 75-octet line folding of RFC 5545 and RFC 6350 is undone before detection, so a card split by a fold in a meeting description or a contact note is found (DLP-1618). A card spread over separate CSV or tab-separated columns (4111,7391,0004,1111) is not joined: joining numeric cells would make every table of 4-digit values a card candidate, which on GovDocs1 it did (DLP-1627) | documented_limitations::a_card_split_by_calendar_or_contact_line_folding_is_found |
Base64, hex or percent-encoding standing free in a text, CSV, JSON, XML or YAML file. Base64, inline or wrapped as base64 prints it, is decoded and read when what it decodes to is text (DLP-1601). At least 20 characters, and 95% printable once decoded, so a key, a hash, a picture or a long word adds nothing. Across 8,001 ordinary files (this repository and the host's /usr/share/doc, /etc and Python libraries) it decoded 16 blocks, and the one detector hit was a real base64-wrapped OpenSSH private key. Since DLP-1599 a hex run of 24 digits or more is decoded the same way, and a percent-encoded token when it escapes at least four letters or digits, which a real URL never does (one that escapes only spaces is already readable and is not copied again); across the same 8,001 files these two added 36 short blocks and no detector hit. Not applied to HTML, whose data: URIs are read as files, nor to text taken out of a document | encoded_text::a_card_base64_encoded_in_a_text_file_or_a_wrapped_csv_dump_is_found, …::a_card_in_hex_or_percent_encoded_in_a_url_is_found, …::base64_of_a_picture_adds_no_text_and_raises_nothing, encoded_text::tests (in the crate) |
| DOCX, XLSX, PPTX | ooxml::tests::a_word_document_yields_its_text, …::a_spreadsheet_reads_its_shared_strings, …::a_presentation_reads_its_slides_and_notes. A workbook's sheets are read cell by cell in their columns, so a header sits over the value under it as in a text sheet (DLP-2245), and its print header and footer are read with their format codes removed (DLP-2248, xlsx_header_footer.rs). A Word watermark's text, which is VML WordArt in a string attribute, is read as a line of the part it sits in (DLP-2249, docx_watermark.rs): xlsx_fin_column::a_fin_header_outside_column_a_is_found_in_a_workbook_as_in_a_text_sheet, …::a_card_whose_groups_sit_in_separate_cells_is_not_one_card, …::a_label_held_only_by_the_shared_strings_table_is_still_read. A slide master's or layout's text box is read as the deck's text, and a placeholder's prompt is not (DLP-2250): pptx_master::a_classification_box_on_the_master_is_found_as_the_marking, ooxml::tests::a_text_box_drawn_on_the_master_is_read_and_its_placeholder_prompt_is_not |
The same as a macro-enabled file, a template or a slide show (.docm, .dotx, .xlsm, .pptm, .ppsx and the rest), an ODF template, and a document renamed to .zip or saved with no extension: the package's own main part (or ODF mimetype) decides, not the name (DLP-1594). A family name is not a rename; a Word document named .zip is | office_documents::a_card_split_across_runs_is_found_in_a_macro_enabled_document_a_template_and_a_renamed_one, …::a_macro_enabled_name_is_honest_and_a_document_named_zip_is_a_rename |
.xlsb, the binary workbook (DLP-1598): its sheets' number, string and formula-result cells and its shared strings are read from their BIFF12 records, and a shape or chart is read as in .xlsx. Its comments, pivot caches and external links are binary records this does not read. Built from MS-XLSB and pyxlsb's record layout and tested on built records; no Excel-written .xlsb has been run through it yet | office_documents::a_card_in_a_binary_workbook_is_found_as_a_number_cell_and_as_a_shared_string, xlsb::tests |
XPS and OpenXPS print files (DLP-1605), recognised by their .fpage parts: each page's glyph runs are read in document order, a space between runs on one baseline and a new line when the baseline moves, so a card printed as two runs is found. Images on the page go to OCR like Office media. Neither .xps nor .oxps is a rename any more | office_documents::a_card_printed_to_xps_across_two_glyph_runs_is_found_and_the_file_is_not_a_rename, xps::tests |
An Excel pivot cache (the copy of the source rows that outlives a deleted source sheet) and external-link cached values are read (DLP-1394) | office_documents::rows_held_only_in_a_pivot_cache_or_an_external_link_are_read |
Text that lives outside the body's own parts (DLP-1593): a shape or text box on an Excel sheet (xl/drawings/), an Excel threaded comment, SmartArt in any of the three (diagrams/dataN.xml), and a chart's cached labels, series names and values (charts/chartN.xml), which are all that is left when the chart's workbook is linked or gone. A diagram's rendered copy and a chart's colour and style parts are not read | office_documents::a_card_in_a_sheets_text_box_a_threaded_comment_smartart_or_a_charts_cache_is_found |
A file embedded in a document (a workbook pasted into a memo, an ODF Object N/, an RTF \objdata object since DLP-1390, a file attached to a PDF since DLP-1391, a file inlined in an HTML page as a data: URI since DLP-1393) is read as a file, one level deeper, and one that cannot be read is named (DLP-1389). In a pre-2007 .doc, .xls or .ppt an embedded Word document, workbook or deck is read since DLP-1665: its main stream sits in the file's own ObjectPool/, MBD.../ or Object.../ storage and is read after the document's own, the same way. Until then only the top-level stream was read, and a card in a table pasted into a Word 97 memo was never scanned. A file dropped into an Office document as an icon, an OLE Package (\x01Ole10Native), is read as a file one level deeper since DLP-1668, in a .doc, .xls or .ppt and in a .docx, .xlsx or .pptx's oleObject.bin alike, and its label is kept as its name. The same holds for an RTF document's OLE1 object of class Package since DLP-1673: a dropped .csv, .txt or .pdf had no signature for the RTF object search to find, so it was named as not unwrapped instead of read. Only a stream that parses as a package to the byte is unwrapped. Other servers keep their own data under the same stream name (all sixteen in GovDocs1 batch 006 were bitmaps, clip art and drawings), and those are left alone | cfb_text::tests::a_document_embedded_in_a_legacy_office_file_is_read_too, …::a_file_dropped_into_a_document_as_a_package_is_read_and_a_picture_is_not, office_documents::a_file_dropped_into_a_word_document_as_a_package_is_found, …::a_csv_dropped_into_an_rtf_document_is_read_not_just_named, office_documents::a_card_in_a_workbook_embedded_in_a_word_document_is_found, …::an_embedded_file_past_the_depth_limit_or_unreadable_is_named, …::an_object_embedded_in_an_opendocument_file_is_read, …::a_card_in_a_workbook_embedded_in_an_rtf_document_is_found, …::a_card_in_a_file_attached_to_a_pdf_is_found |
| Document properties | ooxml::tests::document_properties_are_read |
| ODF - text, spreadsheet, presentation | a_card_number_in_an_odt_is_found, a_spreadsheet_cell_and_a_slide_note_are_both_read (packages/detector-engine/tests/odf_and_rtf.rs) |
| RTF | an_rtf_is_recognised_from_its_magic_bytes_not_its_name, text_in_a_header_reaches_the_detectors, a_formatting_change_inside_a_card_number_leaves_the_card_whole, a_revision_run_boundary_inside_a_word_does_not_split_it (packages/detector-engine/tests/odf_and_rtf.rs) |
HTML, and an HTML mail body: scanned as its source, plus the lines a browser joins across inline elements (<span>4111</span><span>7391</span> reads as one number). A block element such as a table cell ends the line, so separate cells are not joined (DLP-1526) | html_inline_elements::a_card_with_each_group_in_its_own_span_is_found, …::table_cells_are_separate_values_and_are_not_joined_into_one, saved_email::an_html_body_with_a_card_split_across_spans_is_read_as_it_renders |
| PDF, uncompressed and Flate | extract::tests::simple_uncompressed_pdf_text_is_extracted, …::a_flate_compressed_pdf_stream_is_read |
PDF filter chains: ASCIIHex, ASCII85 and RunLength, alone or before Flate or LZW, applied in order (DLP-1588); a chain ending in DCT is a picture for OCR, and any other filter is counted unread | pdf_beyond_page_text::a_page_under_ascii85_over_flate_is_read, …::a_page_under_asciihex_ascii85_or_runlength_alone_is_read, …::a_stream_under_a_filter_this_build_cannot_decode_is_said_not_read, pdf_filters::tests |
A PDF's raster images that are not JPEGs (Flate, JBIG2, CCITT, JPX) are counted as images not read, never decoded and read as page text, and a stencil mask is not counted (DLP-1626). On GovDocs1 000859.pdf, 7,214 Flate images had been read as text, and a digit run in their pixels matched as a card number; a scan whose only content is such images is now reported likely scanned, not read. XMP metadata keeps what a person wrote (title, description, author, keywords, custom properties) and drops machine values: EXIF and TIFF tags, document ids, history, base64 thumbnails, under both the xmp and the pre-2006 xap prefixes; a thumbnail's base64 had matched as an IBAN | extract::tests::a_flate_image_is_counted_as_an_image_and_its_pixels_are_never_read_as_text, pdf_text::tests::xmp_keeps_what_a_person_wrote_and_drops_machine_values |
An image drawn inline in a page's content stream (BI ... ID <data> EI) is not text (DLP-1654): its data is skipped to the EI that ends it, and the text before and after it on the page is read. Two ArcGIS map exports in GovDocs1 draw every tile inline in ASCII85; the data's ( and < bytes were read as strings, and each 450 KB map extracted to about 470 KB of base-85 noise with eight "email addresses" in it, and now to 6 to 7 KB of its own labels. Inline image data is not OCR'd or counted as an image not read | pdf_text::tests::an_inline_images_data_is_not_text_and_the_text_around_it_still_is, pdf_beyond_page_text::an_inline_images_data_is_not_read_as_page_text |
A PDF drawn in fonts that carry a /ToUnicode map, which is every Type0 Identity-H font: Chrome's and Edge's "Save as PDF", and Word or LibreOffice for any text outside WinAnsi (Azerbaijani, Russian, Turkish). Each page's fonts are found through its resources (inherited ones too, and objects packed in object streams), and each string is read through the map of the font Tf selected (DLP-1620). Until then the strings were glyph ids read as bytes, and a card in them was noise. A line a layout engine breaks at every kerned glyph is also read as drawn, without added spaces. Embedded font programs and CMap streams are no longer read as text. Not read through a map: text inside a form XObject, and a simple font with a custom /Differences encoding and no map | pdf_fonts::a_page_printed_by_chrome_is_read_as_its_characters_and_the_card_and_iban_are_found, …::the_glyph_ids_and_the_embedded_fonts_are_not_read_as_text, pdf_cmap::tests, pdf_text::tests |
A PDF encrypted with only an owner password, which restricts printing or copying and which every viewer opens without asking (DLP-1606): the standard security handler is applied with the empty user password, revisions 2 to 6 (RC4 40 to 128, AES-128, AES-256), and its streams and strings are decrypted and read. A PDF whose empty password does not authenticate needs a real one; it stays coverage encrypted and unread, and no password is ever guessed. Fixtures written by qpdf | pdf_encryption::an_owner_password_only_pdf_is_read_at_every_revision_and_is_not_called_encrypted, …::a_pdf_that_needs_a_password_to_open_stays_encrypted_and_unread, …::the_document_title_an_encrypted_pdf_keeps_in_its_info_dictionary_is_read |
PDF text beyond the strings a page draws (DLP-1524): hex strings are decoded like literal ones; a line drawn a glyph per operator is also read joined; a filled form field's /V, a comment's /Contents and /RC, the document information (/Title, /Subject, /Author, /Keywords, UTF-16 included) and an XMP packet are read. A form value its appearance stream also draws is counted twice, as one unique value. Strings inside a compressed object stream are read as that stream's text | pdf_beyond_page_text::a1_a_filled_form_field_value_is_found and the other six of YUSIF's shapes, pdf_text::tests |
A table a browser prints to PDF, every cell in its own text object: each row is read as a line and each cell boundary as a tab, so a header cell such as FIN supports the column under it (DLP-2256). A heading drawn a glyph at a time above the table, as Chromium and Edge draw an <h3>, keeps its own line instead of running into the header row. Measured on Chromium 151 prints only; an Edge print is the endpoint's check | pdf_text::tests::a_table_drawn_cell_by_cell_reads_one_row_per_line_and_a_tab_per_cell, pdf_text::tests::a_line_drawn_a_glyph_at_a_time_keeps_its_line_break_before_the_next_line, office_documents::a_heading_drawn_a_glyph_at_a_time_above_a_pdf_table_leaves_its_header_row_alone |
A PDF stream with fewer than half its bytes printable, and no BT text object, is not read as page text (DLP-2295). Sample data and font programs with no /Type or /Subtype had their ( and < bytes read as strings, and an address-shaped run in them came out as an email the document does not contain. On GovDocs1 000146.pdf the extracted text falls from 4,238,485 bytes to 387,607, and no email was reported for that file before or after. A stream at least half printable is read as before | extract::tests::a_binary_stream_with_no_text_object_is_not_read_as_page_text, pdf_beyond_page_text::a_colour_profile_beside_an_image_only_page_does_not_make_it_read |
| ZIP containers | archive::tests::text_inside_an_archive_is_found |
Two tests worth reading twice, because they are about detection surviving the format rather than the format parsing:
ooxml::tests::a_number_split_across_runs_survives- Word splits a typed card number across runs whenever the typist paused, and an extractor that emits runs separately hands the detector sixteen fragments and no match. That test split only at group boundaries. A split inside a group was still missed untilDLP-1681: a space was written at every tag, so "411173" and "9100041111" in two runs read as two numbers. Runs are now joined as Word draws them, with run properties, spell-check and grammar markers, bookmarks, comment and permission ranges, inserted revisions and hyperlinks drawing nothing between them. A spreadsheet's rich-text runs, a slide's runs and OpenDocument spans join the same way. Paragraphs, tabs, breaks, cells, OpenDocument's space element and deleted text still separate (ooxml::tests::runs_join_as_word_draws_them_and_real_boundaries_still_separate,…::a_rich_text_cell_a_slide_run_and_an_opendocument_span_join_too,office_documents::a_card_word_split_inside_a_digit_group_is_found).an_iban_split_across_control_words_is_found- the same hazard in RTF, where the splitting is done by control words rather than by formatting.
And two about not finding things that are not there:
ooxml::tests::field_codes_are_not_treated_as_contentand…::styles_and_themes_are_not_read- a document's machinery is not its text, and reading it manufactures matches.a_font_table_does_not_become_document_text,a_picture_group_is_skipped_ rather_than_read_as_hex- the RTF equivalents. Hex-encoded image bytes read as text are an excellent source of false card numbers.
2. Nothing is silently empty
The property the whole epic rests on: a file the extractor cannot read is reported with a reason, never returned as empty and never counted as clean.
| Claim | Proved by |
|---|---|
| An unsupported format is reported with a reason | extract::tests::unsupported_formats_are_reported_with_a_reason_not_as_empty |
| A file claiming to be a DOCX and is not says so | extract::tests::a_file_that_claims_to_be_a_docx_and_is_not_says_so |
| An extension mismatch is detected | extract::tests::extension_mismatch_is_detected |
| A malformed PDF neither hangs nor panics | extract::tests::malformed_pdf_does_not_hang_or_panic |
| A PDF stream that will not inflate is counted, not crashed on | extract::tests::a_pdf_stream_that_will_not_inflate_is_counted_not_crashed |
| A compressed PDF needing more than Flate says so honestly | extract::tests::compressed_pdf_is_honestly_reported_as_needing_the_service |
| An ODT that is not a ZIP is reported, not treated as clean | a_file_named_odt_that_is_not_a_zip_is_reported_not_treated_as_clean |
| An RTF with no readable text is reported, not called clean | an_rtf_with_no_readable_text_is_reported_not_called_clean |
| A truncated RTF returns what it read rather than failing | a_truncated_rtf_returns_what_it_read_instead_of_failing |
| Binary data is not mistaken for text | extract::tests::binary_data_is_not_mistaken_for_text |
| A document that is mostly scanned pages is not reported clean | detector_runtime::tests::a_document_that_is_mostly_scanned_pages_is_not_reported_clean (in dlp-agent-core) |
3. The limits
| Limit | Proved by |
|---|---|
| Oversized input is refused | extract::tests::oversized_input_is_refused |
| Output cap truncates and reports it | extract::tests::output_cap_truncates_and_reports_it |
| A decompression bomb is refused without reading it | archive::tests::a_decompression_bomb_is_refused_without_reading_it |
| An absurd declared ratio is refused before any work | archive::tests::an_absurd_declared_ratio_is_refused_before_any_work |
| A lying header does not get past the budget | archive::tests::a_lying_header_does_not_get_past_the_budget |
| Too many entries stops and says so | archive::tests::too_many_entries_stops_and_says_so |
| Nesting beyond the limit is reported, not ignored | archive::tests::nesting_beyond_the_limit_is_reported_not_ignored |
| The budget is shared across nesting levels | archive::tests::the_budget_is_shared_across_nesting_levels |
| A path-traversal entry name is reported | archive::tests::a_path_traversal_entry_name_is_reported, …::traversal_detection_covers_the_forms_that_appear |
| Deep RTF group nesting stops at the limit | deep_group_nesting_stops_at_the_limit_rather_than_running_away |
| The OOXML text budget is respected | ooxml::tests::the_text_budget_is_respected |
| Parts are read in a deterministic order | ooxml::tests::parts_are_read_in_a_deterministic_order |
Two of those are the ones that matter most. The budget is shared across nesting levels - a per-level budget is not a budget, it is a multiplier. And a bomb is refused without reading it, from the declared ratio, because an extractor that has to decompress a file to find out it should not have is already the denial of service.
tests/archive_bombs.sh runs the same ground end to end, and four fuzz targets - extract_any, extract_pdf, extract_rtf, archive_inspect - run against these parsers on every pull request and for hours on a Sunday.
4. Bounded by bytes, and now by time
PERFORMANCE_BUDGET.md said "Cancellation | Every extraction takes a deadline | Implemented" before one existed. ScanLimits::total_time_budget_ms and per_detector_time_budget_ms bound matching; nothing in extract.rs or archive.rs took an Instant, and no extraction could be cancelled.
What that left exposed was narrow and real: a parser with quadratic behaviour on a small pathological file is inside every byte limit and still slow, and the fuzz targets look for panics and hangs rather than for slowness.
ExtractLimits::time_budget_ms closes it (DLP-263). Ten seconds by default, which is a backstop against a pathological file rather than a throughput control - the slowest real extraction measured here is under a second, and a budget that fired on an ordinary large spreadsheet would trade coverage for a theoretical stall. Zero means no budget, so a configuration that forgot to set one behaves as it did before rather than refusing to read anything.
Cooperative, not a watchdog. The budget is checked at the top of the loops that can run long - container entries, archive nesting, RTF group descent, PDF stream iteration - so extraction yields rather than being stopped from outside. A thread that killed a parse would leave a half-built structure behind and could not say what had been read. The RTF tokeniser touches every byte, so it reads the clock every few thousand characters rather than every one.
One deadline for the whole extraction, started once and shared with every nested parse: an archive inside an archive inside an archive gets one budget, not three.
Running out of time is reported the way every other limit is - Unread::out_of_time, which makes the document not clean, and a reason that names the clock rather than blaming a malformed file. A file that stopped early because it was slow is precisely the case where "nothing was found" must not be reported as "there is nothing here".
5. Absent, and where it is recorded
| Format | State |
|---|---|
.doc, .xls, .ppt - the pre-2007 binary formats, and Outlook .msg | Read since DLP-1161 (cfb_text): a .msg field by field with its attachments, and an attachment it could not read (past the depth limit, a format this build does not read, or itself only partly read) named so the message is partially read, not clean (DLP-1384); the Office formats by text-run recovery, plus the numbers an .xls stores as binary cells (DLP-1308) and the cached numeric result of a formula cell (DLP-1591). Before an .xls workbook is read for text runs, its printer settings (DLP-1632) and the records that hold a cell's structure and no text are blanked: blank and numeric cells, shared-string indexes, row and cell offsets, formats and column widths (DLP-1660). Ten formatted blank cells had read "UUUUUUUUUU", and row offsets read as an IPv6 address 48 times in one GovDocs1 workbook. The numbers are read from the unblanked bytes, and formulas are kept because their tokens can hold a string. Across batch 006's 54 workbooks the text read fell by 12% and no other finding changed. A .ppt is read the same way from its text records only (DLP-1663): slide and notes text, comments, hyperlinks, headers and footers, and a shape's name, alt text and WordArt from its OfficeArt properties. Its font tables, geometry, pictures and embedded objects are blanked first, and across batch 006's 43 decks the text read fell from 2.44 MB to 0.93 MB. The words lost were font names, animation attribute names, a theme's XML namespace and noise. A stream that does not parse as PowerPoint 97 records, such as a PowerPoint 95 file, is read whole as before. A .doc is still read by text runs over its whole stream, by decision: reading it through its piece table instead was measured on batch 006's 73 Word 97 files and gave 5% less text, and what it dropped was mostly text deleted in an earlier fast save but still in the file ("contin", "togethe"), which a document that leaves the building still carries. CJK text is not recovered; a password-protected document is reported as encrypted |
Saved e-mail, .eml (MIME) | Read since DLP-1366 (mime_message): recognised by its header block, not its extension. Headers, every part, base64 and quoted-printable decoded, charsets honoured, a forwarded message/rfc822 followed, and each non-text attachment read by the same extractor one level deeper. A part in another transfer encoding (uuencode) is named and makes the file partially read. Until then a base64 attachment reached the detectors still encoded, and a card attached to a saved message read as clean. Encoded-word headers (=?utf-8?B?...?=, B and Q) are decoded before they are read since DLP-1385 |
| Images and scanned pages - OCR | Read, where the endpoint has a helper. Tesseract with aze, tur, eng and rus runs inside the sandbox of ADR-0012, is never linked into the agent, and both a standalone image and the images inside a document are routed to it (DLP-231). In a PDF, image-only pages are reported by page number even when other pages contain text (DLP-2204); the typed pages are still scanned. A document whose scanned page has been read stops being reported partly scanned. A finding read out of a photograph is flagged ocr_derived and its confidence reduced - noisy text is not the same evidence as typed text. An endpoint with no helper reports the image as requiring OCR, as before |
| Encrypted documents | Absent where opening needs a password, and nothing can read them. An encrypted ZIP entry, a password-protected Office document, a PDF that needs a password to open and an AES-encrypted 7z are recognised and reported as coverage encrypted (DLP-1130, DLP-1297); other forms are reported as unscannable, not skipped |
PDF LZWDecode and DCTDecode streams | LZWDecode is decoded like FlateDecode (DLP-1309). DCTDecode is a JPEG: counted as an image, so the document is reported partly scanned, and Windows runs no OCR |
| Archives | ZIP, tar, gzip, bzip2, xz, zstd, 7z and CAB are read (DLP-1177, DLP-1477, DLP-2230), under one set of limits. Every entry that is not itself an archive is read by the same extractor as on disk, so a file reads the same zipped or not: a document (PDF, legacy Office, .msg) since DLP-1386, and text (a saved .eml with its attachments, HTML with its character references, RTF) since DLP-1387. Before that a zipped PDF was skipped silently and zipped text was read raw. An entry in a format this endpoint does not read is named, so the archive is partial; an image entry is not read and not named; a 7z declaring a decoder dictionary above 64 MiB is refused unread. A CAB's stored, MSZIP and LZX folders are read; its Quantum folders are named as refusals, so the archive is partial (DLP-2230, DLP-2274). .rar is absent: RAR's decoder licence is not assessed. An xz stream that would need more than 96 MiB of decoder memory is refused unread |
None of these is a silent gap: each is reported per file, which is the property that makes declining them acceptable.
6. What is not covered
rtf.rshas no#[cfg(test)]module of its own; its seventeen tests live inpackages/detector-engine/tests/odf_and_rtf.rsas integration tests. That is a fine place for them and is noted only because a reader grepping the module for tests will find none.- No Windows run by this audit. These are portable parsers with no
cfg(windows)branch; thewindows-agentCI job runs the same tests there.