Classification Engines — Purview: Discovery and Catalog

A scan can tell you that a file, table, column, or message exists. It can collect names, locations, owners, sizes, schemas, and timestamps. That is useful, but it is not yet understanding. Classification is the step that says this column probably contains credit card numbers, this folder appears to hold résumés, or this document looks like a loan agreement. In Microsoft Purview, that recognition work is done by different classification engines, and the expert move is not to use the most sophisticated one every time. It is to choose the simplest engine that matches the shape of the problem.

Classification starts where scanning stops

Scanning is discovery. It crawls a data source and records technical metadata such as names, paths, schemas, columns, file types, and sometimes sampled content. Classification is interpretation. It looks at the scanned content and metadata and applies a meaningful category, such as credit card number, passport number, confidential project code, legal contract, or résumé.

That difference matters because governance decisions depend on classification quality. A data catalog entry is more useful when it says that a column contains national identification numbers rather than merely saying it is a string field named ID. A data loss prevention policy is safer when it can distinguish a real payment card number from a random sixteen-digit value. A records or legal process is faster when contract-like documents can be found even if their file names are unhelpful.

Scanning discovers technical metadata; classification interprets its meaning.

Built-in sensitive information types

The first engine to understand is the built-in sensitive information type, often shortened to SIT. A sensitive information type is a detector for a known kind of sensitive data. Microsoft Purview includes more than two hundred built-in sensitive information types for common categories such as credit card numbers, national identity numbers, tax identifiers, passport numbers, bank account numbers, and other patterned identifiers.

These detectors are not usually simple text searches. A strong sensitive information type combines several signals. Pattern matching checks whether the value has the right shape, often with a regular expression, which is a compact rule for matching text patterns. Checksum validation checks whether the number is mathematically plausible for that identifier type. Keyword proximity looks for nearby words such as card, account, passport, or expiry to confirm that the surrounding context fits the match.

This combination reduces false positives. A sixteen-digit number in isolation may be a product code, a test value, or a payment card number. A sixteen-digit number that passes a card checksum and appears near the words credit card is much more likely to be meaningful.

✅

Use these first. If Purview already has a built-in sensitive information type for the data you need to find, start there before building anything custom.

A strong sensitive information type combines several signals, not one text search.

Custom pattern-based classifiers

Built-in sensitive information types cover common identifiers, but every large organisation also has private formats that no vendor can know in advance. These might include internal customer reference numbers, proprietary project codes, case-file identifiers, trading desk codes, employee investigation numbers, or manufacturing batch formats.

For these cases, an organisation can create its own pattern-based classifier, often implemented as a custom sensitive information type. The technique is similar to a built-in sensitive information type: define the pattern, add supporting keywords, and where possible add validation logic that rejects values that merely look similar.

The important test is whether the data has a repeatable shape. If the internal account reference always begins with three letters, then a dash, then eight digits, then a check character, a custom pattern can work well. If the thing you are trying to detect is a general kind of document, such as a negotiation memo or employment contract, a pattern-based approach will be brittle.

Good fit: an internal account code with a fixed prefix, length, and check digit.
Good fit: a case number that always follows a department-year-sequence format.
Poor fit: a legal contract that can vary in wording, layout, length, and file name.
Poor fit: a résumé, because the category is defined by overall content rather than one fixed token.

Trainable classifiers

The third engine is the trainable classifier. Instead of looking for a fixed text pattern, it learns from examples. An administrator provides documents that genuinely belong to a category, such as confirmed legal contracts, and separate examples that do not belong to that category. The classifier then learns statistical signals that help distinguish one group from the other.

This is the right tool for unstructured content, which means content that does not follow one rigid format. A credit card number has a shape. A contract does not. Contracts can be short or long, scanned or digital, written by different firms, and organised under different headings. Still, after seeing enough examples, a model can learn that certain phrases, structures, clauses, and combinations of language make a document more contract-like than not.

Trainable classifiers are powerful because they can generalise to new documents they have never seen. They are demanding because they depend on the quality of the training set. A small, biased, stale, or inconsistently labelled set of examples can produce a classifier that appears confident while being wrong in production.

⚠️

Do not skip validation. Always test a trainable classifier against documents kept out of the training set before using its results in production policy.

Choosing the right engine is a design decision

The practical question is not which engine is most advanced. It is which engine fits the evidence available in the data. If the sensitive value is a common, structured identifier, a built-in sensitive information type is usually fastest, most explainable, and easiest to maintain. If the value is structured but unique to the organisation, use a custom pattern-based classifier. If the category is defined by the overall character of a document rather than a fixed string, consider a trainable classifier.

This choice affects cost, delivery time, explainability, and risk. A chief data officer may care that the classification program produces consistent results across business units. A security architect may care that a data loss prevention rule does not block ordinary business activity because of false positives. A compliance lead may care that the organisation can explain why a document was treated as sensitive. A platform owner may care that the classifier can be maintained when formats, templates, and business processes change.

A useful rule is to move from simple to complex. Do not train a classifier to find passport numbers. Do not build a custom pattern for a national identifier that Purview already supports. Do not force a regular expression to identify a contract when the real signal is the meaning and structure of the whole document.

Each classification engine fits a different kind of evidence in the data.

Worked example: Halverton Trust Bank

Halverton Trust Bank, a fictional retail and commercial bank, wants better visibility into sensitive information across file shares, databases, and collaboration sites. Its first requirement is to identify payment card numbers and passport numbers. The data governance team uses built-in sensitive information types because these identifiers are common, structured, and already supported.

The second requirement is more local. Halverton uses an internal account-reference code that no built-in detector recognises. The code has a precise format, and business users often write it near phrases such as acct ref or internal account reference. The team creates a custom pattern-based classifier that checks the code shape and requires relevant nearby keywords, reducing the chance that an unrelated value is flagged.

The third requirement is different again. The legal department wants to find loan agreement documents scattered across old shared drives, regardless of file name or exact wording. There is no reliable regular expression for a loan agreement. The team builds a trainable classifier using a curated set of confirmed loan agreements and confirmed non-agreements, then validates it on a separate set before using the results to support review and cleanup.

Quality, maintenance, and trust

Classification is never a one-time exercise. Built-in sensitive information types should be reviewed against current product documentation because vendor capabilities change. Custom classifiers should be revisited when internal formats change, when business units start using new abbreviations, or when false positives appear in the field. Trainable classifiers should be monitored because document templates, language, and business practices drift over time.

Two measures matter in everyday classification work. Precision asks, of the items flagged, how many were actually correct. Recall asks, of the items that should have been found, how many were found. Governance teams often need to balance the two. A strict detector may have high precision but miss edge cases. A loose detector may find more true items but also create more noise.

Trust improves when classification results are sampled, reviewed, and fed back into design. For important policies, especially blocking, quarantine, retention, legal hold, or automated access decisions, treat classification as evidence rather than magic. Know what the engine is good at, know where it fails, and keep humans in the loop where the consequence of a mistake is high.

Review samples from both matched and unmatched content.
Track false positives and false negatives separately.
Retest classifiers when source systems, templates, or business processes change.
Use stricter thresholds for actions that block work or change access.
Document why each engine was chosen so future teams can maintain it.

📌

Classification is evidence. A classifier result should inform a governance decision, but high-impact automation should be tested and monitored carefully.

Key takeaways

Scanning discovers data; classification interprets what the data probably contains.
Built-in sensitive information types are the default choice for common structured identifiers and Purview includes more than two hundred of them.
Custom pattern-based classifiers are best for organisation-specific formats with a repeatable shape.
Trainable classifiers are best for unstructured document categories that cannot be captured reliably with a fixed pattern.
The right engine is the simplest one that fits the evidence: common pattern, private pattern, or learned document category.
Trainable classifiers need curated positive and negative examples, plus validation on documents not used for training.

Detectors trade precision against recall along a strict-to-loose spectrum.

Glossary

Classification: The process of assigning a meaningful category to data, such as credit card number, résumé, or legal contract.
Sensitive information type: A detector for a known category of sensitive data, often shortened to SIT, commonly built from patterns, validation, and contextual evidence.
Regular expression: A compact text rule used to match values with a particular shape, such as a prefix followed by a fixed number of digits.
Checksum validation: A mathematical check that helps confirm whether a matched number is plausible for its identifier type.
Keyword proximity: A technique that requires relevant words to appear near a possible match to reduce false positives.
Custom classifier: An organisation-authored detector used for data patterns that are specific to that organisation.
Trainable classifier: A classifier that learns from positive and negative example documents rather than relying on one fixed text pattern.
Precision: The share of flagged items that are truly correct matches.
Recall: The share of real target items that the classifier successfully finds.