Data Lineage and Data Quality — Purview: Discovery and Catalog

Most arguments about data begin at the wrong end. A dashboard number looks wrong, a customer record seems suspect, or an artificial intelligence model gives an answer nobody can explain. The useful questions are simpler and more disciplined: where did this data come from, what happened to it on the way here, and does it meet the standard required for this decision? In Microsoft Purview, data lineage helps answer the first two questions, while data quality helps answer the third.

The two trust questions

Data lineage and data quality are closely related, but they are not the same thing. Lineage is about movement and transformation. It records how data passes from a source system through jobs, tables, reports, and other assets. Quality is about fitness for use. It tests whether the data is complete, consistent, correctly shaped, accurate, fresh enough, and free of unwanted duplicates.

A number can have perfect lineage and still be wrong. You may know exactly which point-of-sale system produced a revenue figure and which extract, transform, load job moved it into a warehouse, but the source system may have captured the wrong tax code. The reverse is also possible: a data set may look clean today, but without lineage you may not know what upstream change could break it tomorrow.

Microsoft Purview brings these ideas into the governance experience by connecting technical metadata, catalog assets, lineage views, and quality measurements. That connection matters because trust is rarely established by one control. It comes from being able to explain provenance and measure reliability at the same time.

📌

Simple test. If a team cannot say where a data product came from or how well it passed its quality checks, it is not yet a trusted data product.

Lineage tracks how data moves; quality tests whether it is fit to use.

What lineage records

Data lineage is the recorded map of how data flows between systems and how it changes along the way. In a common reporting path, a sales transaction starts in a point-of-sale application, is copied and reshaped by an extract, transform, load (ETL) process, lands in a reporting warehouse, and is then displayed in a Microsoft Power BI report.

Lineage turns that path into something visible. Instead of treating the report as an isolated object, it lets an analyst or data steward move backward from the report to the table, from the table to the pipeline, and from the pipeline to the original source. Each hop narrows the search area when something is wrong.

This is why lineage is so useful for root-cause analysis. Without it, teams often fix the place where the problem was noticed, not the place where the problem began. With lineage, they can ask a better question: which upstream asset first introduced the bad value?

Lineage makes the path from source system to report visible.

What quality measures

Data quality is measured through rules. A rule is a check that tests data against an expected condition and contributes to a score. In Microsoft Purview, quality rules can be provided out of the box, written as custom rules, or, in newer and still-maturing capabilities, suggested by artificial intelligence based on patterns observed in the data.

The useful point is not that every table should score perfectly. Some data is exploratory, some is operationally messy, and some is good enough for one decision but not for another. The useful point is that quality becomes explicit. Instead of saying a customer table is probably unreliable, the team can say which rules failed, how badly they failed, and whether the failure blocks the intended use.

Purview commonly frames quality through dimensions such as completeness, consistency, conformity, accuracy, freshness, and uniqueness. These dimensions give different kinds of defects different names, which makes remediation easier.

Completeness asks whether required values are present.
Consistency asks whether the same fact agrees across places.
Conformity asks whether values match the expected pattern or type.
Accuracy asks whether the data reflects the real-world fact it claims to represent.
Freshness asks whether the data is recent enough for the decision.
Uniqueness asks whether the same record appears more than it should.

Purview frames quality through several measurable dimensions.

How lineage and quality work together

Lineage explains the route. Quality explains the condition of what traveled along that route. Together, they make data problems diagnosable. If a postal code rule fails in a customer table, lineage can show which upstream application, import process, or transformation supplied those postal codes.

This is the difference between detection and repair. A quality rule can detect that ten percent of customer records have postal codes that do not match the expected region format. Lineage can help the team discover whether the issue began in a web form, a batch load, a regional reference table, or a transformation that trimmed characters incorrectly.

The combination also helps with change management. If an upstream table changes, lineage can show which downstream reports, data products, and quality rules may be affected. If a quality score drops after a pipeline update, lineage gives the investigation a map.

✅

Better repair. A quality failure tells you what is wrong; lineage helps you fix it in the place where the error first appeared.

From rules to catalog health

Quality scores are most valuable when they are visible where people already look for data. In Microsoft Purview, data quality results can feed into the Unified Catalog health experience, so quality becomes part of the cataloged view of the data estate rather than a separate spreadsheet maintained by a governance team.

That shift changes the conversation. A data owner can see whether a critical table is improving or degrading. A data consumer can compare two candidate data products before choosing one for analysis. A governance lead can track whether quality work is reducing risk across important domains such as finance, customer, product, or supply chain.

This does not remove the need for judgment. A high score does not prove that data is suitable for every purpose, and a low score does not always mean the data is useless. It means the decision can be made with evidence. The catalog becomes not just a place to find data, but a place to understand its condition.

The decisions leaders and practitioners actually face

The practical value of lineage and quality appears in ordinary decisions. A chief financial officer deciding whether to trust a month-end margin report needs to know whether the figures came from approved systems and whether the relevant checks passed. A data engineer changing a pipeline needs to know which downstream reports and rules could be affected. A product manager launching a personalization feature needs to know whether customer attributes are fit for automated use.

The same evidence supports different choices. Sometimes the right decision is to fix the source system before releasing a dashboard. Sometimes it is to publish the data with a warning and a known limitation. Sometimes it is to narrow the use case, for example allowing regional trend analysis but blocking customer-level automation until address quality improves.

Good governance does not mean every defect stops every project. It means defects are visible, owned, and considered before the decision is made. Lineage and quality make that possible because they turn trust from a feeling into inspectable evidence.

Before approving a report, check whether its important fields trace back to accepted sources.
Before changing a pipeline, inspect lineage to identify downstream assets that may break.
Before publishing a data product, review its quality score and failed rules.
Before using data in automation, decide whether the quality level is acceptable for the harm a wrong answer could cause.

⚠️

Risk-based use. The stricter the business consequence, the more evidence you need from lineage and quality before using the data.

Why artificial intelligence raises the stakes

Artificial intelligence (AI) does not make poor data harmless. It can make poor data more consequential because it can consume more of it, act on it faster, and produce answers that sound polished even when the underlying evidence is weak. A human analyst may pause when a table looks suspicious. An AI-enabled feature may continue unless the surrounding system gives it signals to stop, warn, or limit use.

Consider a retail recommendation service that uses customer location, inventory, and shipping rules. If ten percent of customer postal codes are inconsistent, the defect is not merely a data hygiene issue. It can lead the service to recommend delivery options that are unavailable in the customer’s region, misprice fulfillment, or create support problems downstream.

A sensible habit is to review lineage and quality before an AI project consumes a data product. A weak quality score is not automatically a reason to cancel the project. It is a reason to repair the data, reduce the scope, add safeguards, or state the limitation clearly before the system reaches users.

⚠️

Confident mistakes. AI can turn bad data into bad decisions at a speed and scale that manual review cannot reliably catch.

Key takeaways

Lineage answers where data came from and what systems or transformations it passed through.
Data quality answers whether the data is fit for a particular use, not whether it is perfect in the abstract.
Quality rules measure dimensions such as completeness, consistency, conformity, accuracy, freshness, and uniqueness.
When quality scores appear in the Unified Catalog, governance teams and data consumers can make trust decisions with evidence.
Lineage and quality together support root-cause analysis: detect the defect, then trace where it entered the flow.
AI makes lineage and quality more important because automated systems can amplify bad data quickly and convincingly.

Review quality before an AI project consumes a data product.

Glossary

Data lineage: Data lineage is the recorded path showing how data moves and changes from one system or asset to another.
Data quality: Data quality is the degree to which data is fit for its intended use across measurable dimensions such as completeness and freshness.
Data quality rule: A data quality rule is a configured check that tests data against an expected condition and produces a result or score.
Extract, transform, load: Extract, transform, load is a process that copies data from a source, reshapes or cleans it, and loads it into a destination system.
Root-cause analysis: Root-cause analysis is the practice of tracing a problem back to the place where it actually began.
Unified Catalog: The Unified Catalog is the Microsoft Purview experience for discovering, describing, and understanding governed data assets.
Catalog health: Catalog health is a governance signal that summarizes the condition of cataloged data assets using measures such as quality scores.
Artificial intelligence: Artificial intelligence is software capability that performs tasks commonly associated with human reasoning, prediction, generation, or decision support.