Data cleansing and deduplication

Most external data needs cleaning before use. This guide covers data cleansing and deduplication, turning messy inputs into reliable data.

Available across the EU. DataSupplier sources and delivers this data in all 27 European Union countries — including Germany, France, Spain, Italy, the Netherlands and Poland — and across the EEA, in the format and cadence you need.

Why cleansing is needed

External data arrives with errors, inconsistencies, missing values and duplicates. Cleansing makes it fit for use, and it is often the largest hidden cost in a data project.

Common problems

• Inconsistency: formats and values vary.
• Errors: typos and invalid values.
• Missing data: gaps and nulls.
• Duplicates: repeated records.

Cleansing techniques

Standardisation, validation against rules and reference data, correction, and handling of missing values turn raw data into consistent records. The aim is reliability without distorting the data.

Deduplication

Removing duplicates relies on matching (deterministic and probabilistic) to identify records that refer to the same thing, then merging them carefully to keep the best information.

Doing it without losing information

Aggressive cleansing can erase real signal, so good practice documents what was changed, keeps an audit trail, and is reversible where possible.

In a managed model

A managed partner can cleanse and deduplicate sourced data with documented, auditable transformations.

Cleansing without losing signal

Cleansing is often the largest hidden cost of external data, and the risk is over-cleaning: aggressive correction can erase real signal. Good practice standardises, validates against rules and reference data, corrects, and handles missing values, while documenting every change and keeping an audit trail so transformations are reversible and explainable.

Deduplication done carefully

Removing duplicates relies on matching to identify records for the same entity, then merging to keep the best information. A wrong merge (two distinct entities combined) is harder to detect than a missed duplicate, so conservative thresholds and documented survivorship rules matter.

Sources & further reading

• DAMA-DMBOK: data quality and cleansing.
• ISO/IEC 25012 and ISO 8000: data quality.
• Reference data for validation.
• Internal practice: DataSupplier preparation.