Data labelling and annotation

Supervised AI is only as good as its labels. This guide covers data labelling and annotation, how to get quality, and how to source labels responsibly.

Available across the EU. DataSupplier sources and delivers this data in all 27 European Union countries — including Germany, France, Spain, Italy, the Netherlands and Poland — and across the EEA, in the format and cadence you need.

Why labels make or break models

For supervised learning, labels are the ground truth the model learns from. Inconsistent or wrong labels cap model performance no matter how good the algorithm. Label quality is a first-order concern.

Getting label quality

• Clear guidelines: precise definitions and examples.
• Inter-annotator agreement: measure consistency.
• Review and adjudication: resolve disagreements.
• Edge-case handling: define the hard cases.

Sourcing labels

Labels can come from internal experts, specialist annotation providers, or be derived. Domain expertise matters for technical labels, and quality control is essential whoever does the work.

Privacy and ethics

Annotating personal or sensitive content brings the GDPR and ethical duties into scope, including the welfare of annotators handling difficult content. Aggregation and minimisation help.

Sourcing considerations

Provenance of both data and labels matters, and rights to use the underlying content for labelling must be confirmed.

In a managed model

A managed partner can source data and coordinate quality-controlled labelling with documented provenance.

Getting label quality right

Label quality is set before any labelling begins, by the guidelines. Precise definitions, worked examples and explicit edge-case rules are what produce consistent labels; vague instructions guarantee noise. Measure inter-annotator agreement to quantify consistency, adjudicate disagreements, and feed the resolutions back into the guidelines. For technical or regulated domains, domain expertise among annotators matters as much as process.

Privacy and annotator welfare

Annotating personal or sensitive content brings the GDPR and ethical duties into scope: minimise and, where possible, anonymise content before it reaches annotators, and consider the welfare of people reviewing difficult material. Provenance of both the data and the labels should be documented, and the right to use the underlying content for labelling confirmed.

Sources & further reading

• Industry references on annotation quality and inter-annotator agreement.
• EUR-Lex: Regulation (EU) 2024/1689 (AI Act) data governance.
• EUR-Lex: Regulation (EU) 2016/679 (GDPR).
• Ethics guidance on data annotation work.