Data integration patterns: ETL, ELT and CDC

Sourcing data is only half the job; integrating it into your systems is the other half. This guide explains the main integration patterns and how to apply them to external data feeds.

Available across the EU. DataSupplier sources and delivers this data in all 27 European Union countries — including Germany, France, Spain, Italy, the Netherlands and Poland — and across the EEA, in the format and cadence you need.

Why integration is where value is realised

A dataset delivers nothing until it lands, cleanly and reliably, in the systems that use it. Integration is where external data becomes usable, and where many projects stumble. The pattern you choose shapes cost, freshness and resilience.

ETL vs ELT

ETL (extract, transform, load) transforms data before loading it into the target, useful when the target is rigid or transformations are heavy. ELT (extract, load, transform) loads raw data first and transforms inside a modern warehouse, favoured for flexibility and scale. Most modern stacks lean ELT, but external data often needs transformation on the way in regardless.

Change data capture

Change data capture (CDC) delivers only what changed since the last load, rather than re-sending everything. For large or frequently updated external datasets, CDC cuts cost and latency, though it adds complexity around ordering and deletes.

Batch vs streaming integration

Batch integration suits scheduled feeds; streaming integration suits continuous, event-driven data. The choice should follow the cadence the use case needs, not the other way round.

Integrating external data specifically

External feeds bring extra challenges: schema differences, identifier mismatches and upstream changes outside your control. Robust integration includes schema validation, mapping to your model, and handling for source changes.

In a managed model

A managed partner can deliver external data already mapped to your schema and integration pattern, absorbing upstream variability so your pipeline sees a stable, documented feed.

ETL vs ELT, decided by context

The choice is not fashion but fit. ELT, loading raw data into a modern warehouse and transforming there, suits flexible, large-scale analytics where you want to keep the raw record and iterate on transformations. ETL, transforming before load, still makes sense when the target is rigid, when heavy cleansing must happen before storage, or when only conformed data may land for governance reasons. External data often needs at least light transformation on the way in regardless, to map it to your schema and validate it.

Change data capture for external feeds

For large or frequently updated sources, re-sending everything each cycle is wasteful. Change data capture delivers only inserts, updates and deletes since the last load, cutting cost and latency, at the price of handling ordering, deletes and occasional full reconciliations. For external feeds whose internals you do not control, a periodic full refresh alongside CDC is a pragmatic safety net against missed changes.

Sources & further reading

• DAMA-DMBOK: data integration and interoperability.
• Industry references on ETL/ELT and CDC patterns.
• European Commission: data spaces interoperability.
• Internal practice: DataSupplier integration support.