Batch vs near-real-time vs real-time: choosing cadence

Cadence, how often data arrives, is a design decision with real cost and value implications. Choosing it well means matching freshness to the decision rather than defaulting to "as often as possible". This guide gives a simple framework.

Available across the EU. DataSupplier sources and delivers this data in all 27 European Union countries — including Germany, France, Spain, Italy, the Netherlands and Poland — and across the EEA, in the format and cadence you need.

Why cadence is a design choice

More frequent data is not automatically better; it is more expensive to source, deliver and operate. The right cadence delivers data just fresh enough for the decision it supports, no more.

The cadence spectrum

• One-off: a single extract for a study or model build.
• Historical backfill: deep history to train or baseline.
• Batch: daily, weekly or monthly deliveries.
• Scheduled feeds: regular, automated deliveries.
• Near-real-time: seconds to minutes.
• Real-time: sub-second to seconds, for reactive systems.

Matching cadence to the decision

Strategic and analytical decisions usually need batch or scheduled data; operational and reactive systems need near-real-time or real-time. Map each use case to the slowest cadence that still supports a good decision.

The cost of freshness

Each step up the spectrum adds engineering, monitoring and licensing cost, and may narrow the set of available sources. Freshness should be justified by the value of faster decisions.

Mixing cadences

Most real programmes combine cadences: a historical backfill to start, scheduled batches for steady state, and a near-real-time feed for the few decisions that need it. A managed supply model can deliver several cadences from the same source.

Mapping cadence to decisions

The discipline is to attach a cadence to each decision, not to the data in the abstract. A board reviewing strategy monthly needs monthly data; a buyer hedging daily needs daily prices; a control system balancing a network needs sub-minute feeds. Listing the decisions a dataset supports, and the slowest acceptable cadence for each, usually reveals that one source should be delivered at two or three cadences for different consumers, rather than everything at the fastest.

The cost of freshness

Every step toward real-time adds engineering, monitoring, licensing and sometimes sourcing cost, and can narrow the set of available providers. Freshness should therefore be justified by the value of acting sooner. A useful test: if the data arrived an hour, a day or a week later, what decision would change, and what is that worth? Where the answer is “nothing”, a slower, cheaper cadence is the right choice.

Sources & further reading

• DAMA-DMBOK: data integration and latency concepts.
• European Commission: data spaces and interoperability guidance.
• OASIS: MQTT specification (for low-latency cadence).
• Internal practice: DataSupplier delivery models.