Real-time data: when you actually need it

At some point in almost any conversation about data, the phrase appears: "and we want this in real time". It sounds like healthy ambition — who wants stale data? — but it hides a decision that is almost never made explicitly: how often your data needs to be updated. The comfortable answer is "the more often, the better". The correct answer depends on something very concrete: which decisions would change if the data were fresher. This article is about making that decision properly, because real time is paid for — in money, in fragility and in maintenance — and most of the time it buys nothing.

What people actually mean by "real time"

It helps to start by separating two things that get conflated. Real time, strictly, means the data reflects what is happening now, with a delay of seconds. What most people mean is something else: "when I open the report, I want it to be current". Those are very different requests with very different costs.

The second one is almost always solved by a daily refresh, or even a refresh before the hours when people actually look. If nobody checks the data between seven in the evening and eight in the morning, refreshing it every minute overnight informs no one: it just consumes.

The naked question is worth asking: when you say real time, do you need to react to what is happening this instant, or do you need to stop finding last week's figures in your reports? The first is real time. The second is reasonable freshness, and it is far, far cheaper.

What real time costs (and what the proposal won't show)

The cost of real time is not a line on an invoice: it is a structure. And it grows in three directions.

Technical complexity. Moving data once a night is a solved, robust problem. Moving it continuously demands a different kind of architecture: permanent connections between systems, failure handling in the moment, more parts that can break. Every step up in frequency adds machinery.

Fragility. A process that runs once a day fails, at most, once a day, and there is usually time to fix it before anyone notices. A continuous flow can fail at any hour, and when it does, your "real-time" figure becomes a frozen figure that looks alive — the worst case, because nobody distrusts it.

Maintenance. All of the above has to be watched and sustained for as long as it lives. The frequency you choose today is a maintenance commitment for years to come, not a box ticked once.

None of this is an argument against real time. It is an argument for making real time earn its place.

The right question: which decision changes?

Update frequency is not a property of the data: it is a property of the decision the data feeds. The practical rule is this: data needs to be as fresh as the decision that depends on it, and no fresher.

If the leadership team reviews sales every Monday, a figure updated Monday morning feeds that decision exactly as well as one updated to the second. If cash is reviewed daily, a daily refresh is enough. Nothing that happens between one review and the next changes any decision, because nobody is deciding in that interval.

When is high frequency justified? When someone — a person or an automated process — can react within that same window. Whoever coordinates logistics through the day, whoever handles an incident that degrades by the minute, a process that holds orders when something goes out of range: there, fresh data changes real actions. The test is always the same: if the data arrived an hour earlier, would anyone do anything an hour earlier? If the honest answer is no, that hour of freshness is worth nothing.

There is also a side effect that gets little airtime: continuously watching data that moves continuously invites reacting to noise. The figures of any given day wobble for reasons that mean nothing, and whoever checks them hourly ends up finding trends where there is only chance. For most management decisions, a number that has settled tells the truth better than a number that trembles.

A frequency per decision, not one for everything

A practical consequence follows: there is no such thing as "the" frequency for your company. There is an appropriate frequency for each kind of decision, and several will normally coexist. Day-to-day operational decisions may call for fresh data through the working day; sales follow-up, daily data; margin reviews or the close, monthly and stable data.

Framing it this way has another advantage: it turns a technical conversation ("can the system refresh every five minutes?") into a business conversation ("which decisions do we make, at what rhythm, and what data do they need?"). The first question gets answered by a vendor with a quote. The second can only be answered by the company, and it is the one that keeps you from paying for a feature nobody will use. When a data project starts from that second question, frequency stops being an inherited requirement and becomes a sized decision.

Five questions before asking for real-time data

1. Which concrete decision would change with fresher data, and who makes it?
2. Is there someone (or something) able to react within that window, or would the fresh data wait for the meeting anyway?
3. How often is that report genuinely consulted today?
4. What happens if the continuous flow breaks on a Tuesday at eleven: who finds out, and how?
5. Would it be enough for the data to be current every morning? If the answer is yes, you have your frequency.

Genuine real time exists and has its use cases. But it is a capability you buy for decisions that need it, not an adjective you attach to a project to make it sound modern. Update frequency is a business decision; treat it as one.