Real-time processing
Real-time processing handles arriving events against a defined latency target. Event time, processing time and late data need explicit rules.
Real-time processing handles arriving events continuously against a latency target defined for the application. Instead of waiting for a complete batch, the application reacts to a data stream. The term “real time” alone specifies neither a fixed millisecond limit nor a guaranteed maximum response time.
In a fictional example, a dashboard counts completed export jobs per minute. The completion event for J-31 occurs at 10:04 but reaches processing only at 10:07. Both times use the same time zone. An event-time calculation assigns it to minute 10:04; a processing-time calculation assigns it to minute 10:07. They answer different questions.
Balance a quick display with complete results
If the display counts by event time, it needs a rule for late arrivals: is a previously displayed minute total corrected, or is the late event recorded separately? Waiting longer can include more delayed events but postpones the result. A watermark marks the assumed progress of event time. Events with earlier timestamps can still arrive afterwards.
Define where latency measurement starts and ends, how old the oldest waiting record is and how backlog becomes visible. Timestamps and clock synchronisation also belong in a design that depends on reliable time assignment. A regularly refreshed dashboard does not by itself prove that its underlying events have already been processed completely.
Sources
Related terms
- Batch processingBatch processing handles a defined set of data or jobs together in one run.
- BackpressureBackpressure adjusts incoming data to the receiver’s processing capacity through signals to slow down or pause.
- WebhookA webhook sends an HTTP notification to another system when a specified event occurs.
- Server-sent eventsServer-sent events carry text notifications from server to client over HTTP, for example about the progress of a background job.
- Entity tagAn ETag identifies a particular representation version of an HTTP resource and supports conditional reads or changes.
- Cross-Origin Resource SharingCORS controls browser access to responses across origin boundaries. It replaces neither API login nor permission checks.
Your project with ASES
Describe your project to us. We will get back to you with an initial assessment.