Skip Navigation

Topics & Messages

Streaming Messages Live

Watch new messages arrive as they are written, wait for a matching message with a text filter, get a sound for each one, and keep long streams light.

A live stream shows a topic's new messages as they're written, until you stop it. Use it to watch an application produce, to check that a message you just sent arrived, or to wait for one particular message, such as an error or a given order ID, without searching again and again. A stream is one of the search types of the Messages tab, so it uses the same form as Searching Messages.

Starting a live stream

  1. Open the topic's Messages tab

    Select the topic in the topics list and open its Messages tab.

  2. Choose Stream Live

    Set Search type to Stream Live, and adjust the rest of the form if you need to (below).

  3. Click Search

    Click Search (or press Enter). The table empties, the form is greyed out and a Stop Stream button starts pulsing at the bottom right of the table. New messages appear as they arrive.

The Messages tab of the orders topic on the Storefront Dev cluster while a live stream runs. The search form is greyed out with Search type Stream Live. The table lists new JSON order messages from partitions 0, 1 and 2, newest at the top, one row selected. A badge at the bottom left reads Showing: 32, Received: 32, Received size: 3.7 KB, and a red Stop Stream button sits at the bottom right. 1 2 3 4
A stream on orders after about 25 seconds: 32 new messages, newest at the top.
  1. Stream Live, the last of the search types.
  2. Search and the rest of the form are greyed out while the stream runs: to change a setting, stop the stream first.
  3. Messages shown, and received since the stream started (search statistics).
  4. Stop Stream, shown only while a stream runs (Stopping a stream).

What a stream reads

A stream starts at the current end of each partition, so it shows only messages written after you click Search. The topic above already held 245 messages; none of them are in the table. To see what's already there, run a Newest search first. KafkIO reads without a consumer group, so a stream commits no offsets and never shows up on the Consumers page.

The rest of the form applies as it does to a search:

  • Partition(s): only the ticked partitions are followed (Choosing partitions).
  • Key deserializer and Value deserializer decode each message (Deserialization & Formats). The Schema-Registry warning triangle described for a search isn't reliable here: a stream started after a search of that topic doesn't check for it.
  • Text filter: only matching messages are shown (below).
  • Max page/poll size and Key/message trim limit the memory a long stream uses (Busy topics and long streams).
  • Collapse after search folds the form away once the first message arrives.

Dump Topic To File (under the arrow beside Search) isn't available for Stream Live: a stream has no end to write up to.

Reading the table while it streams

The newest message is always at the top, and clicking a column header doesn't sort while the stream runs. You can select a message and read it in the detail pane below the table as usual (Viewing Messages): the selection stays on that message as new ones arrive above it.

Waiting for a specific message

Combine Stream Live with a Text filter and the table stays empty until a matching message is written. That's the way to wait for something you're expecting: an order ID, a customer, an error code in a header, a "status": "CANCELLED". The filter works exactly as for a search (case-sensitive plain text or a regular expression, in the key, value and/or headers): see Filtering by text.

A live stream on orders with Text filter CANCELLED and only Value ticked. The table holds six messages, all with status CANCELLED, newest at the top. The badge reads Showing: 6, Received: 70, Received size: 8.1 KB. The selected message is shown in the detail pane, where the word CANCELLED is highlighted and the search box reads 1 of 1 found. 1 2 3
Streaming only the cancelled orders: 6 matches out of 70 new messages.
  1. The filter text, CANCELLED.
  2. Only Value is ticked, so keys and headers aren't searched.
  3. Showing counts the matches, Received every message read: the badge tells you the stream is alive even while nothing matches.

The filter text is also put in the detail pane's search box, so the match is highlighted when you select a message. To be told when a match arrives without watching the screen, turn on notification sounds.

Sound notifications

The arrow on Stop Stream opens a menu with one item, Notification sounds. Tick it and KafkIO plays a short beep for every message added to the table, which, with a text filter, means every match.

The Stop Stream button's menu open below it, with a ticked Notification sounds item. 1 2
Notification sounds, turned on.
  1. The arrow on Stop Stream opens the menu; the button itself stops the stream.
  2. Notification sounds: on or off for this Messages tab.
  • The menu is only there while a stream runs, so start the stream first. Sounds are off for every new Messages tab; once turned on, they stay on for later streams in the same tab.
  • On a busy topic the beeps run together and some are skipped: sounds are meant for a filtered stream where matches are rare.
  • KafkIO uses your computer's default audio output; if no sound device is available, nothing plays.

Push queries in the ksqlDB editor have the same option (Push queries).

Stopping a stream

Click Stop Stream. The messages stay in the table, the badge shows the final counts, the form can be edited again, and column sorting works once more. Starting another stream, or a search, clears the table and starts afresh.

A stream keeps running when you look elsewhere: select another topic, another section or another cluster tab and it carries on in the background; come back to the topic's Messages tab to see what arrived and to stop it. This is handy with a duplicated tab: leave a stream running on one topic in its own tab while you search or produce in another.

Warning: nothing outside the topic's Messages tab shows that a stream is running, and closing a tab only stops the stream of the topic selected in it at the time. Click Stop Stream when you're done with a stream, rather than leaving the topic, so it doesn't keep reading in the background until you quit KafkIO.

If Kafka returns an error while streaming (for example, you lose permission to read the topic), the stream stops and KafkIO shows Error streaming messages with the reason. The messages already received stay in the table.

Busy topics and long streams

A stream can run for hours, so KafkIO keeps it from growing without limit:

  • The table keeps about the page size. For a stream, Max page/poll size is the most rows the table holds (5,000 by default). Past that, the oldest rows are dropped in batches of a tenth of the page size, so Showing stays around the page size while Received keeps counting. Pick a larger page size to keep more history, a smaller one to save memory.
  • Large messages are trimmed. Key/message trim (500 KB by default) caps how much of each key and value is kept, as for a search (Trimming large messages). On a topic with big messages, or a modest machine, a smaller trim keeps a long stream light.
  • About 60 messages a second are shown. KafkIO paces the stream so the table stays readable. If a topic receives more than that, the stream falls behind and the newest row is older than it looks (check the Time column). No messages are skipped. To follow a busy topic, narrow the stream with a text filter or fewer partitions: only matches count towards the limit.

KafkIO also does less work for a stream that isn't on screen, so leaving one running in another tab costs little.