Message statistics in Node-RED debugger

[bartbutenaers]

Hey guys,

Here a design proposal after a short discussion with Nick this morning. In the past I have developed two separate nodes to collect statistics about messages:

1. The node-red-contrib-msg-speed measures at which rate messages pass through it. For example to measure the frames per second of IP camera's.
2. The node-red-contrib-msg-size measures the size of the messages that pass through it. For example to measure the byte size of my IP camera images.

At the time being there was no Node-RED debugger, so it was rather obvious to develop custom nodes to measure such message-based statistics. However now it would make more sense to integrate such functionality in the core. And then I can mark my both nodes as obsolete...

Description of my nodes

I will try to explain below a little background information about both nodes, and explain my design decisions. Because I got a lot of feedback from real life use cases from users, which have been implemented over time. Then you can determine whether the basic concept would be reusable in the debugger somehow.

1. Moving window instead of single counter

To measure the speed (= rate) of messages, I started with a basic counter: count how much messages arrive during the last second, and then send an output every second with the count in the payload. That works fine when you have N messages per second, like e.g. IP cam images.

However in some use cases the message traffic might stop for several seconds, and then start again. In that case you might want to know the speed per minute or per hour. Or some users asked a more custom range, like the message rate per 7 minutes. However in such cases a single counter does not work well anymore. Suppose we need to measure the number of messages per minute. You could count the number of messages during the last minute, and output the counter after that minute. Which means you know the speed only one minute later, which is not sufficient in a lot of use cases. It is like driving with your car on the highway, and you know only one hour later how much km/hour you have been driving during the last hour. Not optimal ;-)

You can solve that by measuring the speed per hour in a moving window, and output the result (= statistic over the 1 hour window) every second. Which means you get every second the speed of the last hour.

2. Need for two separate frequencies

So I ended up with two separate frequencies:

1. The measurement frequency: e.g. calculate the messages per hour, per minute, per second, ...
2. The output frequency: e.g. an output message (containing the statistic result) every second, minute, ...

These frequencies determine the lenght of the moving window, i.e. they determine how much counters need to be stored in memory.

Suppose you want to measure the speed per minute, and you want to get the speed result every second. This means you need to count the messages per second, and you need 60 counters. As a result the moving window has a length of 60 counters. After every second, you go to the next counter and you output the sum of the 60 previous counters (because those contain the total number of messages that have arrived during the last minute):

3. Circular buffer

I needed an easy way to have a moving window of N counters. That became very easily by using the circular-buffer NPM package, which allows you to create a ring of N cells. Every time a value is written in a cell, the pointer moves to the next cell and so on. As soon as you have moved around the entire ring, it will simply continue at the first element where you had originally started. Which means you simply overwrite the previous values in the ring:

So no need to re-allocate memory, or cleanup memory or whatever...

And the advantage of this particular npm package is, that you can not only store numbers but also Javascript objects. Which means you can e.g. store an object containing multiple statistics about that time interval (max msg size, average msg size, speed, ...).

4. Speed up calculations

So it works like this:

1. For every msg that arrives, a counter is incremented.
2. After one second is complete, the counter in stored in the current ring cell.
3. The pointer then moves automatically to the next ring cell.
4. Send a message containing the sum of the statistics in the previous 60 ring cells.

However step 4 will involve a lot of calculations, if you would need to start looping all cells to calculate the sum. Especially when you measure e.g. the speed per hour, because then you have to loop 3600 cells.

However you can do it very fast like this:

1. Initialize at the start the statistics in all ring cells to 0.
2. Keep somewhere a global counter X, which contains the sum of the speed statistic in all the ring cells. So initially that global counter will contain zero.
3. Keep another counter Y where you count all messages that have arrived during the last second.
4. Before you store that counter Y in the ring, read the current original counter Z from the ring cell that will be overwritten.
5. Global counter X = global counter X - counter Z (that will be removed from the ring )
6. Global counter X = global counter X + counter Y (that will be added to the ring)
7. Insert the counter Y into the ring, which means that counter Z will be overwritten
8. Send global counter X in the output message

That way global counter X will always contain the sum of all the cell counters, by doing 1 addition and 1 subtraction every second.

5. Extra statistic (size)

Last year I wanted to measure the msg size statistics. Because the name "node-red-contrib-msg-speed" was not really suitable to add size measurements, I decided at the end to create a separate node "node-red-contrib-msg-size". However it would have been better if I had a "node-red-contrib-msg-statistics" node, which calculated both the speed and the size. But I could not rename my old node anymore...

Inmho I would use a single circular buffer in the debugger, that contains in each cell an object. That object can contain multiple statistics (speed, size, ...) if the user has activated them.

P.S. Since my both nodes needed a similar circular buffer behind the scenes, I have created a separate nr-msg-statistics package that both nodes use.

5. Topic-dependent

There were a lot of feature requests to have topic-dependent statistics. Because I had created the nr-msg-statistics package, it was very easy to implement this. Indeed I keep a map with:

• key is the topic
• value is the circular buffer instance (from the nr-msg-statistics package)

6. Startup period

Suppose your messages arrive at a rather stable rate, and you want the rate per minute. However in the beginning the global counter will only contain the sum of the first seconds. Which means you will get a low rate during the first seconds.
Only after 1 minute the rate will become stable. Which means you have a startup period of 1 minute, which might be annoying in some use cases.
To solve that, you can activate an option to 'estimate' the rate during the startup period. This feature simply works like this:
estimated rate = current rate sum * total number of cells / used number of cells

Integration proposal

1. (de)activate statistics

Based on this Discourse discussion, I thought it might be perhaps useful if - e.g. on the config screen of the output in the debugger - a user could specify which statistics he wants to activate:

• Activate msg size statistic
• Activate msg rate statistic
• ...

And then the runtime keeps a circular buffer for that input/output, in which it stores all the selected statistics. Don't know if it is possible to (de)activate this without a Deploy?

2. Customizable

Not sure whether it would be useful to allow developers somehow to create custom msg statistics? For my IP camera's the rate and size are important. But perhaps for other use cases people might want to develop other msg statistics? Can't think of an example, so perhaps it is not worth the work...

3. Show the results in the debugger

I assume it can be useful to show the statistics in an editableList in the sidebar, and near the input/outputs in the flow editor.

4. A statistic node

I like my nodes like they work at this moment. Their output contains the statistic result, which I can easily display in a chart. Or I can inject them into my flow, for example to trigger an alarm when the rate exceeds some limits.

Therefore it would be nice if there would be a Statistic-node, which does something similar (based on the statistics recorded by the debugger). Would be nice if it could support two modes:

• Similar to the Catch node, which means you can select for which inputs/outputs it needs to send output messages
• Similar to the current size/speed contrib nodes, which means you can use them inline.

Hopefully this makes sense...
Don't hesitate to ask for further explanation!

Bart