Control the speed when publishing messages

[stiangrindvoll]

In some cases a consumer has specific needs or limits were the consumption rate would need to be constant or at least more controllable. By adding limits to the publishing speed, we will be able to support the specific needs of the consumer.

[spier]

Why? :)

[TafThorne]

I could do with this too.

The service I am trying to test expects about 1 message a second. Adding a pacing option might help me more closely replicate what I record. As the messages I am dealing with contain timestamps there is some checking to ensure the time is passing at believable rate.

[TafThorne]

I will see if I can get together a working change for next week. Looks simple enough to add some kind of delay in the Publish function.

[spier]

@TafThorne how would you want the CLI options to look like?

Maybe similar to what the load testing tool vegeta does? i.e.

  -rate value
    	Number of requests per time unit (default 50/1s)

Also what did you mean by this?

ensure the time is passing at believable rate

[TafThorne]

Your suggested interface would be fine. 1 message per second is what I need right now but being able to configure that would be helpful. The time thing is more about my application. As the client gets time in the messages being sent it checks that the included time stamps are monotonic and roughly arrive at the rate they claim. E.g. it will reject a message for 01:02:01 if the previous message included a timestamp of 01:02:02 as it is considered stale. Similarly it would reject a message of 01:02:15 if the 01:02:02 message arrived only 1 second ago (the is some logic to wait till things are regular too but that is the basic concept) So long as I can put messages in in a given order and they come out in a given order, the pace should be enough for all the above to work.

…

On Sat, 11 May 2019, 14:07 Sebastian Spier, ***@***.***> wrote: @TafThorne <https://github.com/TafThorne> how would you want the CLI options to look like? Maybe similar to what the load testing tool vegeta <https://github.com/tsenart/vegeta> does? i.e. -rate value Number of requests per time unit (default 50/1s) Also what did you mean by this? ensure the time is passing at believable rate — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEPGNW4PW5TGZTQ5G4X3G3PU3AIHANCNFSM4FHN5AOA> .

[spier]

Sounds great! @stiangrindvoll and I actually work for the same company, so if you need any help I can walk over to him and make sure he sees this thread :)

@TafThorne would you be willing to take a first pass at this feature and the send a PR, even if its in a semi-finished state? Then we can help to bring it over the finish line.

[TafThorne]

Sure I can give coding something as go and submit a PR. Might get done this week.

…

On Mon, 13 May 2019, 15:14 Sebastian Spier, ***@***.***> wrote: Sounds great! @stiangrindvoll <https://github.com/stiangrindvoll> and I actually work for the same company, so if you need any help I can walk over to him and make sure he sees this thread :) @TafThorne <https://github.com/TafThorne> would you be willing to take a first pass at this feature and the send a PR, even if its in a semi-finished state? Then we can help to bring it over the finish line. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEPGNQ6GV42IGUVXKRCG63PVFZVTANCNFSM4FHN5AOA> .

[stevenklassen8376]

I have a need that is similar, but not identical to, this request. Specifically I need to simulate something close to a real-time system. That is, I need a playback that uses about the same timing as the recordings. Essentially I would like something where I specify the speed as a multiple of real time such as 1x, 2x, 3x, and so on as well as a "as fast as possible" which would essentially be what it does now.

[stevenklassen8376]

Looks like my request could not be handled in the general sense since even if the timestamp header is set, it does not seem to get recorded. I just ran a test where I made sure I set the timestamp, and I get the following when I consume the message in Pika:

<BasicProperties(['content_type=application/json', 'timestamp=1558966913'])>

but these values do not seem to get stored in the recording.

$ gunzip - < recordings/1_messages_1.tgz | pax
pax: Unrecognized header keyword: SCHILY.xattr.RABBITIO.amqp.routingkey
1b98c374-6930-47c9-b901-cb64faa907ba.rio

So, unless I'm missing something, there isn't enough information recorded in order to perform a realtime simulation.

[stiangrindvoll]

@stevenklassen8376 To try to understand your usecase:

• Store original amqp timestamp
• Use this timestamp to calculate the delay between messages
• Publish based on this delay (1x)
• Ability to speed it up 1x, 2x, 3x

This seem to assume that messages are in chronological order already.

This is a little more complicated than just adding an overall rate or delay, where the timestamp is not needed. And deserves a separate issue to this one i feel.

[stevenklassen8376]

Understood. I have actually gone with another tool so don't add a new issue on my account.

[stiangrindvoll]

What tool did you find to solve your use case @stevenklassen8376 ?

[stevenklassen8376]

https://github.com/jandelgado/rabtap

It doesn't solve it either, but is closer so I've added the feature request there.