[JS] How to handle multiple concurrent messages

[Benjiiim]

In the Teams AI Library with JS, how to handle the scenario where a user sends multiple messages to the bot without waiting for the first response?

When trying with a new project scaffolded with the Teams Toolkit, I'm hitting the following error from the memoryStorage class, in Test Tool or in Teams:
"Storage: error writing \"msteams/00000000-0000-0000-0000-00000000000011/conversations/personal-chat-id\" due to eTag conflict."

May I add @corinagum to the discussion? 😇

[lifan0127]

Here is a workaround I used: #191

Credit: @nemesv

https://github.com/nemesv/teams-ai/blob/history_repro/js/samples/04.ai.a.teamsChefBot/src/index.ts#L95

[Benjiiim]

Thanks @lifan0127, that's an interesting way to workaround the issue.
I am surprised that it's up to the developer to handle this issue and that the framework doesn't manage it. This seems like a problem affecting all projects, regardless of the type of storage used, doesn't it?

[Benjiiim]

Adding to the thread the storage implementations the team provided in the questBot sample that was removed with #961 : LastWriterWinsStore

[corinagum]

@Benjiiim hey, I did see this discussion. I haven't had time to dig up the info but there is previous conversation related to eTag conflict already answered by stevnic somewhere in this repo. Feel free to search for it, otherwise I will link it when I have the few minutes to find it myself.

[corinagum]

By 'processed by the bot', do you mean you're waiting for LLM responses? In terms of UX, LLMs generally don't let you even send a new message until the response from the previous messages has been fully loaded. Providing an experience like that would be a teams client feature and outside the scope of our library. Last write wins is the general policy we've adopted to handle this problem without blocking a user from sending another message before the first has been responded to.

There are plenty of botbuilder samples showcasing how to use different types of storage. Please check those out. This stance is not new to our library; botbuilder does the same, and we've provided similar responses to customers in the past.

[Benjiiim]

By 'processed by the bot', do you mean you're waiting for LLM responses

Indeed.

Providing an experience like that would be a teams client feature and outside the scope of our library.

I can understand.

Last write wins is the general policy we've adopted to handle this problem without blocking a user from sending another message before the first has been responded to.

I don't yet understand everything this entails (still learning), but I'm now wondering if the MemoryStorage class (used in most or all Teams AI Library samples) is supposed to implement this policy? If it does, is it normal to receive the above error related to the eTag conflict?

[corinagum]

MemoryStorage was never intended to be used for persistent/production code memory. It's a way to quickly set up a bot with non-persistent memory so that a user doesn't have to deploy storage just to get the bot up and running. Otherwise, users would have to deploy some kind of storage every time they wanted to check out one of our samples.

To elaborate, in this case you are waiting for an LLM response which takes longer than most bot responses would generally take. If using an LLM's UI, the user wouldn't even be able to send another message to cause conflicts. In this case, eTag helps keep track of threads, but utilizing those eTags is dependent on storage and developer implementation choices. This is why most samples (exception: samples directly related to storage) generally bypass this topic via MemoryStorage because it would become very opinionated.

Receiving this error would be expected in this case because the scenario of multiple messages sent to a bot that only processes one at a time has not been handled. Setting up something like a queue would need persistent storage. A decision on how to handle a conflict like this is up to the developer.

[Benjiiim]

Thanks a lot for your detailed answer, that helps a lot and is greatly appreciated.

Let's take a step back, if I may.

From my experience, users are sending multiple messages without waiting for the bot to answer the first one.
That was the case with "last-gen bots" but this will be even more common in this new LLM world, for various reasons including the fact that more time might be needed for the bot to answer.
I'll give you a simple example below, where the LLM asks two pieces of information to the user who will answer with two messages most of the time (most users don't use multiline messages). The second message might arrive before the bot has answered the first one.

I understand you don't own the Teams UX but knowing that this behavior is a real issue in this new LLM world we're navigating in, I'm wondering if something can be done on your side to fix or mitigate this?
These are ideas from my (not-enough-educated) mind:

• Collaborate with the right team to find the best approach to tackle this, the same way you have done great work with AI-Generated label or feedback loop.
  • Don't process a turn if the user continues typing after sending a first message (that's how humans behave in my experience)
  • Abort a turn processing if a new message has been sent by the user and feed the LLM with both (or more) messages to provide a better answer in the next turn
  • Don't begin a new turn processing while a previous one is still being processed (lock mechanism)
  • Disable the compose UI or send button while the bot is typing
  • Any better/simpler idea?
• At least, find a way to reconcile the history in the state to eventually feed the LLM with every user inputs. That won't fix the issue but at least the LLM will have the full history during the next turn. The last write wins approach doesn't feel right for storing the conversation history in this LLM world.
• At the very least, provide better out of the box storage implementations in your library that respect the last write wins policy (eg: write the last value and log a warning instead of throwing an exception). That won't fix the user issue either as the LLM won't be fed with every user inputs but that will help the developers to learn and use the library without the need to write custom storage implementations.
• At the very very least ;-), document what a developer should do to mitigate the issue and provide a sample doing that (this discussion being one part of this solution).

That being said, thanks again for your time and your explanations. I now understand why the exception is raised and I will find a way to bypass it with a custom storage implementation, while still hoping that a solution can be found for the underlying problem. :-)

With the default MemoryStorage or BlobsStorage implementation (crash):

With a custom storage class implementing the last write wins approach by ignoring the eTag conflict (bot ignoring one message):

[corinagum]

The problem of managing conflicts like this far predates the Teams AI SDK, and unless we were to move away from the BotBuilder dependency entirely, this is the system that is set in place. The history of the decision behind how MemoryStorage is handled is long and complicated, but ultimately it can be summed up that Bot Framework and now Teams AI has always recommended developers to manage conflicts themselves.
There are so many considerations related to activity conflicts that directly affect how a bot operates that it would be impossible for us to make choices that would work for a majority or even notable percentage of developers. Once you scale to multiple distributed bots, that adds even more considerations for developers to weigh, such as cost, redundancy, and availability.

To respond in specifics:

Don't process a turn if the user continues typing after sending a first message (that's how humans behave in my experience)

Typing activities within the Bot Framework schema are only from the bot. Several channels (including Teams) have the concept of the typing activity from the user, but the bot either doesn't receive or know how to handle those activities. (Also note that typing indicators in Teams is handled very differently than the Bot Framework schema, meaning there is a mismatch already on typing activity behavior. As an example, today I've also discussed the typing indicator in another bug and the current theory is that the difference in schema and how Teams handles typing activities is causing a race condition)

Abort a turn processing if a new message has been sent by the user and feed the LLM with both (or more) messages to provide a better answer in the next turn
&
Don't begin a new turn processing while a previous one is still being processed (lock mechanism)

This is a design decision that historically we have chosen not to make for the developer, as outlined at the beginning of this reply. It's what we encourage users to implement themselves if that system works for them. Please note that the etag conflict is within the bot's scope but not the LLM's. Further, assuming a queue is set up, what if the user's second message said 'Wait, ignore my last message"? That would mean a processing delay for the first message, the second message, and then waiting for the bot to determine how/which activity to reply to. That's not any logic that we can predetermine for the customer.

Bringing LLM back into the equation - I think there's a misunderstanding on how LLM calls are handled. Parallel calls to the LLM are allowed, albeit they are rate limited. However, those two calls will be handled separately, regardless of how the bot handles the conflict. So if I said "Reply back yes", then in another message said "Nevermind, reply back no", the LLM will still respond to both API calls. There's no way to provide the LLM a better context with two messages instead of one - they are separate calls.

As I've said, this debate has gone on for years, and there is plenty of online discussion and open-source sharing of different solutions to the problem. I invite you to take a look at those to find whatever solution works best for you.