How should event stream consumers handle out-of-order sequence numbers?

[snarfed]

https://atproto.com/specs/event-stream#sequence-numbers says:

Sequence numbers are always positive integers (non-zero), and increase monotonically, but otherwise have flexible semantics. They may contain arbitrary gaps. For example, they might be timestamps.

I assume "increase monotonically" means that, for a given event stream connection, servers should emit messages with sequence numbers in monotonically increasing order, ie they shouldn't send frames with seq 1, then 3, then 2.

If so, ok. I'm curious though, if that does happen, how should an event stream consumer handle it? Specifically, for subscribeRepos, if a BGS sees commits with out-of-order seqs like that, commit 3 probably includes MST nodes (if not prevs) that refer to blocks in missing commit 2, so it might not be able to fully construct and use the repo at commit 3.

If/when that happens, is the BGS expected to continue using the repo at commit 1 until it later receives commit 2, at which point it has all of the commits it needs, so it then advances its copy of the repo to commit 3 and continue on? Or is it ok for the BGS to break and be unable to recover that repo and have it stuck at commit 1 indefinitely?

[DavidBuchanan314]

I plan to handle this case the same as if 2 had been dropped entirely.

When 3 comes in, I won't necessarily know that a 2 ever existed or is missing, but I will notice that I'm missing MST and/or record blocks. When I notice that, I'll start trying to catch up "out-of-band" using getBlocks requests, recursively.

Depending on when 2 finally arrives, it won't be telling me any new information at all, so I can safely ignore it. (I won't even attempt to make use out of out-of-order events, I will simply drop then unconditionally)

The logic to do this ought to come "for free" as part of handling recovery after extended outages, beyond the firehose cursor history window.

[DavidBuchanan314]

If all else fails, or you just want to keep things simple, you could always grab a fresh copy of the repo via getRepo

[snarfed]

Ah, getBlocks is a good point! I still wonder about others, eg subscribeLabels, which doesn't have the same links or CRUD operations for filling in missing labels manually, but that's just academic, I'm not actually working with labels.

If all else fails, or you just want to keep things simple, you could always grab a fresh copy of the repo via getRepo

True! I'm thinking as a server, not as a client. Basically, what's the impact of out-order sequence numbers in subscribeRepos, which helps determine how hard I have to work to guarantee in-order. 😁

[dholms]

If you're emitting events, you should work very hard to guarantee in-order and consistent sequence numbers for events.

Our services will just ignore any out of order event.

However, this is a good sign that something went wrong with the providing service.

Services may:

• investigate the bad event and determine if they missed some blocks, perhaps falling back to a getRepo or getBlocks
• restart the stream from the last in-order cursor (and bail if it happens again)

Persistent failures suggest a faulty service, and I would expect consumers to stop subscribing to them. Similar to publishing poorly signed commits

If the providing service is out of order because they reset their cursor (ie cursor 1,000,000 -> 1,000,001 -> 1 -> 2 ...), then:

• the consumer, on reconnect, would provide their last valid cursor (1,000,001)
• the provider should send a FutureCursor error frame - this tells the consumer that their saved cursor is no longer valid
• the consumer should start at some earlier cursor (for instance 0) or with no cursor parameter

[dholms]

Ah re-read your message - a dropped event is less severe than an out of order event.

There's no expectation that you remain on commit 1 after receiving commit 3 until you get commit 2.

I would encourage services to do as they see fit.

A feed generator (which makes no claim to a complete view of the network) may not care & just not bother indexing the operations in commit 2.

An appview or BGS (which does try to maintain a complete view of a well defined slice of the network) will attempt to fill in the missing blocks using either getBlocks or getRepo with the since param

(our BGS attempts to identify gaps & do the above^^)

[snarfed]

Thanks! This all makes sense. Definitely nice that at least for subscribeRepos, BGSes and other consumers can proactively fill in gaps with getBlocks etc so they don't get stuck.

[dholms]

Yup totally 👍

One goal of BGS is to smooth over errata & noise from various PDS implementations/faults. Altho of course, it only does that up to a point. And a PDS that is making indexing more difficult than it needs to be may stop being crawled