Is there a plan to increase the rate limits, now that the traffic increased by an order of magnitude?

[imax9000]

My moderation thingy has some simple rules that it checks for each DID is sees on firehose, caching the outcome for 24h.

It needs to call getActorProfile, which is rate-limited to 10 requests per second. Which means that it can keep track of at most 864k daily active users.

Is there a plan to increase the rate limits to keep up with the traffic? What about when it increases by another order of magnitude?

Otherwise my only option is to increase caching duration, meaning that required moderation actions will be delayed further.

[imax9000]

(Side note: when the traffic grows by another order of magnitude it will be ~300Mbit/s per firehose consumer. Federation when?)

[DavidBuchanan314]

Which definition of federation solves this?

[imax9000]

One where, instead of receiving all or none of the global traffic, it is possible to only receive inbound traffic of your federated instance.

That is, only checking the records that your users see, regardless of how many are flying around between other instances.

[bnewbold]

It is quite difficult to compute and serve that kind of federation. It isn't something we are working on or on our protocol roadmap. If somebody figures out an efficient and reliable way to do it at scale, that would be great!

And to be clear it is keeping track of which PDS instances might possibly care about which commits or repos, and providing those routing feeds in an efficient way. Letting the subscriber declare which repos they want is much easier, but is still a lot of work (IPC) for the Relay, and also has pragmatic issues if the subscribers wants to declare more than a couple thousand repos (DIDs) when re-subscribing.

The entire relay architecture is currently designed around subscriptions being "stateless".

[imax9000]

Don't worry, I know that federation is not anywhere on your roadmap and it was just an effective marketing buzzword; I'm just pulling your leg.

[snarfed]

If you're already consuming the full firehose, you can just pull the actor profile records out of that as well as DIDs, right?

If that's prohibitive, https://github.com/ericvolp12/jetstream might make it easier/cheaper.

[imax9000]

https://github.com/ericvolp12/jetstream won't do anything useful here. Profile info is seen on firehose only when it changes, and yes, keeping it in a local storage and updating from firehose is an option I'm considering.

But that doesn't change the fact that rate limits are no longer adequate for the traffic volume and network size.

[bnewbold]

We probably do need to bump a bunch of these rate-limits, particularly around handle lookups using HTTPS well-known for *.bsky.social.

Could you say more about your use of getActorProfile? Are you just using it to fetch labels on the account? It should be possible to subscribe to label feeds directly. We should make it easier to discover/enumerate labelers without having to backfill the network, for example by querying an appview.

[imax9000]

I'm running custom matching rules on the information present in the profile. I don't particularly care about labels, since fetching all of them myself is not very resource-intensive.