How to Crawl or Backfill data for a Search or Recommendation system?

[FaizShah]

Hi,

I was looking through the dev docs and it’s not clear to me what is the intended way for devs to acquire a training set to build Search or Recommendation applications.

What I see from the docs is you have the backfill window in the event stream but it explicitly says it’s not intended to be used to replay the history especially from the start: https://atproto.com/specs/event-stream

Then I see in the HTTP api there is a way to list all the repos com.atproto.sync.listRepos without auth but then you in order to com.atproto.repo.listRecords in the repo it requires auth. Additionally this would be done as rate limited http calls instead of as a bulk operation like an event stream replay.

What is the intended way for developers to acquire a training set for Search or Recommendation applications?

[FaizShah]

Documentation here: https://docs.bsky.app/docs/advanced-guides/backfill

It seems that enumerating it using the API via listerepos getrepo is the official solution. Seems inefficient with no batch op tho would prefer to replay the stream.

[DavidBuchanan314]

If you do this with enough concurrency then network bandwidth is the bottleneck, so I'm not sure I'd call it inefficient per se