How much MST history should a PDS store?

[DavidBuchanan314]

The com.atproto.sync.getRepo API takes a since parameter, allowing a consumer to "catch up" from a previous repo revision.

IIUC, In order to implement this intellegently, a PDS must retain MST blocks (but not necessarily record blocks) from some number of prior repo revisions. (but it would also be correct, although less efficient, for a PDS to just return the whole repo in its current state)

I assume there's no hard requirement on this, but what are the expectations of how much MST history a PDS should store, and for how long? And, what is the current behavior of the official PDS package?

[DavidBuchanan314]

Ah, maybe a PDS doesn't need to retain prior MST states explicitly, but simply remember which commit each block was created by. I need to think about this some more...

[bnewbold]

Informally (aka, this isn't a spec-defining answer), I think it would be acceptable for minimal implementations to just not really implement this and do something like return the entire repo CAR file for any since other than the current revision. Obviously inefficient and not best practice, but I don't know if we intend to require that implementations keep track of history?

It is explicitly the case that PDS implementations don't need to keep around old deleted records. That is, if you request since "last wednesday"; a record ABC was created last thursday; then ABC deleted friday; and it is monday today; the PDS shouldn't return the record ABC in the response. It "should" just return the new records, and any MST nodes needed to fill in the tree.

I believe the PDS reference implementation does indeed keep track of each block with the rev it was introduced in, and does a simple query to find all the "new" blocks (which haven't been deleted).

I don't think we define expected behavior yet around duplicate records. Eg, if you create a new record at a new rkey, but it is the exact same record bytes (by CID) as a previously existing record, should it be included in the since CAR file? Seems like this could go either way.