Possible APIs for multi-format support

[sgwilym]

In order for Earthstar to support documents associated with big blobs, I'll be adding a new document format.

Whatever this format will look like, we'll want to use it alongside the existing es.4 document format so that people can keep using the documents they've written already.

This means Earthstar replicas need to support multiple format validators when ingesting and setting documents. This is quite a big change, as the codebase currently assumes all documents will follow the es.4 document format at every level.

I started experimenting with different implementations of this a few days ago. I have two different approaches so far, each with different trade-offs:

Omnivorous multi-format support

With this approach, users can provide whichever formats they want, even their own. Constructing an replica looks like this:

const myReplica = new Replica({
  validators: [ValidatorEs4, ValidatorEsBlob],
  driver: myMemoryDriver
});

Pros

• This provides good typings for Typescript users. This replica will be able to infer the arguments and return types of the replica's querying and setting methods.
• Because you can choose which validators to provide, you can make it so that your replica does not accept certain kinds of documents.

Cons

• A Peer will only be able to accept replicas of the same type. If you have a replica which only deals with es.4, and one which deals with es.4 and esblob, you will need to construct separate Peers to manage them.
• Peers will need to know their type when you construct them. This will require providing type arguments upon initialisation (e.g. new Peer<typeof ValidatorEs4>() or arguments it can infer the type from (e.g. new Peer([ValidatorEs4])). Maybe peers could take a onCreateShare callback, so that you could add replicas to a peer using just a share address?
• This approach means generic typings need to be added to nearly every class in the codebase. These generic typings are quite hard to understand and read, and I worry how they will affect the codebase's maintainability and approachability.

Curated multi-format support

In this approach the formats Earthstar replicas use have been decided up front for you . Constructing an replica looks like this:

const myReplica = new Replica({
  driver: myMemoryDriver
});

Pros

• Simpler replica construction, fewer concepts to learn.
• Replicas always support 'core' formats developed by the Earthstar project.
• No complex generic types in the codebase.

Cons

• You can't choose which formats of documents you want to accept or not. You use what is chosen for you by the Earthstar project.
• You can't use your own formats.

---

I really like the omnivorous approaches' flexibility. But speaking as the sole maintainer of the Earthstar project, the complexity it adds to the codebase concerns me. And no-one has ever expressed any interest in using Earthstar with their own format.

The curated approach is more proscriptive, but makes the API and concepts simpler. The implementation is significantly simpler. And it doesn't close the door to adopting true omnivorous approach in the future, e.g. if Typescript's generic typings get better.

I know which way I'm leaning, but I'm interested if anyone feels one way is better than the other.

[johanbove]

How would the supported formats be selected in the "curated format support" option? cf. "You use what is chosen for you by the Earthstar project." Would that mean waiting for a new release if some other format gets in (through voting) ?

[sgwilym]

Yes. For example the next major release of Earthstar would come with bundled support for es.4 and whatever the format supporting blobs is.

I don't expect there will be new formats very often, and when they do come it'll be with substantial new features (e.g. blobs, read-only shares).