Microblogging schema

[sgwilym]

I'm all tied up with Earthstar core protocol stuff, and I really miss making ordinary people can use.

But maybe if we put our heads together we can keep things from hitting a standstil on that front? One usecase I think Earthstar could do well is microblogging (i.e. sharing small bits of text + media), so you can have multiple private twitters/instagrams.

I've visited this idea before with the Lobby app (source). You'd add different shares to the app, and get an overview of all the latest posts from them.

The format was basically just this: you'd set documents with the plaintext of your message, and write it to a path with the template /lobby/~{authorAddress}/{timestamp}.txt. And the posts could be ephemeral too, so they'd disappear after a set amount of time.

There are a few features I'd like to add to a future microblogging convention:

• Media attachments
• Rich text support
• Replies

These microblogs wouldn't be constrained to Earthstar clients, either: you could configure replica servers to repost them to Twitter or Mastodon so you could go all POSSE. Or maybe you surface them to your personal website.

An image of the old Lobby app posts being synced over to my personal website

---

So what's this discussion about?

1. How do we structure microblogging documents and query them to include all these features?
2. How do we codify the design we settle upon? Is there just a spec someplace, or is there also a JS module people can use for their own CLI, browser, native apps?
3. Can we add a microblogging UI to https://tools.earthstar-project.org?! My dream for that subdomain is to make a suite of friendly tools that show all the things Earthstar can do, with nice switching between different apps and shares.4.
4. How about some off-the-shelf replica server extensions for the POSSE stuff, at least?

I'd love to learn some patterns we could export to doing this for other applications e.g. user profiles.

And also to just have some fun brainstorming what could be. It's important to really do stuff, and it's important to imagine what could be.

[sbutler-gh]

I love the fat marker sketch I'm seeing there @sgwilym! (Fan of Shape Up?)

That's actually the first thought that came to mind. I could be interested in a process, that instead of necessarily starting with the form of microblogging / existing solutions, starts with a process to uncover what we want (and the affordances, the elements) and building from there. Basically, if we're building from ground zero already, it can't hurt!

[sgwilym]

I love the fat marker sketch I'm seeing there @sgwilym! (Fan of Shape Up?)

Thanks! I enjoyed Shape Up, and have never understood how anyone could start thinking about UI with Sketch or Figma.

I could be interested in a process, that instead of necessarily starting with the form of microblogging / existing solutions, starts with a process to uncover what we want (and the affordances, the elements) and building from there. Basically, if we're building from ground zero already, it can't hurt!

Sure! Let's just keep in mind that someone using Earthstar is already going to have to absorb a lot of new concepts.

I like microblogging because it's an established format with a lot of UI precedent. That said, I think this conversation can be very useful in the context of 'how is it different when you do this with people you trust rather than the public?' Does stuff get bigger, more detailed?

[sgwilym]

If you’re writing an application with Earthstar, you’re probably going to use a document schema.

The name ‘document schema’ makes it sound fancy, but it’s just a pattern you use for organising your documents so you can easily get them.

For example, a document schema for keeping recipes could just be that your recipes all have paths that follow this template:

/recipes/{RECIPE_NAME}.md

So the document schema is just:

• Prefix document paths with /recipes/
• End document paths with .md
• Store Markdown content in them.

This is a really simple example that would be suited to a share with a single user, interacting with their data through something like Earthstar’s filesystem sync.

But it wouldn’t work well with lots of users. Why?

Firstly, there’s no permissions in these paths. Anyone can edit anyone else’s recipes, and maybe not everyone’s cool with that. This is something we can remedy with Earthstar’s ‘owned paths’ concept.

Secondly, each recipe’s data is being stored non-atomically in a single document: the title; ingredients; steps; all of it. This can be a problem with many users working together, as Earthstar has no fancy conflict resolution, and uses the rule of ‘last write wins’. If one user were to edit the tile of the recipe, and another user edited the list of ingredients at the same time, one users edits would be clobbered. But we can reduce the likelihood of data conflicts by breaking up data into separate, atomic documents. This also makes it easier to build clients which view and edit this data.

So you could say designing a good document schema is about considering three things:

• Permissions: Who’s allowed to write to a document? We don’t want people to edit each other’s blogs.
• Querying: How do we structure the paths so that we can query different groups of data from a replica? We’d like to be able to get something like all the replies of a microblog in a single query.
• Avoding data conflicts: how can we split data up to reduce the risk of users’ updates stepping on each other? If two people reply to a microblog at the same time, we don’t want to lose one due to Earthstar’s latest-wins conflict resolution.

[sgwilym]

I'm going to apply the stuff I said about document schemas to sketch out a microblogging schema for Earthstar.

Here's a few assumptions about what features microblogging should have:

• Posts can be written.
• Media can be attached to posts (images, video, music etc)
• Replies can be made to posts
• Media can also be attached to replies

Posts can be written

The template for a new microblog root post:

/microblog/roots/~{AUTHOR_ADDR}/{TIMESTAMP}/root.md

Let's break this template into segments:

1. microblog path segment: This is just us saying 'everything starting with this segment is related to microblog data.
2. roots: Everything beginning with roots is the first post of a potential microblog thread. It also makes it easy for us to get all threads by using a pathStartsWith: "/microblog/roots" query in Earthstar.
3. ~{AUTHOR_ADDR}: When a tilde is put in front of an Earthstar identity's public address in a path, then Earthstar knows that only that identity can write to this path. We don't want people being able to edit each other's posts, after all. AUTHOR_ADDRis replaced with the actual public address (@gwil.xxx...) of the author.
4. TIMESTAMP: This is the timestamp in milliseconds of when the post was created. This is good enough as an ID, as most authors will not be making multiple posts per millisecond. We're going to use this as a way to group other things related to this thread root. Using a timestamp also means we know when the post was originally made without storing that in the doc someplace. We can't use the document timestamp, because it changes whenever you edit it.
5. root.md: This bit on the end of the path means that it's the actual text content of the post itself. The .md is there to suggest it's Markdown formatted (maybe? Or is .txt good enough?)

Media can be attached to posts

The template for attachments to a microblog post:

/microblog/roots/~{AUTHOR_ADDR}/{TIMESTAMP}/attachments/*

The first four segments are the same as the one above. But now we have two new segments instead of root.md:

1. attachments: We know that all paths starting with this will be an document representing an attachment.
2. *: Some string representing attached media. Like dirge.mp3, tornado.gif, or brook.png.

Replies can be made to posts

The template for replies to a microblog:

/microblog/threads/--{ROOT_AUTHOR_ADDR}/{ROOT_TIMESTAMP}/~{AUTHOR_ADDR}/{TIMESTAMP}/reply.md

This path starts with microblog too. But there the similarities end:

1. threads: Everything starting with this is related to the replies, rather than the root of, a post.
2. --{ROOT_AUTHOR_ADDR}: This is the public address of the identity which made the root of the thread. Instead of being prefixed with a tilde, it's prefixed with --, so that other identities can post to this path.
3. ROOT_TIMESTAMP: The timestamp value of the post being replied to.
4. ~{AUTHOR_ADDRESS}: This is the public address of the author of the reply, prefixed with a tilde so only they can write to this path.
5. TIMESTAMP: The timestamp of the author's reply in milliseconds. Also makes it easy for us to know when a reply was authored just by using its path.
6. reply.md: The bit indicating what the format of the data written to this document is, and what's it for. It's probably a reply formatted in markdown.

Media can also be attached to replies

The template for replies to a microblog:

/microblog/threads/--{ROOT_AUTHOR_ADDR}/{ROOT_TIMESTAMP}/~{AUTHOR_ADDR}/{TIMESTAMP}/attachments/*

The last feature. The first five parts of this template are the same as the last, but using the same format for root attachments at the end:

1. attachments: We know that all paths starting with this will be an document representing an attachment.
2. *: Some string representing attached media. Like dirge.mp3, tornado.gif, or brook.png.

---

This is kind of based on what we learned about structuring paths in the Letterbox app: you want to be able to grab structurally similar things (e.g. replies) in as few queries as possible, and with as little filtering and sorting of paths as possible.

This is admittedly a weakness of designing schemas like this for Earthstar: the schema often has assumptions about how the UI will work in it (e.g. we will get all root threads first, then use the values we learnt from that to find replies, etc.). There are open issues about addressing this.

And that's not even getting into other stuff you need for a good UI, like indexing and pagination — which we'll probably have to do client-side in some way.