Rhythm Cafe V3 Planning

[auburnsummer]

v1 is the original google sheet

v2 is what we have now. After GAE started rate limiting my google sheet I had to quickly move off it. So we get what we have now, but the system won't be appropriate for a more dynamic level system.

v3...??

Design Goals

• Support live updating of metadata associated with each level (i.e. likes, kudos, rankings, whatever we're doing, but I need the technical capability to do so). When metadata of a level is updated, that change should be visible in the site within seconds.
  • The creator of the level should be able to turn on/off metrics
• Support a level being a successor to another level. This should retain the same metadata but be a different level.
• Support level commenting w/ moderation
  • The creator of the level should be able to turn on/off comments
• Support the ability to add levels to collections like AO3
• Reuse code from v2 wherever possible.
• Continue to support 3rd party integrations (levelsync, etc.).
• Continue to achieve reasonable latency via use of distributed systems
  • i.e. I want to learn more about distributed systems and this is a good way to learn. technically V2 is a distributed system but a very simple one

[auburnsummer]

Database

All info in RCV3 uses a single SQLite database. V2 has a split system (orchard.db and status.db), V3 will just have one SQLite database for everything.

Each individual API server will have its own copy of the database, but only the primary server will be writable.

Reasoning:

• RCV3 is expected to be primarily read heavy, and SQLite is very good at reading.
• Existing code uses SQLite and this will allow better reuse of existing code.
• SQLite will allow existing integrations (datasette, etc.) to work

Schema changes:

• Level

  • Primary key should be derived from a simple hash of the .rdlevel file, instead of the current more complicated id system
  • Add authors_raw

• Status

  • Primary key for Status is called alias, which originates as a synthetic uuid
  • alias is intended to be changed by the user
  • alias is also the user-facing level id (the site, discord bot, etc. should show alias as the "level id"), not the underlying Level primary key
  • Also has a foreign key to a Level:
    • All Statuses have an associated Level
    • A Level can have up to 1 Status
  • Also has a foreign key to a User

• LevelLevelSuccessor

  • join table for Levels, indicates when a level is a successor to another one.
  • The operation for making a level is successor is basically:
    • Update the Status to point to the new Level
    • Add a row to LevelLevelSuccessor for the Level

• User

  • A discord user
  • id is the primary key and the discord snowflake
  • possibly other info here as well, we generally should not store usernames etc, just fetch them from API as needed
  • e.g. opt-out status

• Comment ??

• Collection ??

erDiagram
    Level {
        string rdlevel_hash
    }
    Status {
        string alias
    }
    User {
        string id
    }
    Status |o -- || Level : has 
    Level  |o -- o| Level : successor
    Status }o -- || User : posts

Loading

[auburnsummer]

API Server

An API server serves api.rhythm.cafe. There's a few of these geodistributed around the world. What's on an API server?

• Front Door

  • A Caddy daemon that handles HTTP ingress and outgress.
  • Performs routing from specified routes to the other services.
  • If the request is not HEAD/GET/OPTIONS (i.e. PUT/POST/DELETE/PATCH), and this API server is not the primary server, it replays the request to the primary server and returns the response from the primary server.
  • fly.io we can use https://fly.io/docs/reference/dynamic-request-routing to do this.

• SQLite

  • The SQLite database from the above post, this has a copy of it.
  • The databases are replicated from primary to replicas automatically. This is probably via https://github.com/superfly/litefs

• SQLite Backup Service

  • Only runs on the primary server
  • Periodically vacuums the SQLite database, gzips it, chucks that onto B2
  • We can recover the system from this if a catastrophic failure occurs
  • We expose an endpoint that does it and that endpoint is called by a periodic github action

• Search Engine

  • e.g. Typesense or Meilisearch?
  • Changes to SQLite synchronise to here.
  • I will need to do it via a trigger in SQLite itself, because changes in SQLite occur through LiteFS and not the application layer

• Datasette

  • Stock Datasette instance exposing the SQLite database.
  • Read-only.

• Discord Bot

  • An interaction based Discord bot
  • Specifically this must have at least two new message-context commands:
    • add - add levels in a post via attachments
      • anyone can use this! unless the user has opted out
    • delete - delete levels in a post (via iid?)
  • Allows users to do basic level management in Discord

• The API itself

  • Level management, change alias, delete level, etc.
  • Authentication tokens are Discord JWTs?

[auburnsummer]

Peer Review

PR will use an issue-tracking system. When a level is added to the primary database, it creates a issue in the issue tracking system for the level. The issue can be closed with a tag to either approve/nonrefer the level. A bot observes when this happens and calls the appropriate webhooks.

Investigation for what system to use is ongoing.

[auburnsummer]

Frontend and Admin interface

The Frontend and Admin interface are probably the same thing. The Admin interface is hidden unless you're logged in with Discord and you have the correct roles. Roles are defined in Discord and fetched via the Discord API.

I'll probably use a fancy SSR framework for the frontend, utilising existing components whereever possible. Research ongoing.

Frontend:

• view levels
• filter levels
• manage your own levels
• add levels to collections?

Admin interface:

• view/edit arbitrary levels
• some logging system needed as well