Update documentation across repo for new docs.t2bot.io site (WIP) (#530)

* Update documentation across repo for new docs.t2bot.io site (WIP)

* Use a published alias

CHANGELOG.md

@@ -55,7 +55,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ### Mandatory Configuration Change
 
-**Please see [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/upgrading/130.html) for details.**
+**Please see [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/v1.3.3/upgrading/130.html) for details.**
 
 ### Security Fixes
 
@@ -89,12 +89,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Changed
 
 * **Mandatory configuration change**: You must add datastore IDs to your datastore configuration, as matrix-media-repo will no longer manage datastores for you.
-* If compiling `matrix-media-repo`, note that new external dependencies are required. See [the docs](https://docs.t2bot.io/matrix-media-repo/installing/method/compilation.html).
+* If compiling `matrix-media-repo`, note that new external dependencies are required. See [the docs](https://docs.t2bot.io/matrix-media-repo/v1.3.3/installing/method/compilation.html).
   * Docker images already contain these dependencies. 
 * Datastores no longer use the `enabled` flag set on them. Use `forKinds: []` instead to disable a datastore's usage.
 * Per-user upload quotas now do not allow users to exceed the maximum values, even by 1 byte. Previously, users could exceed the limits by a little bit.
 * Updated to Go 1.19, then Go 1.20 in the same release cycle.
-* New CGO dependencies are required. See [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/installing/method/compilation.html) for details.
+* New CGO dependencies are required. See [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/v1.3.3/installing/method/compilation.html) for details.
 * Logs are now less noisy by default.
 * Connected homeservers must support at least Matrix 1.1 on the Client-Server API. Servers over federation are not affected.
 * The example Grafana dashboard has been updated.

README.md

@@ -1,30 +1,23 @@
 # matrix-media-repo
 
-matrix-media-repo is a highly customizable multi-domain media repository for Matrix. Intended for medium to large environments
-consisting of several homeservers, this media repo de-duplicates media (including remote media) while being fully compliant
-with the specification. 
+MMR is a highly configurable multi-homeserver media repository for Matrix. It is an optional component of your homeserver 
+setup, and recommended only for large individual servers or hosting providers with many servers.
 
-Smaller/individual homeservers can still make use of this project's features, though it may be difficult to set up or have 
-higher than expected resource consumption - please do your research before deploying this as this project may not be useful
-for your environment.
+**If you're looking for an S3 connector, please consider using [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider) instead.**
 
-For help and support, visit [#mediarepo:t2bot.io](https://matrix.to/#/#mediarepo:t2bot.io). Administrator documentation
-can be found on [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/).
+Smaller homeservers can still set this up, though may find it difficult to deploy or use. A high level of knowledge regarding
+the Matrix homeserver stack is assumed.
 
-## Installing
+## Documentation and support
 
-For installation instructions, see [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/installing/index.html).
+Matrix room: [#media-repo:t2bot.io](https://matrix.to/#/#media-repo:t2bot.io)
 
-## Deployment
-
-For deployment information, see [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/deployment/index.html).
+Documentation: [docs.t2bot.io](https://docs.t2bot.io/matrix-media-repo/)
 
 ## Developers
 
-To properly run the media repo in a development setting, it must be compiled manually
-once to ensure the assets are set up correctly: follow the 
-[compilation steps](https://docs.t2bot.io/matrix-media-repo/installing/method/compilation.html)
-posted on docs.t2bot.io.
+MMR requires compiling at least once before it'll run in a development setting. See the [compilation steps](https://docs.t2bot.io/matrix-media-repo/unstable/installation/compile)
+before continuing.
 
 This project offers a development environment you can use to test against a client and homeserver.
 
@@ -34,7 +27,7 @@ As a first-time setup, run:
 docker run --rm -it -v ./dev/synapse-db:/data -e SYNAPSE_SERVER_NAME=localhost -e SYNAPSE_REPORT_STATS=no matrixdotorg/synapse:latest generate
 ```
 
-Then you can run `docker-compose -f dev/docker-compose.yaml up` to always bring the service online. The homeserver will 
+Then you can run `docker compose -f dev/docker-compose.yaml up` to always bring the service online. The homeserver will 
 be behind an nginx reverse proxy which routes media requests to `http://host.docker.internal:8001`. To test accurately, 
 it is recommended to add the following homeserver configuration to your media repo config:
 
@@ -61,86 +54,3 @@ for database management - use at your own risk.
 
 **Note**: Running the Go tests requires Docker, and may pollute your cached images with tons of layers. It is suggested to
 clean these images up manually from time to time, or rely on an ephemeral build system instead.
-
-## Importing media from synapse
-
-Media is imported by connecting to your synapse database and downloading all the content from the homeserver. This is so 
-you have a backup of the media repository still with synapse. **Do not point traffic at the media repo until after the 
-import is complete.**
-
-**Note**: the database options provided on the command line are for the Synapse database. The media repo will use the 
-connection string in the media-repo.yaml config when trying to store the Synapse media.
-
-**Note**: If you want to avoid building the import binaries yourself, you can find them included in every [release](https://github.com/turt2live/matrix-media-repo/releases) and in the Docker container image (in `/usr/local/bin/`)
-
-1. Build the media repo (as stated above)
-2. Edit/setup `media-repo.yaml` per the install instructions above
-3. Run `bin/import_synapse`. The usage is below. 
-    ```
-    Usage of import_synapse:
-      -baseUrl string
-            The base URL to access your homeserver with (default "http://localhost:8008")
-      -config string
-            The path to the media repo configuration (with the database section completed) (default "media-repo.yaml")
-      -dbHost string
-            The PostgresSQL hostname for your Synapse database (default "localhost")
-      -dbName string
-            The name of your Synapse database (default "synapse")
-      -dbPassword string
-            The password for your Synapse's PostgreSQL database. Can be omitted to be prompted when run
-      -dbPort int
-            The port for your Synapse's PostgreSQL database (default 5432)
-      -dbUsername string
-            The username for your Synapse's PostgreSQL database (default "synapse")
-      -migrations string
-            The absolute path the media repo's migrations folder (default "./migrations")
-      -serverName string
-            The name of your homeserver (eg: matrix.org) (default "localhost")
-      -workers int
-            The number of workers to use when downloading media. Using multiple workers risks deduplication not working as efficiently. (default 1)
-    ```
-    Assuming the media repository, postgres database, and synapse are all on the same host, the command to run would look something like: `bin/import_synapse -serverName myserver.com -dbUsername my_database_user -dbName synapse`
-4. Wait for the import to complete. The script will automatically deduplicate media.
-5. Point traffic to the media repository.
-
-## Export and import user data
-
-The admin API for this is specified in [docs/admin.md](./docs/admin.md), though they can be difficult to use for scripts.
-The `bin/gdpr_export` and `bin/gdpr_import` binaries do the process for you, and do so in memory but against the real
-media repo database and datastores - this moves the resource intensiveness to the binary you're running instead of the
-media repo instance, but still makes reads and writes to your database and datastores. For example, when exporting a 
-user's data the binary will pull all the data locally and write it to disk for you, but during that process the user's
-export is accessible via the main media repo too. The export is deleted if the binary is successful at exporting the 
-data.
-
-**Note**: Imports done through this method can affect other homeservers! For example, a user's data export could contain
-an entry for a homeserver other than their own, which the media repo will happily import. Always validate the manifest
-of an import before running it!
-
-Ensuring you have your media repo config available, here's the help for each binary:
-
-```
-Usage of gdpr_export:
-  -config string
-        The path to the configuration (default "media-repo.yaml")
-  -destination string
-        The directory for where export files should be placed (default "./gdpr-data")
-  -entity string
-        The user ID or server name to export
-  -migrations string
-        The absolute path for the migrations folder (default "./migrations")
-  -templates string
-        The absolute path for the templates folder (default "./templates")
-```
-
-```
-Usage of gdpr_import:
-  -config string
-        The path to the configuration (default "media-repo.yaml")
-  -directory string
-        The directory for where the entity's exported files are (default "./gdpr-data")
-  -migrations string
-        The absolute path for the migrations folder (default "./migrations")
-  -verify
-        If set, no media will be imported and instead be tested to see if they've been imported already
-```

common/import_cmdline/ask_machine_id.go

@@ -4,12 +4,13 @@ import (
 	"fmt"
 	"os"
 
+	"github.com/turt2live/matrix-media-repo/common/version"
 	"github.com/turt2live/matrix-media-repo/util/ids"
 	"golang.org/x/term"
 )
 
 func AskMachineId() {
-	fmt.Println("The importer runs as a MMR worker and needs to have a dedicated MACHINE_ID. See https://docs.t2bot.io/matrix-media-repo/deployment/horizontal_scaling.html for details on what a MACHINE_ID is.")
+	fmt.Printf("The importer runs as a MMR worker and needs to have a dedicated MACHINE_ID. See https://docs.t2bot.io/matrix-media-repo/%s/deployment/horizontal_scaling for details on what a MACHINE_ID is.", version.DocsVersion)
 	if !term.IsTerminal(int(os.Stdin.Fd())) {
 		fmt.Println("Please specify a MACHINE_ID environment variable.")
 		os.Exit(2)

common/runtime/init.go

@@ -55,7 +55,7 @@ func LoadDatastores() {
 	for id, found := range dsMap {
 		if !found {
 			logrus.Errorf("No configured datastore for ID %s found - please check your configuration and restart.", id)
-			logrus.Error("See https://docs.t2bot.io/matrix-media-repo/upgrading/130.html for details")
+			logrus.Errorf("See https://docs.t2bot.io/matrix-media-repo/%s/upgrading/130 for details", version.DocsVersion)
 			fatal = true
 		}
 	}

common/version/version.go

@@ -10,6 +10,9 @@ import (
 var GitCommit string
 var Version string
 
+// DocsVersion The version number used by docs.t2bot.io links throughout the application runtime
+const DocsVersion = "v1.3.3"
+
 func SetDefaults() {
 	build, infoOk := debug.ReadBuildInfo()
 

docs/releasing.md

@@ -0,0 +1,25 @@
+# Releasing MMR
+
+MMR is released whenever the changelog *feels* worthy of a release.
+
+## Prerequisites
+
+1. Ensure `CHANGELOG.md` is up-to-date and has consistent language.
+2. Ensure tests pass on main branch.
+
+## Release
+
+1. Update `version/version.go#DocsVersion` to point to the about-to-be-released version.
+2. In the [docs.t2bot.io repo](https://github.com/t2bot/docs.t2bot.io):
+   1. Rename `/content/matrix-media-repo/unstable` to `/content/matrix-media-repo/v1.3.4` (if releasing v1.3.4).
+   2. Run `npm run build` (after `npm install` if required).
+   3. Copy `/site/matrix-media-repo/v1.3.4` to `/old_versions/matrix-media-repo/v1.3.4`.
+   4. Rename `/content/matrix-media-repo/v1.3.4` back to `/content/matrix-media-repo/unstable`.
+   5. Commit and push changes.
+3. Update the links and headers in `CHANGELOG.md`.
+4. Commit any outstanding changes.
+5. Create a git tag for `v1.3.4` and push both the tag and main branch.
+6. On the GitHub releases page, create a new release for the pushed tag. The title is the tag name, and the content is the relevant changelog section.
+7. Build binaries for Windows and Linux, then attach them to the GitHub release.
+8. Publish the release.
+9. Ensure the Docker image is built or building, then announce the release in the MMR Matrix room.