Document development and changelog management practices (#6315)

.github/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+A changelog describes the changes that are made to a project. Usually a changelog is written for developers. In FAForever the changelog is orientated towards the community. Because of that some technical details that may be relevant to developers are not in the changelog. Usually the changelog references the pull requests that are responsible. We encourage contributors to document the technical details in the description and/or conversation of a pull request.
+
+## Changelog folder
+
+All changelogs can be found in the [changelog](../changelog/) folder. Each game version has a separate file with the changelog notes in them. Each changelog file closely matches with a corresponding [release note](https://github.com/FAForever/fa/releases). The release note is better formatted by GitHub and we encourage you to read the release notes instead.
+
+## Changelog snippets
+
+We use snippets to reduce the burden on maintainers to write an accurate changelog and at the same time enable contributors to describe the changes of a pull request. A contributor is encouraged to create a snippet before a pull request is merged. All snippets reside in the [snippets folder](../changelog/snippets/). We use a [workflow](./workflows/changelog.yaml) to compile the snippets into a typical changelog file. This changelog file can then be tweaked, spell checked and be used as (intermediate) release notes.
+
+### Format of a snippet
+
+All current snippets can be found in the [snippets folder](../changelog/snippets/). A snippet has two relevant aspects:
+
+- The structure of the name of the file
+- The content of the file
+
+The structure of the file name is `XXX.ABCDE.md`, where `XXX` is one of the snippet types and `ABCD` is the pull request number. The available snippet types are `fix`, `features`, `balance`, `graphics`, `ai`, `performance` or `other`. The content of a snippet is similar to a commit message. The first line is a title that starts with the relevant pull requests and a concise description of the changes, as an example: ` - (#PR1, #PR2, ...) <concise description of changes>` . The remainder of the file can be used to provide additional and more detailed information about the changes. The file should be formatted using a Markdown formatter, one example is the use of [prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode).
+
+## Sources and inspiration
+
+We did not come up with this approach ourselves. We took inspiration from similar solutions of projects that experienced similar problems:
+
+- [Issue of the PrefectHQ project](https://github.com/PrefectHQ/prefect/issues/2311)
+- [Towncrier](https://github.com/twisted/towncrier)

.github/DEPLOYMENT.md

@@ -8,6 +8,15 @@ In this repository we can deploy to three different environments:
 
 All three branches originate from the `develop` branch, which is the default branch of the remote on Github. Pushing commits towards any of the deployment branches is sufficient to trigger a deployment to the given game type. \
 
+## Deployment procedures for the FAF game type
+
+The following (manual) steps are relevant to create a valid deployment to the FAF game type.
+
+- (1) Update the game version in [mod_info.lua](../mod_info.lua) and [version.lua](../lua/version.lua).
+- (2) Update the game executable. This needs to be done by a server administrator. This is only required when there are changes to the executable.
+- (3) Update the changelog in [changelog.md](/CHANGELOG.md) and [changelogData.lua](../lua/ui/lobby/changelogData.lua).
+- (4) Update the game version in [changelogData.lua](../lua/ui/lobby/changelogData.lua).
+
 ## Automated deployments
 
 There are three workflows to help with deployment:

.github/DEVELOPMENT.md

@@ -0,0 +1,116 @@
+# Development
+
+This document contains a wide range of tips and tricks surrounding the development of the Lua code of the FAForever project. It can help you setup the development environment. It can help you with understand what is, and is not available to you in the Lua environment of Supreme Commander. It is however not a guide on how to write Lua code. And it is not a guide on programming in general. And it is also not a guide on how Git and/or GitHub works.
+
+## Tooling
+
+Everything works and breaks with your tooling. In this section we explain what has worked best so far.
+
+### Lua development
+
+We recommend the following tooling for development of Supreme Commander:
+
+- [Visual Studio Code](https://code.visualstudio.com/) as your interactive development environment (IDE).
+- [Github Desktop](https://github.com/apps/desktop) or [Github CLI](https://git-scm.com/) as your tool to interact with Git.
+
+For Visual Studio Code we recommend the following extensions:
+
+- [FA Lua extension](https://github.com/FAForever/fa-lua-vscode-extension/releases): introduces intellisense - absolutely vital to development.
+- [Gitlens](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens): useful for seeing who made what change.
+- [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode): useful for formatting.
+- [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker): useful to prevent common spelling mistakes.
+
+### Batch processing of blueprints
+
+We recommend the following tooling in addition of the tooling used for development of Supreme Commander:
+
+- [Brew WikiGen](https://github.com/The-Balthazar/BrewWikiGen) that allows for batch processing of blueprints.
+- [Lua 5.4](https://www.lua.org/download.html) required for the Brew WikiGen to work.
+
+There is a `Run.lua` file inside the Brew WikiGen source files. It represents the configuration of the tool. Copy the file and update the following fields:
+
+- `WikiGeneratorDirectory` needs to reference the folder where the Brew WikiGen is located. Requires a trailing `/`.
+- `EnvironmentData.location` needs to reference the checked out fa repository. Requires a trailing `/`.
+- `EnvironmentData.RebuildBlueprints` should be `true`.
+- `EnvironmentData.lua` needs to reference the checked out fa repository. Requires a trailing `/`.
+- `EnvironmentData.LOC` needs to reference the checked out fa repository. Requires a trailing `/`.
+- `EnvironmentData.PostModBlueprints` needs to reference the name of a function that is in the scope of [blueprints.lua](../lua/system/Blueprints.lua). This function is provided all the blueprint values that are loaded in a similar fashion to how the game provides the blueprint files. 
+
+And the following fields should be empty:
+
+- `EnvironmentData.ExtraData`
+- `ModDirectories`
+
+Depending on what blueprints you'd like to rebuild you'll need to update `RebuildBlueprintOptions.RebuildBpFiles` and `EnvironmentData.LoadExtraBlueprints`. You should now be able to batch process all the blueprint files using the functions provided in `EnvironmentData.PostModBlueprints` by calling your run file with the Lua compiler that you installed. You can use [#6279](https://github.com/FAForever/fa/pull/6279) and [#6274](https://github.com/FAForever/fa/pull/6274) as an example on how to prepare the functionality in [blueprints.lua](../lua/system/Blueprints.lua).
+
+### Automation via GitHub Actions
+
+We recommend the following tooling in addition of the tooling used for development of Supreme Commander:
+
+- [Act](https://github.com/nektos/act): allows you to run the average GitHub action on your local machine.
+- [Docker](https://www.docker.com/products/docker-desktop/): required for Act to work.
+- [Github CLI](https://github.com/cli/cli): required to authenticate yourself for Act to work.
+
+You can verify the tooling is installed and available by running `gh --version`, `act --version` and `docker --version` in the command line.
+
+#### Specifics for Act
+
+The tool `act` only works on workflows that have the `push` event. Temporarily add the `push` event to the workflow that you want to test if it is missing.
+
+```bash
+         # Container to use                                        # Workflow to debug              # Token to authorize (optional)    # Do not pull the docker image each time
+    act -P 'ubuntu-latest=ghcr.io/catthehacker/ubuntu:act-latest' -W '.github/workflows/test.yaml' -s GITHUB_TOKEN="$(gh auth token)" -p=false
+```
+
+You can find all the Docker images that work with Act [on Github](https://github.com/catthehacker/docker_images).
+
+## Specifics to the Lua environment
+
+There are various specifics but you're usually actively discouraged to use them. The reason for this is simple: using them breaks all the tooling surrounding the game.
+
+- The operator `^` is the bit-wise XOR operator and **not** the typical power operator, which is `math.pow`.
+- The operator `|` is the bit-wise OR operator.
+- The operator `&` is the bit-wise AND operator.
+- The operators `>>` and `<<` are the bit-wise shift operators.
+- The operator `!=` is an alternative to `~=` to check for inequality.
+- The syntax `#` is an alternative to `--` for creating comments.
+- The statement `continue` exists, which works like you'd expect in other languages with the `continue` keyword.
+
+The one exception that can be used to [improve the performance of the game](https://github.com/FAForever/fa/issues/4539) is this:
+
+- The `{h&a&}` is new syntax to create a table with a pre-allocated hash and array sections. The value `h` pre-allocates `math.pow(2, h)` entries in the hash section of a table. The value `a` pre-allocates `a` entries in the array section of a table. 
+
+Due to safety concerns various modules and/or functions that are part of the default Lua library are not available. This primarily applies to the entire `io` and `os` modules, which is only available during the initialisation phase of the game. [Interfacing with a C package](https://www.lua.org/pil/8.2.html) is also not available. In general anything that would provide access outside of the sandbox of the game is not available. There are some alternatives such as `DiskFindFiles` and `DiskGetFileInfo` that provide basic access to files that are made accessible during the initialisation phase of the game.
+
+### Lua contexts
+
+There are various Lua contexts in Supreme Commander. Each context is isolated from all the other contexts. This is intentional, especially for the session related contexts as changes to the simulation that are not synchronized to all users in a session  will cause a desync. All contexts have [access to a shared package of globals](../engine/Core.lua). 
+
+- (1) Initialisation context
+
+This is run at the start of the game. It is responsible for running the init files, such as [init_faf.lua](../init_faf.lua). Unlike other contexts the `io` and `os` modules are available. 
+
+- (2) Blueprint loading context
+
+This is run when preparing a game session. It is responsible for loading and processing all the blueprint files. The [globalInit.lua](../lua/globalInit.lua) is run to initialize the context and then proceeds to call functions in [blueprints.lua](../lua/system/Blueprints.lua) to process the blueprints.
+
+- (3) Main menu UI context
+
+This is run (as a separate instance) during the splash screen, during the main menu (including the lobby). It is responsible for a lot of the UI functionality. The [userInit.lua](../lua/userInit.lua) is run to initialize the context and all [user globals](../engine/User.lua) are available.
+
+- (4) Session UI context
+
+This is run when a game session has started. It is responsible for a lot of the UI functionality. The [sessionInit.lua](../lua/SessionInit.lua) is run to initialize the context and all [user globals](../engine/User.lua) are available. You can use [Sim Callbacks](../lua/SimCallbacks.lua) to pass and synchronize information to the session sim context. In general, all user globals that (indirectly) interact with the simulation is input and synchronized between users. 
+
+- (5) Session sim context
+
+This is run when a game session has started. It is responsible for all the Lua interactions in the simulation and all [sim globals](../engine/Sim.lua) are available. The [simInit.lua](../lua/simInit.lua) is run to initialize the context. You can use [UserSync.lua](../lua/UserSync.lua) to pass information to the Session UI context.
+
+<!-- 
+## Writing high performing Lua code for Supreme Commander
+
+It goes without saying that premature optimisation is the root of all evil in the world. But Supreme Commander is not like the world. There is some common hygiene that you can apply to make your code a magnitude faster and more readable at the same time.
+
+In Supreme Commander all Lua code is read, parsed and transpiled into bytecode that represent instructions. This happens when a module is [imported](../lua/system/import.lua) for the first time. The instructions are then executed by an interpreter. Unlike the [average compiler](https://en.wikipedia.org/wiki/Optimizing_compiler), an interpreter (and specifically a Lua interpreter) takes your code extremely literal. The instructions directly map to the syntax of the Lua script. You can learn more about what instructions exist by reading chapter 7 of [The implementation of Lua 5.0](https://www.lua.org/doc/jucs05.pdf). You can evaluate the instructions that make up a function using `debug.listcode`.
+
+<todo> -->

README-russian.md

@@ -16,7 +16,7 @@ Forged Alliance Forever — это [проект](https://github.com/FAForever)
 Список изменений
 ---------
 
-Вот полный [список изменений](changelog.md). Существует [альтернативный журнал изменений](http://patchnotes.faforever.com/), особенно для патчей баланса, в удобном для пользователя виде.
+Вот полный [список изменений](./.github/CHANGELOG.md). Существует [альтернативный журнал изменений](http://patchnotes.faforever.com/), особенно для патчей баланса, в удобном для пользователя виде.
 
 Помощь сообществу
 ------------

README.md

@@ -12,39 +12,15 @@ Forged Alliance Forever is a vibrant, community-driven [project](https://github.
 - Log in using the account you registered and host a game with AIs and/or players, queue up for matchmaker or upload your own content to our vaults for other players to enjoy.
 - Engage with the community through the [forums](https://forum.faforever.com/) using the account you registered or join us on the official [Discord server](https://discord.gg/mXahVSKGVb).
 
-## Changelog
+## Information for contributors
 
-The changelog is split across multiple files. Each file represents a release. You can find them in the [changelog](./changelog/) folder. Some changes are also described in an [alternative format](http://patchnotes.faforever.com/) that makes them easier to read for the community.
+All contributions are expected to apply the [code of conduct](https://www.faforever.com/rules) to create a healthy and sustainable contributor environment.
 
-## Contributing
+- [Changelog](./.github/CHANGELOG.md): documentation about how we manage changelogs and release notes.
+- [Development](./.github/DEVELOPMENT.md): documentation about development including how to setup a development environment and various best practices.
+- [Deployment](./.github/DEVELOPMENT.md): documentation about deployment to the FAF, FAF Beta Balance and FAF Develop game types.
 
-There are instructions [in English](setup/setup-english.md) and [in Russian](setup/setup-russian.md) to help you set up a development environment. It is important that you discuss your contributions beforehand. You can do this by making a comment on an existing issue or, if it doesn't exist yet, by opening a new issue. Not all pull requests are merged by default. It is important that the changes align with the vision of the project.
-
-There's an [extensive guide](./setup/setup-english.md) to help you setup a debug environment. Everyone that contributes must adhere to the [rules](https://www.faforever.com/rules) and apply the [contribution guidelines](https://forum.faforever.com/topic/2051/faf-contribution-guidelines) as they discuss issues with other community members.
-
-## Deployment
-
-There are three branches branches that deployable and are available for players to play on. The deployment procedure is automated. Pushing commits to one of these branches is sufficient to trigger the deployment. 
-
-- (1) `deploy/faf` is the production branch that maps to the FAF gametype.
-- (2) `deploy/fafdevelop` is a development branch that maps to the FAF Develop gametype.
-- (3) `deploy/fafbeta` is a development branch that maps to the FAF Beta Balance gametype.
-
-### Deployment of a development branch
-
-There are no requirements to a deployment to a development branch. A push of a commit is sufficient to trigger a deployment. The development branches are unaware of history. For example, A replay will always start using the last deployment. All replays that used a previous deployment when the game was played is guaranteed to desync.
-
-### Deployment of the production branch
-
-There are various requirements when deploying to production:
-
-- (1) Update the game version in [mod_info.lua](/mod_info.lua) and [version.lua](/lua/version.lua).
-- (2) Update the game executable. This needs to be done by a server administrator. This is only required when there are changes to the executable.
-- (3) Update the changelog in [changelog.md](/changelog.md) and [changelogData.lua](/lua/ui/lobby/changelogData.lua).
-- (4) Update the game version in [changelogData.lua](/lua/ui/lobby/changelogData.lua).
-- (5) Create a new release with the changelog that points to a commit on the [develop]() branch
-
-Specifically steps (1), (2) and (5) are required to create a functioning deployment where even replays can retrieve the proper configuration to prevent desyncs.
+In addition, there is an [extensive guide](./setup/setup-english.md) to help you setup your development environment. This guide is due to be replaced by the development file.
 
 ## Related repositories
 

annotation.md

@@ -1,4 +1,3 @@
-
 # Annotating
 
 All future contributions should be properly annotated. With it [our Lua extension](https://github.com/FAForever/fa-lua-vscode-extension) is able to provide intellisense support for our game, mod and campaign developers. You can find all the supported annotations on the [official repository](https://github.com/sumneko/lua-language-server/wiki/EmmyLua-Annotations). Note that the annotations supports markdown.
@@ -44,7 +43,7 @@ end
 --- | ------------------ | ----------- |
 --- | ReadyVariable      | `ScenarioInfo.VarTable[ReadyVariable]` Set when all units are on the transports
 --- | WaitVariable       | `ScenarioInfo.VarTable[WaitVariable]` Needs to be set before the transports can leave
---- | LandingList        | (REQUIRED or LandingChain)               
+--- | LandingList        | (REQUIRED or LandingChain)
 --- | LandingChain       | (REQUIRED or LandingList)
 --- | TransportReturn    | Location for transports to return to (they will attack with the land units if this isn't set)
 --- | TransportChain     | (REQUIRED or TransportRoute)