Skip to content

Commit

Permalink
docs: add markdown language specifiers for code blocks
Browse files Browse the repository at this point in the history
  • Loading branch information
jasonschroeder-sfdc authored and werkt committed Oct 24, 2023
1 parent dfa5937 commit f6459d1
Show file tree
Hide file tree
Showing 5 changed files with 77 additions and 59 deletions.
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ All commandline options override corresponding config settings.

Run via

```
docker run -d --rm --name buildfarm-redis -p 6379:6379 redis:5.0.9
```shell
$ docker run -d --rm --name buildfarm-redis -p 6379:6379 redis:5.0.9
redis-cli config set stop-writes-on-bgsave-error no
```

### Bazel Buildfarm Server

Run via

```
bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- <logfile> <configfile>
```shell
$ bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- <logfile> <configfile>

Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```
Expand All @@ -40,8 +40,8 @@ Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=

Run via

```
bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- <logfile> <configfile>
```shell
$ bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- <logfile> <configfile>

Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml

Expand All @@ -53,9 +53,9 @@ Ex: bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm

To use the example configured buildfarm with bazel (version 1.0 or higher), you can configure your `.bazelrc` as follows:

```
```shell
$ cat .bazelrc
build --remote_executor=grpc://localhost:8980
$ build --remote_executor=grpc://localhost:8980
```

Then run your build as you would normally do.
Expand All @@ -67,20 +67,20 @@ Buildfarm uses [Java's Logging framework](https://docs.oracle.com/javase/10/core
You can use typical Java logging configuration to filter these results and observe the flow of executions through your running services.
An example `logging.properties` file has been provided at [examples/logging.properties](examples/logging.properties) for use as follows:

```
bazel run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```shell
$ bazel run //src/main/java/build/buildfarm:buildfarm-server -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```

and

```
bazel run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
``` shell
$ bazel run //src/main/java/build/buildfarm:buildfarm-shard-worker -- --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties $PWD/examples/config.minimal.yml
```

To attach a remote debugger, run the executable with the `--debug=<PORT>` flag. For example:

```
bazel run //src/main/java/build/buildfarm:buildfarm-server -- --debug=5005 $PWD/examples/config.minimal.yml
```shell
$ bazel run //src/main/java/build/buildfarm:buildfarm-server -- --debug=5005 $PWD/examples/config.minimal.yml
```


Expand Down
32 changes: 16 additions & 16 deletions _site/docs/configuration/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ has_children: true

Minimal required:

```
```yaml
backplane:
redisUri: "redis://localhost:6379"
queues:
Expand Down Expand Up @@ -38,7 +38,7 @@ For an example configuration containing all of the configuration values, see `ex

Example:

```
```yaml
digestFunction: SHA1
defaultActionTimeout: 1800
maximumActionTimeout: 1800
Expand Down Expand Up @@ -79,7 +79,7 @@ worker:

Example:

```
```yaml
server:
instanceType: SHARD
name: shard
Expand All @@ -96,7 +96,7 @@ server:

Example:

```
```yaml
server:
grpcMetrics:
enabled: false
Expand All @@ -114,7 +114,7 @@ server:

Example:

```
```yaml
server:
caches:
directoryCacheMaxEntries: 10000
Expand All @@ -132,7 +132,7 @@ server:

Example:

```
```yaml
server:
admin:
deploymentEnvironment: AWS
Expand All @@ -151,14 +151,14 @@ server:

Example:

```
```yaml
server:
metrics:
publisher: log
logLevel: INFO
```

```
```yaml
server:
metrics:
publisher: aws
Expand Down Expand Up @@ -207,7 +207,7 @@ server:

Example:

```
```yaml
backplane:
type: SHARD
redisUri: "redis://localhost:6379"
Expand All @@ -224,7 +224,7 @@ backplane:

Example:

```
```yaml
backplane:
type: SHARD
redisUri: "redis://localhost:6379"
Expand Down Expand Up @@ -262,7 +262,7 @@ backplane:
| realInputDirectories | List of Strings, _external_ | | A list of paths that will not be subject to the effects of linkInputDirectories setting, may also be used to provide writable directories as input roots for actions which expect to be able to write to an input location and will fail if they cannot |
| gracefulShutdownSeconds | Integer, 0 | | Time in seconds to allow for operations in flight to finish when shutdown signal is received |

```
```yaml
worker:
port: 8981
publicName: "localhost:8981"
Expand All @@ -279,7 +279,7 @@ worker:

Example:

```
```yaml
worker:
capabilities:
cas: true
Expand All @@ -296,7 +296,7 @@ worker:

Example:

```
```yaml
worker:
sandboxSettings:
alwaysUse: true
Expand All @@ -313,7 +313,7 @@ worker:

Example:

```
```yaml
worker:
dequeueMatchSettings:
acceptEverything: true
Expand All @@ -333,7 +333,7 @@ worker:

Example:

```
```yaml
worker:
storages:
- type: FILESYSTEM
Expand Down Expand Up @@ -361,7 +361,7 @@ worker:
Example:
```
```yaml
worker:
executionPolicies:
- name: test
Expand Down
26 changes: 17 additions & 9 deletions _site/docs/execution/execution_policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This policy type specifies that a worker should prepend a single path, and a num

This example will use the buildfarm-provided executable `as-nobody`, which will upon execution demote itself to a `nobody` effective process owner uid, and perform an `execvp(2)` with the remaining provided program arguments, which will subsequently execute as a user that no longer matches the worker process.

```
```yaml
# default wrapper policy application
worker:
executionPolicies:
Expand Down Expand Up @@ -50,32 +50,37 @@ These wrappers are used for detecting actions that rely on time. Below is a dem
This addresses two problems in regards to an action's dependence on time. The 1st problem is when an action takes longer than it should because it's sleeping unnecessarily. The 2nd problem is when an action relies on time which causes it to eventually be broken on master despite the code not changing. Both problems are expressed below as unit tests. We demonstrate a time-spoofing mechanism (the re-writing of syscalls) which allows us to detect these problems generically over any action. The objective is to analyze builds for performance inefficiency and discover future instabilities before they occur.

### Issue 1 (slow test)
```

```bash
#!/bin/bash
set -euo pipefail
echo -n "testing... "
sleep 10;
echo "done"
```

The test takes 10 seconds to run on average.
```
bazel test --runs_per_test=10 --config=remote //cloud/buildfarm:sleep_test

```shell
$ bazel test --runs_per_test=10 --config=remote //cloud/buildfarm:sleep_test
//cloud/buildfarm:sleep_test PASSED in 10.2s
Stats over 10 runs: max = 10.2s, min = 10.1s, avg = 10.2s, dev = 0.0s
```

We can check for performance improvements by using the `skip-sleep` option.
```
bazel test --runs_per_test=10 --config=remote --remote_default_exec_properties='skip-sleep=true' //cloud/buildfarm:sleep_test

```shell
$ bazel test --runs_per_test=10 --config=remote --remote_default_exec_properties='skip-sleep=true' //cloud/buildfarm:sleep_test
//cloud/buildfarm:sleep_test PASSED in 1.0s
Stats over 10 runs: max = 1.0s, min = 0.9s, avg = 1.0s, dev = 0.0s
```

Now the test is 10x faster. If skipping sleep makes an action perform significantly faster without affecting its success rate, that would warrant further investigation into the action's implementation.

### Issue 2 (future failing test)
```

```bash
#!/bin/bash
set -euo pipefail
Expand All @@ -89,12 +94,15 @@ echo "Times change."
date
exit -1;
```

The test passes today, but will it pass tomorrow? Will it pass a year from now? We can find out by using the `time-shift` option.
```
bazel test --test_output=streamed --remote_default_exec_properties='time-shift=31556952' --config=remote //cloud/buildfarm:future_fail

```shell
$ bazel test --test_output=streamed --remote_default_exec_properties='time-shift=31556952' --config=remote //cloud/buildfarm:future_fail
INFO: Found 1 test target...
Times change.
Mon Sep 25 18:31:09 UTC 2023
//cloud/buildfarm:future_fail FAILED in 18.0s
```

Time is shifted to the year 2023 and the test now fails. We can fix the problem before others see it.
25 changes: 15 additions & 10 deletions _site/docs/execution/execution_properties.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,37 +76,42 @@ Despite being given 1 core, they see all of the cpus and decide to spawn that ma

**Standard Example:**
This test will succeed when env var TESTVAR is foobar, and fail otherwise.
```

```shell
#!/bin/bash
[ "$TESTVAR" = "foobar" ]
```
```
./bazel test \

```shell
$ ./bazel test \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
FAIL
```

```
./bazel test --remote_default_exec_properties='env-vars={"TESTVAR": "foobar"}' \
```shell
$ ./bazel test --remote_default_exec_properties='env-vars={"TESTVAR": "foobar"}' \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
PASS
```

**Template Example:**
If you give a range of cores, buildfarm has the authority to decide how many your operation actually claims. You can let buildfarm resolve this value for you (via [mustache](https://mustache.github.io/)).
```
```bash
#!/bin/bash
[ "$MKL_NUM_THREADS" = "1" ]
```
```
./bazel test \

```shell
$ ./bazel test \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
FAIL
```
```
./bazel test \

```shell
$ ./bazel test \
--remote_default_exec_properties='env-vars="MKL_NUM_THREADS": "{{limits.cpu.claimed}}"' \
--remote_executor=grpc://127.0.0.1:8980 --noremote_accept_cached --nocache_test_results \
//env_test:main
Expand Down
25 changes: 15 additions & 10 deletions _site/docs/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ Let's start with a bazel workspace with a single file to compile into an executa
Create a new directory for our workspace and add the following files:

`main.cc`:
```

```c
#include <iostream>

int main( int argc, char *argv[] )
Expand All @@ -35,7 +36,8 @@ int main( int argc, char *argv[] )
```
`BUILD`:
```
```starlark
cc_binary(
name = "main",
srcs = ["main.cc"],
Expand Down Expand Up @@ -118,15 +120,18 @@ That `2 remote` indicates that your compile and link ran remotely. Congratulatio
## Container Quick Start

To bring up a minimal buildfarm cluster, you can run:

```shell
$ ./examples/bf-run start
```
./examples/bf-run start
```

This will start all of the necessary containers at the latest version.
Once the containers are up, you can build with `bazel run --remote_executor=grpc://localhost:8980 :main`.

To stop the containers, run:
```
./examples/bf-run stop

```shell
$ ./examples/bf-run stop
```

## Next Steps
Expand All @@ -137,8 +142,8 @@ We've started our worker on the same host as our server, and also the same host

You can now easily launch a new Buildfarm cluster locally or in AWS using an open sourced [Buildfarm Manager](https://github.com/80degreeswest/bfmgr).

```
wget https://github.com/80degreeswest/bfmgr/releases/download/1.0.7/bfmgr-1.0.7.jar
java -jar bfmgr-1.0.7.jar
Navigate to http://localhost
```shell
$ wget https://github.com/80degreeswest/bfmgr/releases/download/1.0.7/bfmgr-1.0.7.jar
$ java -jar bfmgr-1.0.7.jar
$ open http://localhost
```

0 comments on commit f6459d1

Please sign in to comment.