Skip to content

Commit

Permalink
docs(docker compose): update from docker-compose
Browse files Browse the repository at this point in the history
  • Loading branch information
lmeyerov committed Oct 28, 2024
1 parent 2ebd781 commit 094ae69
Show file tree
Hide file tree
Showing 17 changed files with 54 additions and 53 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
* Reorganized documentation into thematic areas
* Update RAPIDS base image
* Update RAPIDS Python env name to `base` from `rapids`
* Updated most docker compose command references to `docker compose` from `docker-compose`

### Infra

Expand Down
4 changes: 2 additions & 2 deletions docs/app-config/configure-investigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ Via `data/pivot-db/config/config.json`:

After setting these, restart your server:

* Full: `[email protected] : /var/graphistry $ docker-compose stop && docker-compose up -d`
* Pivot: `[email protected] : /var/graphistry $ docker-compose stop nginx pivot && docker-compose up -d`
* Full: `[email protected] : /var/graphistry $ docker compose stop && docker compose up -d`
* Pivot: `[email protected] : /var/graphistry $ docker compose stop nginx pivot && docker compose up -d`


# Schema
Expand Down
8 changes: 4 additions & 4 deletions docs/app-config/configure-ontology.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ See below for the list of built-in types they map to.
## Define custom ontologies

1. Edit `data/investigations/config/config.json` as per below
2. Restart docker service `pivot`: `docker-compose restart pivot`
2. Restart docker service `pivot`: `docker compose restart pivot`

Generally, you can limit the amount of work by mapping custom column names to built-in types, and thereby reuse their preconfigured settings.

Expand Down Expand Up @@ -73,7 +73,7 @@ For example, to create a new node type `ip`,

2. Restart the pivot service:

```[email protected]:/var/graphistry $ docker-compose stop pivot nginx && docker-compose up -d```
```[email protected]:/var/graphistry $ docker compose stop pivot nginx && docker compose up -d```

### Override default node/edge titles

Expand Down Expand Up @@ -128,7 +128,7 @@ For example, to recognize `src_ip` and `dest_ip` columns as both generating `ip`

2. Restart the pivot service:
```
[email protected]:/var/graphistry $ docker-compose stop pivot nginx && docker-compose up -d
[email protected]:/var/graphistry $ docker compose stop pivot nginx && docker compose up -d
```

## Built-in types
Expand Down Expand Up @@ -190,7 +190,7 @@ You can put any regular expression here:
Graphistry tries to detect syntax error, and upon one, logs the error and stops. To see what is going on:

`docker ps` <- see if `pivot` is unhealthy or in a restart loop
`docker-compose logs pivot` <- see the precise error message
`docker compose logs pivot` <- see the precise error message

2. Satisfactory configuration

Expand Down
8 changes: 4 additions & 4 deletions docs/app-config/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ For visualizations to be embeddable in different origin sites, enable `COOKIE_SE
COOKIE_SAMESITE=None
```

... then restart: `docker-compose up -d --force-recreate --no-deps nexus`
... then restart: `docker compose up -d --force-recreate --no-deps nexus`


### Setup free Automatic TLS
Expand Down Expand Up @@ -182,8 +182,8 @@ Custom TLS setups often fail due to the certificate, OS, network, Caddy config,
* Test the certificate
* Test a [standalone Caddy static file server](https://www.baty.net/2018/using-caddy-for-serving-static-content/)
* ... Including on another box, if OS/network issues are suspected
* Check the logs of `docker-compose logs -f -t caddy nginx`
* Test whether the containers are up and ports match via `docker-compose ps`, `curl`, and `curl` from within a docker container (so within the docker network namespace)
* Check the logs of `docker compose logs -f -t caddy nginx`
* Test whether the containers are up and ports match via `docker compose ps`, `curl`, and `curl` from within a docker container (so within the docker network namespace)

If problems persist, please reach out to your Graphistry counterparts. Additional workarounds are possible.

Expand Down Expand Up @@ -281,7 +281,7 @@ SPLUNK_HOST=...

2. Restart `graphistry`, or at least the `pivot` service:

`docker-compose stop && docker-compose up -d` or `docker-compose stop nginx pivot && docker-compose up -d`
`docker compose stop && docker compose up -d` or `docker compose stop nginx pivot && docker compose up -d`

3. Test

Expand Down
18 changes: 9 additions & 9 deletions docs/commands.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Top commands

Graphistry supports advanced command-line administration via standard `docker-compose`, `.yml` / `.env` files, and `caddy` reverse-proxy configuration.
Graphistry supports advanced command-line administration via standard `docker compose`, `.yml` / `.env` files, and `caddy` reverse-proxy configuration.

## Login to server

Expand All @@ -18,17 +18,17 @@ All likely require `sudo`. Run from where your `docker-compose.yml` file is loca
| TASK | COMMAND | NOTES |
|--: |:--- |:--- |
| **Install** | `docker load -i containers.tar.gz` | Install the `containers.tar.gz` Graphistry release from the current folder. You may need to first run `tar -xvvf my-graphistry-release.tar.gz`. |
| **Start <br>interactive** | `docker-compose up` | Starts Graphistry, close with ctrl-c |
| **Start <br>daemon** | `docker-compose up -d` | Starts Graphistry as background process |
| **Start <br>namespaced (concurrent)** | `docker-compose -p my_unique_namespace up` | Starts Graphistry in a specific namespace. Enables running multiple independent instances of Graphistry. NOTE: Must modify Caddy service in `docker-compose.yml` to use non-conflicting public ports, and likewise change global volumes to be independent. |
| **Stop** | `docker-compose stop` | Stops Graphistry |
| **Start <br>interactive** | `docker compose up` | Starts Graphistry, close with ctrl-c |
| **Start <br>daemon** | `docker compose up -d` | Starts Graphistry as background process |
| **Start <br>namespaced (concurrent)** | `docker compose -p my_unique_namespace up` | Starts Graphistry in a specific namespace. Enables running multiple independent instances of Graphistry. NOTE: Must modify Caddy service in `docker-compose.yml` to use non-conflicting public ports, and likewise change global volumes to be independent. |
| **Stop** | `docker compose stop` | Stops Graphistry |
| **Restart (soft)** | `docker restart <CONTAINER>` | Soft restart. May also need to restart service `nginx`. |
| **Restart (hard)** | `docker up -d --force-recreate --no-deps <CONTAINER>` | Restart with fresh state. May also need to restart service `nginx`. |
| **Reset** | `docker-compose down -v && docker-compose up -d` | Stop Graphistry, remove all internal state (including the user account database!), and start fresh . |
| **Status** | `docker-compose ps`, `docker ps`, and `docker status` | Status: Uptime, healthchecks, ... |
| **Reset** | `docker compose down -v && docker compose up -d` | Stop Graphistry, remove all internal state (including the user account database!), and start fresh . |
| **Status** | `docker compose ps`, `docker ps`, and `docker status` | Status: Uptime, healthchecks, ... |
| **GPU Status** | `nvidia-smi` | See GPU processes, compute/memory consumption, and driver. Ex: `watch -n 1.5 nvidia-smi`. Also, `docker run --rm -it nvidia/cuda:11.5.0-base-ubuntu20.04 nvidia-smi` for in-container test. |
| **1.0 API Key** | docker-compose exec streamgl-vgraph-etl curl "http://0.0.0.0:8080/api/internal/provision?text=MYUSERNAME" | Generates API key for a developer or notebook user (1.0 API is deprecated)|
| **Logs** | `docker-compose logs <CONTAINER>` | Ex: Watch all logs, starting with the 20 most recent lines: `docker-compose logs -f -t --tail=20 forge-etl-python` . You likely need to switch Docker to use the local json logging driver by deleting the two default managed Splunk log driver options in `/etc/docker/daemon.json` and then restarting the `docker` daemon (see below). |
| **1.0 API Key** | docker compose exec streamgl-vgraph-etl curl "http://0.0.0.0:8080/api/internal/provision?text=MYUSERNAME" | Generates API key for a developer or notebook user (1.0 API is deprecated)|
| **Logs** | `docker compose logs <CONTAINER>` | Ex: Watch all logs, starting with the 20 most recent lines: `docker compose logs -f -t --tail=20 forge-etl-python` . You likely need to switch Docker to use the local json logging driver by deleting the two default managed Splunk log driver options in `/etc/docker/daemon.json` and then restarting the `docker` daemon (see below). |
| **Create Users** | Use Admin Panel (see [Create Users](tools/user-creation.md)) or `etc/scripts/rest` |
| **Restart Docker Daemon** | `sudo service docker restart` | Use when changing `/etc/docker/daemon.json`, ... |
| **Jupyter shell**| `docker exec -it -u root graphistry_notebook_1 bash` then `source activate rapids` | Use for admin tasks like global package installs |
2 changes: 1 addition & 1 deletion docs/debugging/debug-faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Visualization page never returns or Nginx "504 Gateway Time-out" due to services
* Often with first-ever container launch
* Likely within 60s of launch
* Can happen even after static homepage loads
* In `docker-compose up` logs (or `docker logs ubuntu_central_1`):
* In `docker compose up` logs (or `docker logs ubuntu_central_1`):
* "Error: Server at maximum capacity...
* "Error: Too many users...
* "Error while assigning...
Expand Down
2 changes: 1 addition & 1 deletion docs/debugging/performance-tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ See also [deployment planning](../planning/deployment-planning.md) and [hw/sw pl
* Check for both memory compute, and network consumption, and by which process
* Check logs for potential errors
* System: Standard OS logs
* App: `docker-compose logs`
* App: `docker compose logs`
* Log level impacts performance
* TRACE: Slow due to heavy CPU <> GPU traffic
* DEBUG: Will cause large log volumes that require rotation
Expand Down
4 changes: 2 additions & 2 deletions docs/install/cloud/aws_marketplace.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,9 @@ Many `ssh` clients may require you to first run `chmod 400 my_key.pem` or `chmod

Graphistry leverages `docker-compose` and the AWS Marketplace AMI preconfigures the `nvidia` runtime for `docker`.

```
```bash
cd ~/graphistry
sudo docker-compose ps
sudo docker compose ps
```

=>
Expand Down
2 changes: 1 addition & 1 deletion docs/install/cloud/azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ For steps involving an IP address, see needed IP value at Azure console in `Over

Install docker-compose:

```
```bash
sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```
Expand Down
2 changes: 1 addition & 1 deletion docs/install/on-prem/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Note: In previous versions (< `v2.35`), the file was `containers.tar`
**2. Launch** from the folder with `docker-compose.yml` if not already up, and likely using `sudo`:

```bash
docker-compose up -d
docker compose up -d
```

Note: Takes 1-3 min, and around 5 min, `docker ps` should report all services as `healthy`
Expand Down
4 changes: 2 additions & 2 deletions docs/install/on-prem/manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Skip almost all of these steps by instead running through [AWS Marketplace](../c

* **Start from an Nvidia instace**
<br>You can skip most of the steps by starting with an Nvidia NGC or Tensorflow instance.
* These still typically require installing `docker-compose` (and testing it), setting `/etc/docker/daemon.json` to default to the `nvidia-docker` runtime, and restarting `docker` (and testing it). See end of [RHEL 7.6](rhel_7_6_setup.md) and [Ubuntu 18.04 LTS](ubuntu_18_04_lts_setup.md) sample scripts for install and test instruction.
* These still typically require installing `docker compose` (and testing it), setting `/etc/docker/daemon.json` to default to the `nvidia-docker` runtime, and restarting `docker` (and testing it). See end of [RHEL 7.6](rhel_7_6_setup.md) and [Ubuntu 18.04 LTS](ubuntu_18_04_lts_setup.md) sample scripts for install and test instruction.
* **Start from raw Ubuntu/RHEL**
<br>You can build from scratch by picking a fully unconfigured starting point and following the [RHEL 7.6](rhel_7_6_setup.md) and [Ubuntu 18.04 LTS](ubuntu_18_04_lts_setup.md) On-Prem Sample instructions. Contact Graphistry staff for automation script assistance if also applicable.

Expand Down Expand Up @@ -84,6 +84,6 @@ docker load -i containers.tar

## 5. Start

Launch with `docker-compose up`, and stop with `ctrl-c`. To start as a background daemon, use `docker-compose up -d`.
Launch with `docker compose up`, and stop with `ctrl-c`. To start as a background daemon, use `docker compose up -d`.

Congratulations, you have installed Graphistry!
2 changes: 1 addition & 1 deletion docs/install/on-prem/vGPU.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ A *baremetal OS* (no hypervisor) or *passthrough driver* (hypervisor with non-vG
* Graphistry already automatically uses all GPUs exposed to it, primarily for scaling to more user sessions
* New APIs are starting to use multi-GPUs for acceleration as well
* Multiple Graphistry installs
* You can launch concurrent instances of Graphistry using docker: `docker-compose up -p my_unique_namespace_123`
* You can launch concurrent instances of Graphistry using docker: `docker compose up -p my_unique_namespace_123`
* You can configure docker to use different GPUs or share the same ones
* Isolate Graphistry from other GPU software
* Docker allows picking which GPUs + CPUs are used
Expand Down
26 changes: 13 additions & 13 deletions docs/install/testing-an-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Most of the testing and inspection is standard for Docker-based web apps: `docke

* To test your base Docker environment for GPU RAPIDS, see the in-depth GPU testing section below.

* For logs throughout your session, you can run `docker-compose logs -f -t --tail=1` and `docker-compose logs -f -t --tail=1 SOME_SERVICE_NAME` to see the effects of your activities. Modify `custom.env` to increase `GRAPHISTRY_LOG_LEVEL` and `LOG_LEVEL` to `DEBUG` for increased logging, and `/etc/docker/daemon.json` to use log driver `json-file` for local logs.
* For logs throughout your session, you can run `docker compose logs -f -t --tail=1` and `docker compose logs -f -t --tail=1 SOME_SERVICE_NAME` to see the effects of your activities. Modify `custom.env` to increase `GRAPHISTRY_LOG_LEVEL` and `LOG_LEVEL` to `DEBUG` for increased logging, and `/etc/docker/daemon.json` to use log driver `json-file` for local logs.

NOTE: Below tests use the deprecated 1.0 REST upload API.

Expand Down Expand Up @@ -54,9 +54,9 @@ f0bc21b5bda2 compose_redis_1 0.05% 6.781MiB / 31.27GiB
| Jupyter notebooks | `notebook` (heavy) |
| Dashboards | `graph-app-kit-public`, `graph-app-kit-private` |

* It is safe to reset any individual container **except** `postgres`, which is stateful: `docker-compose up -d --force-recreate --no-deps <some_stateless_services>`
* It is safe to reset any individual container **except** `postgres`, which is stateful: `docker compose up -d --force-recreate --no-deps <some_stateless_services>`

* For any unhealthy container, such as stuck in a restart loop, check `docker-compose logs -f -t --tail=1000 that_service`. To further diagnose, potentially increase the system log level (edit `data/config/custom.env` to have `LOG_LEVEL=DEBUG`, `GRAPHISTRY_LOG_LEVEL=DEBUG`) and recreate + restart the unhealthy container
* For any unhealthy container, such as stuck in a restart loop, check `docker compose logs -f -t --tail=1000 that_service`. To further diagnose, potentially increase the system log level (edit `data/config/custom.env` to have `LOG_LEVEL=DEBUG`, `GRAPHISTRY_LOG_LEVEL=DEBUG`) and recreate + restart the unhealthy container

* Check `data/config/custom.env` has system-local keys (ex: `STREAMGL_SECRET_KEY`) with fallback to `.env`

Expand All @@ -77,7 +77,7 @@ f0bc21b5bda2 compose_redis_1 0.05% 6.781MiB / 31.27GiB
* If points still do not load, or appear and freeze, likely issues with GPU init (driver) or websocket (firewall)
* Can also be because preloaded datasets are unavailable: not provided, or externally mounted data sources
* In this case, use ETL test, and ensure clustering runs for a few seconds (vs. just initial pageload)
* Check `docker-compose logs -f -t --tail=1` and `docker ps` in case config or GPU driver issues, especially for GPU services listed above
* Check `docker compose logs -f -t --tail=1` and `docker ps` in case config or GPU driver issues, especially for GPU services listed above
* Upon failures, see below section on GPU testing

## 4a. Test 1.0 API uploads, Jupyter, and the PyGraphistry client API
Expand All @@ -87,7 +87,7 @@ Do via notebook if possible, else `curl`
* Get a 1.0 API key by logging into your user's dashboard, or generating a new one using host access:

```
docker-compose exec central curl -s http://localhost:10000/api/internal/provision?text=MYUSERNAME
docker compose exec central curl -s http://localhost:10000/api/internal/provision?text=MYUSERNAME
```

* Install PyGraphistry and check recent version number (Latest: https://pypi.org/project/graphistry/), or use the provided `/notebook` install:
Expand Down Expand Up @@ -152,7 +152,7 @@ If you cannot do **3a**, test from the host via `curl` or `wget`:
Login and get the API key from your dashboard homepage, or run the following:

```
docker-compose exec central curl -s http://localhost:10000/api/internal/provision?text=MYUSERNAME
docker compose exec central curl -s http://localhost:10000/api/internal/provision?text=MYUSERNAME
```

* Upload your 1.0 API data using the key
Expand Down Expand Up @@ -190,8 +190,8 @@ Nodes: x y
* Set each config in one go so you can test more quickly, vs start/stop.
* Run
```
docker-compose stop
docker-compose up
docker compose stop
docker compose up
```


Expand All @@ -203,7 +203,7 @@ docker-compose up
#### 5c.ii Connector - Splunk

* Edit `data/custom/custom.env` for `SPLUNK_HOST`, `SPLUNK_PORT`, `SPLUNK_USER`, `SPLUNK_KEY`
* Restart the `/pivot` service: `docker-compose restart pivot`
* Restart the `/pivot` service: `docker compose restart pivot`
* In `/pivot/connectors`, the `Live Connectors` should list `Splunk`, and clicking `Status` will test logging in
* In `Investigations and Templates`, create a new investigation:
* Create and run one pivot:
Expand Down Expand Up @@ -244,18 +244,18 @@ Cloud:
* In Route53/DNS: Assign a domain to your IP, ex: `mytest.graphistry.com`
* Modify `data/config/Caddyfile` to use your domain name
* Unlikely: If needed, run `DOMAIN=my.site.com ./scripts/letsencrypt.sh` and `./gen_dhparam.sh`
* Restart `docker-compose restart caddy`, check pages load
* Restart `docker compose restart caddy`, check pages load
* Try a notebook upload with `graphistry.register(...., protocol='https')`

## 7. Quick Testing and Test GPU

Most of the below tests can be automatically run by `cd etc/scripts && ./test-gpu.sh`:
* Checks `nvidia-smi` works in your OS
* Checks `nvidia-smi` works in Docker, including runtime defaults used by `docker-compose`
* Checks `nvidia-smi` works in Docker, including runtime defaults used by `docker compose`
* Checks Nvidia RAPIDS can successfully create CUDA contexts and run a simple on-GPU compute and I/O task of `1 + 1 == 2`

`docker ps` reports no "unhealthy", "restarting", or prolonged "starting" services:
* check `docker-compose logs`, `docker-compose logs <service>`, `docker-compose logs -f -t --tail=100 <service>`
* check `docker compose logs`, `docker compose logs <service>`, `docker compose logs -f -t --tail=100 <service>`
* unhealthy `streamgl-gpu`, `forge-etl-python` on start: likely GPU driver issue
* GPU is not the default runtime in `/etc/docker/deamon.json` (`docker info | grep Default`)
* `OpenlCL` Initialization error: GPU drivers insufficently setup
Expand All @@ -279,7 +279,7 @@ Reports available GPUs (public base image) <- tests Graphistry container CUDA ve
``docker run --rm -it --entrypoint=/bin/bash graphistry/etl-server-python:`cat VERSION`-11.8 -c "source activate base && python3 -c \"import cudf; print(cudf.DataFrame({'x': [0,1,2]})['x'].sum())\""``
Tests Nvidia RAPIDS (VERSION is your Graphistry version)
* `docker run graphistry/cljs:1.1 npm test` reports success <-- tests driver versioning, may be a faulty test however
* If running in a hypervisor, ensure `RMM_ALLOCATOR=default` in `data/config/custom.env`, and check the startup logs of `docker-compose logs -f -t --tail=1000 forge-etl-python` that `cudf` / `cupy` are respecting that setting (`LOG_LEVEL=INFO`)
* If running in a hypervisor, ensure `RMM_ALLOCATOR=default` in `data/config/custom.env`, and check the startup logs of `docker compose logs -f -t --tail=1000 forge-etl-python` that `cudf` / `cupy` are respecting that setting (`LOG_LEVEL=INFO`)
* Health checks
* CLI: Check `docker ps` for per-service status, may take 1-2min for services to connect and warm up
* Per-service checks run every ~30s after a ~1min initialization delay, with several retries before capped restart
Expand Down
Loading

0 comments on commit 094ae69

Please sign in to comment.