From fc0230c98ea2245464e28c445ebb6906670b5f4c Mon Sep 17 00:00:00 2001 From: Zoe Braiterman Date: Wed, 17 Jul 2024 20:05:39 -0400 Subject: [PATCH] Update readme (#617) --- README.md | 119 +++++++++++++++++++++++++++++------------------------- 1 file changed, 64 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index d8cf334a..e1fec45f 100644 --- a/README.md +++ b/README.md @@ -4,16 +4,16 @@ This docker image provides a simple way to run stellar-core and horizon locally **Looking for instructions for how to run stellar-core or horizon in production? Take a look at the docs [here](https://developers.stellar.org/docs/run-core-node/).** -This image provides a default, non-validating, ephemeral configuration that should work for most developers. By configuring a container using this image with a host-based volume (described below in the "Usage" section) an operator gains access to full configuration customization and persistence of data. +This image provides a default, non-validating, ephemeral configuration that should work for most developers. By configuring a container using this image with a host-based volume (described below in the "Usage" section) an operator gains access to full configuration customization and persistence of data. The image uses the following software: -- Postgresql 12 is used for storing both stellar-core and horizon data +- [PostgreSQL](https://www.postgresql.org) 12 is used for storing both stellar-core and horizon data. - [stellar-core](https://github.com/stellar/stellar-core) - [horizon](https://github.com/stellar/go/tree/master/services/horizon) - [friendbot](https://github.com/stellar/go/tree/master/services/friendbot) - [soroban-rpc](https://github.com/stellar/soroban-rpc/tree/main/cmd/soroban-rpc) -- Supervisord is used from managing the processes of the services above +- [Supervisord](http://supervisord.org) is used from managing the processes of the above services. ## Usage @@ -21,11 +21,11 @@ To use this project successfully, you should first decide a few things: First, decide whether you want your container to be part of the public, production Stellar network (referred to as the _pubnet_) or the test network (called testnet) that we recommend you use while developing software because you need not worry about losing money on the testnet. Alternatively, choose to run a local network (called local) which allows you to run your own acellerated private Stellar network for testing. -Next, you must decide whether you will use a docker volume or not. When not using a volume, we say that the container is in _ephemeral mode_, that is, nothing will be persisted between runs of the container. _Persistent mode_ is the alternative, which should be used in the case that you need to either customize your configuration (such as to add a validation seed) or would like avoid a slow catchup to the Stellar network in the case of a crash or server restart. We recommend persistent mode for anything besides a development or test environment. +Next, you must decide whether you will use a docker volume or not. When not using a volume, we say that the container is in _ephemeral mode_, that is, nothing will be persisted between runs of the container. _Persistent mode_ is the alternative, which should be used in the case that you need to either customize your configuration (such as to add a validation seed) or would like avoid a slow catchup to the Stellar network in the case of a crash or server restart. We recommend persistent mode for anything besides a development or test environment. -Finally, you must decide what ports to expose. The software in these images listen on 4 ports, each of which you may or may not want to expose to the network your host system is connected to. A container that exposes no ports isn't very useful, so we recommend at a minimum you expose the horizon http port. See the "Ports" section below for a more nuanced discussion regarding the decision about what ports to expose. +Finally, you must decide what ports to expose. The software in these images listen on 4 ports, each of which you may or may not want to expose to the network your host system is connected to. A container that exposes no ports isn't very useful, so we recommend at a minimum you expose the horizon http port. See the "Ports" section below for a more nuanced discussion regarding the decision about what ports to expose. -After deciding on the questions above, you can setup your container. Please refer to the appropriate section below based upon what mode you will run the container in. +After deciding on the questions above, you can setup your container. Please refer to the appropriate section below based upon what mode you will run the container in. ### Network Options @@ -54,12 +54,13 @@ In local network mode, you can optionally pass: - `--protocol-version {version}` to run a specific protocol version (defaults to latest version). - `--limits {limits}` to configure specific Soroban resource limits to one of: - - `default` leaves limits set extremely low which is stellar-core's default configuration - - `testnet` sets limits to match those used on testnet (the default quickstart configuration) - - `unlimited` sets limits to the maximum resources that can be configured + - `default` leaves limits set extremely low which is stellar-core's default configuration + - `testnet` sets limits to match those used on testnet (the default quickstart configuration) + - `unlimited` sets limits to the maximum resources that can be configured -__Note: The `--enable` options behaves differently in local network mode, see [Service Options](#service-otions) for more details.__ +**Note: The `--enable` options behaves differently in local network mode, see [Service Options](#service-otions) for more details.** The network passphrase of the network defaults to: + ``` Standalone Network ; February 2017 ``` @@ -67,6 +68,7 @@ Standalone Network ; February 2017 Set the network passphrase in the SDK or tool you're using. If an incorrect network passphrase is used in clients signing transactions, the transactions will fail with a bad authentication error. The root account of the network is fixed to: + ``` Public Key: GBZXN7PIRZGNMHGA7MUUUF4GWPY5AYPV6LY4UV2GL6VJGIQRXFDNMADI Secret Key: SC5O7VZUXDJ6JBDSZ74DSERXL7W3Y5LTOAMRF7RQRL3TAGAPS7LUVG3L @@ -77,7 +79,7 @@ The root account is derived from the network passphrase and if the network passp In local network mode a ledger occurs every one second and so transactions are finalized faster than on deployed networks. -*Note*: The local network in this container is not suitable for any production use as it has a fixed root account. Any private network intended for production use would also required a unique network passphrase. +_Note_: The local network in this container is not suitable for any production use as it has a fixed root account. Any private network intended for production use would also required a unique network passphrase. ### Service Options @@ -94,7 +96,9 @@ To run only select services, simply specify only those services. For example, to ``` --enable rpc ``` -__Note: In `--local` mode the `core` service always runs no matter what options are passed, the `friendbot` faucet service runs whenever `horizon` is running, and `horizon` is run when `rpc` is requested so that friendbot is available.__ + +**Note: In `--local` mode the `core` service always runs no matter what options are passed, the `friendbot` faucet service runs whenever `horizon` is running, and `horizon` is run when `rpc` is requested so that friendbot is available.** + ### Faucet (Friendbot) Stellar development/test networks use friendbot as a faucet for the native asset. @@ -102,6 +106,7 @@ Stellar development/test networks use friendbot as a faucet for the native asset When running in local, testnet, and futurenet modes friendbot is available on `:8000/friendbot` and can be used to fund a new account. For example: + ``` $ curl http://localhost:8000/friendbot?addr=G... ``` @@ -113,6 +118,7 @@ _Note: In local mode a local friendbot is running. In testnet and futurenet mode The RPC Server will be avaialble on port 8000 of the container, and the base URL path for Soroban RPC will be `http://:8000/rpc`. This endpoint uses [JSON-RPC](https://www.jsonrpc.org/specification) protocol. Refer to example usages in [soroban-example-dapp](https://github.com/stellar/soroban-example-dapp). To enable soroban rpc admin endpoint for access to metrics and [Go pprof (profiling)](https://pkg.go.dev/net/http/pprof), include the `--enable-soroban-rpc-admin-endpoint` flag, the HTTP endpoint will be listening on container port 6061, which can be exposed with standard docker port rule `-p "6061:6061"`, the published endpoints are: + ``` http://:6061/metrics http://:6061/debug/pprof/ @@ -139,6 +145,7 @@ You can deploy the quickstart image to DigitalOcean by clicking the button below After clicking the button above, the deployment can be configured to deploy a different variant of the image, or join a different network such as `testnet` or `futurenet` by changing environment variables. Some example configurations that can be used are: + - Local network matching pubnet: `IMAGE`: `stellar/quickstart:latest` `NETWORK`: `local` @@ -149,7 +156,7 @@ Some example configurations that can be used are: `IMAGE`: `stellar/quickstart:testnet` `NETWORK`: `testnet` -*Disclaimer*: The DigitalOcean server is publicly accessible on the Internet. Do not put sensitive information on the network that you would not want someone else to know. Anyone with access to the network will be able to use the root account above. +_Disclaimer_: The DigitalOcean server is publicly accessible on the Internet. Do not put sensitive information on the network that you would not want someone else to know. Anyone with access to the network will be able to use the root account above. ### Building Custom Images @@ -161,14 +168,15 @@ built with. - `TAG`: The docker tag to assign to the build. Default `dev`. - `CORE_REF`: The git reference of stellar-core to build. - `CORE_CONFIGURE_FLAGS`: The `CONFIGURE_FLAGS` to configure the stellar-core -build with. Typically include `--disable-tests`, and to enable the next protocol -version that is still in development, add -`--enable-next-protocol-version-unsafe-for-production`. + build with. Typically include `--disable-tests`, and to enable the next protocol + version that is still in development, add + `--enable-next-protocol-version-unsafe-for-production`. - `HORIZON_REF`: The git reference of stellar-horizon to build. - `FRIENDBOT_REF`: The git reference of stellar-friendbot to build. - `SOROBAN_RPC_REF`: The git reference of soroban-rpc to build. For example: + ``` make build \ TAG=future \ @@ -181,24 +189,23 @@ make build \ ### Background vs. Interactive containers -Docker containers can be run interactively (using the `-it` flags) or in a detached, background state (using the `-d` flag). Many of the example commands below use the `-it` flags to aid in debugging but in many cases you will simply want to run a node in the background. It's recommended that you use the use [the tutorials at docker](https://docs.docker.com/engine/tutorials/usingdocker/) to familiarize yourself with using docker. +Docker containers can be run interactively (using the `-it` flags) or in a detached, background state (using the `-d` flag). Many of the example commands below use the `-it` flags to aid in debugging but in many cases you will simply want to run a node in the background. It's recommended that you use the use [the tutorials at docker](https://docs.docker.com/engine/tutorials/usingdocker/) to familiarize yourself with using docker. ### Ephemeral mode -Ephermeral mode is provided to support development and testing environments. Every time you start a container in ephemeral mode, the database starts empty and a default configuration file will be used for the appropriate network. +Ephermeral mode is provided to support development and testing environments. Every time you start a container in ephemeral mode, the database starts empty and a default configuration file will be used for the appropriate network. -Starting an ephemeral node is simple, just craft a `docker run` command to launch the appropriate image but *do not mount a volume*. To craft your docker command, you need the network name you intend to run against and the flags to expose the ports your want available (See the section named "Ports" below to learn about exposing ports). Thus, launching a testnet node while exposing horizon would be: +Starting an ephemeral node is simple, just craft a `docker run` command to launch the appropriate image but _do not mount a volume_. To craft your docker command, you need the network name you intend to run against and the flags to expose the ports your want available (See the section named "Ports" below to learn about exposing ports). Thus, launching a testnet node while exposing horizon would be: ```shell $ docker run --rm -it -p "8000:8000" --name stellar stellar/quickstart --testnet ``` -As part of launching, an ephemeral mode container will generate a random password for securing the postgresql service and will output it to standard out. You may use this password (provided you have exposed the postgresql port) to access the running postgresql database (See the section "Accessing Databases" below). - +As part of launching, an ephemeral mode container will generate a random password for securing the postgresql service and will output it to standard out. You may use this password (provided you have exposed the postgresql port) to access the running postgresql database (See the section "Accessing Databases" below). ### Persistent mode -In comparison to ephemeral mode, persistent mode is more complicated to operate, but also more powerful. Persistent mode uses a mounted host volume, a directory on the host machine that is exposed to the running docker container, to store all database data as well as the configuration files used for running services. This allows you to manage and modify these files from the host system. +In comparison to ephemeral mode, persistent mode is more complicated to operate, but also more powerful. Persistent mode uses a mounted host volume, a directory on the host machine that is exposed to the running docker container, to store all database data as well as the configuration files used for running services. This allows you to manage and modify these files from the host system. Note that there is no guarantee that the organization of the files of the volume will remain consistent between releases of the image, that occur on every commit to the stellar/quickstart repository. At anytime new files may be added, old files removed, or dependencies and references between them changed. For this reason persistent mode is primarily intended for running short lived test instances for development. If consistency is required over any period of time use [image digest references] to pin to a specific build. @@ -210,9 +217,9 @@ Starting a persistent mode container is the same as the ephemeral mode with one $ docker run --rm -it -p "8000:8000" -v "/home/scott/stellar:/opt/stellar" --name stellar stellar/quickstart --testnet ``` -The `-v` option in the example above tells docker to mount the host directory `/home/scott/stellar` into the container at the `/opt/stellar` path. You may customize the host directory to any location you like, simply make sure to use the same value every time you launch the container. Also note: an absolute directory path is required. The second portion of the volume mount (`/opt/stellar`) should never be changed. This special directory is checked by the container to see if it is mounted from the host system which is used to see if we should launch in ephemeral or persistent mode. +The `-v` option in the example above tells docker to mount the host directory `/home/scott/stellar` into the container at the `/opt/stellar` path. You may customize the host directory to any location you like, simply make sure to use the same value every time you launch the container. Also note: an absolute directory path is required. The second portion of the volume mount (`/opt/stellar`) should never be changed. This special directory is checked by the container to see if it is mounted from the host system which is used to see if we should launch in ephemeral or persistent mode. -Upon launching a persistent mode container for the first time, the launch script will notice that the mounted volume is empty. This will trigger an interactive initialization process to populate the initial configuration for the container. This interactive initialization adds some complications to the setup process because in most cases you won't want to run the container interactively during normal operation, but rather in the background. We recommend the following steps to setup a persistent mode node: +Upon launching a persistent mode container for the first time, the launch script will notice that the mounted volume is empty. This will trigger an interactive initialization process to populate the initial configuration for the container. This interactive initialization adds some complications to the setup process because in most cases you won't want to run the container interactively during normal operation, but rather in the background. We recommend the following steps to setup a persistent mode node: 1. Run an interactive session of the container at first, ensuring that all services start and run correctly. 2. Shut down the interactive container (using Ctrl-C). @@ -220,53 +227,53 @@ Upon launching a persistent mode container for the first time, the launch script ## Regarding user accounts -Managing UIDs between a docker container and a host volume can be complicated. At present, this image simply tries to create a UID that does not conflict with the host system by using a preset UID: 10011001. Currently there is no way to customize this value. All data produced in the host volume be owned by 10011001. If this UID value is inappropriate for your infrastructure we recommend you fork this project and do a find/replace operation to change UIDs. We may improve this story in the future if enough users request it. +Managing UIDs between a docker container and a host volume can be complicated. At present, this image simply tries to create a UID that does not conflict with the host system by using a preset UID: 10011001. Currently there is no way to customize this value. All data produced in the host volume be owned by 10011001. If this UID value is inappropriate for your infrastructure we recommend you fork this project and do a find/replace operation to change UIDs. We may improve this story in the future if enough users request it. ## Ports The image exposes one main port through which services provide their APIs: -| Port | Service | Description | -|-------|---------------------------------|----------------------| -| 8000 | horizon, soroban-rpc, friendbot | main http port | +| Port | Service | Description | +| ---- | ------------------------------- | -------------- | +| 8000 | horizon, soroban-rpc, friendbot | main http port | The image also exposes a few other ports that most developers do not need, but area available: -| Port | Service | Description | -|-------|---------------------------------|----------------------| -| 5432 | postgresql | database access port | -| 6060 | horizon | admin port | -| 6061 | soroban-rpc | admin port | -| 11625 | stellar-core | peer node port | -| 11626 | stellar-core | main http port | -| 11725 | stellar-core (horizon) | peer node port | -| 11726 | stellar-core (horizon) | main http port | -| 11825 | stellar-core (soroban-rpc) | peer node port | -| 11826 | stellar-core (soroban-rpc) | main http port | +| Port | Service | Description | +| ----- | -------------------------- | -------------------- | +| 5432 | postgresql | database access port | +| 6060 | horizon | admin port | +| 6061 | soroban-rpc | admin port | +| 11625 | stellar-core | peer node port | +| 11626 | stellar-core | main http port | +| 11725 | stellar-core (horizon) | peer node port | +| 11726 | stellar-core (horizon) | main http port | +| 11825 | stellar-core (soroban-rpc) | peer node port | +| 11826 | stellar-core (soroban-rpc) | main http port | ### Security Considerations -Exposing the network ports used by your running container comes with potential risks. While many attacks are preventable due to the nature of the stellar network, it is extremely important that you maintain protected access to the postgresql server that runs within a quickstart container. An attacker who gains write access to this DB will be able to corrupt your view of the stellar network, potentially inserting fake transactions, accounts, etc. +Exposing the network ports used by your running container comes with potential risks. While many attacks are preventable due to the nature of the stellar network, it is extremely important that you maintain protected access to the postgresql server that runs within a quickstart container. An attacker who gains write access to this DB will be able to corrupt your view of the stellar network, potentially inserting fake transactions, accounts, etc. -It is safe to open the horizon http port. Horizon is designed to listen on an internet-facing interface and provides no privileged operations on the port. At the same time admin port should only be exposed to a trusted network, as it provides no security itself. +It is safe to open the horizon http port. Horizon is designed to listen on an internet-facing interface and provides no privileged operations on the port. At the same time admin port should only be exposed to a trusted network, as it provides no security itself. -The HTTP port for stellar-core should only be exposed to a trusted network, as it provides no security itself. An attacker that can make requests to the port will be able to perform administrative commands such as forcing a catchup or changing the logging level and more, many of which could be used to disrupt operations or deny service. +The HTTP port for stellar-core should only be exposed to a trusted network, as it provides no security itself. An attacker that can make requests to the port will be able to perform administrative commands such as forcing a catchup or changing the logging level and more, many of which could be used to disrupt operations or deny service. -The peer port for stellar-core however can be exposed, and ideally would be routable from the internet. This would allow external peers to initiate connections to your node, improving connectivity of the overlay network. However, this is not required as your container will also establish outgoing connections to peers. +The peer port for stellar-core however can be exposed, and ideally would be routable from the internet. This would allow external peers to initiate connections to your node, improving connectivity of the overlay network. However, this is not required as your container will also establish outgoing connections to peers. ## Accessing and debugging a running container -There will come a time when you want to inspect the running container, either to debug one of the services, to review logs, or perhaps some other administrative tasks. We do this by starting a new interactive shell inside the running container: +There will come a time when you want to inspect the running container, either to debug one of the services, to review logs, or perhaps some other administrative tasks. We do this by starting a new interactive shell inside the running container: ``` $ docker exec -it stellar /bin/bash ``` -The command above assumes that you launched your container with the name `stellar`; Replace that name with whatever you chose if different. When run, it will open an interactive shell running as root within the container. +The command above assumes that you launched your container with the name `stellar`; Replace that name with whatever you chose if different. When run, it will open an interactive shell running as root within the container. ### Restarting services -Services within the quickstart container are managed using [supervisord](http://supervisord.org/index.html) and we recommend you use supervisor's shell to interact with running services. To launch the supervisor shell, open an interactive shell to the container and then run `supervisorctl`. You should then see a command prompt that looks like: +Services within the quickstart container are managed using [supervisord](http://supervisord.org/index.html) and we recommend you use supervisor's shell to interact with running services. To launch the supervisor shell, open an interactive shell to the container and then run `supervisorctl`. You should then see a command prompt that looks like: ```shell horizon RUNNING pid 143, uptime 0:01:12 @@ -290,27 +297,28 @@ You can learn more about what commands are available by using the `help` command ### Viewing logs -Logs can be found within the container at the path `/var/log/supervisor/`. A file is kept for both the stdout and stderr of the processes managed by supervisord. Additionally, you can use the `tail` command provided by supervisorctl. +Logs can be found within the container at the path `/var/log/supervisor/`. A file is kept for both the stdout and stderr of the processes managed by supervisord. Additionally, you can use the `tail` command provided by supervisorctl. Alternatively, to tail all logs into the container's output for all services, append the `--logs` option. ### Accessing databases -The point of this project is to make running stellar's software within your own infrastructure easier, so that your software can more easily integrate with the stellar network. In many cases, you can integrate with horizon's REST API, but often times you'll want direct access to the database either horizon or stellar-core provide. This allows you to craft your own custom sql queries against the stellar network data. - -This image manages two postgres databases: `core` for stellar-core's data and `horizon` for horizon's data. The username to use when connecting with your postgresql client or library is `stellar`. The password to use is dependent upon the mode your container is running in: Persistent mode uses a password supplied by you and ephemeral mode generates a password and prints it to the console upon container startup. +The point of this project is to make running stellar's software within your own infrastructure easier, so that your software can more easily integrate with the stellar network. In many cases, you can integrate with horizon's REST API, but often times you'll want direct access to the database either horizon or stellar-core provide. This allows you to craft your own custom sql queries against the stellar network data. +This image manages two postgres databases: `core` for stellar-core's data and `horizon` for horizon's data. The username to use when connecting with your postgresql client or library is `stellar`. The password to use is dependent upon the mode your container is running in: Persistent mode uses a password supplied by you and ephemeral mode generates a password and prints it to the console upon container startup. ## Example launch commands -Below is a list of various ways you might want to launch the quickstart container annotated to illustrate what options are enabled. It's also recommended that you should learn and get familiar with the docker command. +Below is a list of various ways you might want to launch the quickstart container annotated to illustrate what options are enabled. It's also recommended that you should learn and get familiar with the docker command. + +_Launch an ephemeral local only dev/test network:_ -*Launch an ephemeral local only dev/test network:* ``` $ docker run -d -p "8000:8000" --name stellar stellar/quickstart --local ``` -*Launch an ephemeral testnet node in the foreground:* +_Launch an ephemeral testnet node in the foreground:_ + ``` $ docker run --rm -it \ -p "8000:8000" \ @@ -318,7 +326,8 @@ $ docker run --rm -it \ stellar/quickstart --testnet ``` -*Setup a new persistent node using the host directory `/str`:* +_Setup a new persistent node using the host directory `/str`:_ + ``` $ docker run -it --rm \ -p "8000:8000" \ @@ -329,4 +338,4 @@ $ docker run -it --rm \ ## Troubleshooting -Let us know what you're having trouble with! Open an issue or join us on our public slack channel. +Let us know what you're having trouble with! Open an issue or join us on our public [Discord server](https://discord.com/invite/stellardev).