diff --git a/docs/docs/best-practices.md b/docs/docs/best-practices.md index 56523cf5a4..850431d964 100644 --- a/docs/docs/best-practices.md +++ b/docs/docs/best-practices.md @@ -12,11 +12,14 @@ Passing [package parameters][package-parameterization] via the CLI can get hairy ```bash kurtosis run github.com/user/repo '{"some_param":5,"some_other_param":"My value"}' ``` -1. **When you have a large number of arguments:** put them in a `.json` file and use `--args-file` to slot them into the `kurtosis run` command. E.g.: +1. **When you have a large number of arguments:** put them in a `.yaml` or `.json` file and use `--args-file` to slot them into the `kurtosis run` command. E.g.: + ```bash + kurtosis run github.com/user/repo --args-file my-params.yaml + ``` + or if you prefer JSON: ```bash kurtosis run github.com/user/repo --args-file my-params.json ``` - Choosing the right wait ----------------------- Kurtosis has three different types of waits. Described here are the three, with tips on when to use each: diff --git a/docs/docs/cli-reference/run.md b/docs/docs/cli-reference/run.md index 245e246f50..189418fecc 100644 --- a/docs/docs/cli-reference/run.md +++ b/docs/docs/cli-reference/run.md @@ -77,10 +77,10 @@ run(plan, some_parameter = struct(some_property = "Property value")) `kurtosis run` has additional flags that can further modify its behaviour: -1. The `--args-file` flag can be used to send in a JSON/YAML file, from a local file through the filepath or from remote using the URL, as an argument to the Kurtosis Package. Note that if you pass in package arguments as CLI arguments and via the flag, the CLI arguments will be the one used. +1. The `--args-file` flag can be used to send in a YAML/JSON file, from a local file through the filepath or from remote using the URL, as an argument to the Kurtosis Package. Note that if you pass in package arguments as CLI arguments and via the flag, the CLI arguments will be the one used. For example: ```bash - kurtosis run github.com/kurtosis-tech/ethereum-package --args-file "devnet-5.json" + kurtosis run github.com/kurtosis-tech/ethereum-package --args-file "devnet-5.yaml" ``` or ```bash diff --git a/docs/docs/get-started/quickstart.md b/docs/docs/get-started/quickstart.md index ef16e4710e..e8e8e1f732 100644 --- a/docs/docs/get-started/quickstart.md +++ b/docs/docs/get-started/quickstart.md @@ -48,37 +48,34 @@ Kurtosis environment definitions, referred to here on as [Kurtosis packages][pac You're now going to create a file to house the parameters that you will pass in at runtime when your system starts up. Check out the README for the [ethereum-package][ethereum-package] to see the full list of configuration options and flags available for use. -Create a file titled: `network_params.json` in your home directory, populate it with the below contents, and save your file: -```json -{ - "participants": [{ - "el_client_type": "geth", - "el_client_image": "ethereum/client-go:latest", - "el_client_log_level": "", - "el_extra_params": [], - "cl_client_type": "lighthouse", - "cl_client_image": "sigp/lighthouse:latest", - "cl_client_log_level": "", - "beacon_extra_params": [], - "validator_extra_params": [], - "builder_network_params": null, - "count": 2 - }], - "network_params": { - "network_id": "3151908", - "deposit_contract_address": "0x4242424242424242424242424242424242424242", - "seconds_per_slot": 12, - "slots_per_epoch": 32, - "num_validator_keys_per_node": 64, - "preregistered_validator_keys_mnemonic": "giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete", - "deneb_for_epoch": 500 - }, - "verifications_epoch_limit": 5, - "global_client_log_level": "info", - "mev_type": "none", - // The default value bootstraps a bunch of services like grafana, prometheus, el/cl forkmons, explorers - "additional_services": [], -} +Create a file titled: `network_params.yaml` in your home directory, populate it with the below contents, and save your file: +```yaml +participants: + - el_client_type: geth + el_client_image: ethereum/client-go:latest + el_client_log_level: '' + el_extra_params: [] + cl_client_type: lighthouse + cl_client_image: sigp/lighthouse:latest + cl_client_log_level: '' + beacon_extra_params: [] + validator_extra_params: [] + builder_network_params: + count: 2 +network_params: + network_id: '3151908' + deposit_contract_address: '0x4242424242424242424242424242424242424242' + seconds_per_slot: 12 + slots_per_epoch: 32 + num_validator_keys_per_node: 64 + preregistered_validator_keys_mnemonic: 'giant issue aisle success illegal bike spike + question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy + very lucky have athlete' + deneb_for_epoch: 500 +verifications_epoch_limit: 5 +global_client_log_level: info +additional_services: [] + ``` The arrays in the `participant` object enables you to define the specific Ethereum client types and respective image tags you want to use in your network, alongside any extra parameters for the validator, beacon, or builder as well as some useful flags for the verbosity of log lines. In this example you'll be using the `latest` image tags for the Geth and Lighthouse clients and have specified `2` nodes to be spun up. @@ -90,7 +87,7 @@ Spin up your system! -------------------- Great! You're now ready to bring up your own network. Simply run: ```console -kurtosis run github.com/kurtosis-tech/ethereum-package --args-file ~/network_params.json --enclave eth-network +kurtosis run github.com/kurtosis-tech/ethereum-package --args-file ~/network_params.yaml --enclave eth-network ``` Kurtosis will then begin to spin up your private Ethereum testnet by interpreting the instructions in the Kurtosis package, validating the plan to ensure there are no conflicts or obvious errors, and then finally executes the plan (read more about multi-phase runs [here][multi-phase-runs-reference]). Kurtosis first spins up an isolated, ephemeral environment on your machine called an [enclave][enclaves-reference] where all the services and [files artifacts][files-artifacts-reference] for your system will reside in. Then, those services will be bootstrapped and required files generated to start up the system. @@ -158,7 +155,7 @@ Why Kurtosis packages - from a consumer's perspective Kurtosis was built to make building distributed systems as easy as building a single server app. Kurtosis aims to achieve this by bridging the environment definition author-consumer divide. Tactically, this means making it dead simple for a consumer (like yourself) to pick up an environment definition, spin it up, and deploy it the way you want, where you want - all without needing to know specialized knowledge about how the system works or how to use Kubernetes or Docker. Specifically, this guide showed you: -- ***The power of parameterizability***: as a consumer of the environment definition, having both the knowledge and means to configure the system to spin up the way you need it is incredibly valuable - a big reason why Kurtosis packages are meant to be parameterized. In this guide, you created the `network_params.json` file which contained your preferences for how the network should look and passed them in to Kurtosis with relative ease. The author of the package need only define the arguments and flags available for a consumer, and Kurtosis handles the rest once those are passed in at runtime. +- ***The power of parameterizability***: as a consumer of the environment definition, having both the knowledge and means to configure the system to spin up the way you need it is incredibly valuable - a big reason why Kurtosis packages are meant to be parameterized. In this guide, you created the `network_params.yaml` file which contained your preferences for how the network should look and passed them in to Kurtosis with relative ease. The author of the package need only define the arguments and flags available for a consumer, and Kurtosis handles the rest once those are passed in at runtime. - ***Portable and easy to wield***: a major contributor to the author-consumer divide comes from the knowledge gap between the author and consumer regarding the infrastruture and tools needed to instantiate a system. Understanding how Kubernetes works, what Bash script to use at which step, and working with Docker primitivies are all pain points we believe Kurtosis alleviates. In this guide, you installed Kurtosis and ran a single command to get your system up and running. This same command will work anywhere, over Docker or on Kubernetes, locally or on remote infrastructure. We believe this portability and ease of use are requirements for bridging the author-consumer divide. There are many other reasons why we believe Kurtosis is the right tool for bridging the author-consumer divide. Check out the [next guide][how-to-set-up-postgres-guide] to experience the workflow for a package author and how Kurtosis improves the developer experience for an environment definition author. diff --git a/docs/docs/guides/how-to-compose-your-own-testnet.md b/docs/docs/guides/how-to-compose-your-own-testnet.md index 6b25a66770..7fa4875531 100644 --- a/docs/docs/guides/how-to-compose-your-own-testnet.md +++ b/docs/docs/guides/how-to-compose-your-own-testnet.md @@ -42,18 +42,19 @@ To begin, create and `cd` into a directory to hold your files: ```bash mkdir my-testnet && cd my-testnet ``` -Next, create a file called `network_params.json` in that folder with the following contents: -```json -{ - "preregistered_validator_keys_mnemonic": "giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete", - "num_validator_keys_per_node": 64, - "network_id": "3151908", - "deposit_contract_address": "0x4242424242424242424242424242424242424242", - "seconds_per_slot": 12, - "genesis_delay": 10, - "capella_fork_epoch": 2, - "deneb_fork_epoch": 500 -} +Next, create a file called `network_params.yaml` in that folder with the following contents: +```yaml +preregistered_validator_keys_mnemonic: 'giant issue aisle success illegal bike spike + question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy + very lucky have athlete' +num_validator_keys_per_node: 64 +network_id: '3151908' +deposit_contract_address: '0x4242424242424242424242424242424242424242' +seconds_per_slot: 12 +genesis_delay: 10 +capella_fork_epoch: 2 +deneb_fork_epoch: 500 + ``` The contents above will be used to define the specific parameters with which to start the network with. @@ -66,10 +67,13 @@ Awesome. You have just created the beginnings of your first [Kurtosis package](. ### 2. Import dependencies Now that you have a local project to house your definition and some parameters to start the network with, its time to actually build the network. First, create a Starlark file called `main.star` and add the following three lines: ```python +import yaml + # main.star geth = import_module("github.com/kurtosis-tech/geth-package/lib/geth.star") lighthouse = import_module("github.com/kurtosis-tech/lighthouse-package/lib/lighthouse.star") -network_params = json.decode(read_file("./network_params.json")) +with open("./network_params.yaml") as stream: + network_params = yaml.safe_load(stream) ``` In the first two lines, you're using [Locators](../concepts-reference/locators.md) to import in `geth.star` and `lighthouse.star` files from Github, making them available to use in your testnet definition. These files themselves are environment definitions that can be used to bootstrap and start up a Geth execution layer client and a Lighthouse consensus layer client as part of your testnet - which is exactly what you will do next. @@ -78,7 +82,7 @@ In the first two lines, you're using [Locators](../concepts-reference/locators.m Feel free to check out the [`geth.star`](https://github.com/kurtosis-tech/geth-package/blob/main/lib/geth.star) and [`lighthouse.star`](https://github.com/kurtosis-tech/lighthouse-package/blob/main/lib/lighthouse.star) to understand how they work. At a high level, the definition instructs Kurtosis to generate genesis data, set up pre-funded accounts, and then launches the client using the client container images. ::: -Finally, we are converting the local `network_params.json` file into a format that can be used in your environment definition using [`json.decode()`](https://bazel.build/rules/lib/core/json#decode) and [`read_file()`](../starlark-reference/read-file.md). +Finally, we are converting the local `network_params.yaml` file into a format that can be used in your environment definition using [`yaml.safe_load()`](https://pyyaml.org/wiki/PyYAMLDocumentation#Loader). This is then saved into a variable called `network_params` which you will use later on. ### 3. Define how your testnet gets built Now that you have all the necessary dependencies, you can start writing the function that will instantiate the network. Within your `main.star` file, add the following 3 new lines: @@ -97,9 +101,12 @@ With some execution layer genesis data in hand, you will now bootstrap the node! ```python # main.star +import yaml + geth = import_module("github.com/kurtosis-tech/geth-package/lib/geth.star") lighthouse = import_module("github.com/kurtosis-tech/lighthouse-package/lib/lighthouse.star") -network_params = json.decode(read_file("./network_params.json")) +with open("./network_params.yaml") as stream: + network_params = yaml.safe_load(stream) def run(plan): # Generate genesis, note EL and the CL needs the same timestamp to ensure that timestamp based forking works @@ -175,7 +182,7 @@ Otherwise, we've got some neat ideas for what you can do next. If you need a han * Deploy your node in a Kubernetes cluster for collaborative work and scale it out to multiple nodes! Check out our docs for how to do so [here](https://docs.kurtosis.com/k8s/). * Simulate MEV workflows by importing the [MEV Package](https://github.com/kurtosis-tech/mev-package) into your testnet definition. The MEV package deploys and configures the Flashbots suite of products to your local Ethereum testnet and includes: MEV-Boost, MEV-Flood, and MEV-relay, and any required dependencies (postgres & redis). Here's a full example of this set up [here](https://github.com/kurtosis-tech/geth-lighthouse-mev-package). * Connect other infrastructure (oracles, relayers, etc) to the network by adding more to your `main.star` file! Remember, this is an environment definition and you can import any pre-existing packages that you may find useful. Here are a [few examples](https://github.com/kurtosis-tech/awesome-kurtosis/tree/main) -* Deploy your dApp onto the local network! Hardhat can be used to do so by using the given RPC URL & the `network_id` defined in the `network_params.json` you wrote at the beginning. In your case, the `network_id` should be: `3151908`. A more thorough example of this workflow can also be found [here](./how-to-local-eth-testnet.md). +* Deploy your dApp onto the local network! Hardhat can be used to do so by using the given RPC URL & the `network_id` defined in the `network_params.yaml` you wrote at the beginning. In your case, the `network_id` should be: `3151908`. A more thorough example of this workflow can also be found [here](./how-to-local-eth-testnet.md). We're currently building out more components of [`eth-kurtosis`](https://github.com/kurtosis-tech/eth-kurtosis/tree/main), which serves as an index of plug-and-play components for Ethereum private testnets. We're building support for more clients - so let us know if there's something you would love to see added to the index! diff --git a/docs/docs/guides/how-to-full-mev-with-eth2-package.md b/docs/docs/guides/how-to-full-mev-with-eth2-package.md index 18794b7d78..2df1723478 100644 --- a/docs/docs/guides/how-to-full-mev-with-eth2-package.md +++ b/docs/docs/guides/how-to-full-mev-with-eth2-package.md @@ -51,7 +51,7 @@ Once the network is online, `mev-flood` will deploy UniV2 smart contracts, provi ## Quickstart Leveraging the [`ethereum-package`](https://github.com/kurtosis-tech/ethereum-package) is simple. In this short quickstart, you will: 1. Install Docker & Kurtosis locally. -2. Configure your network using a `.json` file. +2. Configure your network using a `.yaml` file. 3. Run a single command to launch your network with `full MEV`. 4. Visit the website to witness payloads being delivered from the Relayer to the `mev-boost` sidecar connected to each validator (for block proposals). @@ -60,41 +60,39 @@ Leveraging the [`ethereum-package`](https://github.com/kurtosis-tech/ethereum-pa * [Install Kurtosis](https://docs.kurtosis.com/install/#ii-install-the-cli) or [upgrade Kurtosis to the latest version](https://docs.kurtosis.com/upgrade). You can check if Kurtosis is running using the command: `kurtosis version`, which will print your current Kurtosis engine version and CLI version. #### Configure your network -Next, create a file titled: `ethereum-package-params.json` in your working directory and populate it with: -```json -{ - "participants": [{ - "el_client_type": "geth", - "el_client_image": "ethereum/client-go:latest", - "el_client_log_level": "", - "el_extra_params": [], - "cl_client_type": "lighthouse", - "cl_client_image": "sigp/lighthouse:latest", - "cl_client_log_level": "", - "beacon_extra_params": [], - "validator_extra_params": [], - "builder_network_params": null - }], - "network_params": { - "network_id": "3151908", - "deposit_contract_address": "0x4242424242424242424242424242424242424242", - "seconds_per_slot": 12, - "slots_per_epoch": 32, - "num_validator_keys_per_node": 64, - "preregistered_validator_keys_mnemonic": "giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete", - "deneb_for_epoch": 500 - }, - "verifications_epoch_limit": 5, - "global_client_log_level": "info", - "mev_type": "full" -} +Next, create a file titled: `ethereum-package-params.yaml` in your working directory and populate it with: +```yaml +participants: + - el_client_type: geth + el_client_image: ethereum/client-go:latest + el_client_log_level: '' + el_extra_params: [] + cl_client_type: lighthouse + cl_client_image: sigp/lighthouse:latest + cl_client_log_level: '' + beacon_extra_params: [] + validator_extra_params: [] + builder_network_params: null +network_params: + network_id: '3151908' + deposit_contract_address: '0x4242424242424242424242424242424242424242' + seconds_per_slot: 12 + slots_per_epoch: 32 + num_validator_keys_per_node: 64 + preregistered_validator_keys_mnemonic: 'giant issue aisle success illegal bike spike + question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy + very lucky have athlete' + deneb_for_epoch: 500 +verifications_epoch_limit: 5 +global_client_log_level: info +mev_type: full ``` You will use the above file by passing it in at runtime, effectively enabling you to define the way your network should look using parameters. #### Launch the network with `full MEV` Great! You're now ready to bring up your own network. Simply run: ```bash -kurtosis run --enclave eth-network github.com/kurtosis-tech/ethereum-package "$(cat ~/ethereum-package-params.json)" +kurtosis run --enclave eth-network github.com/kurtosis-tech/ethereum-package "$(cat ~/ethereum-package-params.yaml)" ``` Kurtosis will then begin to spin up your private Ethereum testnet with `full MEV`. You will see a stream of text get printed in your terminal as Kurtosis begins to generate genesis files, configure the Ethereum nodes, launch a Grafana and Prometheus instance, and bootstrap the network together with the full suite of MEV products from Flashbots. In ~2 minutes, you should see the following output at the end: ```bash @@ -200,14 +198,14 @@ If there are other use cases you had in mind (e.g. fuzzing the network at the pr ## Conclusion This guide was meant to be quick - we hope it was. To recap, you: - Installed Kurtosis and Docker -- Created a `.json` file that contains the necessary parameters for the network +- Created a `.yaml` file that contains the necessary parameters for the network - Ran a single command to spin up a private Ethereum testnet with MEV infrastructure The `ethereum-package` is available for anyone to use, will work the same way on your local machine or in the cloud, and will run on Docker or Kubernetes. You saw first-hand how packages, effectively environment definitions, are written once and then can be used by anyone in a very trivial way to reproduce the environment. This accelerates developer velocity by enabling engineers to spend less time on configuring and setting up development and testing frameworks, and more time instead on building the unique features and capabilities for their projects. -Additionally, we hope you also enjoyed the parameterizability aspect of Kurtosis Packages. By changing the `ethereum-package-params.json`, you can get a fine-tune your testnet however you see fit. +Additionally, we hope you also enjoyed the parameterizability aspect of Kurtosis Packages. By changing the `ethereum-package-params.yaml`, you can get a fine-tune your testnet however you see fit. We hope this guide was helpful and we'd love to hear from you. Please don't hesitate to share with us what went well, and what didn't, using [`kurtosis feedback`](../cli-reference/feedback.md) from the CLI to file an issue in our [Github](https://github.com/kurtosis-tech/eth-kurtosis/issues) or post your question in our [Github Discussions](https://github.com/kurtosis-tech/kurtosis/discussions). diff --git a/docs/docs/guides/how-to-local-eth-testnet.md b/docs/docs/guides/how-to-local-eth-testnet.md index b84c79ffa9..669307d308 100644 --- a/docs/docs/guides/how-to-local-eth-testnet.md +++ b/docs/docs/guides/how-to-local-eth-testnet.md @@ -181,7 +181,7 @@ Now let’s explore how you can configure the underlying network for testing our Your local Ethereum testnet can be configured to use different EL and CL client pairs, as well as a varying number of nodes, depending on the scenario and specific network configuration you want to develop or test. This means that, once set up, you can spin up a customized local testnet and use it to run the same workflows (deployment, tests, etc.) under various network configurations to ensure everything works as expected. To learn more about the other parameters you can modify, visit this link. -Give it a try! You can pass various configuration options to the `ethereum-package` via a JSON file. This network params JSON file provides the specific configurations that Kurtosis will use to set up the local Ethereum network. +Give it a try! You can pass various configuration options to the `ethereum-package` via a YAML/JSON file. This network params YAML/JSON file provides the specific configurations that Kurtosis will use to set up the local Ethereum network. Take the default configuration file and edit it to spin up two nodes with different EL/CL pairs: * Node 1 with `geth`/`lighthouse` @@ -245,7 +245,7 @@ The `network_params` struct configures the network settings that are used to cre Save your edited params file in any directory you wish (in the example below, it is saved to the desktop) and then use it to run your Kurtosis package by running: ```bash -kurtosis clean -a && kurtosis run --enclave local-eth-testnet github.com/kurtosis-tech/ethereum-package --args-file ~/eth-network-params.json +kurtosis clean -a && kurtosis run --enclave local-eth-testnet github.com/kurtosis-tech/ethereum-package --args-file ~/eth-network-params.yaml ``` :::TIP Note that the `kurtosis clean -a` command is used here to instruct Kurtosis to destroy the old testnet and its contents before starting a new one up.