-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setup ten config library to replace our flags approach #2115
Merged
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
158c828
Initial spec for ten config
BedrockSquirrel 73ccf36
WIP fixing after rebase
BedrockSquirrel c3511ac
WIP
BedrockSquirrel 7194eab
WIP local network still buggered:
BedrockSquirrel 000c78f
WIP some fixes
BedrockSquirrel 648b8b4
fix local testnet launcher
BedrockSquirrel 07623b1
tidying up
BedrockSquirrel ec2ec5e
Tidying and fixes
BedrockSquirrel eab6f52
Fix type cfg
BedrockSquirrel f1f61c9
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel 8f5c248
add the host ID back in for now
BedrockSquirrel dc06319
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel 39b6544
review comments
BedrockSquirrel 4511e92
linting
BedrockSquirrel bb9ac7f
fix dockerfile
BedrockSquirrel 552ef00
fix tests
BedrockSquirrel bf4d073
WIP fixes for node CLI
BedrockSquirrel 35c1538
fix enclave DB flag
BedrockSquirrel 8812b19
node launcher edb path
BedrockSquirrel 1029750
node launcher: missed enclave rpc host
BedrockSquirrel 26c9c37
node launcher: fix host db cfg
BedrockSquirrel 987467f
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel 4cd32e5
testnet launcher: turn on debug namespace by default
BedrockSquirrel 276451d
fix noderunner test
BedrockSquirrel 2d92ad9
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel ca7da4d
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel 4a3362b
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel fd0a45e
review: add comment
BedrockSquirrel a046221
Merge remote-tracking branch 'origin/main' into matt/config-viper
BedrockSquirrel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
tidying up
- Loading branch information
commit 07623b1469e7443c3bffbbe0fd558f7b4149cccd
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
All go processes in the TEN ecosystem will use the config mechanism here to load their configuration. | ||
|
||
We have a TenConfig hierarchical structure that contains typed configuration fields used across all our systems. | ||
|
||
They are defined in a single place, in the Config structs declared in `config.go`. These declarations include their type, | ||
the mapstructure annotation that gives their yaml field name and any comments about their usage. | ||
|
||
For example, `network.chainId` is a field that might be used by multiple processes, but `host.rpc.httpPort` is only used by the host processes. | ||
|
||
This library provides a loading mechanism powered by viper, it reads 0-base-config.yaml to initialise the config and then | ||
overrides fields with any other yaml files provided and finally overrides with environment variables. | ||
|
||
Environment variable keys are the same as the yaml keys but are uppercased and have dots replaced with underscores. | ||
For example, `network.chainId` would be `NETWORK_CHAINID`, and `host.rpc.httpPort` would be `HOST_RPC_HTTPPORT`. | ||
|
||
## Defining a new field | ||
When defining a new field in the Config struct, there are a few things to keep in mind: | ||
|
||
- Environment variable values are only applied when the field is defined in at least one of the yaml files. For this reason, | ||
we want to make sure that all fields are defined in the base config file even if just with trivial or empty values. | ||
|
||
- The ego enclave restricts the environment variables that can be accessed by the enclave process. This means that the enclave | ||
process will not be able to access environment variables that are not whitelisted. This is a security feature of the ego enclave. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice |
||
|
||
The enclave.json file used to produce the signed ego artifact allows environment variables to be specified, either with a hardcoded value | ||
which we will use for fixed constants that are not allowed to change or with a 'fromHost' flag which will allow the enclave to access the | ||
environment variable from the host process. This is useful for configuration values that are allowed to change between deployments. | ||
|
||
So any configuration value that is expected to be set by an environment variable should be whitelisted in the enclave.json file. | ||
|
||
The upshot of this is that defining a new field requires adding it in 2 or 3 places. | ||
The nested Config struct declared from config.go, the 0-base-config.yaml file and, if field is used by the enclave, the enclave.json file. | ||
|
||
## Loading the config | ||
|
||
The loading method allows an ordered list of yaml files to be provided, values in later files will override values in earlier files. | ||
|
||
In practice for production deployments, we expect that the 0-base-config.yaml will be the only file that is always provided, and the rest will be optional. | ||
|
||
As a convention, we're using a prefixes for the files when lists are provided: | ||
`0-base-config.yaml` - the base configuration that is always provided | ||
`1-env-<env-name>.yaml` - environment-specific configuration | ||
`2-node-<node-name>.yaml` - node-specific configuration (e.g. sequencer, validator in integration tests) | ||
`3-network.yaml` - network-specific configuration (e.g. chainId, networkId) | ||
|
||
The loader is not performing any verification on these conventions though, callers can provide files with any names and they are applied in the order provided. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import ( | ||
"fmt" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import ( | ||
"fmt" | ||
|
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import "time" | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import "time" | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import ( | ||
"embed" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import ( | ||
"math/big" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
package config2 | ||
package config | ||
|
||
import ( | ||
gethcommon "github.com/ethereum/go-ethereum/common" | ||
|
Empty file.
Empty file.
Empty file.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this ok for production? ENV variable overrides should be disabled in enclave IMO? Or at least disabled for anything baked into the image