Improving the deployment process

[CyrusVorwald]

Currently the deployment process for ICON Bridge smart contracts and relay relies on many esoteric bash scripts. It requires modifying too many commands depending on what the target environment is, and runs a very high risk of user error.

The deployment process should be very clear, very easy, and very fast. I should be able to deploy to a new EVM chain in under 10 minutes in all of the following cases on any network environment:

• The hub network contracts are already deployed and I am deploying a new chain integration to a pre-existing service
• The hub network contracts are not already deployed and I am deploying the entire solution
• I am deploying an update to one or more contracts on one or more chains
• I am setting up one or more bidirectional relays
• I am deploying an update to all relays

Currently, the hub network is specifically ICON.

I would like to do the following:

• Get rid of /docker, /docker-compose, /devnet, /build
• Create /provisioning directory for all setup
• Create /provisioning/users directory for all identity management. This would be where relay server, relay wallet, and contract owner access and secrets get defined
• Create /provisioning/networks directory for all blockchain provisioning. This would be where kubernetes yaml files define the containers for the local chain environment
• Create /provisioning/relays directory for relay provisioning. This would be where kubernetes yaml files define the containers for the relay
• Create /provisioning/smart-contracts directory for smart contract provisioning. Each execution environment and chain would have its own subfolder, such as /provisioning/smart-contracts/jvm/icon, /provisioning/smart-contracts/evm/bsc, /provisioning/smart-contracts/wasm/near, etc.
• Move all generated files used to track state of deployment into a single file akin to state.json in Uniswap's deployment scripts
• Automatically create or update a cross-chain token list as an artifact of deployment
• Use Bazel for builds if it can be demonstrated that it reduces the amount of time to build by enough to make it worth the time investment
• Use kubernetes for deployment if it can be demonstrated that it reduces the amount of time to deploy by enough to make it worth the time investment
• Use tilt to facilitate build and deployment processes if either of the above 2 points are found to be the case

I would like to know what everyone else's thoughts are and whether I'm missing anything. I get the impression that the e2e framework is doing a lot of this provisioning separately as well, even though it should probably be dependent on this occurring (or calling provisioning on its own when necessary based on config)

[andrii-kl]

Before starting the implementation, I propose to create a model that can implement all the requirements that were outlined above.
It is worth take a look at the Bazel implementation to see if we can extend it to all chains

In my opinion the task is not trivial, it will be quite difficult to generalize all chains. And for this reason, it is worth starting with a plan and a vision in order to jointly come to a better solution.

[manishbista28]

Here are some of the features in a deployment script that can be helpful:

1. Checkpoint: script should resume from where it left off. This is necessary to save transaction cost & time.
2. Utility Scripts: it should be possible to write scripts, which are not part of the deployment process, but are added later on to help with some task (upgrade, configuration). The script should be able to easily fetch deployment artifacts (without hard-coding values).
3. Integrate new chains:
   a) Deploy a new set of smart contracts and link it to existing deployment(s) e.g. integrate SNOW to ICON-BSC
   b) Only link to existing deployment(s). Deployment could have been done by different method. e.g. integrate NEAR to ICON-BSC
4. Deployment strategy:
   a) Should a single docker image do the entire set of steps (build smart contracts, deploy and run relay) OR
   b) Should these steps be separated and let users run them each at a time separately
   The options are important because the former is useful for a beginner but may not useful for production deployments. The latter option requires installing dependencies on local system but is much more flexible and useful for a developer.
5. Differences between chains:
   a) Even if set of chains can share a lot of similarity (e.g. evm chains), the difference in verification method necessitates some difference in the deployment script to fetch data specific to a chain.
   b) Similar chains also have difference in configuration params (e.g. rpc endpoint, coins, etc). It should be easy to specify what's different and share the bulk of what's common.
   c) There might be a difference in the environment to deploy (e.g. localnet, testnet, mainnet). Yet the script should provide sufficient abstraction that a user does not have to dwell around the differences. For example, to deploy to testnet, you fund a wallet using a faucet but do deploy to a localnet, you need to fetch god keys from inside the docker container.
6. Error Handling
   While doing some transactions (especially deployments which take long), the connection is susceptible to get interrupted. Better error handling languages or measures can greatly help avoid difficulties during mainnet deployments.
7. Migration Scripts
   If we are resorting to a different deployment strategy, a set of migration scripts will have to be written to migrate the current deployment artifacts (e.g. _ixh folder and scripts)
8. Modular
   Deployment scripts include a list of methods to interact with smart contracts. These methods should be easily usable through utility scripts (e.g. upgrade scripts) and command line (e.g. transfer token command).

[andrii-kl]

I'd like to suggest next deployment model. This model removes dependencies between different chains and makes workflow straight. Each chain will have independent deployment process.

Directory structure

contracts - This folder stores all btp contracts implementation

deploy - This folder stores deployment scripts for specific chains

deployments - This folder stores publick deployment artifacts after deploy. (contract addresses, transaction hashes, block numbers, abi and so on)

provisioning - This folder for deployment framework. (Python, ....)

keystores - keystore is not part of the project and resides on the user's file system

Deployment process

Case 1: The hub network contracts are already deployed (Icon) and I am deploying a new chain (Bsc) integration to a pre-existing service

1. Deploy Bsc contracts

Example: ./deploy/bsc/bsc_deployment_script -name bscTestNet -ks ../keystores/bsc

2. Register new contracts in the Icon chain

Example: ./deploy/bsc/bsc_register_script -bsc bscTestNet -icon iconLisbon -ksBsc ../keystores/bsc -ksIcon ../keystores/icon

Case 2: The hub network contracts are not already deployed and I am deploying the entire solution

1. Deploy Icon contracts

2. Deploy Bsc contracts

3. Register new contracts in the Icon chain

Case 3: I am deploying an update to one or more contracts on one or more chains

Each chain has own update scripts. We will run them in the order we need.

Case 4: I am setting up one or more bidirectional relays

To run a relay we should have only a relay config file. We can easily do it using existing deployment artifacts from the deployment folder.

Case 5: I am deploying and update to all relays

The process with several relays shouldn't be different.

Also I think it make sense to move the relay functionality into separate repository (icon-bridge-relay / icon-relay). It will help avoid any dependencies between contracts and relay.

[andrii-kl]

The current scripting solution is built for two specific chains, which imposes great restrictions on the expansion of the system

[shreyasbhat0]

Adding to this, we can have a factory method with an interface that has to build, deploy, and state management. The underlying chains will implement these methods from the interface, and do the necessary operations. Also, we can use a single config file to mention all necessary requirements for the workflow.

[CyrusVorwald]

@shreyasbhat0 I think we can leverage Uniswap's deployment scripts for what you are describing.

[shreyasbhat0]

@shreyasbhat0 I think we can leverage Uniswap's deployment scripts for what you are describing.
yes. If we could have a modular way of handling different chains that would be great in terms of implementation

[manishbista28]

Regarding the suggested directory structure, the contracts folder after a deployment will hold artifacts specific to the deployment e.g. truffle will update some metadata file (.oppenzepplin) to store contract addresses.
As such the contracts folder should be bound to deployments folder, e.g inside deployment's sub-directories.

[CyrusVorwald]

A few questions -
What do we have currently that works well and what doesn't work well?
What's the separation in how we use Docker and Bash currently?
How does Bazel fit into all this?

[andrii-kl]

A few questions - What do we have currently that works well and what doesn't work well? What's the separation in how we use Docker and Bash currently? How does Bazel fit into all this?

1. In my opinion we don't have big issue with build part. It's working and doesn't take much dev time, if we can any time limitation we can do it later.
   Icon - we use gradle task to build optimizedJar(s)
   Solidity - The project has already built contracts

2. We don't use Docker for build and deploy processes.

3. I don't have experience with Bazel, I'll take a look at it

[CyrusVorwald]

The question about bazel is directed at @MuhammedIrfan and @shreyasbhat0 since the NEAR integration uses it. If builds are not an issue then I don't think it provides much value.

There are many docker files. What are those doing if we aren't using them for build and deploy?

[andrii-kl]

There are many docker files. What are those doing if we aren't using them for build and deploy?

1. To run local bsc / icon (I don't think many people use it now)

2. GitHub Actions tasks

[CyrusVorwald]

If I do a local build it uses docker for the chains and relay

[andrii-kl]

It's possible to use, but right now easier to use a testnet. So I don't think anyone has used it for a long time.

[manishbista28]

Based on the suggestion above and after some internal discussions, I've created a git issue #775 to decide on an acceptable directory structure.
The suggested changes have been pushed to this feature branch

Please check and let me know for necessary improvements.

The filenames and types(shell or python) can be changed based on which language we agree to write the scripts in.

[manishbista28]

Yes, a single config file will include necessary config per chain (icon, bsc, snow,.) per deployment type (mainnet, testnet, localnet). Relevant script should select which chain(s) and chainType to pick for the current deployment run.

[CyrusVorwald]

I'm good with most of this, except that it needs to be kept dry. There's no reason for there to be a localnet.sh, testnet.sh, and mainnet.sh for example since there are steps that are all the same for all 3.

[CyrusVorwald]

I think the best way to do so would be to inject the chain(s) being deployed to an interface. I don't think bash is capable of this without writing such functionality.

[manishbista28]

The folder names inside /provisioning/{mainnet,testnet,localnet}/ should be distinct for different deployments but the names themselves can be arbitrary. A user will be able to specify the name of this folder that houses the deployment artifacts during the script run.

A user could
a) choose to include network_ids of all the chains involved eg.
/provisioning/deployments/mainnet/0x1icon_0x38bsc_0x228snow

b) choose to include chain_name and deployment version tag e.g.
/provisioning/deployments/testnet/lisbonbsc_v0.0.10
/provisioning/deployments/testnet/lisbonbsc_v0.0.11, etc

Point is

1. folder names can be arbitrary but should be distinct
2. other than being a location identifier, the name itself does not serve any purpose during script run

The folder names in the current commit is meant to only serve as an example to show

1. that there can be multiple folders inside /provisioning/{mainnet,testnet,localnet}/
2. what are the kind of files (deployment artifacts) that these folders should/can contain
   These example folders have no actual content to them (holds empty files) so will remove later once we are aware of their purpose.

[manishbista28]

I'm good with most of this, except that it needs to be kept dry. There's no reason for there to be a localnet.sh, testnet.sh, and mainnet.sh for example since there are steps that are all the same for all 3.

Whatever common between scripts will be shared (imported) through a common script and what's different (if any) can be placed inside relevant custom script. We will have better understanding of the necessity for different files (localnet.sh, etc) once we start with the actual script implementation.
Currently shown empty .sh files inside /provisioning/provisions/ are there to demonstrate where we intend to place task-specific wrapper scripts on the suggested directory layout.

[andrii-kl]

During discussion about the deployment process with @manishbista28, we came to some changes in the structure.
I have marked it on the diagram. (deployments folder)

[andrii-kl]

We can make parallel deployment for different chains, now we do it one by one.

icon ->  deploy bmc - deploy btc
bsc  ->  deploy bmc - deploy btc
snow -> ...

[manishbista28]

yes, there are steps that can be run in parallel and can save time.

If fresh deployments are a repeated and frequent task then this performance enhancement has its benefits.
One of the primary reasons we have been doing frequent deployments is because it takes longer for an existing deployment's relay to initialize than it takes to re-run a fresh deployment. If so, the suggested method will not be solving the actual problem, which is with relay's block sync duration.

We should consider if we're trading simplicity and ease of use for performance here, especially considering who the users of these scripts can be and how likely they are in understanding when steps run in parallel, and when they should not.

If sequential method and the relative delay introduced with it does not impact development operations as much, we should classify concurrent methods as a "nice to have" feature than a requirement and plan accordingly.

[CyrusVorwald]

I do not think that parallel deployment is impactful or worth considering. Let's finalize this Monday, sent out a meeting on the calendar.

[andrii-kl]

If fresh deployments are a repeated and frequent task then this performance enhancement has its benefits. One of the primary reasons we have been doing frequent deployments is because it takes longer for an existing deployment's relay to initialize than it takes to re-run a fresh deployment. If so, the suggested method will not be solving the actual problem, which is with relay's block sync duration.

I partially agree.
Current fresh deployment takes about 30 mins, I think It's quite a lot of time.

[andrii-kl]

I do not think that parallel deployment is impactful or worth considering. Let's finalize this Monday, sent out a meeting on the calendar.

Yes, such a solution complicates the architecture of the application. Therefore, we can start with a simple example. But in this case, you should not wait for the deployment process to be accelerated (which is not critical now).

[andrii-kl]

Let's also discussed the next step. Let's suppose we deployed all chains.
We have chains deployment artifacts.

deployments/mainnet_beta/ ->
    icon_addresses
    icon_state
    bsc_addresses
    bsc_state
    snow_addresses
    snow_state

Or

deployments/mainnet_beta/ ->
  icon / ->
      icon_addresses
      icon_state ?
  bsc / ->
      bsc_addresses
      bsc_state?
  snow / ->
      snow_addresses
      snow_state?

Next step We want to run a relay.

[andrii-kl]

To configure a relay current model use bmr.config.json file.

@manishbista28 @bbist @CyrusVorwald-ICON The discussion points that I can see are:

1. How we will build this file?
   In case when we have more than two chains we can start a relay with any pair of them. It means that we can't build this file in the deployment process.

2. The relay config file has a private information. (key_password)
   I think we should remove all private information from config file. We can pass them as an application variables.

3. The relay config file has a key store object. (key_store)
   I think we should replace it on a link to the keystore file.

[manishbista28]

1. all of the parameters necessary for relay config is present inside state.json. A user should be able to run a bmr config generator script in which he can specify the parameters involved. Example parameters can be chain pairs, key-store separation (removing private information to a json file like how we require it for mainnet deployments)

[andrii-kl]

Let's also discuss a convenient approach to storing private keys from different chains.

The discussion points that I can see are:

1. Keystore location

I think it should be configurable for deployment process.

2. Excessive number of files

Each contract create 3 files.

wallet.json
wallet.json.priv
wallet.secret

Should the current approach be simplified?

4. Keystore structure

Should we put all the keys in one directory or different ones?

keystore/ ->
  icon / ->
      ....
  bsc / ->
      ....
  snow / ->
      ....

[manishbista28]

The current implementation creates keystores for more users than strictly necessary for deployment. The deployer wallet and bmr wallet is absolutely necessary but bmc and bts wallets are not.

It should be possible to have separate wallets for different deployments of the same chain and same deployment type. Example: if I'm creating separate icon-bsc and icon-snow testnet deployments, then it is advisable to use different set of keystores (user wallets) for ICON chain especially if both of these deployments are in use at the same time.

I do not see any harm in saving them all inside a single directory as long as there are not too many files. example location: here.

As to whether the keystore location should be configurable, I have not seen any benefits in doing so.

[manishbista28]

Few useful features specific to scripts:

1. Error Handling: Deploying smart contracts is susceptible to rpc endpoint network issues.
   Current implementation includes retries on the commands which are more susceptible to get network interrupts and retries if so. Secondly, some transaction (e.g. register coin) could get submitted on chain but the connection can get interrupted while waiting for acknowledgement. In this case, we will observe an error (a Failure Response) but what needs to be done on chain (registering coin) will have already been successful. Such issue can lead to discrepancy between state preserved locally and state of contracts on chain.

2. Checkpoint: If a command takes multiple steps and it's costly to re-run it from scratch, then some way to query how far along we've progressed can help especially if there are numerous set of steps and one needs to re-run the command. It makes sense to limit the number of locally saved checkpoints and instead query the state from blockchain itself.

3. Integer Overflows: Since smart contracts deal with large number (uint256), ensure if scripting language and the libraries involved can work with these values. Example: while setting BTP fee for a coin with decimals 21 (NEAR), check if python and javascript libraries can work with these values

4. Irreversible Steps: There are some commands (e.g. register coin) which does not have an equivalent undo command (unregister a coin). It is best to ensure that the script itself makes necessary checks around the arguments involved (e.g. verifying chain ids, coinName format). Else increasing visibility by allowing the user to observe what arguments are being used and making the process interactive can take some risk off of such steps as well.
   Needless to say, verifying input args (example address formats, number bases (hex or decimal), coinName formats is something any step can benefit from especially considering that some of these commands can be run by passing args directly from CLI.

5. Migration Scripts: How convenient will it be to move current deployment artifacts to new deployment standards (example: create local state files)

6. Logging: Add log file that lists down which step was executed, when, arguments passed and response (success, failure) etc.