refactor: isolate faulty channels and retry channel task on faults

[prestwich]

High Level Changes:

• Allows kathy, relayer, and processor to isolate failures at the channel level and retry channel task instead of crashing whole agent if one channel fails
• Note that isolating channel failures is not relevant to the updater (updater only touches home)
• This behavior did not seem desirable for the watcher

Code Changes

• Agent::run no longer borrows &self and instead takes an agent-specific <Agent>Channel struct that defines all data types needed to run one home <> replica channel
• Agent::run_many builds an <Agent>Channel struct and hands this off to an Agent::run task; if the run task errors out, it will log error and try to start it again instead of returning error to top level
• Watcher and updater ignore this pattern, as they must overwrite Agent::run_all

TODO:
[ ] add unit tests to mock faulty RPC
[x] add exponential backoff for retries
[x] metric to track channel number of channel faults

Closes #161

[kekonen]

@prestwich @ltchang2019 I merged the Unit tests, so I believe it is ready to be merged

[luketchang]

@arnaud036 @yourbuddyconner lets run this PR in dev before merging

[arnaud036]

This repo only build a container once the PR is merged to main. We could change this behavior but we should agree on our git flow strategy for CI/CD.

The flow I had in mind was to continuously deploy main to dev when a PR gets merged and deploy to prod once a tag is created. We could also consider deploying to staging eventually and use this environment as an integration env