networking: allow cluster operators to specify their own bridge CNI config

[jrasell]

The Nomad bridge network mode uses CNI to configure the interface via the bridge plugin. The CNI plugin configuration is embedded within the client with a small number of configuration options exposed on the client configuration block.

Whilst the configuration options offer enough scope for some, others would like to update fields which are not exposed. In order to add these in the current method, we would need to add these to the client configuration block. This could create an unwieldily set of config params and could also mean Nomad needs to continually update to accommodate new bridge plugin options.

It could prove useful and more flexible to allow operators to define their own CNI configuration to load for the bridge network, rather than add individual config parameters. This would immediately solve the problems mentioned within this Discuss post and #13352

[NOBLES5E]

Is this being worked on? Right now we need to use our customized build of nomad to support hairpin mode, which is not ideal.

[lgfa29]

Closing this one as completed by #15961.

While this issue mentions other bridge parameters, or a more flexible configuration approaches, we feel like adding more customization to the default bridge may result in unexpected outcomes that are hard for us to debug. The bridge network mode should something predictable and easily reproducible by the team.

Users that require more advanced customization are able to create their own bridge network using CNI. The main downside of this is that Consul Service Mesh requires network_mode = "bridge", but this is a separate feature request that is being tracked in #8953.

Feel free to 👍 and add more comments there.