MTU enhancement

[rbankston]

Describe the solution you'd like
The default MTU within the calico cni provider is for best results when using GCE at 1500. We should update the MTU in this project to match the documentation from Tigera for AWS

Anything else you would like to add:
Updating the MTU helps improve throughput on the cluster.

[detiber]

@rbankston if we do go down this path, then we need to ensure that we are not including any instance types in the template that do not support jumbo frames: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html

[rbankston]

@detiber indeed. We currently don't include any that aren't available to have jumbo frames. Would allowing the mtu size to be set possibly better going forward?

[detiber]

@vincepri I'm interested in your thoughts here. I think we want to be able to better optimize for MTU, but we also need to be careful that we don't make it too easy for users to shoot themselves in the foot.

[vincepri]

The MTU is indeed configured at 1500/1440. The process for changing it is documented here and looks pretty straightforward to change the value.