Handle network issues gracefully

[wsry]

Motivation

Currently, network issues like unstable network may cause task failover which may further lead to reproducing of data. In fact, we can improve the behavior by reconnecting and retransmitting to the remote ShuffleWorker.

Changes

For network issues, the client and server should not fail immediately, instead, if the client can reconnect to the server in some timeout, no failover or data reproducing should be triggered. At the same time, we may need a switch which can disable this feature.

Test

• Unit test.
• Test manually on a cluster.