[Flink-27826] Support training very high dimensional logistic regression

[zhipeng93]

What is the purpose of the change

This PR aims to (1) support training high dimensional logistic regression models with a parameter-server style infrasture, with flink-ml-iteration infra. (2) Abstract infrasture that could be reused by other machine learning algorithms.

Brief change log

• Added WorkerOperator and ServerOperator to support distributed communication among parallel tasks, following the idea of parameter servers.
• Added IterationStages to model iterative training process as a sequence of local computation and global communication process, so as to ease the programming of using flink-ml-iterations.
• Added MLSession to store the information that can be shared among different IterationStages.
• Expanded the model data format of LogisticRegression from a single vector as a several sliced vectors, by adding startIndex and endIndex in model data format.
• Used third party jar fastutil to accelerate the process of primitive collections.
• Added unit test to verify the infra.
• Left several TODOs for later PRs.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)

Documentation

• Does this pull request introduce a new feature? (yes)
• If yes, how is the feature documented? (avaDocs)

[lindong28]

Thanks for the PR. Left some comments below.

[lindong28]

Thanks for the PR. Left some comments below.

[Fanoid]

Hi, @zhipeng93 . As you have mentioned the proposed solution in this PR could be an alternative to implement GBDT in #210, can we make the target applications/scenarios clearer in the description of this PR?

Although the description states the infrastructure can be reused by other ML algorithms, the infra codes seem not easy to be adapted. Many APIs/enums/codes in the infra codes are specifically designed/hard-coded for gradient-based algorithms, like MessageType, model format (double[]), ModelUpdater, codes in ServerOperator, etc. It is very difficult to extend current codes to support POJO messages and POJO model types unless developers make changes in the infra codes.

If the PS infra is targeting to algorithms more than gradient-based algorithms, the infra APIs may need to reflect the considerations on those cases.

[zhipeng93]

This PR introduces too many changes (i.e., communication infra, vectors, algorithms) and it is hard to review. I will open another PR [1] to introduce the communication infra first and address the comments raised in this PR.

[1] #251