Bad performance of storage write

[fmoessbauer]

When flashing an image via mtda-cli -r <host> storage write to remove target, the speed of the copy is significantly lower than when first using scp and then bmaptool copy or dd. This is mainly due to the RPC based streaming of the image:

• serialization / deserialization overhead of the ~1MB chunks
• sequential RPC calls (Once the chunk copy command returns, the next is started. This makes each invocation delayed by at least one roundtrip time)
• Python GIL
  • The progress reporting calls interrupt the copy operations
  • The download operations interrupt the disk write operations

With the current zeroRPC based architecture there is not much room for improvement. Probably the best strategy would be to use zeroRPC only on the control path, but stream the image directly using zeromq. Also the receiver and the writer need to be put into two separate processes to parallelize this.

[fmoessbauer]

With the on-the-fly zstd compression added in #420 , at least for uncompressed images the performance is much better now.