Add parallel convert #36

The convert package implements efficient conversion of qcow2 image to raw sparse image using multiple threads. This will be useful for users of this library that need to work with raw images. Signed-off-by: Nir Soffer <[email protected]>

Using the same benchmarks infrastructure we can run compare the serial read (using io.Reader inteface) and parallel copy using the convert package. It may be useful to change BenchmarkRead() to actually copy the data to a file instead of discarding it. We can do this later. % go test -bench Benchmark/ Benchmark0p/qcow2/read-12 421 2569832 ns/op 104456.41 MB/s 1050513 B/op 39 allocs/op Benchmark0p/qcow2/convert-12 550 2188487 ns/op 122657.99 MB/s 9440266 B/op 60 allocs/op Benchmark0p/qcow2_zlib/read-12 534 2230269 ns/op 120360.10 MB/s 1050511 B/op 39 allocs/op Benchmark0p/qcow2_zlib/read#01-12 570 2126408 ns/op 126238.91 MB/s 9440130 B/op 60 allocs/op Benchmark50p/qcow2/read-12 100 10936881 ns/op 24544.06 MB/s 1181852 B/op 45 allocs/op Benchmark50p/qcow2/convert-12 28 60437272 ns/op 4441.55 MB/s 10157185 B/op 79 allocs/op Benchmark50p/qcow2_zlib/read-12 2 892929271 ns/op 300.62 MB/s 185073236 B/op 43275 allocs/op Benchmark50p/qcow2_zlib/convert-12 6 187644889 ns/op 1430.55 MB/s 194612194 B/op 43346 allocs/op Benchmark100p/qcow2/read-12 60 19555156 ns/op 13727.09 MB/s 1181857 B/op 45 allocs/op Benchmark100p/qcow2/convert-12 22 66297214 ns/op 4048.97 MB/s 10233635 B/op 83 allocs/op Benchmark100p/qcow2_zlib/read-12 1 1775486625 ns/op 151.19 MB/s 368587320 B/op 86425 allocs/op Benchmark100p/qcow2_zlib/convert-12 3 338774583 ns/op 792.37 MB/s 378895709 B/op 86617 allocs/op Signed-off-by: Nir Soffer <[email protected]>

To make room for a new convert sub command, using parallel convert. This is also a better example, having only the relevant argument and separate file for every example command. Example usage: % ./go-qcow2reader-example Usage: ./go-qcow2reader-example COMMAND [OPTIONS...] Available commands: info show image information read read image data and print to stdout % ./go-qcow2reader-example info /tmp/images/test.zlib.qcow2 { "type": "qcow2", "size": 3758096384, ... % time ./go-qcow2reader-example read /tmp/images/test.zlib.qcow2 >/dev/null ./go-qcow2reader-example read /tmp/images/test.zlib.qcow2 > /dev/null 10.05s user 0.35s system 101% cpu 10.279 total Signed-off-by: Nir Soffer <[email protected]>

Add convert subcommand using the new convert package and include the new command in the functional tests. The new example is also good way to benchmark the library with real images as we can see bellow. Testing show significant speedup for compressed images and for unallocated or zero clusters, and smaller speedup for uncompressed images. | image | size | compression | throughput | speedup | |--------------|-----------|-------------|--------------|---------| | Ubuntu 24.04 | 3.5 GiB | - | 6.04 GiB/s | 1.51 | | Ubuntu 24.04 | 3.5 GiB | zlib | 1.62 GiB/s | 5.42 | | Empty image | 100.0 GiB | - | 240.15 GiB/s | 7.32 | Ubuntu 24.04 image in qcow2 format: % hyperfine -w3 "./go-qcow2reader-example read /tmp/images/test.qcow2 >/tmp/tmp.img" \ "./go-qcow2reader-example convert /tmp/images/test.qcow2 /tmp/tmp.img" Benchmark 1: ./go-qcow2reader-example read /tmp/images/test.qcow2 >/tmp/tmp.img Time (mean ± σ): 874.8 ms ± 41.3 ms [User: 64.3 ms, System: 717.5 ms] Range (min … max): 851.9 ms … 985.3 ms 10 runs Benchmark 2: ./go-qcow2reader-example convert /tmp/images/test.qcow2 /tmp/tmp.img Time (mean ± σ): 579.4 ms ± 22.8 ms [User: 90.5 ms, System: 681.2 ms] Range (min … max): 556.0 ms … 631.3 ms 10 runs Summary './go-qcow2reader-example convert /tmp/images/test.qcow2 /tmp/tmp.img' ran 1.51 ± 0.09 times faster than './go-qcow2reader-example read /tmp/images/test.qcow2 >/tmp/tmp.img' Ubuntu 24.04 image in qcow2 compressed format: % hyperfine -w3 -r3 "./go-qcow2reader-example read /tmp/images/test.zlib.qcow2 >/tmp/tmp.img" \ "./go-qcow2reader-example convert /tmp/images/test.zlib.qcow2 /tmp/tmp.img" Benchmark 1: ./go-qcow2reader-example read /tmp/images/test.zlib.qcow2 >/tmp/tmp.img Time (mean ± σ): 11.702 s ± 0.200 s [User: 10.423 s, System: 1.121 s] Range (min … max): 11.533 s … 11.923 s 3 runs Benchmark 2: ./go-qcow2reader-example convert /tmp/images/test.zlib.qcow2 /tmp/tmp.img Time (mean ± σ): 2.161 s ± 0.027 s [User: 10.980 s, System: 1.032 s] Range (min … max): 2.139 s … 2.191 s 3 runs Summary './go-qcow2reader-example convert /tmp/images/test.zlib.qcow2 /tmp/tmp.img' ran 5.42 ± 0.11 times faster than './go-qcow2reader-example read /tmp/images/test.zlib.qcow2 >/tmp/tmp.img' 100 GiB empty sparse image in qcow2 format. Comparing to the read command is not useful since it writes 100 GiB of zeros, so I'm comparing 1 and 8 workers. % hyperfine -w3 "./go-qcow2reader-example convert -workers 1 /tmp/images/test.0p.qcow2 /tmp/tmp.img" \ "./go-qcow2reader-example convert -workers 8 /tmp/images/test.0p.qcow2 /tmp/tmp.img" Benchmark 1: ./go-qcow2reader-example convert -workers 1 /tmp/images/test.0p.qcow2 /tmp/tmp.img Time (mean ± σ): 3.050 s ± 0.023 s [User: 2.991 s, System: 0.054 s] Range (min … max): 3.036 s … 3.107 s 10 runs Benchmark 2: ./go-qcow2reader-example convert -workers 8 /tmp/images/test.0p.qcow2 /tmp/tmp.img Time (mean ± σ): 416.4 ms ± 3.4 ms [User: 3252.7 ms, System: 16.3 ms] Range (min … max): 412.0 ms … 421.9 ms 10 runs Summary './go-qcow2reader-example convert -workers 8 /tmp/images/test.0p.qcow2 /tmp/tmp.img' ran 7.32 ± 0.08 times faster than './go-qcow2reader-example convert -workers 1 /tmp/images/test.0p.qcow2 /tmp/tmp.img' Signed-off-by: Nir Soffer <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add parallel convert #36

Add parallel convert #36

Commits on Oct 25, 2024