refactor: improve the performance of active slot scanning

[Bodong-Yang]

Description

This PR applies the following refinement to speed up the active slot scanning during delta_calculating and preparing:

1. when using thread pool executor, for each worker thread, now pre-allocated buffer area is used throughout the work's life-cycle, data reading from source files are re-using the same buffer area, avoiding unnecessary garbage collection and memory allocation.
2. not waiting for futures saving the future objects and call wait on it, instead after all tasks are scheduled, directly call executor.shutdown(wait=True) to wait, this will saving lots of memory and speed up the overall progress.
3. wait for all tasks instead of wait for each folder being scanned, reduce the idling time of the thread worker.

Check list

• VM test is passed.
• local test is passed.

Test result

The test is executed by update the VM from same image to same image, maximize the re-usable delta.

Used image: https://evaluation.ci.dev.tier4.jp/evaluation/vehicle_catalogs/8cbe2292-5300-4762-8f25-0d53501986a9/firmware_releases/4bbb9de1-c8ab-4d9a-ae39-8eaab2758c29?project_id=tier4_demo.

1. with this PR(otaclient is manually installed.)
   During the active slot scanning and delta calculation, the memory usage is stable around 280MB.
   

2. with v3.7.1
   During the active slot scanning and delta calculation, the memory usage is around 350MB, with up and down.
   

Changes

Behavior changes

Does this PR introduce behavior change(s)?

• No.

Breaking change

Does this PR introduce breaking change?

• No.

Related links & tickets

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	5	0	100%
__main__.py	1	1	0%	16
_utils
__init__.py	31	5	83%	69–71, 86, 88
linux.py	47	11	76%	45–47, 53, 63, 68, 70, 102–103, 127–128
logging.py	29	14	51%	41–44, 49–52, 54–55, 57–58, 62–63
typing.py	15	0	100%
app
__main__.py	1	1	0%	16
common.py	383	39	89%	65, 145, 251, 254–256, 271, 278–280, 346–348, 358, 367–369, 475, 487, 606, 610, 698, 700, 732–734, 736, 762–765, 770, 774–778, 804–805
configs.py	75	0	100%
downloader.py	268	44	83%	70, 83–84, 299, 304, 308, 326–327, 377–381, 400–402, 405–406, 409–410, 431–434, 438–439, 443–444, 448–449, 458, 533–535, 551, 571–573, 577, 579, 582, 587–589
errors.py	112	0	100%
interface.py	5	0	100%
log_setting.py	53	5	90%	54, 56, 65–67
main.py	31	2	93%	45–46
ota_client.py	388	137	64%	57–58, 78, 184–185, 198, 201, 205, 208, 245–248, 260–263, 266–267, 271–276, 286–289, 294–295, 297, 306, 309, 314–315, 318, 324, 326, 329, 371–374, 379, 383, 386, 399–402, 405–412, 415–422, 428–431, 460, 463–464, 466, 469–472, 474–475, 480–481, 484, 498–505, 512, 515–521, 568–571, 579, 615, 620–623, 628–630, 633–634, 636–637, 639–640, 642, 702–703, 706, 714–715, 718, 729–730, 733, 741–742, 745, 756, 775, 802, 821, 839
ota_client_call.py	38	6	84%	42–44, 80–82
ota_client_service.py	27	3	88%	55–57
ota_client_stub.py	394	109	72%	76–78, 80–81, 89–92, 95–97, 101, 106–107, 109–110, 113, 115–116, 119–121, 124–125, 128–130, 135–140, 144, 147–151, 153–154, 162–164, 167, 204–206, 211, 247, 272, 275, 278, 382, 406, 408, 432, 478, 535, 605–606, 645, 664–666, 672–675, 679–681, 688–690, 693, 697–700, 753, 840–842, 849, 879–880, 883–887, 896–905, 912, 918, 921–922, 926, 929
ota_metadata.py	314	32	89%	143, 148, 184–185, 195–196, 199, 211, 269, 279–282, 321–324, 404, 407, 415–417, 430, 439–440, 443–444, 717–718, 721, 725, 728
ota_status.py	14	2	85%	34, 42
update_stats.py	105	2	98%	159, 169
app/boot_control
__init__.py	4	0	100%
_common.py	233	76	67%	73–74, 95–97, 113–114, 134–135, 154–155, 174–175, 194–195, 210–211, 232, 240, 258, 266, 285–286, 289–290, 313, 315–324, 326–335, 337–339, 358, 361, 369, 377, 393–395, 397–402, 495, 500, 505, 618, 622–623, 626, 634, 636–637, 711–712, 722, 740
_grub.py	417	128	69%	216, 264–267, 273–277, 314–315, 322–327, 330–336, 339, 342–343, 348, 350–352, 361–367, 369–370, 372–374, 383–385, 387–389, 468–469, 473–474, 526, 532, 558, 580, 584–585, 600–602, 626–629, 641, 645–647, 649–651, 710–713, 738–741, 764–767, 779–780, 783–784, 819, 825, 845–846, 848, 860, 863, 866, 869, 873–875, 893–896, 924–927, 932–940, 945–953
_jetson_cboot.py	270	214	20%	66–67, 74–75, 93–102, 114, 121–122, 134, 140–141, 151–153, 165–166, 177–178, 181–182, 185–186, 189–193, 196–197, 201–202, 207–208, 210–214, 216–222, 224–225, 230, 233, 236–237, 240, 244–245, 249–250, 254, 257, 260, 264–270, 272–274, 279, 282, 285, 289, 296, 298–301, 314, 317, 321, 323–325, 329, 336, 338, 341, 347–348, 353, 361, 369–371, 380–381, 383–385, 391, 394–396, 400–401, 403, 406, 415–417, 420, 423, 426–431, 433–435, 438, 441, 445–450, 454–456, 461–462, 466–467, 470, 473, 476–477, 480, 483, 488, 491, 494–495, 497, 499, 502, 505, 507–508, 511–515, 520–521, 523, 531–535, 537, 540, 543, 554–555, 560, 570, 573–581, 586–594, 599–607, 613–615, 618, 621
_jetson_common.py	142	66	53%	52, 76, 131–136, 138, 143–145, 150–153, 161–162, 169–170, 175–176, 192–193, 195–197, 200–202, 205, 209, 213, 217–219, 225–226, 228, 261, 287–288, 290–292, 296–299, 301–302, 304–308, 310, 317–318, 321, 323, 333, 336–337, 340, 342
_rpi_boot.py	258	122	52%	84–86, 92–93, 95–97, 99, 102–103, 108–109, 118–119, 123, 125, 129, 133–136, 141–143, 147–150, 174–176, 182–184, 197–199, 205–207, 220–227, 229, 233–235, 238–241, 244–245, 250, 254, 258, 262, 296, 323–325, 335–338, 342–348, 388–390, 432–436, 455–458, 463, 466, 490–493, 498–506, 511–519, 533–536, 542–544, 547
configs.py	46	0	100%
protocol.py	4	0	100%
selecter.py	38	26	31%	44–46, 49–50, 54–55, 58–60, 63, 65, 69, 77–79, 81–82, 84–85, 89, 91–93, 95, 97
app/create_standby
__init__.py	12	5	58%	28–30, 32, 34
common.py	219	43	80%	62, 65–66, 70–72, 74, 78–79, 81, 127, 175–177, 179–181, 183, 186–189, 193, 204, 278–279, 281–286, 299, 354, 357–359, 375–376, 390, 394, 416–417
interface.py	5	0	100%
rebuild_mode.py	89	1	98%	103
app/proto
__init__.py	30	3	90%	37, 44–45
_common.py	398	45	88%	87, 165, 172, 184–186, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 707, 713, 760–763, 765
_ota_metafiles_wrapper.py	84	13	84%	38, 41–43, 113–117, 123–126
_otaclient_v2_pb2_wrapper.py	256	23	91%	86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 586
streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
wrapper.py	3	0	100%
configs
_common.py	8	0	100%
ecu_info.py	57	1	98%	107
proxy_info.py	52	2	96%	88, 90
ota_proxy
__init__.py	36	10	72%	59, 61, 63, 72, 81–82, 102, 104–106
__main__.py	7	7	0%	16–18, 20, 22–23, 25
_consts.py	15	0	100%
cache_control.py	68	4	94%	71, 91, 113, 121
config.py	18	0	100%
db.py	146	15	89%	75, 81, 103, 113, 116, 145–147, 166, 199, 208–209, 229, 258, 300
errors.py	5	0	100%
orm.py	112	10	91%	92, 97, 102, 108, 114, 141–142, 155, 232, 236
ota_cache.py	401	86	78%	98–99, 218, 229, 256–258, 278, 294–295, 297, 320–321, 327, 331, 333, 360–362, 378, 439–440, 482–483, 553, 566–569, 619, 638–639, 671–672, 683, 717–721, 725–727, 729, 731–738, 740–742, 745–746, 750–751, 755, 802, 810–812, 891–894, 898, 901–902, 916–917, 919–921, 925–926, 932–933, 964, 970, 997, 1026–1028
server_app.py	138	39	71%	76, 79, 85, 101, 103, 162, 171, 213–214, 216–218, 221, 226–228, 231–232, 235, 238, 241, 244, 257–258, 261–262, 264, 267, 293–296, 299, 313–315, 321–323
utils.py	23	1	95%	33
TOTAL	5972	1361	77%

Tests	Skipped	Failures	Errors	Time
179	0 💤	0 ❌	0 🔥	4m 56s ⏱️

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud