refactor: refine downloader

[Bodong-Yang]

Description

Test

Tested by from same running image and then OTA to the same image again, but with /home/autoware/autoware.proj folder removed to generate need-to-be-downloaded delta.

Used image: https://evaluation.ci.dev.tier4.jp/evaluation/vehicle_catalogs/8cbe2292-5300-4762-8f25-0d53501986a9/firmware_releases/4bbb9de1-c8ab-4d9a-ae39-8eaab2758c29?project_id=tier4_demo.

1. with PR
   Memory usage is stable at around 383MB

2. with v3.7.1
   Memory usage is stable at around 385MB

Check list

• test file(s) that covers the change(s) is implemented.
• local test is passed.

Changes

Behavior changes

Does this PR introduce behavior change(s)?

• Yes, internal behaivor (will not impact user experience).
• Yes, external behaivor (will impact user experience).
• No.

Previous behavior

Behavior with this PR

Breaking change

Does this PR introduce breaking change?

• Yes.
• No.

Related links & tickets

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	5	0	100%
__main__.py	1	1	0%	16
_utils
__init__.py	31	5	83%	69–71, 86, 88
linux.py	47	11	76%	45–47, 53, 63, 68, 70, 102–103, 127–128
logging.py	29	14	51%	41–44, 49–52, 54–55, 57–58, 62–63
typing.py	15	0	100%
app
__main__.py	1	1	0%	16
common.py	383	39	89%	65, 145, 251, 254–256, 271, 278–280, 346–348, 358, 367–369, 475, 487, 606, 610, 698, 700, 732–734, 736, 762–765, 770, 774–778, 804–805
configs.py	75	0	100%
downloader.py	267	44	83%	70, 83–84, 303, 308, 312, 330–331, 381–385, 404–406, 409–410, 413–414, 435–438, 442–443, 447–448, 452–453, 462, 531–533, 549, 569–571, 575, 577, 580, 585–587
errors.py	112	0	100%
interface.py	5	0	100%
log_setting.py	53	5	90%	54, 56, 65–67
main.py	31	2	93%	45–46
ota_client.py	388	137	64%	57–58, 78, 184–185, 198, 201, 205, 208, 245–248, 260–263, 266–267, 271–276, 286–289, 294–295, 297, 306, 309, 314–315, 318, 324, 326, 329, 371–374, 379, 383, 386, 399–402, 405–412, 415–422, 428–431, 460, 463–464, 466, 469–472, 474–475, 480–481, 484, 498–505, 512, 515–521, 568–571, 579, 615, 620–623, 628–630, 633–634, 636–637, 639–640, 642, 702–703, 706, 714–715, 718, 729–730, 733, 741–742, 745, 756, 775, 802, 821, 839
ota_client_call.py	38	6	84%	42–44, 80–82
ota_client_service.py	27	3	88%	55–57
ota_client_stub.py	394	109	72%	76–78, 80–81, 89–92, 95–97, 101, 106–107, 109–110, 113, 115–116, 119–121, 124–125, 128–130, 135–140, 144, 147–151, 153–154, 162–164, 167, 204–206, 211, 247, 272, 275, 278, 382, 406, 408, 432, 478, 535, 605–606, 645, 664–666, 672–675, 679–681, 688–690, 693, 697–700, 753, 840–842, 849, 879–880, 883–887, 896–905, 912, 918, 921–922, 926, 929
ota_metadata.py	314	32	89%	143, 148, 184–185, 195–196, 199, 211, 269, 279–282, 321–324, 404, 407, 415–417, 430, 439–440, 443–444, 717–718, 721, 725, 728
ota_status.py	14	2	85%	34, 42
update_stats.py	105	2	98%	159, 169
app/boot_control
__init__.py	4	0	100%
_common.py	233	76	67%	73–74, 95–97, 113–114, 134–135, 154–155, 174–175, 194–195, 210–211, 232, 240, 258, 266, 285–286, 289–290, 313, 315–324, 326–335, 337–339, 358, 361, 369, 377, 393–395, 397–402, 495, 500, 505, 618, 622–623, 626, 634, 636–637, 711–712, 722, 740
_grub.py	417	128	69%	216, 264–267, 273–277, 314–315, 322–327, 330–336, 339, 342–343, 348, 350–352, 361–367, 369–370, 372–374, 383–385, 387–389, 468–469, 473–474, 526, 532, 558, 580, 584–585, 600–602, 626–629, 641, 645–647, 649–651, 710–713, 738–741, 764–767, 779–780, 783–784, 819, 825, 845–846, 848, 860, 863, 866, 869, 873–875, 893–896, 924–927, 932–940, 945–953
_jetson_cboot.py	270	214	20%	66–67, 74–75, 93–102, 114, 121–122, 134, 140–141, 151–153, 165–166, 177–178, 181–182, 185–186, 189–193, 196–197, 201–202, 207–208, 210–214, 216–222, 224–225, 230, 233, 236–237, 240, 244–245, 249–250, 254, 257, 260, 264–270, 272–274, 279, 282, 285, 289, 296, 298–301, 314, 317, 321, 323–325, 329, 336, 338, 341, 347–348, 353, 361, 369–371, 380–381, 383–385, 391, 394–396, 400–401, 403, 406, 415–417, 420, 423, 426–431, 433–435, 438, 441, 445–450, 454–456, 461–462, 466–467, 470, 473, 476–477, 480, 483, 488, 491, 494–495, 497, 499, 502, 505, 507–508, 511–515, 520–521, 523, 531–535, 537, 540, 543, 554–555, 560, 570, 573–581, 586–594, 599–607, 613–615, 618, 621
_jetson_common.py	142	66	53%	52, 76, 131–136, 138, 143–145, 150–153, 161–162, 169–170, 175–176, 192–193, 195–197, 200–202, 205, 209, 213, 217–219, 225–226, 228, 261, 287–288, 290–292, 296–299, 301–302, 304–308, 310, 317–318, 321, 323, 333, 336–337, 340, 342
_rpi_boot.py	258	122	52%	84–86, 92–93, 95–97, 99, 102–103, 108–109, 118–119, 123, 125, 129, 133–136, 141–143, 147–150, 174–176, 182–184, 197–199, 205–207, 220–227, 229, 233–235, 238–241, 244–245, 250, 254, 258, 262, 296, 323–325, 335–338, 342–348, 388–390, 432–436, 455–458, 463, 466, 490–493, 498–506, 511–519, 533–536, 542–544, 547
configs.py	46	0	100%
protocol.py	4	0	100%
selecter.py	38	26	31%	44–46, 49–50, 54–55, 58–60, 63, 65, 69, 77–79, 81–82, 84–85, 89, 91–93, 95, 97
app/create_standby
__init__.py	12	5	58%	28–30, 32, 34
common.py	215	43	80%	61, 64–65, 69–71, 73, 77–78, 80, 126, 174–176, 178–180, 182, 185–188, 192, 203, 275–276, 278–283, 295, 339, 342–344, 360–361, 375, 379, 402–403
interface.py	5	0	100%
rebuild_mode.py	89	1	98%	103
app/proto
__init__.py	30	3	90%	37, 44–45
_common.py	398	45	88%	87, 165, 172, 184–186, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 707, 713, 760–763, 765
_ota_metafiles_wrapper.py	84	13	84%	38, 41–43, 113–117, 123–126
_otaclient_v2_pb2_wrapper.py	256	23	91%	86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 586
streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
wrapper.py	3	0	100%
configs
_common.py	8	0	100%
ecu_info.py	57	1	98%	107
proxy_info.py	52	2	96%	88, 90
ota_proxy
__init__.py	36	10	72%	59, 61, 63, 72, 81–82, 102, 104–106
__main__.py	7	7	0%	16–18, 20, 22–23, 25
_consts.py	15	0	100%
cache_control.py	68	4	94%	71, 91, 113, 121
config.py	18	0	100%
db.py	146	15	89%	75, 81, 103, 113, 116, 145–147, 166, 199, 208–209, 229, 258, 300
errors.py	5	0	100%
orm.py	112	10	91%	92, 97, 102, 108, 114, 141–142, 155, 232, 236
ota_cache.py	401	86	78%	98–99, 218, 229, 256–258, 278, 294–295, 297, 320–321, 327, 331, 333, 360–362, 378, 439–440, 482–483, 553, 566–569, 619, 638–639, 671–672, 683, 717–721, 725–727, 729, 731–738, 740–742, 745–746, 750–751, 755, 802, 810–812, 891–894, 898, 901–902, 916–917, 919–921, 925–926, 932–933, 964, 970, 997, 1026–1028
server_app.py	138	39	71%	76, 79, 85, 101, 103, 162, 171, 213–214, 216–218, 221, 226–228, 231–232, 235, 238, 241, 244, 257–258, 261–262, 264, 267, 293–296, 299, 313–315, 321–323
utils.py	23	1	95%	33
TOTAL	5967	1361	77%

Tests	Skipped	Failures	Errors	Time
179	0 💤	0 ❌	0 🔥	4m 53s ⏱️

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud