fix(backport v3.8.x): ota_proxy: fix content-encoding header is not passed through in direct proxy mode

[Bodong-Yang]

Introduction

This PR fixes a bug that prevents ota_proxy properly handling request of file being gzip by remote HTTP server. The cause of bug is ota_proxy doesn't include content-encoding header in response back to client when in direct proxy passthrough mode.
The bug is fixed by using multidict when parsing/preparing headers, so multidict<7.0,>=4.5 is added to the direct dep of otaclient. Note that multidict is used by aiohttp, so it is already a indirect dep of otaclient.

Tests

• OTA test with ota_proxy in direct proxy mode(by setting 0 hard space limit).
• OTA test with ota_proxy in caching mode(by setting soft & hard space limit to 100).
• OTA test with full local cache presented.
• OTA test with caching mode and LRU cache rotating being enabled in the middle of OTA.

https://tier4.atlassian.net/browse/RT4-13280

[Bodong-Yang]

Still set the soft and hard limit to a reasonable lower values.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/ota_metadata/legacy
__init__.py	11	0	100%
parser.py	341	40	88%	103, 162, 167, 203–204, 214–215, 218, 230, 288, 298–301, 340–343, 423, 426, 434–436, 449, 458–459, 462–463, 675–676, 686, 688, 691, 718–720, 770, 773–775
types.py	84	13	84%	37, 40–42, 112–116, 122–125
src/ota_proxy
__init__.py	36	10	72%	59, 61, 63, 72, 81–82, 102, 104–106
__main__.py	7	7	0%	16–18, 20, 22–23, 25
_consts.py	17	0	100%
cache_control_header.py	68	4	94%	71, 91, 113, 121
cache_streaming.py	144	22	84%	154–156, 184–186, 211, 225, 229–230, 265–266, 268, 280, 349, 355–356, 359, 367–370
config.py	17	0	100%
db.py	74	18	75%	110, 116, 154, 160–161, 164, 170, 172, 193–200, 202–203
errors.py	5	0	100%
lru_cache_helper.py	47	2	95%	84–85
ota_cache.py	219	63	71%	70–71, 140, 151–152, 184–185, 202, 239–243, 247–249, 251, 253–260, 262–264, 267–268, 272–273, 277, 324, 332–334, 413–416, 430, 433–434, 448–449, 451–453, 457–459, 465, 467–469, 474, 501, 507, 534, 586–588
server_app.py	141	39	72%	79, 82, 88, 107, 111, 170, 179, 221–222, 224–226, 229, 234–235, 238, 241–242, 245, 248, 251, 254, 267–268, 271–272, 274, 277, 303–306, 309, 323–325, 331–333
utils.py	14	0	100%
src/otaclient
__init__.py	5	2	60%	17, 19
__main__.py	1	1	0%	16
log_setting.py	52	5	90%	53, 55, 64–66
src/otaclient/app
__main__.py	1	1	0%	16
configs.py	76	0	100%
errors.py	120	0	100%
interface.py	3	0	100%
main.py	46	5	89%	52–53, 75–77
ota_client.py	381	115	69%	80, 88, 109, 136, 138–139, 142–143, 145–146, 150, 154–155, 160–161, 167, 169, 207–210, 216, 220, 226, 345, 357–358, 360, 369, 372, 377–378, 381, 387, 389–393, 412–415, 418–429, 457–460, 506–507, 511, 513–514, 544–545, 554–561, 568, 571–577, 622–625, 633, 669–671, 676–678, 681–682, 684–685, 687, 745–746, 749, 757–758, 761, 772–773, 776, 784–785, 788, 799, 818, 845, 864, 882
ota_client_stub.py	393	109	72%	75–77, 79–80, 88–91, 94–96, 100, 105–106, 108–109, 112, 114–115, 118–120, 123–124, 127–129, 134–139, 143, 146–150, 152–153, 161–163, 166, 203–205, 210, 246, 271, 274, 277, 381, 405, 407, 431, 477, 534, 604–605, 644, 663–665, 671–674, 678–680, 687–689, 692, 696–699, 752, 841–843, 850, 880–881, 884–888, 897–906, 913, 919, 922–923, 927, 930
update_stats.py	104	9	91%	57, 103, 105, 114, 116, 125, 127, 148, 179
src/otaclient/boot_control
__init__.py	4	0	100%
_common.py	248	112	54%	74–75, 96–98, 114–115, 135–136, 155–156, 175–176, 195–196, 218–220, 235–236, 260–266, 287, 295, 313, 321, 340–341, 344–345, 368, 370–379, 381–390, 392–394, 413, 416, 424, 432, 448–450, 452–457, 550, 555, 560, 673, 677–678, 681, 689, 691–692, 718–719, 721–724, 729, 735–736, 739–740, 742, 749–750, 761–767, 777–779, 783–784, 787–788, 791, 797
_firmware_package.py	94	22	76%	83, 87, 137, 181, 187, 210–211, 214–219, 221–222, 225–230, 232
_grub.py	417	128	69%	217, 265–268, 274–278, 315–316, 323–328, 331–337, 340, 343–344, 349, 351–353, 362–368, 370–371, 373–375, 384–386, 388–390, 469–470, 474–475, 527, 533, 559, 581, 585–586, 601–603, 627–630, 642, 646–648, 650–652, 711–714, 739–742, 765–768, 780–781, 784–785, 820, 826, 846–847, 849, 861, 864, 867, 870, 874–876, 894–897, 925–928, 933–941, 946–954
_jetson_cboot.py	262	262	0%	20, 22–25, 27–29, 35–38, 40–41, 57–58, 60, 62–63, 69, 73, 132, 135, 137–138, 141, 148–149, 157–158, 161, 167–168, 176, 185–189, 191, 197, 200–201, 207, 210–211, 216–217, 219, 225–226, 229–230, 233–235, 237, 243, 248–250, 252–254, 259, 261–264, 266–267, 276–277, 280–281, 286–287, 290–294, 297–298, 303–304, 307, 310–314, 319–322, 325, 328–329, 332, 335–336, 339, 343–348, 352–353, 357, 360–361, 364, 367–370, 372, 375–376, 380, 383, 386–389, 391, 398, 402–403, 406–407, 413–414, 420, 422–423, 427, 429, 431–433, 436, 440, 443, 446–447, 449, 452, 460–461, 468, 478, 481, 489–490, 495–498, 500, 507, 509–511, 517–518, 522–523, 526, 530, 533, 535, 542–546, 548, 560–563, 566, 569, 571, 578, 582–583, 585–586, 588–590, 592, 594, 597, 600, 603, 605–606, 609–613, 617–619, 621, 629–633, 635, 638, 642, 645, 656–657, 662, 672, 675–683, 687–696, 700–709, 713, 715–717, 719–720, 722–723
_jetson_common.py	172	45	73%	132, 140, 288–291, 294, 311, 319, 354, 359–364, 382, 408–409, 411–413, 417–420, 422–423, 425–429, 431, 438–439, 442–443, 453, 456–457, 460, 462, 506–507
_jetson_uefi.py	397	271	31%	127–129, 134–135, 154–156, 161–164, 331, 449, 451–454, 458, 462–463, 465–473, 475, 487–488, 491–492, 495–496, 499–501, 505–506, 511–513, 517, 521–522, 525–526, 529–530, 534, 537–538, 540, 545–546, 550, 553–554, 559, 563–565, 569–571, 573, 577–580, 582–583, 605–606, 610–611, 613, 617, 621–622, 625–626, 633, 636–638, 641, 643–644, 649–650, 653–656, 658–659, 664, 666–667, 675, 678–681, 683–684, 686, 690–691, 695, 703–707, 710–711, 713, 716–720, 723, 726–730, 734–735, 738–743, 746–747, 750–753, 755–756, 763–764, 774–777, 780, 783–786, 789–793, 796–797, 800, 803–806, 809, 811, 816–817, 820, 823–826, 828, 834, 839–840, 859–860, 863, 871–872, 879, 889, 892, 899–900, 905–908, 916–919, 927–928, 940–943, 945, 948, 951, 959, 970–972, 974–976, 978–982, 987–988, 990, 1003, 1007, 1010, 1020, 1025, 1033–1034, 1037, 1041, 1043–1045, 1051–1052, 1057, 1065–1072, 1077–1085, 1090–1098, 1104–1106
_rpi_boot.py	287	134	53%	55, 58, 122–123, 127, 135–138, 152–155, 162–163, 165–166, 171–172, 175–176, 185–186, 224, 230–234, 237, 255–257, 261–263, 268–270, 274–276, 286–287, 290, 293, 295–296, 298–299, 301–303, 309, 312–313, 323–326, 334–338, 340, 342–343, 348–349, 356–362, 393, 395–398, 408–411, 415–416, 418–422, 450–453, 472–475, 480, 483, 501–504, 509–517, 522–530, 547–550, 556–558, 561, 564
configs.py	55	0	100%
protocol.py	4	0	100%
selecter.py	41	29	29%	45–47, 50–51, 55–56, 59–61, 64, 66, 70, 78–80, 82–83, 85–86, 90, 92, 94–95, 97, 99–100, 102, 104
src/otaclient/configs
_common.py	8	0	100%
ecu_info.py	58	1	98%	108
proxy_info.py	52	2	96%	88, 90
src/otaclient/create_standby
__init__.py	12	5	58%	29–31, 33, 35
common.py	224	44	80%	62, 65–66, 70–72, 74, 78–79, 81, 127, 175–177, 179–181, 183, 186–189, 193, 204, 278–279, 281–286, 298, 335, 363, 366–368, 384–385, 399, 403, 425–426
interface.py	5	0	100%
rebuild_mode.py	97	9	90%	93–95, 107–112
src/otaclient_api/v2
api_caller.py	39	6	84%	45–47, 83–85
api_stub.py	17	0	100%
types.py	256	23	91%	86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 586
src/otaclient_common
__init__.py	34	8	76%	42–44, 61, 63, 69, 76–77
common.py	156	18	88%	47, 202, 205–207, 222, 229–231, 297–299, 309, 318–320, 366, 370
downloader.py	199	10	94%	107–108, 126, 153, 369, 424, 428, 516–517, 526
linux.py	61	15	75%	51–53, 59, 69, 74, 76, 108–109, 133–134, 190, 195–196, 198
logging.py	29	1	96%	55
persist_file_handling.py	118	18	84%	113, 118, 150–152, 163, 192–193, 228–232, 242–244, 246–247
proto_streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
proto_wrapper.py	398	48	87%	87, 165, 172, 184–186, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 707, 713, 760–763, 765, 803–805
retry_task_map.py	105	5	95%	158–159, 161, 181–182
typing.py	25	3	88%	69–70, 72
TOTAL	6323	1692	73%

Tests	Skipped	Failures	Errors	Time
217	0 💤	0 ❌	0 🔥	12m 51s ⏱️

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
65.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud