feat: use file_table and resource_table as sqlite3 db

[Bodong-Yang]

Introduction

Note

This PR is split from the dev PR #453 . This PR only introduces the file_table sqlite3 table implementation, but not yet integrates the file_table into otaclient code.

This PR introduces file_table as sqlite3 db with three tables, naming as follow:

1. ft_regular: table contains all regular file entries.
2. ft_non_regular: table contains all non-regular file entries, including symlinks and chardev(with 0,0 devnum).
3. ft_dir: table contains all directory entries.

Also a new resource_table sqlite3 db with one table is introduced:

1. rs_table: table contains all the unique file entries(by sha256digest) and the necessary metadata of the corresponding resource.

The simple-sqlite3-orm python package(which is developed and maintained by me also) is used to implement the table definitions and ORMs.

For each tables definition(in file_table._table module), helper methods to prepare the entries onto the target(standby slot mount point during OTA) are implemented.
Comparing to previous implementation, xattrs and chardev files(required by supporting storing docker images on system image) are supported now.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/ota_metadata/file_table
__init__.py	4	0	100%
_orm.py	16	0	100%
_table.py	80	8	90%	170, 188–193, 195
_types.py	31	4	87%	47, 54–56
src/ota_metadata/legacy
__init__.py	1	0	100%
parser.py	335	48	85%	106, 170, 175, 211–212, 222–223, 226, 238, 289–291, 295–298, 324–327, 396, 399, 407–409, 422, 431–432, 435–436, 601–603, 653–654, 657, 685–686, 689–690, 692, 696, 698–699, 753, 756–758
types.py	84	13	84%	37, 40–42, 112–116, 122–125
src/ota_metadata/utils
cert_store.py	86	8	90%	58–59, 73, 87, 91, 102, 123, 127
src/ota_proxy
__init__.py	15	6	60%	48, 50, 52, 61, 71–72
__main__.py	8	8	0%	16, 18–20, 22, 24–25, 27
_consts.py	17	0	100%
cache_control_header.py	68	4	94%	71, 91, 113, 121
cache_streaming.py	144	13	90%	211, 225, 229–230, 265–266, 268, 280, 349, 367–370
config.py	20	0	100%
db.py	80	18	77%	103, 109, 167, 173–174, 177, 183, 185, 209–216, 218–219
errors.py	5	0	100%
external_cache.py	28	20	28%	31, 35, 40–42, 44–45, 48–49, 51–53, 60, 63–65, 69–72
lru_cache_helper.py	48	2	95%	95–96
ota_cache.py	234	61	73%	71–72, 143, 146–147, 159–160, 192–193, 210, 231, 250–254, 258–260, 262, 264–271, 273–275, 278–279, 283–284, 288, 335, 343–345, 418, 471–473, 477–479, 485, 487–489, 494, 520–522, 557–559, 586, 592, 607
server_app.py	141	39	72%	79, 82, 88, 107, 111, 170, 179, 221–222, 224–226, 229, 234–235, 238, 241–242, 245, 248, 251, 254, 267–268, 271–272, 274, 277, 303–306, 309, 323–325, 331–333
utils.py	14	0	100%
src/otaclient
__init__.py	5	2	60%	17, 19
__main__.py	1	1	0%	16
_logging.py	51	33	35%	43–44, 46–47, 49–54, 56–57, 59–60, 62–65, 67, 77, 80–82, 84–86, 89–90, 92–96
_otaproxy_ctx.py	43	43	0%	20, 22–30, 32–37, 39, 41–42, 45, 47–51, 54–57, 60–61, 63–64, 66–68, 70, 75–79, 81
_status_monitor.py	185	14	92%	56–57, 169, 172, 192, 195, 211–212, 220, 223, 286, 308, 325–326
_types.py	96	0	100%
_utils.py	30	2	93%	80–81
errors.py	120	1	99%	97
main.py	25	25	0%	17, 19–29, 31–33, 35, 37, 41–42, 44–46, 48–50
ota_core.py	342	139	59%	121, 123–124, 128–129, 131–133, 137–138, 143–144, 150, 152, 211–214, 337, 369–370, 372, 381, 384, 389–390, 393, 399, 401–405, 412, 418, 453–456, 459–470, 473–476, 512–515, 531–532, 536–537, 603–610, 615, 618–625, 650–651, 657, 661–662, 668, 693–695, 697, 737, 759, 786–788, 797–803, 817–823, 825–826, 831–832, 840, 842, 848, 850, 856, 858, 862, 868, 870, 876, 879–881, 891–892, 903–905, 907–908, 910, 912–913, 918, 920, 925
src/otaclient/boot_control
__init__.py	4	0	100%
_firmware_package.py	93	22	76%	82, 86, 136, 180, 186, 209–210, 213–218, 220–221, 224–229, 231
_grub.py	418	127	69%	214, 262–265, 271–275, 312–313, 320–325, 328–334, 337, 340–341, 346, 348–350, 359–365, 367–368, 370–372, 381–383, 385–387, 466–467, 471–472, 524, 530, 556, 578, 582–583, 598–600, 624–627, 639, 643–645, 647–649, 708–711, 736–739, 762–765, 777–778, 781–782, 817, 823, 843–844, 846, 871–873, 891–894, 919–922, 929–932, 937–945, 950–957
_jetson_cboot.py	261	261	0%	20, 22–25, 27–29, 35–40, 42, 58–60, 62, 64–65, 71, 75, 134, 137, 139–140, 143, 150–151, 159–160, 163, 169–170, 178, 187–191, 193, 199, 202–203, 209, 212–213, 218–219, 221, 227–228, 231–232, 235–237, 239, 245, 250–252, 254–256, 261, 263–266, 268–269, 278–279, 282–283, 288–289, 292–296, 299–300, 305–306, 309, 312–316, 321–324, 327, 330–331, 334, 337–338, 341, 345–350, 354–355, 359, 362–363, 366, 369–372, 374, 377–378, 382, 385, 388–391, 393, 400, 404–405, 408–409, 415–416, 422, 424–425, 429, 431, 433–435, 438, 442, 445, 448–449, 451, 454, 462–463, 470, 480, 483, 491–492, 497–500, 502, 509, 511–513, 519–520, 524–525, 528, 532, 535, 537, 544–548, 550, 562–565, 568, 571, 573, 580, 590–592, 594, 596, 599, 602, 605, 607–608, 611–615, 619–621, 623, 631–635, 637, 640, 644, 647, 658–659, 664, 674, 677–683, 687–693, 697–706, 710–718, 722, 724, 726–728
_jetson_common.py	172	45	73%	132, 140, 288–291, 294, 311, 319, 354, 359–364, 382, 408–409, 411–413, 417–420, 422–423, 425–429, 431, 438–439, 442–443, 453, 456–457, 460, 462, 506–507
_jetson_uefi.py	404	274	32%	124–126, 131–132, 151–153, 158–161, 328, 446, 448–451, 455, 459–460, 462–470, 472, 484–485, 488–489, 492–493, 496–498, 502–503, 508–510, 514, 518–519, 522–523, 526–527, 531, 534–535, 537, 542–543, 547, 550–551, 556, 560–561, 564, 568–570, 572, 576–579, 581–582, 604–605, 609–610, 612, 616, 620–621, 624–625, 632, 635–637, 640, 642–643, 648–649, 652–655, 657–658, 663, 665–666, 674, 677–680, 682–683, 685, 689–690, 694, 702–706, 709–710, 712, 715–719, 722, 725–729, 733–734, 737–742, 745–746, 749–752, 754–755, 762–763, 773–776, 779, 782–785, 788–792, 795–796, 799, 802–805, 808, 810, 815–816, 819, 822–825, 827, 833, 838–839, 858–859, 862, 870–871, 878, 888, 891, 898–899, 904–907, 915–918, 926–927, 939–942, 944, 947, 950, 958, 969–971, 973–975, 977–981, 986–987, 989, 1002, 1006, 1009, 1019, 1024, 1032–1033, 1036, 1040, 1042–1044, 1050–1051, 1056, 1064–1069, 1074–1079, 1084–1092, 1097–1104, 1112–1114
_ota_status_control.py	102	11	89%	117, 122, 127, 240, 244–245, 248, 255, 257–258, 273
_rpi_boot.py	287	133	53%	53, 56, 120–121, 125, 133–136, 150–153, 158–159, 161–162, 167–168, 171–172, 181–182, 222, 228–232, 235, 253–255, 259–261, 266–268, 272–274, 284–285, 288, 291, 293–294, 296–297, 299–301, 307, 310–311, 321–324, 332–336, 338, 340–341, 346–347, 354, 357–362, 393, 395–398, 408–411, 415–416, 418–422, 450–453, 472–475, 501–504, 509–517, 522–529, 544–547, 554–557, 565–567
_slot_mnt_helper.py	10	0	100%
configs.py	51	0	100%
protocol.py	6	0	100%
selecter.py	41	29	29%	44–46, 49–50, 54–55, 58–60, 63, 65, 69, 77–79, 81–82, 84–85, 89, 91, 93–94, 96, 98–99, 101, 103
src/otaclient/configs
__init__.py	17	0	100%
_cfg_configurable.py	47	0	100%
_cfg_consts.py	47	1	97%	97
_common.py	8	0	100%
_ecu_info.py	56	4	92%	59, 64–65, 112
_proxy_info.py	50	5	90%	84, 86–87, 89, 100
cfg.py	19	0	100%
src/otaclient/create_standby
__init__.py	13	1	92%	36
common.py	226	44	80%	59, 62–63, 67–69, 71, 75–76, 78, 126, 174–176, 178–180, 182, 185–188, 192, 203, 279–280, 282–287, 299, 339, 367, 370–372, 388–389, 403, 407, 429–430
interface.py	7	0	100%
rebuild_mode.py	115	10	91%	98–100, 119, 150–155
src/otaclient/grpc/api_v2
ecu_status.py	145	7	95%	117, 142, 144, 275, 347–348, 384
ecu_tracker.py	54	54	0%	17, 19–22, 24–30, 32–33, 35, 46–47, 50, 52, 58–61, 63, 65, 67–70, 77, 81–84, 88–89, 91, 93, 95–103, 107–108, 110, 112–115
main.py	41	41	0%	17, 19–24, 26–27, 29, 32, 39, 41–42, 44–45, 47–48, 50–55, 57–59, 61, 64, 70, 72–73, 76–77, 79–82, 84–85, 87
servicer.py	116	95	18%	57–61, 63–64, 66–67, 73–77, 81–82, 87, 90, 94–96, 100–102, 110–112, 115–119, 128–138, 145, 151, 154–156, 167–169, 172–174, 179, 186–189, 192, 196–197, 202, 205, 209–211, 215–217, 225–226, 229–233, 242–251, 258, 264, 267–269, 274–275, 278
types.py	44	2	95%	78–79
src/otaclient_api/v2
api_caller.py	39	6	84%	45–47, 83–85
types.py	256	32	87%	61, 64, 67–70, 86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 578, 585–586, 588
src/otaclient_common
__init__.py	34	15	55%	42–44, 61, 63, 68–77
_io.py	64	1	98%	41
cmdhelper.py	13	0	100%
common.py	106	10	90%	148, 151–153, 168, 175–177, 271, 275
downloader.py	199	10	94%	107–108, 126, 153, 369, 424, 428, 516–517, 526
linux.py	61	15	75%	51–53, 59, 69, 74, 76, 108–109, 133–134, 190, 195–196, 198
logging.py	29	1	96%	55
persist_file_handling.py	118	18	84%	113, 118, 150–152, 163, 192–193, 228–232, 242–244, 246–247
proto_streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
proto_wrapper.py	398	57	85%	87, 134–141, 165, 172, 184–186, 189–190, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 692, 707, 713, 762–763, 765, 803–805
retry_task_map.py	129	9	93%	134–135, 153–154, 207–208, 210, 230–231
shm_status.py	95	21	77%	79–80, 83–84, 105, 120–122, 134, 139, 156–160, 169–170, 172, 179, 192, 204
typing.py	31	4	87%	48, 97–98, 100
TOTAL	6825	1885	72%

Tests	Skipped	Failures	Errors	Time
242	0 💤	0 ❌	0 🔥	12m 30s ⏱️

[airkei]

The simple-sqlite3-orm python package(which is developed and maintained by me also) is used to implement the table definitions and ORMs.

Just comment. If possible, I think it will be better to move/fork the repo under tier4 organization. In future, someone might confuse why we are using the repo without knowing the background.

[Bodong-Yang]

Just comment. If possible, I think it will be better to move/fork the repo under tier4 organization. In future, someone might confuse why we are using the repo without knowing the background.

This package is designed for general purpose use, licensed under Apace license. It has regular and stable release strategy, aligning with semantic versioning. It is not just for otaclient.
So I think it is not needed to fork/move this package under tier4 organization 🙏

[Bodong-Yang]

Also I have published this package to pypi, so every version of this package is available on pypi and can be easily installed by pip.

[airkei]

Thank you. Overall, LGTM. left some minor comments.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
73.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud