[Refinement] create_standby: refine copy_tree to support container mode

[Bodong-Yang]

Description

During implementing the support for container mode, I found that copy_tree.py module(which is for handling preserving files from active slot to standby slot) doesn't support container mode(a.k.a, it expects that the src path is always rooted at /).

Note

The changes in this PR are cherry-picked from the container mode implementation working branch.

This PR implements a drop-in replacement for copy_tree module at create_standby.common.PersistFilesHandler, which supports handling files preserving under container mode and simplify the implementation.

Check list

• test file(s) that covers the change(s) is implemented.
• local test is passed.
• VM test is passed.

Changes

1. copy_tree module is re-implemented as create_standby.common.PersistFilesHandler,
2. uid/gid mapping logic now is implemented in utils.unix package.

Behavior changes

Does this PR introduce behavior change(s)?

• No.

Breaking change

Does this PR introduce breaking change?

• No.

[Bodong-Yang]

The previous test_copy_tree.py is moved to create_standby/test_persis_files_handling.py, with test setup now as follow:
creating two folders(src, dst) to simulate active slot and to-be-updated standby slot. Specific files/dirs under src are required to be preserved from src to dst, which specified by the canonical path(a.k.a, path rooted at '/').

For example, we create src/A/B/C and want to preserve A/B/C to dst, then we will setup test as follow:

1. create_files will create src and dst folders as active slot and standby slot,
2. create_files will create A/B/C under src folder,
3. PersistFilesHandler is setup with test passwd/group db and src_rootfs=src, dst_rootfs=dst,
4. PersistFilesHandler.preserve_persist_entry is called with params: _persist_entry="/A/B/C",
5. check src/A/B/C against dst/A/B/C.

Important

All test assertion cases are all unchanged(i.e., uid/gid mapping check, mode check , etc), besides the rules of src as active slot and dst as standby slot are applied.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
otaclient/app
common.py	269	15	94%	56, 186, 201, 276–278, 288, 297–299, 405, 417, 505, 536, 540
configs.py	10	0	100%
downloader.py	263	44	83%	68, 81–82, 297, 302, 306, 324–325, 375–379, 398–400, 403–404, 407–408, 429–432, 436–437, 441–442, 446–447, 456, 531–533, 549, 569–571, 575, 577, 580, 585–587
ecu_info.py	59	7	88%	76–77, 90, 95, 118–119, 128
errors.py	113	0	100%
interface.py	5	0	100%
log_setting.py	26	7	73%	27–28, 59, 61–63, 69
main.py	42	4	90%	48–49, 64, 67
ota_client.py	376	130	65%	52–53, 73, 175–176, 189, 192, 196, 199, 238–241, 253–256, 259–260, 264–269, 279–282, 326–329, 334, 338, 341, 354–357, 360–367, 370–377, 383–386, 413, 416–417, 419, 422–425, 427–428, 433–434, 437, 451–458, 465, 468–474, 521–524, 535, 571, 576–579, 584–586, 589–590, 592–593, 595–596, 598, 659–660, 663, 671, 675, 678, 689–690, 693, 701, 705, 708, 719, 738, 749–750, 767, 786, 790–791, 806
ota_client_call.py	40	6	85%	44–46, 82–84
ota_client_service.py	39	5	87%	60–61, 72–74
ota_client_stub.py	394	108	72%	78–80, 82–83, 91, 94–96, 99–101, 105–107, 109–110, 113–116, 118, 121–122, 125–127, 132–137, 141, 144–148, 150–151, 159–161, 164, 206–208, 213, 248, 273, 276, 279, 383, 409, 411, 437, 487, 549, 619–620, 659, 679–681, 687–690, 694–696, 703–705, 708, 712–715, 772, 859–861, 868, 898–899, 902–906, 915–924, 931, 937, 940–941, 945, 948
ota_metadata.py	315	31	90%	145, 150, 186–187, 197–198, 201, 213, 271, 304–306, 323–326, 406, 409, 417–419, 432, 441–442, 445–446, 719–720, 723, 727, 730
ota_status.py	14	2	85%	33, 41
proxy_info.py	47	5	89%	79, 81, 84, 119, 126
update_stats.py	105	2	98%	160, 170
otaclient/app/boot_control
__init__.py	4	0	100%
_cboot.py	293	142	51%	80–84, 86–88, 97–100, 102–104, 108, 110, 116–117, 119, 123, 130, 134, 138, 142–146, 150–154, 161–163, 167–169, 172–173, 176–177, 179, 181–183, 186–190, 193–197, 201, 206–210, 213, 215–216, 219, 225, 228, 231, 234, 237, 240, 243, 246–247, 250–251, 254, 256–258, 261–262, 277, 280, 320, 334–337, 350–353, 363–364, 366, 374, 413, 430–433, 442–445, 450–452, 461, 463–464, 466–467, 470, 477, 507–510, 544–547, 552–554, 559–563, 568–574
_common.py	370	148	60%	64–66, 68–70, 72–73, 82–83, 87–88, 96–97, 108–109, 114–119, 124–129, 138, 147, 151, 155, 159, 173–174, 176, 186–188, 190, 195, 199, 209–211, 213, 221–222, 230–231, 233, 244–248, 252–256, 268–270, 282–284, 295–296, 299–303, 306, 308–309, 319–323, 327–331, 333–340, 359, 363, 366, 375, 392, 395, 401–403, 411–417, 512, 517, 522, 581–582, 636, 640–641, 644, 652, 654–655, 711–712, 715, 729, 734–735, 737, 745, 748, 751–753, 756, 765, 825–828, 849–852, 868–869, 878
_errors.py	47	19	59%	43–46, 50–52, 59–67, 69, 89–90
_grub.py	405	114	71%	216, 264–267, 273–277, 314–315, 322–327, 330, 332, 335–337, 342, 344–346, 355–357, 359–361, 370–372, 374–376, 455–456, 460–461, 513, 519, 545, 564, 568–569, 584–586, 617, 621–623, 625–627, 682, 685–688, 707–710, 735–738, 768–771, 813, 819, 839–840, 842, 854, 857, 860, 863, 867–869, 887–890, 912–915, 920–928, 933–941
_rpi_boot.py	272	125	54%	84–86, 92–93, 95, 98, 101, 104, 109, 117–118, 120, 126, 130–133, 138–140, 144–147, 172–174, 180–182, 197–199, 205–207, 221–228, 230, 234, 238–239, 243–246, 249–250, 255, 259, 263, 267, 301, 331, 334–336, 344, 353–356, 366–369, 373–380, 424–426, 471, 474–477, 494–497, 502, 505, 526–529, 534–542, 547–555, 569–572, 578–580, 583
configs.py	106	5	95%	57–58, 177, 256, 276
firmware.py	31	5	83%	60, 62, 73–75
protocol.py	4	0	100%
selecter.py	38	27	28%	42–44, 46, 49–50, 52–54, 57, 59, 64–66, 74–76, 78–79, 81–83, 85–87, 89, 91
otaclient/app/create_standby
__init__.py	12	5	58%	29–31, 33, 35
common.py	326	110	66%	82, 85–86, 90–92, 94, 96–98, 101, 103, 151, 162, 199–201, 203–205, 207, 210–213, 217, 228, 301–303, 305–310, 312–322, 325–329, 331, 334, 356–357, 361, 365–366, 369–371, 377–378, 382, 384–389, 394–397, 401–402, 406, 414–417, 419, 424–426, 428, 431–432, 434, 440–441, 572, 574, 606–608, 610, 636–639, 644, 648–652, 678–679
interface.py	6	0	100%
rebuild_mode.py	95	4	95%	81, 94–95, 122
otaclient/app/proto
__init__.py	31	3	90%	37, 44–45
_common.py	407	47	88%	85, 163, 170, 182–184, 203, 208, 219, 256, 262, 267, 298, 302, 306, 403, 464, 471, 474, 494, 501, 503, 529, 535, 538, 540, 565, 571, 574, 576, 605, 609, 611, 625, 642, 670, 673, 677, 680, 708, 714, 761–764, 766, 797
_ota_metafiles_wrapper.py	84	16	80%	36–42, 112–116, 122–125
_otaclient_v2_pb2_wrapper.py	266	37	86%	87, 90–93, 132, 175, 183, 197, 207, 210–211, 213, 258, 261, 264–265, 285, 305, 385, 452, 505, 513–515, 519–520, 522, 525–526, 529–530, 551, 559, 573, 581, 595
streamer.py	43	8	81%	32, 47, 65–66, 71, 80–81, 99
wrapper.py	4	0	100%
otaclient/ota_proxy
__init__.py	31	6	80%	58–59, 61, 70, 79–80
_consts.py	15	0	100%
cache_control.py	68	4	94%	71, 91, 113, 121
config.py	18	0	100%
db.py	146	15	89%	75, 81, 103, 113, 116, 145–147, 166, 199, 208–209, 229, 258, 300
errors.py	10	0	100%
orm.py	113	10	91%	91, 96, 101, 107, 113, 140–141, 154, 231, 235
ota_cache.py	404	88	78%	95–96, 215, 226, 253–255, 275, 291–292, 294, 317–318, 324, 328, 330, 359–363, 379, 440–441, 483–484, 554, 567–570, 620, 639–640, 672–673, 684, 718–722, 726–728, 730, 732–739, 741–743, 746–747, 751–752, 756, 803, 811–813, 892–895, 899, 902–903, 917–918, 920–922, 926–927, 933–934, 965, 971, 998, 1027–1029
server_app.py	138	39	71%	75, 78, 84, 100, 102, 161, 170, 212–213, 215–217, 220, 225–227, 230–231, 234, 237, 240, 243, 256–257, 260–261, 263, 266, 292–295, 298, 312–314, 320–322
utils.py	23	1	95%	31
TOTAL	5927	1344	77%

Tests	Skipped	Failures	Errors	Time
172	0 💤	0 ❌	0 🔥	4m 57s ⏱️

[obi-t4]

LGTM 👍