Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TC-IDM-10.7: Implement command checking test #36708

Closed
wants to merge 9 commits into from
Closed

TC-IDM-10.7: Implement command checking test #36708

wants to merge 9 commits into from

Conversation

cecille
Copy link
Contributor

@cecille cecille commented Dec 4, 2024 •
edited
Loading

Please see #36716 for why the OTA requestor command is skipped in CI. The goal of this PR is not to fix the OTA requestor issue, but to catch it in cert.

https://github.com/CHIP-Specifications/chip-test-plans/pull/4798 for test plan

@cecille
TC-IDM-10.7: Implement command checking test
3e78f34 
Note: Currently failing because of the OTA requestor.
@semanticdiff-com SemanticDiff.com
Copy link

semanticdiff-com bot commented Dec 4, 2024 •
edited
Loading

Review changes with  SemanticDiff

Changed Files
File Status
  src/python_testing/matter_testing_infrastructure/chip/testing/spec_parsing.py  47% smaller
  src/python_testing/TC_AccessChecker.py  5% smaller
  src/python_testing/TestSpecParsingSupport.py  3% smaller

@github-actions github-actions bot added tests matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 labels Dec 4, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024 •
edited
Loading

PR #36708: Size comparison from e0fc29a to 3e78f34

Full report (69 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
platform target config section e0fc29a 3e78f34 change % change
bl602 lighting-app bl602+mfd+littlefs+rpc FLASH 1352942 1352942 0 0.0
RAM 104104 104104 0 0.0
bl702 lighting-app bl702+eth FLASH 651536 651536 0 0.0
RAM 25345 25345 0 0.0
bl702+wifi FLASH 829120 829120 0 0.0
RAM 14085 14085 0 0.0
bl706+mfd+rpc+littlefs FLASH 1057314 1057314 0 0.0
RAM 23933 23933 0 0.0
bl702l lighting-app bl702l+mfd+littlefs FLASH 978688 978688 0 0.0
RAM 16572 16572 0 0.0
cc13x4_26x4 lighting-app LP_EM_CC1354P10_6 FLASH 839576 839576 0 0.0
RAM 123656 123656 0 0.0
lock-ftd LP_EM_CC1354P10_6 FLASH 825116 825116 0 0.0
RAM 125536 125536 0 0.0
pump-app LP_EM_CC1354P10_6 FLASH 771864 771864 0 0.0
RAM 114004 114004 0 0.0
pump-controller-app LP_EM_CC1354P10_6 FLASH 756052 756052 0 0.0
RAM 114220 114220 0 0.0
cc32xx air-purifier CC3235SF_LAUNCHXL FLASH 630686 630686 0 0.0
RAM 205808 205808 0 0.0
lock CC3235SF_LAUNCHXL FLASH 669298 669298 0 0.0
RAM 205952 205952 0 0.0
cyw30739 light CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 681249 681249 0 0.0
RAM 78716 78716 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 701101 701101 0 0.0
RAM 81348 81348 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 701101 701101 0 0.0
RAM 81348 81348 0 0.0
CYW930739M2EVB-02 unknown 2040 2040 0 0.0
FLASH 658037 658037 0 0.0
RAM 73784 73784 0 0.0
light-switch CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 617809 617809 0 0.0
RAM 71692 71692 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 637437 637437 0 0.0
RAM 74244 74244 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 637437 637437 0 0.0
RAM 74244 74244 0 0.0
lock CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 637201 637201 0 0.0
RAM 74708 74708 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 656917 656917 0 0.0
RAM 77260 77260 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 656917 656917 0 0.0
RAM 77260 77260 0 0.0
thermostat CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 613669 613669 0 0.0
RAM 68796 68796 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 633521 633521 0 0.0
RAM 71436 71436 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 633521 633521 0 0.0
RAM 71436 71436 0 0.0
efr32 lock-app BRD4187C FLASH 932116 932116 0 0.0
RAM 160180 160180 0 0.0
BRD4338a FLASH 745920 745912 -8 -0.0
RAM 233312 233312 0 0.0
window-app BRD4187C FLASH 1024480 1024480 0 0.0
RAM 128284 128284 0 0.0
esp32 all-clusters-app c3devkit DRAM 95344 95344 0 0.0
FLASH 1543262 1543262 0 0.0
IRAM 82542 82542 0 0.0
m5stack DRAM 116296 116296 0 0.0
FLASH 1550006 1550006 0 0.0
IRAM 117039 117039 0 0.0
linux air-purifier-app debug unknown 4720 4720 0 0.0
FLASH 2713581 2713581 0 0.0
RAM 129768 129768 0 0.0
all-clusters-app debug unknown 5560 5560 0 0.0
FLASH 6005600 6005600 0 0.0
RAM 523512 523512 0 0.0
all-clusters-minimal-app debug unknown 5456 5456 0 0.0
FLASH 5343408 5343408 0 0.0
RAM 242568 242568 0 0.0
bridge-app debug unknown 5440 5440 0 0.0
FLASH 4682894 4682894 0 0.0
RAM 218368 218368 0 0.0
chip-tool debug unknown 5992 5992 0 0.0
FLASH 12845682 12845682 0 0.0
RAM 582314 582314 0 0.0
chip-tool-ipv6only arm64 unknown 21312 21312 0 0.0
FLASH 10979696 10979696 0 0.0
RAM 633232 633232 0 0.0
fabric-admin debug unknown 5816 5816 0 0.0
FLASH 11249747 11249747 0 0.0
RAM 582690 582690 0 0.0
fabric-bridge-app debug unknown 4696 4696 0 0.0
FLASH 4508584 4508584 0 0.0
RAM 205536 205536 0 0.0
fabric-sync debug unknown 4936 4936 0 0.0
FLASH 5605301 5605301 0 0.0
RAM 472536 472536 0 0.0
lighting-app debug+rpc+ui unknown 6104 6104 0 0.0
FLASH 5619729 5619729 0 0.0
RAM 228760 228760 0 0.0
lock-app debug unknown 5376 5376 0 0.0
FLASH 4732248 4732248 0 0.0
RAM 204744 204744 0 0.0
ota-provider-app debug unknown 4752 4752 0 0.0
FLASH 4358002 4358002 0 0.0
RAM 198416 198416 0 0.0
ota-requestor-app debug unknown 4688 4688 0 0.0
FLASH 4497042 4497042 0 0.0
RAM 203000 203000 0 0.0
shell debug unknown 4248 4248 0 0.0
FLASH 3028477 3028477 0 0.0
RAM 160456 160456 0 0.0
thermostat-no-ble arm64 unknown 9520 9520 0 0.0
FLASH 4102480 4102480 0 0.0
RAM 243040 243040 0 0.0
tv-app debug unknown 5704 5704 0 0.0
FLASH 5956293 5956293 0 0.0
RAM 596000 596000 0 0.0
tv-casting-app debug unknown 5288 5288 0 0.0
FLASH 11052605 11052605 0 0.0
RAM 692120 692120 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 FLASH 917380 917380 0 0.0
RAM 143292 143292 0 0.0
nrf7002dk_nrf5340_cpuapp FLASH 889868 889868 0 0.0
RAM 141479 141479 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 FLASH 851548 851548 0 0.0
RAM 142200 142200 0 0.0
nxp contact k32w0+release FLASH 585264 585264 0 0.0
RAM 71072 71072 0 0.0
mcxw71+release FLASH 599800 599800 0 0.0
RAM 63168 63168 0 0.0
light k32w0+release FLASH 612204 612204 0 0.0
RAM 70464 70464 0 0.0
k32w1+release FLASH 686344 686344 0 0.0
RAM 48792 48792 0 0.0
lock mcxw71+release FLASH 762688 762688 0 0.0
RAM 70828 70828 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 FLASH 1646068 1646068 0 0.0
RAM 212088 212088 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 FLASH 1553836 1553836 0 0.0
RAM 208888 208888 0 0.0
light cy8ckit_062s2_43012 FLASH 1469108 1469108 0 0.0
RAM 200872 200872 0 0.0
lock cy8ckit_062s2_43012 FLASH 1466836 1466836 0 0.0
RAM 225232 225232 0 0.0
qpg lighting-app qpg6105+debug FLASH 663736 663736 0 0.0
RAM 105400 105400 0 0.0
lock-app qpg6105+debug FLASH 621572 621572 0 0.0
RAM 99860 99860 0 0.0
stm32 light STM32WB5MM-DK FLASH 484492 484492 0 0.0
RAM 144864 144864 0 0.0
telink bridge-app tlsr9258a FLASH 682594 682594 0 0.0
RAM 91200 91200 0 0.0
contact-sensor-app tlsr9528a_retention FLASH 623004 623004 0 0.0
RAM 31432 31432 0 0.0
light-app-ota-compress-lzma-shell-factory-data tl3218x FLASH 764510 764510 0 0.0
RAM 50212 50212 0 0.0
light-switch-app-ota-compress-lzma-shell-factory-data tlsr9528a FLASH 710424 710424 0 0.0
RAM 73496 73496 0 0.0
lighting-app-ota-factory-data tlsr9118bdk40d FLASH 627530 627530 0 0.0
RAM 142132 142132 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d FLASH 813458 813458 0 0.0
RAM 99676 99676 0 0.0
tizen all-clusters-app arm unknown 4980 4980 0 0.0
FLASH 1731836 1731836 0 0.0
RAM 90724 90724 0 0.0
chip-tool-ubsan arm unknown 10792 10792 0 0.0
FLASH 1796559 1796559 0 0.0
RAM 7838600 7838600 0 0.0

@cecille cecille marked this pull request as ready for review December 4, 2024 20:26
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2024 •
edited
Loading

PR #36708: Size comparison from e0fc29a to eb73c7c

Full report (69 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
platform target config section e0fc29a eb73c7c change % change
bl602 lighting-app bl602+mfd+littlefs+rpc FLASH 1352942 1352942 0 0.0
RAM 104104 104104 0 0.0
bl702 lighting-app bl702+eth FLASH 651536 651536 0 0.0
RAM 25345 25345 0 0.0
bl702+wifi FLASH 829120 829120 0 0.0
RAM 14085 14085 0 0.0
bl706+mfd+rpc+littlefs FLASH 1057314 1057314 0 0.0
RAM 23933 23933 0 0.0
bl702l lighting-app bl702l+mfd+littlefs FLASH 978688 978688 0 0.0
RAM 16572 16572 0 0.0
cc13x4_26x4 lighting-app LP_EM_CC1354P10_6 FLASH 839576 839576 0 0.0
RAM 123656 123656 0 0.0
lock-ftd LP_EM_CC1354P10_6 FLASH 825116 825116 0 0.0
RAM 125536 125536 0 0.0
pump-app LP_EM_CC1354P10_6 FLASH 771864 771864 0 0.0
RAM 114004 114004 0 0.0
pump-controller-app LP_EM_CC1354P10_6 FLASH 756052 756052 0 0.0
RAM 114220 114220 0 0.0
cc32xx air-purifier CC3235SF_LAUNCHXL FLASH 630686 630686 0 0.0
RAM 205808 205808 0 0.0
lock CC3235SF_LAUNCHXL FLASH 669298 669298 0 0.0
RAM 205952 205952 0 0.0
cyw30739 light CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 681249 681249 0 0.0
RAM 78716 78716 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 701101 701101 0 0.0
RAM 81348 81348 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 701101 701101 0 0.0
RAM 81348 81348 0 0.0
CYW930739M2EVB-02 unknown 2040 2040 0 0.0
FLASH 658037 658037 0 0.0
RAM 73784 73784 0 0.0
light-switch CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 617809 617809 0 0.0
RAM 71692 71692 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 637437 637437 0 0.0
RAM 74244 74244 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 637437 637437 0 0.0
RAM 74244 74244 0 0.0
lock CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 637201 637201 0 0.0
RAM 74708 74708 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 656917 656917 0 0.0
RAM 77260 77260 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 656917 656917 0 0.0
RAM 77260 77260 0 0.0
thermostat CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 613669 613669 0 0.0
RAM 68796 68796 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 633521 633521 0 0.0
RAM 71436 71436 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 633521 633521 0 0.0
RAM 71436 71436 0 0.0
efr32 lock-app BRD4187C FLASH 932116 932116 0 0.0
RAM 160180 160180 0 0.0
BRD4338a FLASH 745920 745816 -104 -0.0
RAM 233312 233312 0 0.0
window-app BRD4187C FLASH 1024480 1024480 0 0.0
RAM 128284 128284 0 0.0
esp32 all-clusters-app c3devkit DRAM 95344 95344 0 0.0
FLASH 1543262 1543262 0 0.0
IRAM 82542 82542 0 0.0
m5stack DRAM 116296 116296 0 0.0
FLASH 1550006 1550006 0 0.0
IRAM 117039 117039 0 0.0
linux air-purifier-app debug unknown 4720 4720 0 0.0
FLASH 2713581 2713581 0 0.0
RAM 129768 129768 0 0.0
all-clusters-app debug unknown 5560 5560 0 0.0
FLASH 6005600 6005600 0 0.0
RAM 523512 523512 0 0.0
all-clusters-minimal-app debug unknown 5456 5456 0 0.0
FLASH 5343408 5343408 0 0.0
RAM 242568 242568 0 0.0
bridge-app debug unknown 5440 5440 0 0.0
FLASH 4682894 4682894 0 0.0
RAM 218368 218368 0 0.0
chip-tool debug unknown 5992 5992 0 0.0
FLASH 12845682 12845682 0 0.0
RAM 582314 582314 0 0.0
chip-tool-ipv6only arm64 unknown 21312 21312 0 0.0
FLASH 10979696 10979696 0 0.0
RAM 633232 633232 0 0.0
fabric-admin debug unknown 5816 5816 0 0.0
FLASH 11249747 11249747 0 0.0
RAM 582690 582690 0 0.0
fabric-bridge-app debug unknown 4696 4696 0 0.0
FLASH 4508584 4508584 0 0.0
RAM 205536 205536 0 0.0
fabric-sync debug unknown 4936 4936 0 0.0
FLASH 5605301 5605301 0 0.0
RAM 472536 472536 0 0.0
lighting-app debug+rpc+ui unknown 6104 6104 0 0.0
FLASH 5619729 5619729 0 0.0
RAM 228760 228760 0 0.0
lock-app debug unknown 5376 5376 0 0.0
FLASH 4732248 4732248 0 0.0
RAM 204744 204744 0 0.0
ota-provider-app debug unknown 4752 4752 0 0.0
FLASH 4358002 4358002 0 0.0
RAM 198416 198416 0 0.0
ota-requestor-app debug unknown 4688 4688 0 0.0
FLASH 4497042 4497042 0 0.0
RAM 203000 203000 0 0.0
shell debug unknown 4248 4248 0 0.0
FLASH 3028477 3028477 0 0.0
RAM 160456 160456 0 0.0
thermostat-no-ble arm64 unknown 9520 9520 0 0.0
FLASH 4102480 4102480 0 0.0
RAM 243040 243040 0 0.0
tv-app debug unknown 5704 5704 0 0.0
FLASH 5956293 5956293 0 0.0
RAM 596000 596000 0 0.0
tv-casting-app debug unknown 5288 5288 0 0.0
FLASH 11052605 11052605 0 0.0
RAM 692120 692120 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 FLASH 917380 917380 0 0.0
RAM 143292 143292 0 0.0
nrf7002dk_nrf5340_cpuapp FLASH 889868 889868 0 0.0
RAM 141479 141479 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 FLASH 851548 851548 0 0.0
RAM 142200 142200 0 0.0
nxp contact k32w0+release FLASH 585264 585264 0 0.0
RAM 71072 71072 0 0.0
mcxw71+release FLASH 599800 599800 0 0.0
RAM 63168 63168 0 0.0
light k32w0+release FLASH 612204 612204 0 0.0
RAM 70464 70464 0 0.0
k32w1+release FLASH 686344 686344 0 0.0
RAM 48792 48792 0 0.0
lock mcxw71+release FLASH 762688 762688 0 0.0
RAM 70828 70828 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 FLASH 1646068 1646068 0 0.0
RAM 212088 212088 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 FLASH 1553836 1553836 0 0.0
RAM 208888 208888 0 0.0
light cy8ckit_062s2_43012 FLASH 1469108 1469108 0 0.0
RAM 200872 200872 0 0.0
lock cy8ckit_062s2_43012 FLASH 1466836 1466836 0 0.0
RAM 225232 225232 0 0.0
qpg lighting-app qpg6105+debug FLASH 663736 663736 0 0.0
RAM 105400 105400 0 0.0
lock-app qpg6105+debug FLASH 621572 621572 0 0.0
RAM 99860 99860 0 0.0
stm32 light STM32WB5MM-DK FLASH 484492 484492 0 0.0
RAM 144864 144864 0 0.0
telink bridge-app tlsr9258a FLASH 682594 682594 0 0.0
RAM 91200 91200 0 0.0
contact-sensor-app tlsr9528a_retention FLASH 623004 623004 0 0.0
RAM 31432 31432 0 0.0
light-app-ota-compress-lzma-shell-factory-data tl3218x FLASH 764510 764510 0 0.0
RAM 50212 50212 0 0.0
light-switch-app-ota-compress-lzma-shell-factory-data tlsr9528a FLASH 710424 710424 0 0.0
RAM 73496 73496 0 0.0
lighting-app-ota-factory-data tlsr9118bdk40d FLASH 627530 627530 0 0.0
RAM 142132 142132 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d FLASH 813458 813458 0 0.0
RAM 99676 99676 0 0.0
tizen all-clusters-app arm unknown 4980 4980 0 0.0
FLASH 1731836 1731836 0 0.0
RAM 90724 90724 0 0.0
chip-tool-ubsan arm unknown 10792 10792 0 0.0
FLASH 1796559 1796559 0 0.0
RAM 7838600 7838600 0 0.0

@cecille
Copy link
Contributor Author

cecille commented Dec 5, 2024

re-drafting this - as pointed out in the test plans, we can play some games here with the ACL in order to ensure we're not doing anything with side effects. At that point, doing the full ACE test is a relatively straightforward extension that covers the same functionality. So I'm going to change this to do that instead.

@cecille cecille marked this pull request as draft December 5, 2024 16:15
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 5, 2024 •
edited
Loading

PR #36708: Size comparison from e0fc29a to 21e2552

Full report (3 builds for cc32xx, stm32)
platform target config section e0fc29a 21e2552 change % change
cc32xx air-purifier CC3235SF_LAUNCHXL FLASH 630686 631026 340 0.1
RAM 205808 205824 16 0.0
lock CC3235SF_LAUNCHXL FLASH 669298 669622 324 0.0
RAM 205952 205968 16 0.0
stm32 light STM32WB5MM-DK FLASH 484492 484696 204 0.0
RAM 144864 144880 16 0.0

@cecille cecille closed this Dec 11, 2024
@cecille cecille deleted the IDM_10_7 branch December 11, 2024 20:35
@cecille
Copy link
Contributor Author

cecille commented Dec 11, 2024

whoopsie doodles - changed the branch name thinking this would stay open. Lesson learned. Ah well. Will open a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 review - pending tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@cecille