tests: pm: Adding tests based on new approach to PM testing. #80692

gbarkadiusz · 2024-10-31T14:30:53Z

Introduces two scripts abstract_power_monitor.py and pwsh_stm32l562.py.
Adds three new power management pytests based on a new approach.

SETUP

TESTS

pm.states: Verifies the transition to all available power states on stm32l562e_dk.

During the test, pytest detects each step and calculates the RMS (Root Mean Square) current for every detected state. The test then compares the calculated RMS values against the expected values, which have been determined manually through experimental observation

pm.residency_time: Tests the residency time of each power state.
This test evaluates the residency time of different power states on the stm32l562e_dk board.

pm.wakeup_timer: Tests the RTC alarm’s ability to wake the CPU from the lowest power state.

Each test calculates RMS current values for each detected state and compares them to expected values, which were manually set based on experimental data.

To Reproduce

Configure the stm32l562e_dk board according to the instructions provided above.
Connect both USB ports of the board to your host system. The board will appear as two separate targets:
Power Monitor: e.g., /dev/ttyACM0 or /dev/serial/by-id/usb-STMicroelectronics_PowerShield__Virtual_ComPort_in_FS_Mode__FFFFFFFEFFFF-if00
Target: e.g., /dev/ttyACM1 or /dev/serial/by-id/usb-STMicroelectronics_STLINK-V3_004##############39-if02
Copy the path of the Power Monitor and update the <path_to_power_monitor> in the tests/subsys/pm/power_states/testcase.yaml file under the pytest_args section as follows:
pytest_args: [--powershield=<path_to_power_monitor>]
Run the following command to execute the test:
twister --device-testing --device-serial <path_to_target> -p stm32l562e_dk -s <test_name> -x=CONFIG_BOOT_DELAY=500

hakehuang · 2024-11-25T07:35:36Z

tests/subsys/pm/power_states/pytest/test_power_states.py

+    data = current_measurement_output
+    # Empirical RMS values in mA with descriptive keys
+    rms_expected = {
+        "k_cpu_idle":57.0,  # K_cpu_idle


can we move those expected data to a yaml file for different platform?

Of course we can, the good hint. Thank you.

PS: Btw, Please take a look. #80989. Feel free to share ideas or warries. :)

hakehuang · 2024-11-25T07:40:33Z

tests/subsys/pm/power_states/src/main.c

+	while (1) {
+		printk("\nGoing to k_cpu_idle.\n");
+		k_msleep(400);
+		printk("\nWake Up.\n");


can we use this tests\subsys\pm\power_mgmt_soc\src\power_mgmt.c, instead of the sleep time?

I don't know yet, let me see.

Yep, get the information from the device tree is the way to go imho, otherwise it will not be portable and the previous comment is useless.

@ceolin I added the use of DTS, is this what you meant?

hakehuang · 2024-11-25T08:30:50Z

tests/subsys/pm/power_residency_time/pytest/current_analyzer.py

@@ -0,0 +1,30 @@
+# Copyright (c) 2024 Intel Corporation.


would this be better to move this function to scripts/pm/pm_helper.py, and rms is not the only implement, e.g. we can have average mean

gbarkadiusz · 2025-01-10T13:50:31Z

@hakehuang I've moved the hardcoded expected value to the appropriate yaml file. Please take a look.

hakehuang

looks good

erwango · 2025-01-16T14:34:49Z

tests/subsys/pm/power_residency_time/testcase.yaml

+      pytest_args: [--powershield=<path_to>/STMicroelectronics_PowerShield__Virtual_ComPort,
+                    --expected-values=<path_to>/stm32l562e_dk.yaml]


IIUC these need to be set manually before testing, right ?

Any way we can spare the need for manual update ?
At least, <path_to>/stm32l562e_dk.yaml is rather static, so it looks reachable.
For <path_to>/STMicroelectronics_PowerShield__Virtual_ComPort maybe a dedicated twister configuration could be used or added ?

bjarki-andreasen · 2025-01-24T14:53:58Z

I believe the approach in this PR is too highly coupled to pm power states, a specific power monitor, and a specific test case.

Measuring current/power or anything else is "generic", its measured in SI units, and can be performed by an "endless" variation of probes. I compare measuring power with "any probe" to flashing a board with "any runner", be it jlink, openocd, pyocd, or whatever.

I believe it can and should be as simple to define a testcase requiring power measurements as this
testcase.yaml:

tests:
  pm.power_states:
    harness: probe
    harness_config:
      measurements:
        - name: k_cpu_idle:
          rms: 56.0
        - name: state_0:
          rms: 56.0

The measurements could be anything, a mode, some peripheral being on, like a comparator in low power mode:

        - name: comp_low_power:
          rms: 56.0
        - name: comp_high_power:
          rms: 71.0

and that specific probes, like runners, should be defined like --probe stm_powershield and --probe-port /dev/ttyACM0 if required, to specify which probe to use for the test. See https://docs.zephyrproject.org/latest/develop/test/twister.html#running-tests-on-hardware for example. Any testsuite can be run for any board, with whatever flasher is available and supported.

This is important, as the test case should not depend on any specific probe, if one developer has a JouleScope, another has a Nordic PPK2, and a third has an stm_powershield, they should all be able to run the same testcase. We are measuring the same "generic" properties after all.

This PR keeps a bunch of this local, using pytest, but I don't think that is a nice, scalable approach to power measurements.

nashif · 2025-01-28T14:21:46Z

I believe the approach in this PR is too highly coupled to pm power states, a specific power monitor, and a specific test case.

agree this could be abstracted to support future probes and other measurements. We need to start somewhere though.

I believe it can and should be as simple to define a testcase requiring power measurements as this testcase.yaml:
tests:
  pm.power_states:
    harness: probe
    harness_config:
      measurements:
        - name: k_cpu_idle:
          rms: 56.0
        - name: state_0:
          rms: 56.0
The measurements could be anything, a mode, some peripheral being on, like a comparator in low power mode:
        - name: comp_low_power:
          rms: 56.0
        - name: comp_high_power:
          rms: 71.0

Ok, probably probe is a bit too generic for a harness name, but we could use a dedicated harness name for power measurements and still keep the underlying implementation using pytest and use the harness config for any probe related options.
Also moving expected results from a dedicated yaml file to testcase.yaml is fine.

@gbarkadiusz can you look into this? AFAIK that should be possible with the current implementation while allowing for future addition of probes.

gbarkadiusz · 2025-01-30T13:50:39Z

@nashif @bjarki-andreasen Regarding the proposal for a separate harness for power measurement, IIUC, Is it right to create a new harness that duplicates the functionality of pytest (which would lead to code duplication)? In fact, we can use the existing pytest harness to achieve our goal. We can simply extend its configuration to include additional details about the probe and port being used.

bjarki-andreasen · 2025-01-30T14:33:21Z

@nashif @bjarki-andreasen Regarding the proposal for a separate harness for power measurement, IIUC, Is it right to create a new harness that duplicates the functionality of pytest (which would lead to code duplication)? In fact, we can use the existing pytest harness to achieve our goal. We can simply extend its configuration to include additional details about the probe and port being used.

pytest is a generic framework, any harness could theoretically be implemented with pytest I believe, a custom harness would be purpose built, its less a question of duplication than it is simplicity of use :)

ceolin

It is looking good, I think that is a good start, even being target specific.

We need to start somewhere :) Overall comments but nothing major.

Thanks a lot for this effort.

ceolin · 2025-01-31T18:57:38Z

scripts/pm/power_monitor_stm32l562e_dk/AbstractPowerMonitor.py

+
+        Returns:
+            List[float]: An array of measured current values in amperes.
+        """


Wondering if that isn't generic enough to be the interface for this sort of test and not specific for stm32l56e_dk

This allows us to easily extend the system to support other platforms without needing to rewrite the entire test logic. The goal is to keep the test interface flexible and adaptable to different devices.

ceolin · 2025-01-31T18:59:28Z