New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NPU][IE-MDK] Load testing skip config at runtime #29056

Open

sbutnari wants to merge 7 commits into openvinotoolkit:master from sbutnari:new_runtime_skip_config

+968 −1

Contributor

sbutnari commented Feb 18, 2025 •

edited

Loading

Details:

Adds a mechanism to load and parse an xml file containing all test skips
Defines skip enabling "rules": a user can select if a skip is applied depending on OS, Platform number or Backend type
Rules are optional. Users can negate a rule by using "!" and they can add as many rules as needed
Skip filters and rules are grouped in the following format:

<skip_config>
    <message> Provide reason why these tests are skipped </message>
    <enable_rules> <-- rules section starts here -->
        <backend>LEVEL0</backend>
        <backend>IMD</backend>
        <device>3720</device>
        <device>4000</device>
        <operating_system>windows</operating_system>
        <operating_system>linux</operating_system>
    </enable_rules>
    <filters> <-- skip filters section starts here -->
        <filter>skip this test 1</filter>
        <filter>skip this test 2</filter>
        <filter>skip this test 3</filter>
    </filters>
</skip_config>

Functionality is enabled by setting OV_NPU_TESTS_SKIP_CONFIG_FILE environment variable with the correct path, the file will be loaded and parsed.
However, if the path is left empty the application will use the legacy skip filters as configured in skip_tests_config.cpp

Tickets:

EISW-142588

sbutnari requested review from a team as code owners

February 18, 2025 16:44

github-actions bot added category: build category: NPU labels

sbutnari added 4 commits

February 19, 2025 11:22


          Add mechanism to read .xml and populate the skip configuration

010e2d4


          Add new environment variable: OV_NPU_TESTS_SKIP_CONFIG_FILE

407caea


          Create skip_tests.xml and populate it with latest skipped tests

59dd10e


          Update cmakelists, install skip_test.xml in build folder

f49eac1

sbutnari force-pushed the new_runtime_skip_config branch from 4cacc49 to f49eac1 Compare

February 19, 2025 09:22

Contributor Author

sbutnari commented Feb 19, 2025

build_jenkins

pereanub reviewed

View reviewed changes

src/plugins/intel_npu/tests/functional/shared_tests_instances/skip_tests_config.cpp

+                          // Multiple Backends, Devices, OSes can be selected
+                          // If "!" is found, then rule is inverted
+                          pugi::xml_node enableRules = skipConfigRule.child("enable_rules");
+                          bool ruleFlag = false;

Contributor

pereanub Feb 19, 2025

Why not set it to true here if the default value is true?

src/plugins/intel_npu/tests/functional/shared_tests_instances/skip_tests_config.cpp

+                          pugi::xml_node enableRules = skipConfigRule.child("enable_rules");
+                          bool ruleFlag = false;
+                          if (!enableRules.empty()) {
+                              bool backendRuleFlag = false;

Contributor

pereanub Feb 19, 2025

Same question here

src/plugins/intel_npu/tests/functional/shared_tests_instances/skip_tests_config.cpp

+                              }
+                              } else {
+                                  // Rule empty, default to true
+                                  deviceRuleFlag = true;

Contributor

pereanub Feb 19, 2025

Same

src/plugins/intel_npu/tests/functional/shared_tests_instances/skip_tests_config.cpp

+                                  deviceRuleFlag = true;
+                              }
+                              bool operatingSystemRuleFlag = false;

Contributor

pereanub Feb 19, 2025

same

src/plugins/intel_npu/tests/functional/shared_tests_instances/skip_tests_config.cpp Outdated Show resolved Hide resolved

src/plugins/intel_npu/tests/functional/CMakeLists.txt

    
            @@ -66,3 +69,7 @@ install(
          
                RUNTIME DESTINATION tests

                        COMPONENT tests

                        EXCLUDE_FROM_ALL)

              install(FILES ${SKIP_CONFIG_PATH}

Contributor

pereanub Feb 19, 2025

@XinWangIntel, could you please have another look here?

sbutnari added 3 commits

February 20, 2025 19:55


          Update skip congfig .xml

f7e699c


          Added loging, fixed backend conditions

55ec8c9


          clang fix

b4c3ebd

Signed-off-by: Sorin Butnariu <[email protected]>

Contributor Author

sbutnari commented Feb 20, 2025

build_jenkins

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: build category: NPU