Merge branch 'master' into develop

reframe-hpc · Dec 9, 2024 · f71044a · f71044a
2 parents 7930201 + 4c67d49
commit f71044a
Show file tree

Hide file tree

Showing 18 changed files with 153 additions and 61 deletions.
diff --git a/docs/index.rst b/docs/index.rst
@@ -21,28 +21,30 @@ Publications
 Presentations & Talks
 ---------------------
 
-* Slides [`pdf <https://users.ugent.be/~kehoste/eum24/008_eum24_ReFrame.pdf>`__][`talk <https://www.youtube.com/live/uSEeU-VJf6k?si=YB19mFpG6aAEBOgH>`__] @ `9th EasyBuild User Meeting 2024 <https://easybuild.io/eum24/>`__.
-* Slides [`part 1 <https://docs.google.com/presentation/d/1GmO2Uf29SaLg36bPB9g9eeaKMN-bLlDJ5IvLGLQJfD8/edit?usp=share_link>`__][`part 2 <https://drive.google.com/file/d/1gZwch0BPc1wDEkMwbM4vxCpMzWIx-Lo1/view?usp=sharing>`__][`talk <https://youtu.be/0ApEKc185Bw>`__] @ `8th EasyBuild User Meeting 2023 <https://easybuild.io/eum23/>`__.
-* Slides [`pdf <https://drive.google.com/file/d/1vmaWyRHgtq3DrYhSCVBzR8U5ErKbxGNf/view?usp=sharing>`__] @ `7th EasyBuild User Meeting 2022 <https://easybuild.io/eum22/>`__.
-* Slides [`pdf <https://drive.google.com/file/d/1kNZu1QNBDDsbKarzwNWYjTGKgOukg-96/view?usp=sharing>`__] @ `6th EasyBuild User Meeting 2021 <https://easybuild.io/eum21/>`__.
-* Slides [`pdf <https://drive.google.com/open?id=1W7R5lfRkXvBpVDSZ7dVBadk_d3K4dFrS>`__] @ `5th EasyBuild User Meeting 2020 <https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting>`__.
-* Slides [`pdf <https://drive.google.com/open?id=1Z3faPh9OSSXvlLHL07co3MRRn443dYsY>`__] @ `HPC System Testing BoF <https://sc19.supercomputing.org/session/?sess=sess324>`__, SC'19.
-* Slides [`pdf <https://drive.google.com/open?id=1JOFqY3ejbR1X5kTn_IZyp1GlCd2ZZS58>`__] @ `HUST 2019 <https://sc19.supercomputing.org/session/?sess=sess116>`__, SC'19.
-* Slides [`pdf <https://drive.google.com/open?id=1iwg1I48LVaWhhZCZIYPJSi3hdFLRcuhi>`__] @ `HPC Knowledge Meeting '19 <https://hpckp.org/>`__.
-* Slides [`pdf <https://fosdem.org/2019/schedule/event/reframe/attachments/slides/3226/export/events/attachments/reframe/slides/3226/FOSDEM_2019.pdf>`__] & `Talk <https://fosdem.org/2019/schedule/event/reframe/>`__ @ `FOSDEM'19 <https://fosdem.org/2019/>`__.
-* Slides [`pdf <https://indico.cism.ucl.ac.be/event/4/contributions/24/attachments/30/62/ReFrame_EUM_2019.pdf>`__] @ `4th EasyBuild User Meeting <https://github.com/easybuilders/easybuild/wiki/4th-EasyBuild-User-Meeting>`__.
-* Slides [`pdf <https://drive.google.com/open?id=1bSykDrl1e2gPflf4jFJ8kfe_SZAtrJ_Q>`__] @ `HUST 2018 <https://sc18.supercomputing.org/>`__, SC'18.
-* Slides [`pdf <https://github.com/eth-cscs/UserLabDay/blob/master/2018/slides/ci_and_regression/ReFrame_CI.pdf>`__] @ `CSCS User Lab Day 2018 <https://github.com/eth-cscs/UserLabDay>`__.
-* Slides [`pdf <https://drive.google.com/open?id=1sZhibvUlGlT670aOHPdMlWFffWptYzLX>`__] @ `HPC Advisory Council 2018 <http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/>`__.
-* Slides [`pdf <https://drive.google.com/open?id=1EyJ-siupkgLeVT54A4WlFpQtrJaU0xOy>`__] @ `SC17 <https://sc17.supercomputing.org/>`__.
-* Slides [`pdf <https://cug.org/proceedings/cug2017_proceedings/includes/files/pap122s2-file2.pdf>`__] @ `CUG 2017 <https://cug.org/cug-2017/>`__.
+* [`slides <https://drive.google.com/file/d/1cwlIipJtJoD-0xGcDMxbL4FQe3xtwAen/view?usp=sharing>`__] "Introduction to ReFrame," CINECA visit, Jun 2024.
+* [`slides <https://users.ugent.be/~kehoste/eum24/008_eum24_ReFrame.pdf>`__][`recording <https://www.youtube.com/live/uSEeU-VJf6k?si=YB19mFpG6aAEBOgH>`__] "Recent Advances in ReFrame," `9th EasyBuild User Meeting 2024 <https://easybuild.io/eum24/>`__.
+* [`slides <https://docs.google.com/presentation/d/1GmO2Uf29SaLg36bPB9g9eeaKMN-bLlDJ5IvLGLQJfD8/edit?usp=share_link>`__][`recording <https://youtu.be/0ApEKc185Bw>`__] "Recent Advances in ReFrame," `8th EasyBuild User Meeting 2023 <https://easybuild.io/eum23/>`__.
+* [`slides <https://drive.google.com/file/d/1gZwch0BPc1wDEkMwbM4vxCpMzWIx-Lo1/view?usp=sharing>`__][`recording <https://youtu.be/0ApEKc185Bw?si=uxa_3QhMjTagS2to&t=1485>`__] "Embracing ReFrame Programmable Configurations," `8th EasyBuild User Meeting 2023 <https://easybuild.io/eum23/>`__.
+* [`slides <https://drive.google.com/file/d/1vmaWyRHgtq3DrYhSCVBzR8U5ErKbxGNf/view?usp=sharing>`__] "ReFrame Update," `7th EasyBuild User Meeting 2022 <https://easybuild.io/eum22/>`__.
+* [`slides <https://drive.google.com/file/d/1kNZu1QNBDDsbKarzwNWYjTGKgOukg-96/view?usp=sharing>`__] "Writing powerful HPC regression tests with ReFrame," `6th EasyBuild User Meeting 2021 <https://easybuild.io/eum21/.>`__
+* [`slides <https://drive.google.com/open?id=1W7R5lfRkXvBpVDSZ7dVBadk_d3K4dFrS>`__] "ReFrame: A Framework for Writing Regression Tests for HPC Systems," `5th EasyBuild User Meeting 2020 <https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting>`__.
+* [`slides <https://drive.google.com/open?id=1Z3faPh9OSSXvlLHL07co3MRRn443dYsY>`__] "Enabling Continuous Testing of HPC Systems using ReFrame," `HPC System Testing BoF <https://sc19.supercomputing.org/session/?sess=sess324>`__, SC'19.
+* [`slides <https://drive.google.com/open?id=1JOFqY3ejbR1X5kTn_IZyp1GlCd2ZZS58>`__] "Enabling Continuous Testing of HPC Systems using ReFrame," `HUST 2019 <https://sc19.supercomputing.org/session/?sess=sess116>`__, SC'19.
+* [`slides <https://drive.google.com/open?id=1iwg1I48LVaWhhZCZIYPJSi3hdFLRcuhi>`__] "ReFrame: A Tool for Enabling Regression Testing and Continuous Integration for HPC Systems," `HPC Knowledge Meeting '19 <https://hpckp.org/>`__.
+* [`slides <https://fosdem.org/2019/schedule/event/reframe/attachments/slides/3226/export/events/attachments/reframe/slides/3226/FOSDEM_2019.pdf>`__][`recording <https://fosdem.org/2019/schedule/event/reframe/>`__] "ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems," `FOSDEM'19 <https://fosdem.org/2019/>`__.
+* [`slides <https://indico.cism.ucl.ac.be/event/4/contributions/24/attachments/30/62/ReFrame_EUM_2019.pdf>`__] "ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems," `4th EasyBuild User Meeting <https://github.com/easybuilders/easybuild/wiki/4th-EasyBuild-User-Meeting>`__.
+* [`slides <https://drive.google.com/open?id=1bSykDrl1e2gPflf4jFJ8kfe_SZAtrJ_Q>`__] "ReFrame: A Regression Testing and Continuous Integration Framework for HPC systems," `HUST 2018 <https://sc18.supercomputing.org/>`__, SC'18.
+* [`slides <https://github.com/eth-cscs/UserLabDay/blob/master/2018/slides/ci_and_regression/ReFrame_CI.pdf>`__] "Regression Testing and Continuous Integration with ReFrame," `CSCS User Lab Day 2018 <https://github.com/eth-cscs/UserLabDay>`__.
+* [`slides <https://drive.google.com/open?id=1sZhibvUlGlT670aOHPdMlWFffWptYzLX>`__] "ReFrame: A Regression Testing Framework Enabling Continuous Integration of Large HPC Systems," `HPC Advisory Council 2018 <http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/>`__.
+* [`slides <https://drive.google.com/open?id=1EyJ-siupkgLeVT54A4WlFpQtrJaU0xOy>`__] "ReFrame: A Regression Testing Tool for HPC Systems," Regression testing BoF, `SC17 <https://sc17.supercomputing.org/>`__.
+* [`slides <https://cug.org/proceedings/cug2017_proceedings/includes/files/pap122s2-file2.pdf>`__] "ReFrame: A regression framework for checking the health of large HPC systems" `CUG 2017 <https://cug.org/cug-2017/>`__.
 
 
 Webinars & Tutorials
 --------------------
 
-* "ReFrame – Efficient System and Application Performance Testing," CSCS Webinar, Aug. 29, 2022 [`slides <https://drive.google.com/file/d/1nOS_daleR79ZB1IaToVdW5mDpJQYRcY2/view?usp=sharing>`__] [`recording <https://youtu.be/NDxlKATEcQk>`__] [`demo run <https://asciinema.org/a/517693>`__].
-* Tutorial at 6th EasyBuild User Meeting 2021 [`YouTube <https://youtube.com/playlist?list=PLhnGtSmEGEQjySVEPTUSLpewpOWwX5mjb>`__]
+* [`slides <https://drive.google.com/file/d/1nOS_daleR79ZB1IaToVdW5mDpJQYRcY2/view?usp=sharing>`__][`recording <https://youtu.be/NDxlKATEcQk>`__][`demo run <https://asciinema.org/a/517693>`__] "ReFrame – Efficient System and Application Performance Testing," CSCS Webinar, Aug. 29, 2022.
+* [`recording <https://youtube.com/playlist?list=PLhnGtSmEGEQjySVEPTUSLpewpOWwX5mjb>`__] "ReFrame Tutorial," 6th EasyBuild User Meeting 2021.
 
 
 Papers

diff --git a/docs/manpage.rst b/docs/manpage.rst
@@ -1043,7 +1043,7 @@ The way the tests are generated and how they interact with the test filtering op
 
    Parameterize a test on an existing variable.
 
-   This option will create a new test with a parameter named ``$VAR`` with the values given in the comma-separated list ``VAL0,VAL1,...``.
+   The test will behave as if the variable ``VAR`` was a parameter taking the values ``VAL0,VAL1,...``.
    The values will be converted based on the type of the target variable ``VAR``.
    The ``TEST.`` prefix will only parameterize the variable ``VAR`` of test ``TEST``.
 

diff --git a/docs/tutorial.rst b/docs/tutorial.rst
@@ -2135,8 +2135,3 @@ Finally, a stored session can be deleted using the :option:`--delete-stored-sess
    reframe --delete-stored-sessions=47e8d98f-e2b9-4019-9a41-1c44d8a53d1b
 
 Deleting a session will also delete all its test cases from the database.
-
-
-.. tip::
-
-   You can disable results storage by either setting ``RFM_ENABLE_RESULTS_STORAGE=0`` or by setting the :attr:`storage.enable <config.storage.enable>` configuration parameter to ``False``.
diff --git a/reframe/__init__.py b/reframe/__init__.py
@@ -6,7 +6,11 @@
 import os
 import sys
 
+<<<<<<< HEAD
 VERSION = '4.8.0-dev.0'
+=======
+VERSION = '4.7.1'
+>>>>>>> master
 INSTALL_PREFIX = os.path.normpath(
     os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 )

diff --git a/reframe/core/pipeline.py b/reframe/core/pipeline.py
@@ -1646,7 +1646,9 @@ def _resolve_fixtures(self):
             # registered under the same fixture class. So the loop below must
             # also inspect the fixture data the instance was registered with.
             for fixt_name, fixt_data in registry[f.cls].items():
-                if f.scope != fixt_data.scope:
+                if fixt_data.variables != f.variables:
+                    continue
+                elif f.scope != fixt_data.scope:
                     continue
                 elif fixt_data.variant_num not in target_variants:
                     continue

diff --git a/reframe/core/schedulers/slurm.py b/reframe/core/schedulers/slurm.py
@@ -351,6 +351,7 @@ def filternodes(self, job, nodes):
         option_parser.add_argument('-w', '--nodelist')
         option_parser.add_argument('-C', '--constraint')
         option_parser.add_argument('-x', '--exclude')
+        self.log(f'Filtering by Slurm options: {" ".join(options)}')
         parsed_args, _ = option_parser.parse_known_args(options)
         reservation = parsed_args.reservation
         partitions = parsed_args.partition
@@ -363,7 +364,7 @@ def filternodes(self, job, nodes):
         else:
             nodes = {node for node in nodes if not node.in_state('RESERVED')}
 
-        self.log(f'[F] Filtering nodes by reservation={reservation}: '
+        self.log(f'Filtering nodes by reservation={reservation}: '
                  f'available nodes now: {len(nodes)}')
 
         if partitions:
@@ -377,27 +378,27 @@ def filternodes(self, job, nodes):
             )
             partitions = {default_partition} if default_partition else set()
             self.log(
-                f'[F] No partition specified; using {default_partition!r}'
+                f'No partition specified; using {default_partition!r}'
             )
 
         nodes = {n for n in nodes if n.partitions >= partitions}
-        self.log(f'[F] Filtering nodes by partition(s) {partitions}: '
+        self.log(f'Filtering nodes by partition(s) {partitions}: '
                  f'available nodes now: {len(nodes)}')
         if constraints:
             nodes = {n for n in nodes if n.satisfies(constraints)}
-            self.log(f'[F] Filtering nodes by constraint(s) {constraints}: '
+            self.log(f'Filtering nodes by constraint(s) {constraints}: '
                      f'available nodes now: {len(nodes)}')
 
         if nodelist:
             nodelist = nodelist.strip()
             nodes &= self._get_nodes_by_name(nodelist)
-            self.log(f'[F] Filtering nodes by nodelist: {nodelist}: '
+            self.log(f'Filtering nodes by nodelist: {nodelist}: '
                      f'available nodes now: {len(nodes)}')
 
         if exclude_nodes:
             exclude_nodes = exclude_nodes.strip()
             nodes -= self._get_nodes_by_name(exclude_nodes)
-            self.log(f'[F] Excluding node(s): {exclude_nodes}: '
+            self.log(f'Excluding node(s): {exclude_nodes}: '
                      f'available nodes now: {len(nodes)}')
 
         return nodes
@@ -711,17 +712,29 @@ def is_down(self):
         return not self.is_avail()
 
     def satisfies(self, slurm_constraint):
+        def _replacemany(s, replacements):
+            for src, dst in replacements:
+                s = s.replace(src, dst)
+
+            return s
+
         # Convert the Slurm constraint to a Python expression and evaluate it,
         # but restrict our syntax to accept only AND or OR constraints and
-        # their combinations
-        if not re.match(r'^[\w\d\(\)\|\&]*$', slurm_constraint):
+        # their combinations; to properly treat `-` in constraints we need to
+        # convert them to valid Python identifiers before evaluating the
+        # constraint.
+        if not re.match(r'^[\-\w\d\(\)\|\&]*$', slurm_constraint):
             return False
 
-        names = {grp[0]
-                 for grp in re.finditer(r'(\w(\w|\d)*)', slurm_constraint)}
-        expr = slurm_constraint.replace('|', ' or ').replace('&', ' and ')
-        vars = {n: True for n in self.active_features}
-        vars.update({n: False for n in names - self.active_features})
+        names = {
+            grp[0] for grp in re.finditer(r'[\-\w][\-\w\d]*', slurm_constraint)
+        }
+        expr = _replacemany(slurm_constraint,
+                            [('-', '_'), ('|', ' or '), ('&', ' and ')])
+        vars = {n.replace('-', '_'): True for n in self.active_features}
+        vars.update({
+            n.replace('-', '_'): False for n in names - self.active_features
+        })
         try:
             return eval(expr, {}, vars)
         except BaseException:

diff --git a/reframe/frontend/cli.py b/reframe/frontend/cli.py
@@ -1041,7 +1041,8 @@ def restrict_logging():
         with exit_gracefully_on_error('failed to retrieve test case data',
                                       printer):
             printer.info(jsonext.dumps(reporting.testcase_info(
-                options.describe_stored_testcases, namepatt
+                options.describe_stored_testcases,
+                namepatt, options.filter_expr
             ), indent=2))
             sys.exit(0)
 
@@ -1637,9 +1638,12 @@ def module_unuse(*paths):
             if options.max_retries and runner.stats.failed(run=0):
                 printer.retry_report(report)
 
-            # Print a failure report if we had failures in the last run
+            # Print a failure report in case of failures.
+            # If `--duration` or `--reruns` is used then take into account
+            # all runs, else (i.e., `--max-retries`) only the last run.
             success = True
-            if runner.stats.failed():
+            runid = None if options.duration or options.reruns else -1
+            if runner.stats.failed(run=runid):
                 success = False
                 printer.failure_report(
                     report,
@@ -1733,6 +1737,12 @@ def module_unuse(*paths):
             sys.exit(1)
 
         sys.exit(0)
+    except errors.RunSessionTimeout as err:
+        printer.warning(f'run session stopped: {err}')
+        if not success:
+            sys.exit(1)
+        else:
+            sys.exit(0)
     except (Exception, KeyboardInterrupt, errors.ReframeFatalError):
         exc_info = sys.exc_info()
         tb = ''.join(traceback.format_exception(*exc_info))

diff --git a/reframe/frontend/loader.py b/reframe/frontend/loader.py
@@ -193,17 +193,31 @@ def load_from_file(self, filename, force=False):
 
         try:
             dirname = os.path.dirname(filename)
-            with osext.change_dir(dirname):
-                with util.temp_sys_path(dirname):
-                    if os.path.exists(os.path.join(dirname, '__init__.py')):
-                        # If the containing directory is a package,
-                        # import it, too.
-                        parent = util.import_module_from_file(dirname).__name__
-                    else:
-                        parent = None
 
+            # Load all parent modules of test file
+            parents = []
+            while os.path.exists(os.path.join(dirname, '__init__.py')):
+                parents.append(os.path.join(dirname))
+                dirname = os.path.split(dirname)[0]
+
+            parent_module = None
+            for pdir in reversed(parents):
+                with osext.change_dir(pdir):
+                    with util.temp_sys_path(pdir):
+                        package_path = os.path.join(pdir, '__init__.py')
+                        parent_module = util.import_module_from_file(
+                            package_path, parent=parent_module
+                        ).__name__
+
+            # Now load the actual test file
+            if not parents:
+                pdir = dirname
+
+            with osext.change_dir(pdir):
+                with util.temp_sys_path(pdir):
                     return self.load_from_module(
-                        util.import_module_from_file(filename, force, parent)
+                        util.import_module_from_file(filename, force,
+                                                     parent_module)
                     )
         except Exception:
             exc_info = sys.exc_info()

diff --git a/reframe/frontend/testgenerators.py b/reframe/frontend/testgenerators.py
@@ -77,12 +77,12 @@ def _generate_tests(testcases, gen_fn):
 @time_function
 def distribute_tests(testcases, node_map):
     def _rfm_pin_run_nodes(obj):
-        nodelist = getattr(obj, '$nid')
+        nodelist = getattr(obj, '.nid')
         if not obj.local:
             obj.job.pin_nodes = nodelist
 
     def _rfm_pin_build_nodes(obj):
-        pin_nodes = getattr(obj, '$nid')
+        pin_nodes = getattr(obj, '.nid')
         if obj.build_job and not obj.local and not obj.build_locally:
             obj.build_job.pin_nodes = pin_nodes
 
@@ -99,9 +99,9 @@ def _rfm_set_valid_systems(obj):
                 'valid_systems': [partition.fullname],
                 # We add a partition parameter so as to differentiate the test
                 # in case another test has the same nodes in another partition
-                '$part': builtins.parameter([partition.fullname],
+                '.part': builtins.parameter([partition.fullname],
                                             loggable=False),
-                '$nid': builtins.parameter(
+                '.nid': builtins.parameter(
                     [[n] for n in node_map[partition.fullname]],
                     fmt=util.nodelist_abbrev, loggable=False
                 )
@@ -113,7 +113,7 @@ def _rfm_set_valid_systems(obj):
                 # will not be overwritten by a parent post-init hook
                 builtins.run_after('init')(_rfm_set_valid_systems),
             ]
-        ), ['$part', '$nid']
+        ), ['.part', '.nid']
 
     return _generate_tests(testcases, _make_dist_test)
 
@@ -127,10 +127,10 @@ def _make_repeat_test(testcase):
         return make_test(
             cls.__name__, (cls,),
             {
-                '$repeat_no': builtins.parameter(range(num_repeats),
+                '.repeat_no': builtins.parameter(range(num_repeats),
                                                  loggable=False)
             }
-        ), ['$repeat_no']
+        ), ['.repeat_no']
 
     return _generate_tests(testcases, _make_repeat_test)
 
@@ -164,7 +164,7 @@ def _make_parameterized_test(testcase):
                 )
                 continue
 
-            body[f'${var}'] = builtins.parameter(values, loggable=False)
+            body[f'.{var}'] = builtins.parameter(values, loggable=False)
 
         def _set_vars(self):
             for var in body.keys():

diff --git a/reframe/utility/__init__.py b/reframe/utility/__init__.py
@@ -866,7 +866,7 @@ def count_digits(n):
     '''
 
     num_digits = 1
-    while n > 10:
+    while n >= 10:
         n /= 10
         num_digits += 1