Volume type standard

Standards/scs-0114-v1-volume-type-standard.md

@@ -0,0 +1,137 @@
+---
+title: Volume Type Standard
+type: Standard
+status: Draft
+track:  IaaS
+---
+
+## Introduction
+
+A volume is a virtual drive that is to be used by an instance (i. e., a virtual machine). With OpenStack,
+each volume is created per a type that determines basic features of the volume as provided by the backend,
+such as encryption, replication, or quality of service. As of the writing of this document, presence or absence of these
+features can not be discovered with full certainty by non-privileged users via the OpenStack API.
+
+### Glossary
+
+The following special terms are used throughout this standard document:
+| Term | Meaning |
+|---|---|
+| volume | OpenStack resource, virtual drive which usually resides in a network storage backend |
+| volume feature | A certain feature a volume can possess |
+| aspect | Part of a volume type that will activate a corresponding feature in a created volume |
+| AZ | Availability Zone |
+| Volume QoS | Quality of Service object for Volumes |
+
+## Motivation
+
+As an SCS user, I want to be able to create volumes with certain common features, such as encryption or
+replication, and to do so in a standardized manner as well as programmatically.
+This standard outlines a way of formally advertising these common aspects for a volume type to
+non-privileged users, so that the most suitable volume type can be discovered and selected easily — both by
+the human user and by a program.
+
+## Design Considerations
+
+All considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)
+
+### Systematic Description of Volume Types
+
+To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. As for the time this standard is created, there is no way for users to discover all aspects through OpenStack commands. Therefore the aspects, that are fulfilled within a volume type, should be stated in the beginning of the **description** of a volume type in the following manner:
+
+`[scs:aspect1, aspect2, ..., aspectN]...`
+
+The mentioned aspects MUST be sorted alphebetically and every aspect should only be mentioned to the maximal amount of one.
+
+### Standardized Aspects
+
+The following table shows which aspects are considered in this standard. The third column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled:
+
+| Aspect | Requirement | standardized description | comment |
+| ---- | ---- | ------ | ------ |
+| Encryption | **Recommended** | **"[scs:encrypted]"** | volume is encrypted |
+| Replication | **Recommended** | **"[scs:replicated]"** | volume is replicated to avoid data loss in a case of hardware failure |
+
+It is possible to use multiple of those aspects within one volume type. There don't have to be different volume types for each aspect.
+For instance, one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard.
+
+## DEFAULT volume type
+
+There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. This standard does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`.
+
+## REQUIRED volume types
+
+Currently, this standard will not require volume types with certain specification.
+
+## RECOMMENDED volume types
+
+This standard recommends to have one or more volume types, that feature encryption and replication.
+
+## OPTIONAL volume types
+
+Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the Cloud Service Providers, if they want to use some of these other aspects.
+
+## Implementation Details
+
+### Encryption
+
+Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. Additionally to be discoverable by users, the description should start with an aspect list such as `[scs:encrypted]` (potentially with additional aspects). It should look like this example:
+
+```text
+openstack volume type show LUKS
++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Field              | Value                                                                                                                                                        |
++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| access_project_ids | None                                                                                                                                                         |
+| description        | [scs:encrypted] This volume uses LUKS-encryption                                                                                                             |
+| id                 | d63307fb-167a-4aa0-9066-66595ea9fb21                                                                                                                         |
+| is_public          | True                                                                                                                                                         |
+| name               | LUKS                                                                                                                                                         |
+| qos_specs_id       | None                                                                                                                                                         |
++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+### Replication
+
+Replication states whether or not there are multiple replicas of a volume. Thus, it answers the question whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved:
+
+1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type.
+2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users.
+
+To fulfill this recommendation, the description should start with an aspect list such as `[scs:replicated]` (potentially with additional aspects).
+
+### Example
+
+One volume type that is configured as an encrypted volume type in a ceph backend, with automated replication would fit both recommendations and will be enough to comply to this part of the volume type standard.
+
+It should look like the following part:
+
+```text
++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Field              | Value                                                                                                                                                        |
++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| access_project_ids | None                                                                                                                                                         |
+| description        | [scs:encrypted, replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used.        |
+| id                 | d63307fb-167a-4aa0-9066-66595ea9fb21                                                                                                                         |
+| is_public          | True                                                                                                                                                         |
+| name               | hdd-three-replicas-LUKS                                                                                                                                      |
+| properties         |                                                                                                                                                              |
+| qos_specs_id       | None                                                                                                                                                         |
++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+## Related Documents
+
+- corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)
+
+## Conformance Tests
+
+The script `/Tests/iaas/volume-types/volume-types-check.py` connects to an OpenStack environment and tests
+the following:
+
+- for each volume type: if its description starts with `[scs:....]`, then this prefix is a feature list
+  (sorted, each entry at most once), and each entry is one of the possible features described here,
+- the recommended volume types are present (otherwise, a WARNING is produced).
+
+The return code is zero precisely when the test could be performed and the conditions are satisfied.
+Otherwise, detailed errors and warnings are output to stderr.

Tests/iaas/volume-types/volume-types-check.py

@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+"""Volume types checker
+
+Check given cloud for conformance with SCS standard regarding
+volume types, to be found under /Standards/scs-0112-v1-volume-types.md
+
+Return code is 0 precisely when it could be verified that the standard is satisfied.
+Otherwise the return code is the number of errors that occurred (up to 127 due to OS
+restrictions); for further information, see the log messages on various channels:
+    CRITICAL  for problems preventing the test to complete,
+    ERROR     for violations of requirements,
+    INFO      for violations of recommendations,
+    DEBUG     for background information and problems that don't hinder the test.
+"""
+from collections import Counter, defaultdict
+import getopt
+import logging
+import os
+import re
+import sys
+
+import openstack
+import openstack.cloud
+
+
+logger = logging.getLogger(__name__)
+RECOGNIZED_FEATURES = ('encrypted', 'replicated')
+
+
+def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")):
+    """Extract feature-list-like prefix
+
+    If given `description` starts with a feature-list-like prefix, return list of features,
+    otherwise None. To be more precise, we look for a string of this form:
+
+    `[scs:`feat1`, `...`, `...featN`]`
+
+    where N >= 1 and featJ is a string that doesn't contain any comma or brackets. We return
+    the list [feat1, ..., featN] of substrings.
+    """
+    if not description:
+        # The description can be None or empty - we need to catch this here
+        return
+    match = pattern.match(description)
+    if not match:
+        return
+    fs = match.group(1)
+    if not fs:
+        return []
+    return [f.strip() for f in fs.split(",")]
+
+
+def test_feature_list(type_name: str, fl: list[str], recognized=RECOGNIZED_FEATURES):
+    """Test given list of features and report errors to error channel"""
+    if not fl:
+        # either None (no feature list) or empty feature list: nothing to check
+        return
+    if fl != sorted(fl):
+        logger.error(f"{type_name}: feature list not sorted")
+    ctr = Counter(fl)
+    duplicates = [key for key, c in ctr.items() if c > 1]
+    if duplicates:
+        logger.error(f"{type_name}: duplicate features: {', '.join(duplicates)}")
+    unrecognized = [f for f in ctr if f not in recognized]
+    if unrecognized:
+        logger.error(f"{type_name}: unrecognized features: {', '.join(unrecognized)}")
+
+
+def print_usage(file=sys.stderr):
+    """Help output"""
+    print("""Usage: volume-types-check.py [options]
+This tool checks volume types according to the SCS Standard 0112 "Volume Types".
+Options:
+ [-c/--os-cloud OS_CLOUD] sets cloud environment (default from OS_CLOUD env)
+ [-d/--debug] enables DEBUG logging channel
+""", end='', file=file)
+
+
+class CountingHandler(logging.Handler):
+    def __init__(self, level=logging.NOTSET):
+        super().__init__(level=level)
+        self.bylevel = Counter()
+
+    def handle(self, record):
+        self.bylevel[record.levelno] += 1
+
+
+def main(argv):
+    # configure logging
+    logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.INFO)
+    openstack.enable_logging(debug=False)
+    # count the number of log records per level (used for summary and return code)
+    counting_handler = CountingHandler(level=logging.INFO)
+    logger.addHandler(counting_handler)
+
+    try:
+        opts, args = getopt.gnu_getopt(argv, "c:i:hd", ["os-cloud=", "help", "debug"])
+    except getopt.GetoptError as exc:
+        logger.critical(f"{exc}")
+        print_usage()
+        return 1
+
+    cloud = os.environ.get("OS_CLOUD")
+    for opt in opts:
+        if opt[0] == "-h" or opt[0] == "--help":
+            print_usage()
+            return 0
+        if opt[0] == "-c" or opt[0] == "--os-cloud":
+            cloud = opt[1]
+        if opt[0] == "-d" or opt[0] == "--debug":
+            logging.getLogger().setLevel(logging.DEBUG)
+
+    if not cloud:
+        logger.critical("You need to have OS_CLOUD set or pass --os-cloud=CLOUD.")
+        return 1
+
+    try:
+        logger.debug(f"Connecting to cloud '{cloud}'")
+        with openstack.connect(cloud=cloud, timeout=32) as conn:
+            volume_types = conn.list_volume_types()
+        # collect volume types according to features
+        by_feature = defaultdict(list)
+        for typ in volume_types:
+            fl = extract_feature_list(typ.description)
+            if fl is None:
+                continue
+            logger.debug(f"{typ.name}: feature list {fl!r}")
+            test_feature_list(typ.name, fl)
+            for feat in fl:
+                by_feature[feat].append(typ.name)
+        logger.debug(f"volume types by feature: {dict(by_feature)}")
+        for feat in ('encrypted', 'replicated'):
+            if not by_feature[feat]:
+                logger.warning(f"Recommendation violated: missing {feat} volume type")
+    except BaseException as e:
+        logger.critical(f"{e!r}")
+        logger.debug("Exception info", exc_info=True)
+
+    c = counting_handler.bylevel
+    logger.debug(
+        "Total critical / error / warning: "
+        f"{c[logging.CRITICAL]} / {c[logging.ERROR]} / {c[logging.WARNING]}"
+    )
+    return min(127, c[logging.CRITICAL] + c[logging.ERROR])  # cap at 127 due to OS restrictions
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))