Skip to content

Commit

Permalink
Volume type standard (#351)
Browse files Browse the repository at this point in the history
Create Standard for Volume Types

---------

Signed-off-by: Josephine Seifert <[email protected]>
Co-authored-by: Sven <[email protected]>
Co-authored-by: anjastrunk <[email protected]>
Co-authored-by: Matthias Büchse <[email protected]>
  • Loading branch information
4 people authored Apr 10, 2024
1 parent c95b917 commit f05bdc1
Show file tree
Hide file tree
Showing 2 changed files with 285 additions and 0 deletions.
137 changes: 137 additions & 0 deletions Standards/scs-0114-v1-volume-type-standard.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
---
title: Volume Type Standard
type: Standard
status: Draft
track: IaaS
---

## Introduction

A volume is a virtual drive that is to be used by an instance (i. e., a virtual machine). With OpenStack,
each volume is created per a type that determines basic features of the volume as provided by the backend,
such as encryption, replication, or quality of service. As of the writing of this document, presence or absence of these
features can not be discovered with full certainty by non-privileged users via the OpenStack API.

### Glossary

The following special terms are used throughout this standard document:
| Term | Meaning |
|---|---|
| volume | OpenStack resource, virtual drive which usually resides in a network storage backend |
| volume feature | A certain feature a volume can possess |
| aspect | Part of a volume type that will activate a corresponding feature in a created volume |
| AZ | Availability Zone |
| Volume QoS | Quality of Service object for Volumes |

## Motivation

As an SCS user, I want to be able to create volumes with certain common features, such as encryption or
replication, and to do so in a standardized manner as well as programmatically.
This standard outlines a way of formally advertising these common aspects for a volume type to
non-privileged users, so that the most suitable volume type can be discovered and selected easily — both by
the human user and by a program.

## Design Considerations

All considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)

### Systematic Description of Volume Types

To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. As for the time this standard is created, there is no way for users to discover all aspects through OpenStack commands. Therefore the aspects, that are fulfilled within a volume type, should be stated in the beginning of the **description** of a volume type in the following manner:

`[scs:aspect1, aspect2, ..., aspectN]...`

The mentioned aspects MUST be sorted alphebetically and every aspect should only be mentioned to the maximal amount of one.

### Standardized Aspects

The following table shows which aspects are considered in this standard. The third column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled:

| Aspect | Requirement | standardized description | comment |
| ---- | ---- | ------ | ------ |
| Encryption | **Recommended** | **"[scs:encrypted]"** | volume is encrypted |
| Replication | **Recommended** | **"[scs:replicated]"** | volume is replicated to avoid data loss in a case of hardware failure |

It is possible to use multiple of those aspects within one volume type. There don't have to be different volume types for each aspect.
For instance, one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard.

## DEFAULT volume type

There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. This standard does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`.

## REQUIRED volume types

Currently, this standard will not require volume types with certain specification.

## RECOMMENDED volume types

This standard recommends to have one or more volume types, that feature encryption and replication.

## OPTIONAL volume types

Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the Cloud Service Providers, if they want to use some of these other aspects.

## Implementation Details

### Encryption

Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. Additionally to be discoverable by users, the description should start with an aspect list such as `[scs:encrypted]` (potentially with additional aspects). It should look like this example:

```text
openstack volume type show LUKS
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| access_project_ids | None |
| description | [scs:encrypted] This volume uses LUKS-encryption |
| id | d63307fb-167a-4aa0-9066-66595ea9fb21 |
| is_public | True |
| name | LUKS |
| qos_specs_id | None |
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

### Replication

Replication states whether or not there are multiple replicas of a volume. Thus, it answers the question whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved:

1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type.
2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users.

To fulfill this recommendation, the description should start with an aspect list such as `[scs:replicated]` (potentially with additional aspects).

### Example

One volume type that is configured as an encrypted volume type in a ceph backend, with automated replication would fit both recommendations and will be enough to comply to this part of the volume type standard.

It should look like the following part:

```text
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| access_project_ids | None |
| description | [scs:encrypted, replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used. |
| id | d63307fb-167a-4aa0-9066-66595ea9fb21 |
| is_public | True |
| name | hdd-three-replicas-LUKS |
| properties | |
| qos_specs_id | None |
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

## Related Documents

- corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)

## Conformance Tests

The script `/Tests/iaas/volume-types/volume-types-check.py` connects to an OpenStack environment and tests
the following:

- for each volume type: if its description starts with `[scs:....]`, then this prefix is a feature list
(sorted, each entry at most once), and each entry is one of the possible features described here,
- the recommended volume types are present (otherwise, a WARNING is produced).

The return code is zero precisely when the test could be performed and the conditions are satisfied.
Otherwise, detailed errors and warnings are output to stderr.
148 changes: 148 additions & 0 deletions Tests/iaas/volume-types/volume-types-check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""Volume types checker
Check given cloud for conformance with SCS standard regarding
volume types, to be found under /Standards/scs-0112-v1-volume-types.md
Return code is 0 precisely when it could be verified that the standard is satisfied.
Otherwise the return code is the number of errors that occurred (up to 127 due to OS
restrictions); for further information, see the log messages on various channels:
CRITICAL for problems preventing the test to complete,
ERROR for violations of requirements,
INFO for violations of recommendations,
DEBUG for background information and problems that don't hinder the test.
"""
from collections import Counter, defaultdict
import getopt
import logging
import os
import re
import sys

import openstack
import openstack.cloud


logger = logging.getLogger(__name__)
RECOGNIZED_FEATURES = ('encrypted', 'replicated')


def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")):
"""Extract feature-list-like prefix
If given `description` starts with a feature-list-like prefix, return list of features,
otherwise None. To be more precise, we look for a string of this form:
`[scs:`feat1`, `...`, `...featN`]`
where N >= 1 and featJ is a string that doesn't contain any comma or brackets. We return
the list [feat1, ..., featN] of substrings.
"""
if not description:
# The description can be None or empty - we need to catch this here
return
match = pattern.match(description)
if not match:
return
fs = match.group(1)
if not fs:
return []
return [f.strip() for f in fs.split(",")]


def test_feature_list(type_name: str, fl: list[str], recognized=RECOGNIZED_FEATURES):
"""Test given list of features and report errors to error channel"""
if not fl:
# either None (no feature list) or empty feature list: nothing to check
return
if fl != sorted(fl):
logger.error(f"{type_name}: feature list not sorted")
ctr = Counter(fl)
duplicates = [key for key, c in ctr.items() if c > 1]
if duplicates:
logger.error(f"{type_name}: duplicate features: {', '.join(duplicates)}")
unrecognized = [f for f in ctr if f not in recognized]
if unrecognized:
logger.error(f"{type_name}: unrecognized features: {', '.join(unrecognized)}")


def print_usage(file=sys.stderr):
"""Help output"""
print("""Usage: volume-types-check.py [options]
This tool checks volume types according to the SCS Standard 0112 "Volume Types".
Options:
[-c/--os-cloud OS_CLOUD] sets cloud environment (default from OS_CLOUD env)
[-d/--debug] enables DEBUG logging channel
""", end='', file=file)


class CountingHandler(logging.Handler):
def __init__(self, level=logging.NOTSET):
super().__init__(level=level)
self.bylevel = Counter()

def handle(self, record):
self.bylevel[record.levelno] += 1


def main(argv):
# configure logging
logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.INFO)
openstack.enable_logging(debug=False)
# count the number of log records per level (used for summary and return code)
counting_handler = CountingHandler(level=logging.INFO)
logger.addHandler(counting_handler)

try:
opts, args = getopt.gnu_getopt(argv, "c:i:hd", ["os-cloud=", "help", "debug"])
except getopt.GetoptError as exc:
logger.critical(f"{exc}")
print_usage()
return 1

cloud = os.environ.get("OS_CLOUD")
for opt in opts:
if opt[0] == "-h" or opt[0] == "--help":
print_usage()
return 0
if opt[0] == "-c" or opt[0] == "--os-cloud":
cloud = opt[1]
if opt[0] == "-d" or opt[0] == "--debug":
logging.getLogger().setLevel(logging.DEBUG)

if not cloud:
logger.critical("You need to have OS_CLOUD set or pass --os-cloud=CLOUD.")
return 1

try:
logger.debug(f"Connecting to cloud '{cloud}'")
with openstack.connect(cloud=cloud, timeout=32) as conn:
volume_types = conn.list_volume_types()
# collect volume types according to features
by_feature = defaultdict(list)
for typ in volume_types:
fl = extract_feature_list(typ.description)
if fl is None:
continue
logger.debug(f"{typ.name}: feature list {fl!r}")
test_feature_list(typ.name, fl)
for feat in fl:
by_feature[feat].append(typ.name)
logger.debug(f"volume types by feature: {dict(by_feature)}")
for feat in ('encrypted', 'replicated'):
if not by_feature[feat]:
logger.warning(f"Recommendation violated: missing {feat} volume type")
except BaseException as e:
logger.critical(f"{e!r}")
logger.debug("Exception info", exc_info=True)

c = counting_handler.bylevel
logger.debug(
"Total critical / error / warning: "
f"{c[logging.CRITICAL]} / {c[logging.ERROR]} / {c[logging.WARNING]}"
)
return min(127, c[logging.CRITICAL] + c[logging.ERROR]) # cap at 127 due to OS restrictions


if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))

0 comments on commit f05bdc1

Please sign in to comment.