From 6f03aac092f4ebe85f681fc4ac235b11b9aa82bd Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 25 Jul 2023 14:28:03 +0200 Subject: [PATCH 01/35] Create scs-0112-v1-volume-type-standard.md First draft of Standard, without tests and link to decision document. Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 169 ++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 Standards/scs-0112-v1-volume-type-standard.md diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md new file mode 100644 index 000000000..40bb37f5b --- /dev/null +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -0,0 +1,169 @@ +--- +title: Volume Type Standard +type: Standard +status: Draft +track: IaaS +--- + +## Introduction + +Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes very be backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. + +## Motivation + +We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. And whether the volume will be replicated is a mix between definiton in the volume type and backend specific configuration, but it's visiblity can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are + +## Design Considerations + +All Considerations can be looked up in detail in the Decision Record for the Volume Type Standard. + +| Aspect | Standardize? | Discoverability | other Things | +| ---- | ---- | ---- | ------ | +| encryption | **Recommended** | work needed | extra_spec: encrypted=True/False | +| Backend name | - | - | - | +| AZs | - | - | describe as optional and backend-dependend | +| multiattach | - | yes | describe as optional | +| Replication | **Recommended** | lot of work | either get from backend to OS or as extra_spec defined by deployer | +| Number of Replicas, etc | ? | lot of work | optional, work on it after Replication is standardized | +| Volume QoS | ? | admin only | needs further discussion, should be at least described as optional | + +## DEFAULT volume types + +There is always a default volume type defined in an OpenStack deployment. The SCS will does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. + +The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And the SCS will not make any assumptions about parameters being present in the default volume type. + +## REQUIRED volume types + +Currently the SCS will not require volume types with certain specification. This might change in the future. + +## RECOMMENDED volume types + +The SCS recommends to have one or more volume types, which have the following specifications: + +### Encryption + +Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. And for admins it is also currently possible to see these configurations in a volume type with both list and show commands. + +``` +openstack volume type list --encryption-type ++--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ID | Name | Is Public | Encryption | ++--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| d34b5de0-d7bf-43cf-8a74-3827169d6616 | LUKS | True | cipher='aes-xts-plain64', control_location='front-end', encryption_id='217386bc-1e9b-46a3-9e0e-3ad62c07826c', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor' | ++--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +**TODO**: +Users that don't have admin rights currently cannot see these encryption parameters. We want and need to change this in the OpenStack workflow through adding a property (e.g. "encryption"="true") that is also visible for users. This way we will be able to automatically check whether a volume type with encyrption is present. It should look like this: + +``` +openstack volume type show LUKS ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | None | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | LUKS | +| properties | encrypted='true' + | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +### Replication + +Replication states, whether or not there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved: + +1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. +2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. + +We recommend for now to state the replication in the description or name of a volume type, so users are aware of it + +**TODO:** +We want to find a way to also use the internal extra_spec for replication, when the replication is automatically done by the backend. If this is not possible, we would like to introduce another property, which has to be set by the admin, when setting the volume type. Only after that we will have the possibility to automatically check for a volume type with replication. + +#### OPTIONAL addition: Number of Replicas + +Additionally to the fact, that a volume type is replicated, it OPTIONALLY can be stated in the description or name, how many times the provisioned volume will be replicated. + +### Example + +One volume type that is configured as an encrypted volume type in a ceph backend, with automated replication would fit both recommendations and will be enough to comply to this part of the volume type standard. + +It might look like the following part for administrators: +``` ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | Content will be replicated three times on spinning disks to ensure consistency and availability for your data. LUKS encryption is used. | +| encryption | cipher='aes-xts-plain64', control_location='front-end', created_at='2023-05-05T13:50:34.000000', deleted='False', deleted_at=, encryption_id='217....6c', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor', updated_at= | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | hdd-three-replicas-LUKS | +| properties | + | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +## OPTIONAL volume types + +In this section volume types are described, that we do not want to explicitly recommend, but find them worth listing as optional for deployers. + +### Availability Zones + +Availability Zones are not necessarily used by every deployer of a cloud. The SCS does not want do make it necessary to have AZs for a compatible deployment. But we want to encourage deployer who use multiple AZs to read the following: + +While there can be multiple Compute AZs, there can also be multiple Storage AZs for Volumes. And it might be not quite confusing for users, which volumes can be used in which AZs. To make it even further complicated, there are backends like ceph, which can provide volumes for multiple compute AZs just with some Nova configuration. Therefore we would encourage to use either the property of volume types OR the description of the volume types to describe, in which AZs the volumes based on this type can be used. + +A description for a ceph volume type might look like this: +``` ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | Content will be replicated three times on spinning disks to ensure consistency and availability for your data. The volumes can be used in AZ 1,3,4. | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | hdd-three-replicas-AZ134 | +| properties | + | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +### Multiattach + +A few backends enable providers to allow volumes to be attached to mulitple VMs simultaneously. The OpenStack API can be used to enable the usage of multiattach through the volume type. The property is also automatically shown to users. If a deployer want to use this feature they have to do it in this way. So users will always see whether a volume type can be used for mulitattach volumes or not. Nevertheless due to other problems, that might occur when using the multiattach feature, the SCS will only have this as OPTIONAL. + +### Volume QoS + +Quality of Service for volumes can be defined in volume qos objects. While admins can use these directly on volumes, a user is not able to see these object. Instead there can be ONE volume qos object associated to a volume type, which is then be used on all volumes created from that volume type. Through this indirection, functioning volume types without these associated qos objects and this feature not being heavily used, the SCS will currently only state this as an OPTIONAL volume type feature. + +To make users aware that a volume type includes specific qos options, we reccomend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: +``` ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | Content will be replicated three times on spinning disks to ensure consistency and availability for your data. Iops: read:20k, write:10k | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | hdd-three-replicas-AZ134 | +| properties | + | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +## Related Documents + +TODO: Link Decision Doc here + +## Conformance Tests + +BIG TODO, after Upstream work From 7cccebde290e13dae054914f06a7e4a21ad6e03c Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Thu, 28 Sep 2023 15:43:14 +0200 Subject: [PATCH 02/35] Update scs-0112-v1-volume-type-standard.md refine standard after decision document merged Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 39 ++++++++++++++----- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 40bb37f5b..ac9f9188a 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -9,6 +9,14 @@ track: IaaS Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes very be backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. +### Glossary +The following special terms are used throughout this standard document: +| Term | Meaning | +|---|---| +| volume | OpenStack object, virtual drive which usually resides in a network stoarage backend | +| AZ | Availability Zone | +| Volume QoS | Quality of Service object for Volumes | + ## Motivation We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. And whether the volume will be replicated is a mix between definiton in the volume type and backend specific configuration, but it's visiblity can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are @@ -17,15 +25,20 @@ We want to standardize a few varieties of volume types. While a user can choose All Considerations can be looked up in detail in the Decision Record for the Volume Type Standard. -| Aspect | Standardize? | Discoverability | other Things | +To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard and whether it is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. + +| Aspect | Standardize? | Discoverability | comments | | ---- | ---- | ---- | ------ | -| encryption | **Recommended** | work needed | extra_spec: encrypted=True/False | +| encryption | **Recommended** | admin only | Upstream work for extra_spec: encrypted=True/False needed | | Backend name | - | - | - | -| AZs | - | - | describe as optional and backend-dependend | -| multiattach | - | yes | describe as optional | -| Replication | **Recommended** | lot of work | either get from backend to OS or as extra_spec defined by deployer | -| Number of Replicas, etc | ? | lot of work | optional, work on it after Replication is standardized | -| Volume QoS | ? | admin only | needs further discussion, should be at least described as optional | +| AZs | **Optional** | - | Upstream: setable extra_spec would help | +| multiattach | **Optional** | yes | - | +| Replication | **Recommended** | backend-dependend | Upstream: setable extra_spec would help | +| Number of Replicas, etc | **Optional** | backend-dependend | Upstream: setable extra_spec would help | +| Volume QoS | **Optional** | admin only | depends on the admin only volume qos object | + +In addition it is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but they don't have to be different volume types. +Example: one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. ## DEFAULT volume types @@ -73,6 +86,8 @@ openstack volume type show LUKS +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` +Until this change happend, it will be necessary that deployer state that the volume type is encrypted either in the name or in the description of the volume type. + ### Replication Replication states, whether or not there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved: @@ -80,7 +95,7 @@ Replication states, whether or not there are multiple replicas of a volume. Thus 1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. -We recommend for now to state the replication in the description or name of a volume type, so users are aware of it +We recommend for now to state the replication in the description or name of a volume type, so users are aware of it. **TODO:** We want to find a way to also use the internal extra_spec for replication, when the replication is automatically done by the backend. If this is not possible, we would like to introduce another property, which has to be set by the admin, when setting the volume type. Only after that we will have the possibility to automatically check for a volume type with replication. @@ -162,8 +177,12 @@ To make users aware that a volume type includes specific qos options, we reccome ## Related Documents -TODO: Link Decision Doc here +[Here is the decision record document.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) ## Conformance Tests -BIG TODO, after Upstream work +As there are currently no REQUIRED volume types, we can only look for the RECOMMENDED aspects and a conformance test is not a must. +Furthermore those aspects either are only discoverable by an admin or have to be described in either name or description by the deployer. +And it is also possible that a single volume type can currently fulfill all RECOMMENDED aspects. + +TODO: create meaningful test From 7380a31c6e3a0cce8b47394a4a5ad5c4262111d8 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 29 Sep 2023 11:24:24 +0200 Subject: [PATCH 03/35] Update scs-0112-v1-volume-type-standard.md Design meaningful test case for recommended volume types Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 49 +++++++++++++++---- 1 file changed, 39 insertions(+), 10 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index ac9f9188a..bfca4a854 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -86,7 +86,22 @@ openstack volume type show LUKS +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` -Until this change happend, it will be necessary that deployer state that the volume type is encrypted either in the name or in the description of the volume type. +Until this change happened, it will be necessary that deployer state that the volume type is encrypted in the description of the volume type in the following way: +The description needs to begin with `[encrypted]`, after that any further description is allowed. It should look like this example: + +``` +openstack volume type show LUKS ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | [encrypted] This volume uses LUKS-encryption | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | LUKS | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` ### Replication @@ -95,7 +110,8 @@ Replication states, whether or not there are multiple replicas of a volume. Thus 1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. -We recommend for now to state the replication in the description or name of a volume type, so users are aware of it. +To fulfill this recommentation for now, the deployer needs to state the replication in the description of a volume type in the following way: +The description needs to begin with `[replicated]`, after that any further description is allowed. **TODO:** We want to find a way to also use the internal extra_spec for replication, when the replication is automatically done by the backend. If this is not possible, we would like to introduce another property, which has to be set by the admin, when setting the volume type. Only after that we will have the possibility to automatically check for a volume type with replication. @@ -108,19 +124,32 @@ Additionally to the fact, that a volume type is replicated, it OPTIONALLY can be One volume type that is configured as an encrypted volume type in a ceph backend, with automated replication would fit both recommendations and will be enough to comply to this part of the volume type standard. -It might look like the following part for administrators: +It should look like the following part: ``` +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access_project_ids | None | -| description | Content will be replicated three times on spinning disks to ensure consistency and availability for your data. LUKS encryption is used. | -| encryption | cipher='aes-xts-plain64', control_location='front-end', created_at='2023-05-05T13:50:34.000000', deleted='False', deleted_at=, encryption_id='217....6c', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor', updated_at= | +| description | [encrypted][replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used. | | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | hdd-three-replicas-LUKS | -| properties | - | +| properties | | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +It is OPTIONAL to give more information to users like this: +``` ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | [encrypted:cipher=aes,key_size=256][replicated] Content will be replicated three times and LUKS encryption is used. | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | hdd-three-replicas-LUKS | +| properties | | | qos_specs_id | None | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` @@ -181,8 +210,8 @@ To make users aware that a volume type includes specific qos options, we reccome ## Conformance Tests -As there are currently no REQUIRED volume types, we can only look for the RECOMMENDED aspects and a conformance test is not a must. -Furthermore those aspects either are only discoverable by an admin or have to be described in either name or description by the deployer. +As there are currently no REQUIRED volume types, we can only look for the RECOMMENDED aspects and thus executing a conformance test is not a must. +Furthermore the recommended aspects currently have to be described in the description by the deployer. In future versions we aim to integrate some extra_specs for them in upstream OpenStack And it is also possible that a single volume type can currently fulfill all RECOMMENDED aspects. -TODO: create meaningful test +**TODO:** The current test will check for the presence of `[encrypted]` and `[replicated]` in the description of at least one volume type. From 28acb9eecea3a152e7f793982b95e694c3924249 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 29 Sep 2023 11:47:30 +0200 Subject: [PATCH 04/35] Update scs-0112-v1-volume-type-standard.md edit styling Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 29 ++++++++++--------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index bfca4a854..c2694441b 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -10,6 +10,7 @@ track: IaaS Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes very be backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. ### Glossary + The following special terms are used throughout this standard document: | Term | Meaning | |---|---| @@ -58,7 +59,7 @@ The SCS recommends to have one or more volume types, which have the following sp Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. And for admins it is also currently possible to see these configurations in a volume type with both list and show commands. -``` +```text openstack volume type list --encryption-type +--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Is Public | Encryption | @@ -70,7 +71,7 @@ openstack volume type list --encryption-type **TODO**: Users that don't have admin rights currently cannot see these encryption parameters. We want and need to change this in the OpenStack workflow through adding a property (e.g. "encryption"="true") that is also visible for users. This way we will be able to automatically check whether a volume type with encyrption is present. It should look like this: -``` +```text openstack volume type show LUKS +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | @@ -80,8 +81,7 @@ openstack volume type show LUKS | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | LUKS | -| properties | encrypted='true' - | +| properties | encrypted='true' | | qos_specs_id | None | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` @@ -89,7 +89,7 @@ openstack volume type show LUKS Until this change happened, it will be necessary that deployer state that the volume type is encrypted in the description of the volume type in the following way: The description needs to begin with `[encrypted]`, after that any further description is allowed. It should look like this example: -``` +```text openstack volume type show LUKS +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | @@ -125,7 +125,8 @@ Additionally to the fact, that a volume type is replicated, it OPTIONALLY can be One volume type that is configured as an encrypted volume type in a ceph backend, with automated replication would fit both recommendations and will be enough to comply to this part of the volume type standard. It should look like the following part: -``` + +```text +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -140,7 +141,8 @@ It should look like the following part: ``` It is OPTIONAL to give more information to users like this: -``` + +```text +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -165,7 +167,8 @@ Availability Zones are not necessarily used by every deployer of a cloud. The SC While there can be multiple Compute AZs, there can also be multiple Storage AZs for Volumes. And it might be not quite confusing for users, which volumes can be used in which AZs. To make it even further complicated, there are backends like ceph, which can provide volumes for multiple compute AZs just with some Nova configuration. Therefore we would encourage to use either the property of volume types OR the description of the volume types to describe, in which AZs the volumes based on this type can be used. A description for a ceph volume type might look like this: -``` + +```text +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -174,22 +177,22 @@ A description for a ceph volume type might look like this: | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | hdd-three-replicas-AZ134 | -| properties | - | +| properties | | | qos_specs_id | None | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ### Multiattach -A few backends enable providers to allow volumes to be attached to mulitple VMs simultaneously. The OpenStack API can be used to enable the usage of multiattach through the volume type. The property is also automatically shown to users. If a deployer want to use this feature they have to do it in this way. So users will always see whether a volume type can be used for mulitattach volumes or not. Nevertheless due to other problems, that might occur when using the multiattach feature, the SCS will only have this as OPTIONAL. +A few backends enable providers to allow volumes to be attached to mulitple VMs simultaneously. The OpenStack API can be used to enable the usage of multiattach through the volume type. The property is also automatically shown to users. If a deployer want to use this feature they have to do it in this way. So users will always see whether a volume type can be used for mulitattach volumes or not. Nevertheless due to other problems, that might occur when using the multiattach feature, the SCS will only have this as OPTIONAL. ### Volume QoS Quality of Service for volumes can be defined in volume qos objects. While admins can use these directly on volumes, a user is not able to see these object. Instead there can be ONE volume qos object associated to a volume type, which is then be used on all volumes created from that volume type. Through this indirection, functioning volume types without these associated qos objects and this feature not being heavily used, the SCS will currently only state this as an OPTIONAL volume type feature. To make users aware that a volume type includes specific qos options, we reccomend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: -``` + +```text +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -206,7 +209,7 @@ To make users aware that a volume type includes specific qos options, we reccome ## Related Documents -[Here is the decision record document.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) +[Here is the decision record document.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) ## Conformance Tests From fa964a9ff268cfc846831b3deae46af32facd64c Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 6 Oct 2023 09:49:54 +0200 Subject: [PATCH 05/35] Apply suggestions from code review Co-authored-by: Sven Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index c2694441b..12c29bc87 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -7,14 +7,14 @@ track: IaaS ## Introduction -Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes very be backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. +Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes be very backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. ### Glossary The following special terms are used throughout this standard document: | Term | Meaning | |---|---| -| volume | OpenStack object, virtual drive which usually resides in a network stoarage backend | +| volume | OpenStack object, virtual drive which usually resides in a network storage backend | | AZ | Availability Zone | | Volume QoS | Quality of Service object for Volumes | @@ -43,7 +43,7 @@ Example: one volume type that uses LUKS-encryption with a ceph storage with inhe ## DEFAULT volume types -There is always a default volume type defined in an OpenStack deployment. The SCS will does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. +There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And the SCS will not make any assumptions about parameters being present in the default volume type. @@ -164,7 +164,7 @@ In this section volume types are described, that we do not want to explicitly re Availability Zones are not necessarily used by every deployer of a cloud. The SCS does not want do make it necessary to have AZs for a compatible deployment. But we want to encourage deployer who use multiple AZs to read the following: -While there can be multiple Compute AZs, there can also be multiple Storage AZs for Volumes. And it might be not quite confusing for users, which volumes can be used in which AZs. To make it even further complicated, there are backends like ceph, which can provide volumes for multiple compute AZs just with some Nova configuration. Therefore we would encourage to use either the property of volume types OR the description of the volume types to describe, in which AZs the volumes based on this type can be used. +While there can be multiple Compute AZs, there can also be multiple Storage AZs for Volumes. And it might be quite confusing for users, which volumes can be used in which AZs. To make it even further complicated, there are backends like ceph, which can provide volumes for multiple compute AZs just with some Nova configuration. Therefore we would encourage to use either the property of volume types OR the description of the volume types to describe, in which AZs the volumes based on this type can be used. A description for a ceph volume type might look like this: @@ -190,7 +190,7 @@ A few backends enable providers to allow volumes to be attached to mulitple VMs Quality of Service for volumes can be defined in volume qos objects. While admins can use these directly on volumes, a user is not able to see these object. Instead there can be ONE volume qos object associated to a volume type, which is then be used on all volumes created from that volume type. Through this indirection, functioning volume types without these associated qos objects and this feature not being heavily used, the SCS will currently only state this as an OPTIONAL volume type feature. -To make users aware that a volume type includes specific qos options, we reccomend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: +To make users aware that a volume type includes specific qos options, we recommend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: ```text +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ From 4f00047253e3027d9949fc3fcad85a5c3b943422 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 6 Oct 2023 10:14:31 +0200 Subject: [PATCH 06/35] Update scs-0112-v1-volume-type-standard.md Rephrase a few paragraphs for better understanding Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 12c29bc87..a1af3c1eb 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -20,11 +20,11 @@ The following special terms are used throughout this standard document: ## Motivation -We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. And whether the volume will be replicated is a mix between definiton in the volume type and backend specific configuration, but it's visiblity can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are +We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. While the option with which the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are REQUIRED, RECOMMENDED or only OPTIONAL within an SCS-compatible deployment. ## Design Considerations -All Considerations can be looked up in detail in the Decision Record for the Volume Type Standard. +All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard and whether it is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. @@ -184,11 +184,11 @@ A description for a ceph volume type might look like this: ### Multiattach -A few backends enable providers to allow volumes to be attached to mulitple VMs simultaneously. The OpenStack API can be used to enable the usage of multiattach through the volume type. The property is also automatically shown to users. If a deployer want to use this feature they have to do it in this way. So users will always see whether a volume type can be used for mulitattach volumes or not. Nevertheless due to other problems, that might occur when using the multiattach feature, the SCS will only have this as OPTIONAL. +A few backends enable providers to allow volumes to be attached to mulitple VMs simultaneously. The OpenStack API can be used to enable the usage of multiattach through the volume type. The property is also automatically shown to users. If a deployer wants to use this feature they have to do it in this way. So users will always see whether a volume type can be used for mulitattach volumes or not. Nevertheless due to other problems, that might occur when using the multiattach feature such as the requirement of special file systems and suitable backends, the SCS standard will only view this as OPTIONAL. ### Volume QoS -Quality of Service for volumes can be defined in volume qos objects. While admins can use these directly on volumes, a user is not able to see these object. Instead there can be ONE volume qos object associated to a volume type, which is then be used on all volumes created from that volume type. Through this indirection, functioning volume types without these associated qos objects and this feature not being heavily used, the SCS will currently only state this as an OPTIONAL volume type feature. +Quality of Service parameters for volumes can be defined in a volume qos object. While admins can use these qos objects directly on volumes, a user is not able to see such an object. Instead there can be ONE volume qos object associated to a volume type, which is then be used on all volumes created from that volume type. Through this indirection, volume types also working without these associated qos objects and this feature not being heavily used, the SCS standard will currently only view this as an OPTIONAL volume type feature. To make users aware that a volume type includes specific qos options, we recommend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: From 8c8dc01808ace62ab24cb982655c36dc7dd82308 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Mon, 6 Nov 2023 09:20:20 +0100 Subject: [PATCH 07/35] Update scs-0112-v1-volume-type-standard.md rename probable "encrypted" parameter in volume type properties for consistency Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index a1af3c1eb..b84b2ad11 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -69,7 +69,7 @@ openstack volume type list --encryption-type ``` **TODO**: -Users that don't have admin rights currently cannot see these encryption parameters. We want and need to change this in the OpenStack workflow through adding a property (e.g. "encryption"="true") that is also visible for users. This way we will be able to automatically check whether a volume type with encyrption is present. It should look like this: +Users that don't have admin rights currently cannot see these encryption parameters. We want and need to change this in the OpenStack workflow through adding a property (e.g. "encrypted"="true") that is also visible for users. This way we will be able to automatically check whether a volume type with encyrption is present. It should look like this: ```text openstack volume type show LUKS From 7a1d2ca4fa4b95d7c4108ddb451c1286733ca454 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Mon, 6 Nov 2023 15:14:54 +0100 Subject: [PATCH 08/35] Update Standards/scs-0112-v1-volume-type-standard.md fixing little typo Co-authored-by: Sven Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index b84b2ad11..559d59e0f 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -20,7 +20,7 @@ The following special terms are used throughout this standard document: ## Motivation -We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. While the option with which the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are REQUIRED, RECOMMENDED or only OPTIONAL within an SCS-compatible deployment. +We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of a volume. Encryption of volumes for example is solely decided by the volume type. While the option with which the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are REQUIRED, RECOMMENDED or only OPTIONAL within an SCS-compatible deployment. ## Design Considerations From 59b66c657f0b56855773e2445c4b8e398c25647f Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Thu, 9 Nov 2023 13:34:26 +0100 Subject: [PATCH 09/35] Update Standards/scs-0112-v1-volume-type-standard.md rename object to ressource Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 559d59e0f..97c37dd97 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -14,7 +14,7 @@ Volume Types are used to classify volumes and provide a basic decision for what The following special terms are used throughout this standard document: | Term | Meaning | |---|---| -| volume | OpenStack object, virtual drive which usually resides in a network storage backend | +| volume | OpenStack ressource, virtual drive which usually resides in a network storage backend | | AZ | Availability Zone | | Volume QoS | Quality of Service object for Volumes | From b5e5ed768791adb0661edddd3b24901c58459748 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Thu, 9 Nov 2023 15:25:00 +0100 Subject: [PATCH 10/35] Apply suggestions from code review rephrase a few sentence to better show the current upstream state and what to expect from the default volume type Co-authored-by: anjastrunk <119566837+anjastrunk@users.noreply.github.com> Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 97c37dd97..6516542c8 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -26,7 +26,7 @@ We want to standardize a few varieties of volume types. While a user can choose All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) -To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard and whether it is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. +To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. | Aspect | Standardize? | Discoverability | comments | | ---- | ---- | ---- | ------ | @@ -43,7 +43,7 @@ Example: one volume type that uses LUKS-encryption with a ceph storage with inhe ## DEFAULT volume types -There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. +There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from default volume types. The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And the SCS will not make any assumptions about parameters being present in the default volume type. From 0792426bdb609dce31f65c02b947350d54d1283c Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 10 Nov 2023 08:54:27 +0100 Subject: [PATCH 11/35] Update Standards/scs-0112-v1-volume-type-standard.md remove trailing spaces Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 6516542c8..a4a9bc27d 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -43,7 +43,7 @@ Example: one volume type that uses LUKS-encryption with a ceph storage with inhe ## DEFAULT volume types -There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from default volume types. +There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from default volume types. The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And the SCS will not make any assumptions about parameters being present in the default volume type. From 021a3c2540b0bcd19827af91d7e630dd9f382208 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 28 Nov 2023 15:08:43 +0100 Subject: [PATCH 12/35] Update scs-0112-v1-volume-type-standard.md a little restructuring of the standard. Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 76 ++++++++++--------- 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index a4a9bc27d..5a2af82f5 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -26,21 +26,40 @@ We want to standardize a few varieties of volume types. While a user can choose All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) -To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. - -| Aspect | Standardize? | Discoverability | comments | -| ---- | ---- | ---- | ------ | -| encryption | **Recommended** | admin only | Upstream work for extra_spec: encrypted=True/False needed | -| Backend name | - | - | - | -| AZs | **Optional** | - | Upstream: setable extra_spec would help | -| multiattach | **Optional** | yes | - | -| Replication | **Recommended** | backend-dependend | Upstream: setable extra_spec would help | -| Number of Replicas, etc | **Optional** | backend-dependend | Upstream: setable extra_spec would help | -| Volume QoS | **Optional** | admin only | depends on the admin only volume qos object | +To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. The last column says how the standard is handled, which is currently through the addition of a stadardized phrase at the beginning of the description of a volume type. + +| Aspect | Part of Standard | Discoverability | comments | concluded standardized description | +| ---- | ---- | ---- | ---- | ------ | +| encryption | **Recommended** | admin only | Upstream work for extra_spec: encrypted=True/False needed | **"[encrypted]"** | +| Replication | **Recommended** | backend-dependend | Upstream: setable extra_spec would help | **"[replicated]"** | +| AZs | **Optional** | - | Upstream: setable extra_spec would help | - | +| multiattach | **Optional** | yes | - | - | +| Number of Replicas, etc | **Optional** | backend-dependend | Upstream: setable extra_spec would help | - | +| Volume QoS | **Optional** | admin only | depends on the admin only volume qos object | - | +| Backend name | no | - | - | - | In addition it is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but they don't have to be different volume types. Example: one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. +The current downside is that users without admin rights currently cannot see the recommended aspects. We want and need to change this in the OpenStack workflow through adding discoverable properties (e.g. "encrypted"="true" and "replicated"="true") that are also visible for users. This way we will be able to automatically check whether a volume type with encyrption and or replication is present. It should look like this example for encryption: + +```text +openstack volume type show LUKS ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Field | Value | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| access_project_ids | None | +| description | None | +| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | +| is_public | True | +| name | LUKS | +| properties | encrypted='true' | +| qos_specs_id | None | ++--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ +``` + +We want to find a way to also use the internal extra_spec for replication, when the replication is automatically done by the backend. If this is not possible, we would like to introduce another property, which has to be set by the admin, when setting the volume type. Only after that we will have the possibility to automatically check for a volume type with replication. + ## DEFAULT volume types There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from default volume types. @@ -55,6 +74,14 @@ Currently the SCS will not require volume types with certain specification. This The SCS recommends to have one or more volume types, which have the following specifications: +| Volume Type Nr | Type Name | Volume Type Description | +| ---- | ---- | ---- | +| 1 | LUKS | [encrypted] This volume type creates LUKS encryped volumes | +| 2 | replicated | [replicated] Volumes will be three times replicated | +| 3 | enc-repli| [encrypted][replicated] Volumes will be encrypted and three times replicated | + +The standardized phrases MUST be ordered alphabetically. Having only volume type number 3 would be enough to fulfill all recommandations. + ### Encryption Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. And for admins it is also currently possible to see these configurations in a volume type with both list and show commands. @@ -68,25 +95,7 @@ openstack volume type list --encryption-type +--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` -**TODO**: -Users that don't have admin rights currently cannot see these encryption parameters. We want and need to change this in the OpenStack workflow through adding a property (e.g. "encrypted"="true") that is also visible for users. This way we will be able to automatically check whether a volume type with encyrption is present. It should look like this: - -```text -openstack volume type show LUKS -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Field | Value | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| access_project_ids | None | -| description | None | -| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | -| is_public | True | -| name | LUKS | -| properties | encrypted='true' | -| qos_specs_id | None | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -``` - -Until this change happened, it will be necessary that deployer state that the volume type is encrypted in the description of the volume type in the following way: +As this is not visible for users, it will be necessary that deployers state that the volume type is encrypted in the description of the volume type in the following way: The description needs to begin with `[encrypted]`, after that any further description is allowed. It should look like this example: ```text @@ -113,9 +122,6 @@ Replication states, whether or not there are multiple replicas of a volume. Thus To fulfill this recommentation for now, the deployer needs to state the replication in the description of a volume type in the following way: The description needs to begin with `[replicated]`, after that any further description is allowed. -**TODO:** -We want to find a way to also use the internal extra_spec for replication, when the replication is automatically done by the backend. If this is not possible, we would like to introduce another property, which has to be set by the admin, when setting the volume type. Only after that we will have the possibility to automatically check for a volume type with replication. - #### OPTIONAL addition: Number of Replicas Additionally to the fact, that a volume type is replicated, it OPTIONALLY can be stated in the description or name, how many times the provisioned volume will be replicated. @@ -147,7 +153,7 @@ It is OPTIONAL to give more information to users like this: | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access_project_ids | None | -| description | [encrypted:cipher=aes,key_size=256][replicated] Content will be replicated three times and LUKS encryption is used. | +| description | [encrypted:cipher=aes,key_size=256][replicated: 3 times] Content will be replicated three times and LUKS encryption is used. | | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | hdd-three-replicas-LUKS | @@ -217,4 +223,4 @@ As there are currently no REQUIRED volume types, we can only look for the RECOMM Furthermore the recommended aspects currently have to be described in the description by the deployer. In future versions we aim to integrate some extra_specs for them in upstream OpenStack And it is also possible that a single volume type can currently fulfill all RECOMMENDED aspects. -**TODO:** The current test will check for the presence of `[encrypted]` and `[replicated]` in the description of at least one volume type. +The current test will check for the presence of `[encrypted]` and `[replicated]` in the description of at least one volume type. From f41021154d9eec79e46306b96d4fc74c67b9d5d4 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 28 Nov 2023 15:13:18 +0100 Subject: [PATCH 13/35] Update scs-0112-v1-volume-type-standard.md fix linter errors Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 5a2af82f5..79b874224 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -76,9 +76,9 @@ The SCS recommends to have one or more volume types, which have the following sp | Volume Type Nr | Type Name | Volume Type Description | | ---- | ---- | ---- | -| 1 | LUKS | [encrypted] This volume type creates LUKS encryped volumes | -| 2 | replicated | [replicated] Volumes will be three times replicated | -| 3 | enc-repli| [encrypted][replicated] Volumes will be encrypted and three times replicated | +| 1 | LUKS | "[encrypted] This volume type creates LUKS encryped volumes" | +| 2 | replicated | "[replicated] Volumes will be three times replicated" | +| 3 | enc-repli| "[encrypted][replicated] Volumes will be encrypted and three times replicated" | The standardized phrases MUST be ordered alphabetically. Having only volume type number 3 would be enough to fulfill all recommandations. From 2ac3269730c383107319446332a87b27653a323b Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 28 Nov 2023 15:14:46 +0100 Subject: [PATCH 14/35] Update scs-0112-v1-volume-type-standard.md Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 79b874224..a57fe65da 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -78,7 +78,7 @@ The SCS recommends to have one or more volume types, which have the following sp | ---- | ---- | ---- | | 1 | LUKS | "[encrypted] This volume type creates LUKS encryped volumes" | | 2 | replicated | "[replicated] Volumes will be three times replicated" | -| 3 | enc-repli| "[encrypted][replicated] Volumes will be encrypted and three times replicated" | +| 3 | enc-repli| "[encrypted] [replicated] Volumes will be encrypted and three times replicated" | The standardized phrases MUST be ordered alphabetically. Having only volume type number 3 would be enough to fulfill all recommandations. From c4c4d6b1141601421e9c90bdf56ee15e5e1d3e62 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 1 Dec 2023 10:49:15 +0100 Subject: [PATCH 15/35] Apply suggestions from code review resolve typos Co-authored-by: Sven Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index a57fe65da..4f42694f8 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -26,7 +26,7 @@ We want to standardize a few varieties of volume types. While a user can choose All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) -To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. The last column says how the standard is handled, which is currently through the addition of a stadardized phrase at the beginning of the description of a volume type. +To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. The last column says how the standard is handled, which is currently through the addition of a standardized phrase at the beginning of the description of a volume type. | Aspect | Part of Standard | Discoverability | comments | concluded standardized description | | ---- | ---- | ---- | ---- | ------ | @@ -68,7 +68,7 @@ The parameters of volume types described in this standard do not have to be appl ## REQUIRED volume types -Currently the SCS will not require volume types with certain specification. This might change in the future. +Currently the SCS will not require volume types with certain specification. This will change in the future. ## RECOMMENDED volume types From 4e2f0fd0847da772657d163e64a7b20b8824051b Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 1 Dec 2023 14:38:47 +0100 Subject: [PATCH 16/35] Update scs-0112-v1-volume-type-standard.md Standardize the order of the tags in the description of recommended volume types Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 4f42694f8..2d3ef1e25 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -72,15 +72,15 @@ Currently the SCS will not require volume types with certain specification. This ## RECOMMENDED volume types -The SCS recommends to have one or more volume types, which have the following specifications: +The SCS recommends to have one or more volume types, that satisfy the need for encrpytion and replication. These are all allowed possibilites for those recommended volume types: | Volume Type Nr | Type Name | Volume Type Description | | ---- | ---- | ---- | -| 1 | LUKS | "[encrypted] This volume type creates LUKS encryped volumes" | -| 2 | replicated | "[replicated] Volumes will be three times replicated" | -| 3 | enc-repli| "[encrypted] [replicated] Volumes will be encrypted and three times replicated" | +| 1 | `__DEFAULT__` | `[encrypted][replicated] Volumes will be encrypted and three times replicated` | +| 2 | `LUKS` | `[encrypted] This volume type creates LUKS encryped volumes` | +| 3 | `replicated` | `[replicated] Volumes will be three times replicated` | -The standardized phrases MUST be ordered alphabetically. Having only volume type number 3 would be enough to fulfill all recommandations. +The standardized phrases are `[encrypted]` and `[replicated]`. They MUST appear at the beginning of the description of the volume type and they MUST be ordered alphabetically. Having only volume type number 1 would be enough to fulfill all recommendations. Having only volume types 2 and 3 together would also satisfy all recommendations ### Encryption @@ -120,7 +120,8 @@ Replication states, whether or not there are multiple replicas of a volume. Thus 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. To fulfill this recommentation for now, the deployer needs to state the replication in the description of a volume type in the following way: -The description needs to begin with `[replicated]`, after that any further description is allowed. +If the volume type is not encrypted the description needs to begin with `[replicated]`, after that any further description is allowed. +Otherwise the description needs to begin with `[encrypted][replicated]`. #### OPTIONAL addition: Number of Replicas From ee2678d295ebec7c080f79a879824f151848936c Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 1 Dec 2023 14:40:38 +0100 Subject: [PATCH 17/35] Update scs-0112-v1-volume-type-standard.md remove trailing space Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 2d3ef1e25..5f3a33d81 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -120,7 +120,7 @@ Replication states, whether or not there are multiple replicas of a volume. Thus 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. To fulfill this recommentation for now, the deployer needs to state the replication in the description of a volume type in the following way: -If the volume type is not encrypted the description needs to begin with `[replicated]`, after that any further description is allowed. +If the volume type is not encrypted the description needs to begin with `[replicated]`, after that any further description is allowed. Otherwise the description needs to begin with `[encrypted][replicated]`. #### OPTIONAL addition: Number of Replicas From bc77c1c3642c6e3c56e882adbd189be98995b18d Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 12 Jan 2024 13:46:07 +0100 Subject: [PATCH 18/35] Apply suggestions from code review Co-authored-by: anjastrunk <119566837+anjastrunk@users.noreply.github.com> Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 5f3a33d81..7d1cbd714 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -7,7 +7,7 @@ track: IaaS ## Introduction -Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes be very backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. +Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes be very backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one type. ### Glossary @@ -20,13 +20,13 @@ The following special terms are used throughout this standard document: ## Motivation -We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of a volume. Encryption of volumes for example is solely decided by the volume type. While the option with which the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are REQUIRED, RECOMMENDED or only OPTIONAL within an SCS-compatible deployment. +We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of a volume. Encryption of volumes for example is solely decided by the volume type. While the option with which the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are REQUIRED, RECOMMENDED or only OPTIONAL within a SCS-compatible deployment. ## Design Considerations All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) -To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. The last column says how the standard is handled, which is currently through the addition of a standardized phrase at the beginning of the description of a volume type. +To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. The last column says how this standard is handled, which is currently through the addition of a standardized phrase at the beginning of the description of a volume type. | Aspect | Part of Standard | Discoverability | comments | concluded standardized description | | ---- | ---- | ---- | ---- | ------ | @@ -38,10 +38,10 @@ To test whether a deployment has volume types with certain aspects, the discover | Volume QoS | **Optional** | admin only | depends on the admin only volume qos object | - | | Backend name | no | - | - | - | -In addition it is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but they don't have to be different volume types. +In addition it is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but there don't have to be different volume types. Example: one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. -The current downside is that users without admin rights currently cannot see the recommended aspects. We want and need to change this in the OpenStack workflow through adding discoverable properties (e.g. "encrypted"="true" and "replicated"="true") that are also visible for users. This way we will be able to automatically check whether a volume type with encyrption and or replication is present. It should look like this example for encryption: +The current downside is that users without admin rights currently cannot see the recommended aspects. To be able to automatically check whether a volume type with encyrption and or replication is present, an update or extension of the OpenStack workflow would be required. It should look like this example for encryption: ```text openstack volume type show LUKS @@ -62,7 +62,7 @@ We want to find a way to also use the internal extra_spec for replication, when ## DEFAULT volume types -There is always a default volume type defined in an OpenStack deployment. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from default volume types. +There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`. The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And the SCS will not make any assumptions about parameters being present in the default volume type. From 374e8c5fad9e8bde7500d2846b56c22ac9dc3a0c Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Fri, 12 Jan 2024 15:10:19 +0100 Subject: [PATCH 19/35] Update scs-0112-v1-volume-type-standard.md rephrasing some parts about the volume qos object and deleting references to upstreaming options Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 7d1cbd714..2305df2f4 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -58,8 +58,6 @@ openstack volume type show LUKS +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` -We want to find a way to also use the internal extra_spec for replication, when the replication is automatically done by the backend. If this is not possible, we would like to introduce another property, which has to be set by the admin, when setting the volume type. Only after that we will have the possibility to automatically check for a volume type with replication. - ## DEFAULT volume types There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`. @@ -195,7 +193,7 @@ A few backends enable providers to allow volumes to be attached to mulitple VMs ### Volume QoS -Quality of Service parameters for volumes can be defined in a volume qos object. While admins can use these qos objects directly on volumes, a user is not able to see such an object. Instead there can be ONE volume qos object associated to a volume type, which is then be used on all volumes created from that volume type. Through this indirection, volume types also working without these associated qos objects and this feature not being heavily used, the SCS standard will currently only view this as an OPTIONAL volume type feature. +Quality of Service parameters for volumes can be defined in a `volume qos` object. This is an extra OpenStack ressource, that can be created by administrators having the admin role and are only visible to them. Those admins can directly use such a `volume qos` object to create a volume. To also let user roles benefit of these objects, administrators can associate a single `volume qos` object ot a volume type, which is then be used on all volumes created from that volume type. Through this indirection, volume types are also working without these associated QoS objects and this feature not being heavily used, the SCS standard will currently only view this as an OPTIONAL volume type feature. To make users aware that a volume type includes specific qos options, we recommend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: From 6a5957313fe5e376423a58cb366c693956d9fc45 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Mon, 12 Feb 2024 14:50:57 +0100 Subject: [PATCH 20/35] Update scs-0112-v1-volume-type-standard.md Exclude Optional types and include a topic for standardized description naming. Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-0112-v1-volume-type-standard.md | 151 ++++-------------- 1 file changed, 27 insertions(+), 124 deletions(-) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-0112-v1-volume-type-standard.md index 2305df2f4..b28dd8d62 100644 --- a/Standards/scs-0112-v1-volume-type-standard.md +++ b/Standards/scs-0112-v1-volume-type-standard.md @@ -26,39 +26,27 @@ We want to standardize a few varieties of volume types. While a user can choose All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) -To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. The following table shows, which aspects are considered in this standard. Furthermore the table shows, according to the current upstream implementation, whether the feature is discoverable, and who can discover this. The comments state what can be done to help with the discoverability and other important restrictions. The last column says how this standard is handled, which is currently through the addition of a standardized phrase at the beginning of the description of a volume type. - -| Aspect | Part of Standard | Discoverability | comments | concluded standardized description | -| ---- | ---- | ---- | ---- | ------ | -| encryption | **Recommended** | admin only | Upstream work for extra_spec: encrypted=True/False needed | **"[encrypted]"** | -| Replication | **Recommended** | backend-dependend | Upstream: setable extra_spec would help | **"[replicated]"** | -| AZs | **Optional** | - | Upstream: setable extra_spec would help | - | -| multiattach | **Optional** | yes | - | - | -| Number of Replicas, etc | **Optional** | backend-dependend | Upstream: setable extra_spec would help | - | -| Volume QoS | **Optional** | admin only | depends on the admin only volume qos object | - | -| Backend name | no | - | - | - | - -In addition it is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but there don't have to be different volume types. -Example: one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. +### Systematic Description of Volume Types -The current downside is that users without admin rights currently cannot see the recommended aspects. To be able to automatically check whether a volume type with encyrption and or replication is present, an update or extension of the OpenStack workflow would be required. It should look like this example for encryption: +To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. As for the time this standard is created, there is no way for users to discover all aspects through OpenStack commands. Therefore the aspects, that are fulfilled within a volume type, should be stated in the beginning of the **description** of a volume type in the following manner: -```text -openstack volume type show LUKS -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Field | Value | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| access_project_ids | None | -| description | None | -| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | -| is_public | True | -| name | LUKS | -| properties | encrypted='true' | -| qos_specs_id | None | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -``` +`[scs:aspect1][scs:aspect2]...` + +There is no sorting of aspects required. Every aspect should only be mentioned to the maximal amount of one. + +### Standardized Aspects + +The following table shows, which aspects are considered in this standard. The last column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled: + +| Aspect | Part of Standard | standardized description | +| ---- | ---- | ------ | +| Encryption | **Recommended** | **"[scs:encrypted]"** | +| Replication | **Recommended** | **"[scs:replicated]"** | + +It is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but there don't have to be different volume types. +Example: one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. -## DEFAULT volume types +## DEFAULT volume type There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`. @@ -70,31 +58,13 @@ Currently the SCS will not require volume types with certain specification. This ## RECOMMENDED volume types -The SCS recommends to have one or more volume types, that satisfy the need for encrpytion and replication. These are all allowed possibilites for those recommended volume types: - -| Volume Type Nr | Type Name | Volume Type Description | -| ---- | ---- | ---- | -| 1 | `__DEFAULT__` | `[encrypted][replicated] Volumes will be encrypted and three times replicated` | -| 2 | `LUKS` | `[encrypted] This volume type creates LUKS encryped volumes` | -| 3 | `replicated` | `[replicated] Volumes will be three times replicated` | - -The standardized phrases are `[encrypted]` and `[replicated]`. They MUST appear at the beginning of the description of the volume type and they MUST be ordered alphabetically. Having only volume type number 1 would be enough to fulfill all recommendations. Having only volume types 2 and 3 together would also satisfy all recommendations +The SCS recommends to have one or more volume types, that satisfy the need for encrpytion and replication. ### Encryption -Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. And for admins it is also currently possible to see these configurations in a volume type with both list and show commands. - -```text -openstack volume type list --encryption-type -+--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| ID | Name | Is Public | Encryption | -+--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| d34b5de0-d7bf-43cf-8a74-3827169d6616 | LUKS | True | cipher='aes-xts-plain64', control_location='front-end', encryption_id='217386bc-1e9b-46a3-9e0e-3ad62c07826c', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor' | -+--------------------------------------+-------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -``` +There SHOULD at least be one volume type with the encryption aspect. -As this is not visible for users, it will be necessary that deployers state that the volume type is encrypted in the description of the volume type in the following way: -The description needs to begin with `[encrypted]`, after that any further description is allowed. It should look like this example: +Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. Additionally to be discoverable by users an admin has to edit the description and add `[scs:encrypted]` at the beginning or after another scs aspect. It should look like this example: ```text openstack volume type show LUKS @@ -102,7 +72,7 @@ openstack volume type show LUKS | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access_project_ids | None | -| description | [encrypted] This volume uses LUKS-encryption | +| description | [scs:encrypted] This volume uses LUKS-encryption | | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | LUKS | @@ -112,18 +82,14 @@ openstack volume type show LUKS ### Replication +There SHOULD at least be one volume type with the replication aspect. + Replication states, whether or not there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved: 1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. -To fulfill this recommentation for now, the deployer needs to state the replication in the description of a volume type in the following way: -If the volume type is not encrypted the description needs to begin with `[replicated]`, after that any further description is allowed. -Otherwise the description needs to begin with `[encrypted][replicated]`. - -#### OPTIONAL addition: Number of Replicas - -Additionally to the fact, that a volume type is replicated, it OPTIONALLY can be stated in the description or name, how many times the provisioned volume will be replicated. +To fulfill this recommentation for now, the admin needs to add `[scs:replicated]` at the beginning or after any other scs aspect into the descriotion of the volume type. ### Example @@ -136,23 +102,7 @@ It should look like the following part: | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access_project_ids | None | -| description | [encrypted][replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used. | -| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | -| is_public | True | -| name | hdd-three-replicas-LUKS | -| properties | | -| qos_specs_id | None | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -``` - -It is OPTIONAL to give more information to users like this: - -```text -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Field | Value | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| access_project_ids | None | -| description | [encrypted:cipher=aes,key_size=256][replicated: 3 times] Content will be replicated three times and LUKS encryption is used. | +| description | [scs:encrypted][scs:replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used. | | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | hdd-three-replicas-LUKS | @@ -163,54 +113,7 @@ It is OPTIONAL to give more information to users like this: ## OPTIONAL volume types -In this section volume types are described, that we do not want to explicitly recommend, but find them worth listing as optional for deployers. - -### Availability Zones - -Availability Zones are not necessarily used by every deployer of a cloud. The SCS does not want do make it necessary to have AZs for a compatible deployment. But we want to encourage deployer who use multiple AZs to read the following: - -While there can be multiple Compute AZs, there can also be multiple Storage AZs for Volumes. And it might be quite confusing for users, which volumes can be used in which AZs. To make it even further complicated, there are backends like ceph, which can provide volumes for multiple compute AZs just with some Nova configuration. Therefore we would encourage to use either the property of volume types OR the description of the volume types to describe, in which AZs the volumes based on this type can be used. - -A description for a ceph volume type might look like this: - -```text -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Field | Value | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| access_project_ids | None | -| description | Content will be replicated three times on spinning disks to ensure consistency and availability for your data. The volumes can be used in AZ 1,3,4. | -| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | -| is_public | True | -| name | hdd-three-replicas-AZ134 | -| properties | | -| qos_specs_id | None | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -``` - -### Multiattach - -A few backends enable providers to allow volumes to be attached to mulitple VMs simultaneously. The OpenStack API can be used to enable the usage of multiattach through the volume type. The property is also automatically shown to users. If a deployer wants to use this feature they have to do it in this way. So users will always see whether a volume type can be used for mulitattach volumes or not. Nevertheless due to other problems, that might occur when using the multiattach feature such as the requirement of special file systems and suitable backends, the SCS standard will only view this as OPTIONAL. - -### Volume QoS - -Quality of Service parameters for volumes can be defined in a `volume qos` object. This is an extra OpenStack ressource, that can be created by administrators having the admin role and are only visible to them. Those admins can directly use such a `volume qos` object to create a volume. To also let user roles benefit of these objects, administrators can associate a single `volume qos` object ot a volume type, which is then be used on all volumes created from that volume type. Through this indirection, volume types are also working without these associated QoS objects and this feature not being heavily used, the SCS standard will currently only view this as an OPTIONAL volume type feature. - -To make users aware that a volume type includes specific qos options, we recommend to write it into the description of a volume type, as any association to a volume qos object cannot be seen by normal users: - -```text -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Field | Value | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| access_project_ids | None | -| description | Content will be replicated three times on spinning disks to ensure consistency and availability for your data. Iops: read:20k, write:10k | -| id | d63307fb-167a-4aa0-9066-66595ea9fb21 | -| is_public | True | -| name | hdd-three-replicas-AZ134 | -| properties | - | -| qos_specs_id | None | -+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -``` +Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the CSPs, if they want to use some of these other aspects. ## Related Documents From f616d542f6b94ad3a00bc731d90109690a580299 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Wed, 14 Feb 2024 08:29:37 +0100 Subject: [PATCH 21/35] Rename scs-0112-v1-volume-type-standard.md to scs-XXXX-v1-volume-type-standard.md rename to address Error in CI jobs Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- ...olume-type-standard.md => scs-XXXX-v1-volume-type-standard.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Standards/{scs-0112-v1-volume-type-standard.md => scs-XXXX-v1-volume-type-standard.md} (100%) diff --git a/Standards/scs-0112-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md similarity index 100% rename from Standards/scs-0112-v1-volume-type-standard.md rename to Standards/scs-XXXX-v1-volume-type-standard.md From 8a2b0d1251181605ae133212e78d52e3f2661518 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Wed, 6 Mar 2024 16:27:40 +0100 Subject: [PATCH 22/35] Update scs-XXXX-v1-volume-type-standard.md Edited the introduction, motivation and some minor spelling issues. Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 30 ++++++++++++------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index b28dd8d62..01f99a21c 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -7,7 +7,10 @@ track: IaaS ## Introduction -Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes be very backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one type. +A volume is a virtual drive that is to be used by an instance (i. e., a virtual machine). With OpenStack, +each volume is created per a type that determines basic features of the volume as provided by the backend, +such as encryption, replication, or quality of service. As of the writing of this document, presence or absence of these +features can not be discovered with full certainty by non-privileged users via the OpenStack API. ### Glossary @@ -20,7 +23,11 @@ The following special terms are used throughout this standard document: ## Motivation -We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of a volume. Encryption of volumes for example is solely decided by the volume type. While the option with which the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. In the following part, we want to state, which varieties of volume types are REQUIRED, RECOMMENDED or only OPTIONAL within a SCS-compatible deployment. +As an SCS user, I want to be able to create volumes with certain common features, such as encryption or +replication, and to do so in a standardized manner as well as programmatically. +This standard outlines a way of formally advertising these common features for a volume type to +non-privileged users, so that the most suitable volume type can be discovered and selected easily -- both by +the human user and by a program. ## Design Considerations @@ -30,7 +37,7 @@ All Considerations can be looked up in detail in the [Decision Record for the Vo To test whether a deployment has volume types with certain aspects, the discoverability of the parameters in the volume type has to be given. As for the time this standard is created, there is no way for users to discover all aspects through OpenStack commands. Therefore the aspects, that are fulfilled within a volume type, should be stated in the beginning of the **description** of a volume type in the following manner: -`[scs:aspect1][scs:aspect2]...` +`[scs:aspect1, aspect2, ..., aspectN]...` There is no sorting of aspects required. Every aspect should only be mentioned to the maximal amount of one. @@ -48,17 +55,17 @@ Example: one volume type that uses LUKS-encryption with a ceph storage with inhe ## DEFAULT volume type -There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. The SCS does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`. +There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. This standard does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`. -The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And the SCS will not make any assumptions about parameters being present in the default volume type. +The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And this standard will not make any assumptions about parameters being present in the default volume type. ## REQUIRED volume types -Currently the SCS will not require volume types with certain specification. This will change in the future. +Currently, this standard will not require volume types with certain specification. This will change in the future. ## RECOMMENDED volume types -The SCS recommends to have one or more volume types, that satisfy the need for encrpytion and replication. +This standard recommends to have one or more volume types, that satisfy the need for encrpytion and replication. ### Encryption @@ -102,7 +109,7 @@ It should look like the following part: | Field | Value | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access_project_ids | None | -| description | [scs:encrypted][scs:replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used. | +| description | [scs:encrypted, replicated] Content will be replicated three times to ensure consistency and availability for your data. LUKS encryption is used. | | id | d63307fb-167a-4aa0-9066-66595ea9fb21 | | is_public | True | | name | hdd-three-replicas-LUKS | @@ -122,7 +129,10 @@ Any other aspects of volume types, that can be found in the decision record are ## Conformance Tests As there are currently no REQUIRED volume types, we can only look for the RECOMMENDED aspects and thus executing a conformance test is not a must. -Furthermore the recommended aspects currently have to be described in the description by the deployer. In future versions we aim to integrate some extra_specs for them in upstream OpenStack -And it is also possible that a single volume type can currently fulfill all RECOMMENDED aspects. +Furthermore the recommended aspects currently have to be described in the description by the deployer. +Deployers MUST provided correct information, when adhering to this standard. +The tests will only check the syntactical correctness. +In future versions we aim to integrate some extra_specs for them in upstream OpenStack. +It is also possible that a single volume type can currently fulfill all RECOMMENDED aspects. The current test will check for the presence of `[encrypted]` and `[replicated]` in the description of at least one volume type. From 8620598e77415112f6dcc771e9425d61386cf032 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Wed, 6 Mar 2024 16:37:12 +0100 Subject: [PATCH 23/35] Update scs-XXXX-v1-volume-type-standard.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit change "--" to — Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index 01f99a21c..2069357de 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -26,7 +26,7 @@ The following special terms are used throughout this standard document: As an SCS user, I want to be able to create volumes with certain common features, such as encryption or replication, and to do so in a standardized manner as well as programmatically. This standard outlines a way of formally advertising these common features for a volume type to -non-privileged users, so that the most suitable volume type can be discovered and selected easily -- both by +non-privileged users, so that the most suitable volume type can be discovered and selected easily — both by the human user and by a program. ## Design Considerations From 3ac8ed5568f4b0480a39c1b5a955f6e5257327d9 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 12 Mar 2024 14:14:33 +0100 Subject: [PATCH 24/35] Create volume-types-check.py I incorporated @mbuechse s test and used the new syntax. A test also showed the need to catch empty descriptions for volume types. Now we do have everything together in one PR. Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Tests/iaas/volume-types/volume-types-check.py | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 Tests/iaas/volume-types/volume-types-check.py diff --git a/Tests/iaas/volume-types/volume-types-check.py b/Tests/iaas/volume-types/volume-types-check.py new file mode 100644 index 000000000..97ece4153 --- /dev/null +++ b/Tests/iaas/volume-types/volume-types-check.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Volume types checker +Check given cloud for conformance with SCS standard regarding +volume types, to be found under /Standards/scs-0112-v1-volume-types.md +Return code is 0 precisely when it could be verified that the standard is satisfied. +Otherwise the return code is the number of errors that occurred (up to 127 due to OS +restrictions); for further information, see the log messages on various channels: + CRITICAL for problems preventing the test to complete, + ERROR for violations of requirements, + INFO for violations of recommendations, + DEBUG for background information and problems that don't hinder the test. +""" +from collections import Counter, defaultdict +import getopt +import logging +import os +import re +import sys + +import openstack +import openstack.cloud + + +logger = logging.getLogger(__name__) +RECOGNIZED_FEATURES = ('encrypted', 'replicated') + + +def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")): + """Extract feature-list-like prefix + If given `description` starts with a feature-list-like prefix, return list of features, + otherwise None. To be more precise, we look for a string of this form: + `[scs:`feat1`, `...`, `...featN`]` + where N >= 1 and featJ is a string that doesn't contain any comma or brackets. We return + the list [feat1, ..., featN] of substrings. + """ + if not description: + # The description can be "None" or empty - we need to catch this here + # otherwise we will get a critical Error in the pattern matching + return + match = pattern.match(description) + if not match: + return + fs = match.group(1) + if not fs: + return [] + return fs.split(", ") + + +def test_feature_list(type_name: str, fl: list[str], recognized=RECOGNIZED_FEATURES): + """Test given list of features and report errors to error channel""" + if not fl: + # either None (no feature list) or empty feature list: nothing to check + return + if fl != sorted(fl): + logger.error(f"{type_name}: feature list not sorted") + ctr = Counter(fl) + duplicates = [key for key, c in ctr.items() if c > 1] + if duplicates: + logger.error(f"{type_name}: duplicate features: {', '.join(duplicates)}") + unrecognized = [f for f in ctr if f not in recognized] + if unrecognized: + logger.error(f"{type_name}: unrecognized features: {', '.join(unrecognized)}") + + +def print_usage(file=sys.stderr): + """Help output""" + print("""Usage: entropy-check.py [options] +This tool checks volume types according to the SCS Standard 0112 "Volume Types". +Options: + [-c/--os-cloud OS_CLOUD] sets cloud environment (default from OS_CLOUD env) + [-d/--debug] enables DEBUG logging channel +""", end='', file=file) + + +class CountingHandler(logging.Handler): + def __init__(self, level=logging.NOTSET): + super().__init__(level=level) + self.bylevel = Counter() + + def handle(self, record): + self.bylevel[record.levelno] += 1 + + +def main(argv): + # configure logging + logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.INFO) + openstack.enable_logging(debug=False) + # count the number of log records per level (used for summary and return code) + counting_handler = CountingHandler(level=logging.INFO) + logger.addHandler(counting_handler) + + try: + opts, args = getopt.gnu_getopt(argv, "c:i:hd", ["os-cloud=", "help", "debug"]) + except getopt.GetoptError as exc: + logger.critical(f"{exc}") + print_usage() + return 1 + + cloud = os.environ.get("OS_CLOUD") + for opt in opts: + if opt[0] == "-h" or opt[0] == "--help": + print_usage() + return 0 + if opt[0] == "-c" or opt[0] == "--os-cloud": + cloud = opt[1] + if opt[0] == "-d" or opt[0] == "--debug": + logging.getLogger().setLevel(logging.DEBUG) + + if not cloud: + logger.critical("You need to have OS_CLOUD set or pass --os-cloud=CLOUD.") + return 1 + + try: + logger.debug(f"Connecting to cloud '{cloud}'") + with openstack.connect(cloud=cloud, timeout=32) as conn: + volume_types = conn.list_volume_types() + # collect volume types according to features + by_feature = defaultdict(list) + for typ in volume_types: + fl = extract_feature_list(typ.description) + if fl is None: + continue + logger.debug(f"{typ.name}: feature list {fl!r}") + test_feature_list(typ.name, fl) + for feat in fl: + by_feature[feat].append(typ.name) + logger.debug(f"volume types by feature: {dict(by_feature)}") + for feat in ('encrypted', 'replicated'): + if not by_feature[feat]: + logger.warning(f"Recommendation violated: missing {feat} volume type") + except BaseException as e: + logger.critical(f"{e!r}") + logger.debug("Exception info", exc_info=True) + + c = counting_handler.bylevel + logger.debug( + "Total critical / error / warning: " + f"{c[logging.CRITICAL]} / {c[logging.ERROR]} / {c[logging.WARNING]}" + ) + return min(127, c[logging.CRITICAL] + c[logging.ERROR]) # cap at 127 due to OS restrictions + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) From 03cc149604bdda57ec630ff308fbba7d35d89f6d Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 12 Mar 2024 14:18:41 +0100 Subject: [PATCH 25/35] Update volume-types-check.py restructure formatting of the comment in line 37 /38 Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Tests/iaas/volume-types/volume-types-check.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Tests/iaas/volume-types/volume-types-check.py b/Tests/iaas/volume-types/volume-types-check.py index 97ece4153..712f6a7ae 100644 --- a/Tests/iaas/volume-types/volume-types-check.py +++ b/Tests/iaas/volume-types/volume-types-check.py @@ -34,7 +34,7 @@ def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")) the list [feat1, ..., featN] of substrings. """ if not description: - # The description can be "None" or empty - we need to catch this here + # The description can be "None" or empty - we need to catch this here # otherwise we will get a critical Error in the pattern matching return match = pattern.match(description) From 427a286e3f08e1e8e1190d68804c15926863aa62 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Thu, 14 Mar 2024 09:16:40 +0100 Subject: [PATCH 26/35] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Matthias Büchse Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index 2069357de..3d8eb7bc2 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -31,7 +31,7 @@ the human user and by a program. ## Design Considerations -All Considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) +All considerations can be looked up in detail in the [Decision Record for the Volume Type Standard.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) ### Systematic Description of Volume Types @@ -124,7 +124,7 @@ Any other aspects of volume types, that can be found in the decision record are ## Related Documents -[Here is the decision record document.](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) +- corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) ## Conformance Tests From 504c48c832488177ae445b33b5e0fb3341bdedbb Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Thu, 14 Mar 2024 10:03:23 +0100 Subject: [PATCH 27/35] Update scs-XXXX-v1-volume-type-standard.md Adjust some wording Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index 3d8eb7bc2..90575da74 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -45,13 +45,13 @@ There is no sorting of aspects required. Every aspect should only be mentioned t The following table shows, which aspects are considered in this standard. The last column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled: -| Aspect | Part of Standard | standardized description | +| Aspect | Requirement | standardized description | | ---- | ---- | ------ | | Encryption | **Recommended** | **"[scs:encrypted]"** | | Replication | **Recommended** | **"[scs:replicated]"** | -It is possible to use multiple of those aspects within one volume type. SCS will only ever look, if there is a volume type that has an aspect, but there don't have to be different volume types. -Example: one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. +It is possible to use multiple of those aspects within one volume type. There don't have to be different volume types for each aspect. +For instance, one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. ## DEFAULT volume type @@ -61,11 +61,11 @@ The parameters of volume types described in this standard do not have to be appl ## REQUIRED volume types -Currently, this standard will not require volume types with certain specification. This will change in the future. +Currently, this standard will not require volume types with certain specification. ## RECOMMENDED volume types -This standard recommends to have one or more volume types, that satisfy the need for encrpytion and replication. +This standard recommends to have one or more volume types, that feature encryption and replication. ### Encryption From 4b479d5bc638dbef69ab1b0f318d3c6f179af959 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Mon, 18 Mar 2024 10:50:16 +0100 Subject: [PATCH 28/35] Update scs-XXXX-v1-volume-type-standard.md adjust details: aspect, sorting, comment on table as discussed with @anjastrunk and @mbuechse Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index 90575da74..dd2c40589 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -18,6 +18,8 @@ The following special terms are used throughout this standard document: | Term | Meaning | |---|---| | volume | OpenStack ressource, virtual drive which usually resides in a network storage backend | +| volume feature | A certain feature a volume can possess | +| aspect | part of a volume type that will activate a corresponding feature in a created volume | | AZ | Availability Zone | | Volume QoS | Quality of Service object for Volumes | @@ -25,7 +27,7 @@ The following special terms are used throughout this standard document: As an SCS user, I want to be able to create volumes with certain common features, such as encryption or replication, and to do so in a standardized manner as well as programmatically. -This standard outlines a way of formally advertising these common features for a volume type to +This standard outlines a way of formally advertising these common aspects for a volume type to non-privileged users, so that the most suitable volume type can be discovered and selected easily — both by the human user and by a program. @@ -39,16 +41,16 @@ To test whether a deployment has volume types with certain aspects, the discover `[scs:aspect1, aspect2, ..., aspectN]...` -There is no sorting of aspects required. Every aspect should only be mentioned to the maximal amount of one. +The mentioned aspects MUST be sorted alphebetically and every aspect should only be mentioned to the maximal amount of one. ### Standardized Aspects The following table shows, which aspects are considered in this standard. The last column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled: -| Aspect | Requirement | standardized description | -| ---- | ---- | ------ | -| Encryption | **Recommended** | **"[scs:encrypted]"** | -| Replication | **Recommended** | **"[scs:replicated]"** | +| Aspect | Requirement | standardized description | comment | +| ---- | ---- | ------ | ------ | +| Encryption | **Recommended** | **"[scs:encrypted]"** | used to encrypt the volume | +| Replication | **Recommended** | **"[scs:replicated]"** | volume is replicated to aviod data loss in a case of hardware failure | It is possible to use multiple of those aspects within one volume type. There don't have to be different volume types for each aspect. For instance, one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. @@ -67,9 +69,13 @@ Currently, this standard will not require volume types with certain specificatio This standard recommends to have one or more volume types, that feature encryption and replication. -### Encryption +## OPTIONAL volume types + +Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the CSPs, if they want to use some of these other aspects. + +## Implementation Details -There SHOULD at least be one volume type with the encryption aspect. +### Encryption Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. Additionally to be discoverable by users an admin has to edit the description and add `[scs:encrypted]` at the beginning or after another scs aspect. It should look like this example: @@ -89,8 +95,6 @@ openstack volume type show LUKS ### Replication -There SHOULD at least be one volume type with the replication aspect. - Replication states, whether or not there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved: 1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. @@ -118,10 +122,6 @@ It should look like the following part: +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` -## OPTIONAL volume types - -Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the CSPs, if they want to use some of these other aspects. - ## Related Documents - corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md) From aa68c776ab38f3487b412223860d86771c76d922 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Mon, 18 Mar 2024 10:50:40 +0100 Subject: [PATCH 29/35] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Matthias Büchse Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index dd2c40589..1f9964e03 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -128,11 +128,12 @@ It should look like the following part: ## Conformance Tests -As there are currently no REQUIRED volume types, we can only look for the RECOMMENDED aspects and thus executing a conformance test is not a must. -Furthermore the recommended aspects currently have to be described in the description by the deployer. -Deployers MUST provided correct information, when adhering to this standard. -The tests will only check the syntactical correctness. -In future versions we aim to integrate some extra_specs for them in upstream OpenStack. -It is also possible that a single volume type can currently fulfill all RECOMMENDED aspects. - -The current test will check for the presence of `[encrypted]` and `[replicated]` in the description of at least one volume type. +The script `/Tests/iaas/volume-types/volume-types-check.py` connects to an OpenStack environment and tests +the following: + +- for each volume type: if its description starts with `[scs:....]`, then this prefix is a feature list + (sorted, each entry at most once), and each entry is one of the possible features described here, +- the recommended volume types are present (otherwise, a WARNING is produced). + +The return code is zero precisely when the test could be performed and the conditions are satisfied. +Otherwise, detailed errors and warnings are output to stderr. From 80940e53cb7814bb21b13a103313c1dbf7f3d403 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 19 Mar 2024 09:22:53 +0100 Subject: [PATCH 30/35] Update volume-types-check.py Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Tests/iaas/volume-types/volume-types-check.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/Tests/iaas/volume-types/volume-types-check.py b/Tests/iaas/volume-types/volume-types-check.py index 712f6a7ae..1570b6806 100644 --- a/Tests/iaas/volume-types/volume-types-check.py +++ b/Tests/iaas/volume-types/volume-types-check.py @@ -1,7 +1,9 @@ #!/usr/bin/env python3 """Volume types checker + Check given cloud for conformance with SCS standard regarding volume types, to be found under /Standards/scs-0112-v1-volume-types.md + Return code is 0 precisely when it could be verified that the standard is satisfied. Otherwise the return code is the number of errors that occurred (up to 127 due to OS restrictions); for further information, see the log messages on various channels: @@ -27,14 +29,17 @@ def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")): """Extract feature-list-like prefix + If given `description` starts with a feature-list-like prefix, return list of features, otherwise None. To be more precise, we look for a string of this form: + `[scs:`feat1`, `...`, `...featN`]` + where N >= 1 and featJ is a string that doesn't contain any comma or brackets. We return the list [feat1, ..., featN] of substrings. """ if not description: - # The description can be "None" or empty - we need to catch this here + # The description can be None or empty - we need to catch this here # otherwise we will get a critical Error in the pattern matching return match = pattern.match(description) From fd1eed8e4c06b9f24397e8b9bcde8ab82b4faeab Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 19 Mar 2024 09:26:23 +0100 Subject: [PATCH 31/35] Update volume-types-check.py MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Matthias Büchse Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Tests/iaas/volume-types/volume-types-check.py | 1 - 1 file changed, 1 deletion(-) diff --git a/Tests/iaas/volume-types/volume-types-check.py b/Tests/iaas/volume-types/volume-types-check.py index 1570b6806..21e5767c5 100644 --- a/Tests/iaas/volume-types/volume-types-check.py +++ b/Tests/iaas/volume-types/volume-types-check.py @@ -40,7 +40,6 @@ def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")) """ if not description: # The description can be None or empty - we need to catch this here - # otherwise we will get a critical Error in the pattern matching return match = pattern.match(description) if not match: From 9f6fa4c036d53f1b946b18792b62d472d59306af Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 19 Mar 2024 12:50:04 +0100 Subject: [PATCH 32/35] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fixing typos Co-authored-by: Matthias Büchse Co-authored-by: anjastrunk <119566837+anjastrunk@users.noreply.github.com> Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 18 +++++++++--------- Tests/iaas/volume-types/volume-types-check.py | 4 ++-- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index 1f9964e03..ebe61b489 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -17,9 +17,9 @@ features can not be discovered with full certainty by non-privileged users via t The following special terms are used throughout this standard document: | Term | Meaning | |---|---| -| volume | OpenStack ressource, virtual drive which usually resides in a network storage backend | +| volume | OpenStack resource, virtual drive which usually resides in a network storage backend | | volume feature | A certain feature a volume can possess | -| aspect | part of a volume type that will activate a corresponding feature in a created volume | +| aspect | Part of a volume type that will activate a corresponding feature in a created volume | | AZ | Availability Zone | | Volume QoS | Quality of Service object for Volumes | @@ -45,12 +45,12 @@ The mentioned aspects MUST be sorted alphebetically and every aspect should only ### Standardized Aspects -The following table shows, which aspects are considered in this standard. The last column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled: +The following table shows which aspects are considered in this standard. The third column shows how the description of the volume type has to be adjusted, if the aspect is fulfilled: | Aspect | Requirement | standardized description | comment | | ---- | ---- | ------ | ------ | -| Encryption | **Recommended** | **"[scs:encrypted]"** | used to encrypt the volume | -| Replication | **Recommended** | **"[scs:replicated]"** | volume is replicated to aviod data loss in a case of hardware failure | +| Encryption | **Recommended** | **"[scs:encrypted]"** | volume is encrypted | +| Replication | **Recommended** | **"[scs:replicated]"** | volume is replicated to avoid data loss in a case of hardware failure | It is possible to use multiple of those aspects within one volume type. There don't have to be different volume types for each aspect. For instance, one volume type that uses LUKS-encryption with a ceph storage with inherent replication would fulfill all recommendations of this standard. @@ -71,13 +71,13 @@ This standard recommends to have one or more volume types, that feature encrypti ## OPTIONAL volume types -Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the CSPs, if they want to use some of these other aspects. +Any other aspects of volume types, that can be found in the decision record are OPTIONAL. They SHOULD NOT be referenced in the way this standard describes. Some of them already are natively discoverable by users, while others could be described in the name or description of a volume type. Users should look into the provided volume types of the Cloud Service Providers, if they want to use some of these other aspects. ## Implementation Details ### Encryption -Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. Additionally to be discoverable by users an admin has to edit the description and add `[scs:encrypted]` at the beginning or after another scs aspect. It should look like this example: +Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. Additionally to be discoverable by users, the description should start with an aspect list such as `[scs:encrypted]` (potentially with additional aspects). It should look like this example: ```text openstack volume type show LUKS @@ -95,12 +95,12 @@ openstack volume type show LUKS ### Replication -Replication states, whether or not there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved: +Replication states whether or not there are multiple replicas of a volume. Thus, it answers the question whether the data could survive a node outage. Unfortunately there are two ways replication can be achieved: 1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. -To fulfill this recommentation for now, the admin needs to add `[scs:replicated]` at the beginning or after any other scs aspect into the descriotion of the volume type. +To fulfill this recommendation, the description should start with an aspect list such as `[scs:replicated]` (potentially with additional aspects). ### Example diff --git a/Tests/iaas/volume-types/volume-types-check.py b/Tests/iaas/volume-types/volume-types-check.py index 21e5767c5..444755816 100644 --- a/Tests/iaas/volume-types/volume-types-check.py +++ b/Tests/iaas/volume-types/volume-types-check.py @@ -47,7 +47,7 @@ def extract_feature_list(description, pattern=re.compile(r"\[scs:([^\[\]]*)\]")) fs = match.group(1) if not fs: return [] - return fs.split(", ") + return [f.strip() for f in fs.split(",")] def test_feature_list(type_name: str, fl: list[str], recognized=RECOGNIZED_FEATURES): @@ -68,7 +68,7 @@ def test_feature_list(type_name: str, fl: list[str], recognized=RECOGNIZED_FEATU def print_usage(file=sys.stderr): """Help output""" - print("""Usage: entropy-check.py [options] + print("""Usage: volume-types-check.py [options] This tool checks volume types according to the SCS Standard 0112 "Volume Types". Options: [-c/--os-cloud OS_CLOUD] sets cloud environment (default from OS_CLOUD env) From 311eac3e7eb5e00bc247b4451576a0f02b4134f6 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Tue, 19 Mar 2024 12:51:48 +0100 Subject: [PATCH 33/35] Update scs-XXXX-v1-volume-type-standard.md Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index ebe61b489..34c2ed457 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -100,7 +100,7 @@ Replication states whether or not there are multiple replicas of a volume. Thus, 1. In the configuration of a volume type. It then is visible as extra_spec in the properties of a volume type. 2. Via the used backend. Ceph for example provides automatic replication, that does not need to be specified in the volume type. This is currently not visible for users. -To fulfill this recommendation, the description should start with an aspect list such as `[scs:replicated]` (potentially with additional aspects). +To fulfill this recommendation, the description should start with an aspect list such as `[scs:replicated]` (potentially with additional aspects). ### Example From 50cf7881f8af674a4c216e4523d6ce00abfa4c6f Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Thu, 21 Mar 2024 10:32:38 +0100 Subject: [PATCH 34/35] shortened paragraph of default volume type Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-type-standard.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-XXXX-v1-volume-type-standard.md index 34c2ed457..630ee63ec 100644 --- a/Standards/scs-XXXX-v1-volume-type-standard.md +++ b/Standards/scs-XXXX-v1-volume-type-standard.md @@ -59,8 +59,6 @@ For instance, one volume type that uses LUKS-encryption with a ceph storage with There is always a default volume type defined in an OpenStack deployment. This volume type is created in the setup of cinder and will always be present in any OpenStack deployments under the name `__default__`. This standard does not have any requirements about this volume type at this moment, instead deployers are free to choose what fits best in their environment. Conversely, a cloud user can not expect any specific behavior or properties from volume types named `__default__`. -The parameters of volume types described in this standard do not have to be applied to the chosen default volume type. And this standard will not make any assumptions about parameters being present in the default volume type. - ## REQUIRED volume types Currently, this standard will not require volume types with certain specification. From 3b3f3f01e4c3b5526fafc335a4631af12a03ae29 Mon Sep 17 00:00:00 2001 From: josephineSei <128813814+josephineSei@users.noreply.github.com> Date: Wed, 10 Apr 2024 13:45:36 +0200 Subject: [PATCH 35/35] add correct standard number before merge Signed-off-by: josephineSei <128813814+josephineSei@users.noreply.github.com> --- ...olume-type-standard.md => scs-0114-v1-volume-type-standard.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Standards/{scs-XXXX-v1-volume-type-standard.md => scs-0114-v1-volume-type-standard.md} (100%) diff --git a/Standards/scs-XXXX-v1-volume-type-standard.md b/Standards/scs-0114-v1-volume-type-standard.md similarity index 100% rename from Standards/scs-XXXX-v1-volume-type-standard.md rename to Standards/scs-0114-v1-volume-type-standard.md