Create Gaia-X compliant credential for CSP's as LegalPerson and OpenStack Cloud as "ServiceOffering"

[anjastrunk]

closes #70

[markus-hentsch]

I started reviewing and wanted to some practical tests for the OpenStack part using a DevStack of mine.

The instructions in the README are unlcear. In short, it says to do the following:

cd gx-credential-generator

python3 cli.py openstack <os-cloud>

However, cli.py is located in the "generator" subdirectory but even when navigating to it, it fails:

cd generator/
python3 cli.py openstack devstack

Traceback (most recent call last):
  File "<redacted>/gx-credential-generator/generator/cli.py", line 23, in <module>
    import generator.common.const as const
ModuleNotFoundError: No module named 'generator'

@anjastrunk what is the correct way of invoking the generator?

[mbuechse]

@markus-hentsch If I may... I think you need to run

python3 -m generator.cli openstack devstack

from the root of the repo.

[anjastrunk]

@markus-hentsch If I may... I think you need to run

python3 -m generator.cli openstack devstack

from the root of the repo.

@mbuechse is right. You have to run and should add you own configuration file.

python3 -m generator.cli openstack devstack --config <path-to-your-config>

If you omit --config parameter, configuration file is taken from config/config.yaml sub-directory. I will send you a sample configuration file via PM.

@markus-hentsch You are right. Documentation has to be improved. I will do this within a separate PR.

[mbuechse]

@anjastrunk Will you send me the file as well? Do you plan to get code coverage for discovery back to at least 90? (Then I might wait with the review...)

[markus-hentsch]

@mbuechse is right. You have to run and should add you own configuration file.

python3 -m generator.cli openstack devstack --config <path-to-your-config>

If you omit --config parameter, configuration file is taken from config/config.yaml sub-directory. I will send you a sample configuration file via PM.

@markus-hentsch You are right. Documentation has to be improved. I will do this within a separate PR.

Thanks. I used the new command now and it starts execution.

I have encountered two issues while testing:

Incompatibility with non-compliant images

In my DevStack there are images that are not SCS-compliant and lack some of the expected metadata, leading to an error:

  File ".../gx-credential-generator/generator/discovery/openstack/vm_images_discovery.py", line 290, in _get_operation_system
    if os_image.os_distro.lower() == "arch":
       ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'

I don't think we can expect every image visible to the user to have all the metadata.

Missing output processing and VP

When executing the generator, it seems to print a dump of all generated VCs simply concatenated together.
Piping the output to a file with python3 ... > output.json is not really helping as the file also includes status output of the tool at the top:

"tandc"
"lrn"
"lp"
"cs"
{
  "@context": [
    "https://www.w3.org/2018/credentials/v1",
    "https://w3id.org/security/suites/jws-2020/v1",
    "https://registry.lab.gaia-x.eu/development/api/trusted-shape-registry/v1/shapes/jsonld/trustframework#"
  ],
  "type": "VerifiableCredential",
...

... and the individual VCs are not really separated. My suggestion would be:

1. only print status messages to stdout, not the JSON (the VCs are far too long, depending on the terminal used, you might not be able to scroll back far enough)
2. write each VC into an individual .json file in an output folder
3. additionally write a Verifiable Presentation (VP) .json that includes all VCs like in this template

I think this would improve user experience greatly as the VP could be directly used at the GXDCH Compliance API.

[markus-hentsch]

See #96 (comment)

[mbuechse]

In my DevStack there are images that are not SCS-compliant

I'm not even sure if this is a matter of compliance. If your image is very specific (not a general purpose OS image), it could be reasonable to omit the os_distro property.

edit you are right, the standard seems to require os_distro -- strange, for I have recently encountered an image without that property, and the environment was not considered in violation? I will investigate...

edit 2: okay, the test script only checks public images by default, and the offending image in my case was not public.

[anjastrunk]

Missing output processing and VP

When executing the generator, it seems to print a dump of all generated VCs simply concatenated together. Piping the output to a file with python3 ... > output.json is not really helping as the file also includes status output of the tool at the top ...

You are right and I agree with our approach. I will change this.
update from 20.08.24: done, see cdc80f3

[anjastrunk]

Incompatibility with non-compliant images

In my DevStack there are images that are not SCS-compliant and lack some of the expected metadata, leading to an error:

  File ".../gx-credential-generator/generator/discovery/openstack/vm_images_discovery.py", line 290, in _get_operation_system
    if os_image.os_distro.lower() == "arch":
       ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'

I don't think we can expect every image visible to the user to have all the metadata.

Thanks for bringing this up. You are completely right, we can not assume presence of any optional property. I will change this and look for other occurrence of such type of properties,

[anjastrunk]

@markus-hentsch @mbuechse I increased code coverage and fixed the issues in #96 (comment). Please do a re-review. Thanks

[mbuechse]

First round of review done. So far, mostly very minor things.

[anjastrunk]

@markus-hentsch @mbuechse All good things come in threes... Third round of review ;-)

[markus-hentsch]

I noticed that when generating VCs you are referencing different JSON files for credential id and subject id, for example:

• credential id: <url>/tandc.json
• credential subject id: <url>/tandc_subject.json

I don't think that hosting separate file copies makes sense here. I assume you did that to represent the same asset while keeping identifiers unique (as required by the RFC).

In Gaia-X examples¹, they often use an anchor (e.g. #subject) within the same JSON in order to keep the identifiers unique while still referencing the same file. The anchors do not need to be resolvable. See the corresponding section of my blog post paragraph² for some more context.

I think we should use anchors here as well and reference a single file for each asset only to keep things simple and prevent file copies getting out of sync.
I added suggestions to adjust the affected code parts.

Footnotes

1. https://gitlab.com/gaia-x/lab/workshops/gaia-x-101/-/blob/e0b01980eead64c0a20fec4643659b4c9d9f3331/gaia-x-101.ipynb#L116 ↩

2. https://github.com/SovereignCloudStack/website/blob/381d6ff38d33e4b8697cde010c6f74108aae0f93/_i18n/en/_posts/blog/2024-06-05-demystifying-gaia-x-credentials.md?plain=1#L222-L248 ↩

[markus-hentsch]

Last round of suggestions: some minor remaining CPS typo fixes.

[markus-hentsch]

With the latest commits I can attest that the produced VCs and VPs for the OpenStack side of things are valid as I was able to successfully:

• verify the proof on all generated VCs
• submit each generated VP to the Gaia-X Compliance API and receive a valid compliance VC in turn

Note that I have only been able to verify the OpenStack ServiceOffering generation due to the lack of a suitable Kubernetes cluster.

---

For the record, these are the scripts I used for the VC and VP validations mentioned above:

VC verification script (click to expand ...)

"""Verify proof signature of local credential

Provide full path to a Verifiable Credential (VC) as the only argument.
The VC must contain a "proof" property.
"""

import sys
import json
from utils import (
    verify_credential, get_verification_cert_url_from_did
)

local_cred_path = sys.argv[1]

print(f"Verifying '{local_cred_path}' ...")
with open(local_cred_path, 'r') as f:
    vc_as_json = json.loads(f.read())

verification_cert_url = get_verification_cert_url_from_did(
    vc_as_json.get("proof").get("verificationMethod")
)
verify_credential(json.dumps(vc_as_json), verification_cert_url)
print("Verified.")

VP compliance script (click to expand ...)

"""Issue a local Verifiable Presentation for compliance

Provide full path to a Verifiable Presentation (VP) as the only argument.
Will send the VP to the Gaia-X Compliance API and receive a Verifiable
Credential (VC) attesting the VP's compliance. The received VC's proof
is then additionally verified at the end of the script.
"""

import sys
import json
import requests
from utils import (
    verify_credential, get_verification_cert_url_from_did
)

GXDCH_COMPLIANCE_API_URL = \
    "https://compliance.lab.gaia-x.eu/v1-staging/api/credential-offers"

local_pres_path = sys.argv[1]

print(f"Submitting '{local_pres_path}' ...")
with open(local_pres_path, 'r') as f:
    vp_as_json = json.loads(f.read())

compliance_response = requests.post(
    GXDCH_COMPLIANCE_API_URL,
    json.dumps(vp_as_json)
)

if compliance_response.status_code == 201:
    compliance = compliance_response.json()
else:
    raise Exception(
        "Unable to submit to compliance: " +
        compliance_response.text
    )

print("Verifying received VC proof ...")

# verify the proof received by the Compliance API
verification_cert_url = get_verification_cert_url_from_did(
    compliance.get("proof").get("verificationMethod")
)
verify_credential(json.dumps(compliance), verification_cert_url)
print("Verified.")
print()
print("Received valid compliance VC:")
print(json.dumps(compliance, indent=2, sort_keys=True))

utils.py library with utility functions (click to expand ...)

import os
import requests
import json
from jwcrypto import jwk, jws
from utils import sha256_normalized_vc, normalize


def resolve_did(did_ref):
    # did_ref example:
    #   did:web:compliance.lab.gaia-x.eu:v1-staging#X509-JWK2020

    # DID Resolver
    if did_ref.count(':') == 2:
        _, _, fqdn = did_ref.split(':', 2)
        if '#' in fqdn:
            fqdn, _ = fqdn.split('#')
        did_url = f"https://{fqdn}/.well-known/did.json"
    else:
        _, _, fqdn, version = did_ref.split(':', 3)
        if '#' in version:
            version, _ = version.split('#')
        did_url = f"https://{fqdn}/{version}/.well-known/did.json"
    # ---

    did_response = requests.get(did_url)
    did_json = did_response.json()

    return did_json


def get_verification_cert_url_from_did(did_ref):
    if not did_ref.startswith("did:web:"):
        raise Exception(
            f"Not a supported DID to retrieve cert: '{did_ref}'"
        )
    did_json = resolve_did(did_ref)
    x5u_url = did_json.get("verificationMethod")[0]\
                      .get("publicKeyJwk")\
                      .get("x5u")
    return x5u_url


def verify_credential(credential_json_str, cert_url):

    verifiable_credential = json.loads(credential_json_str)

    # Retrieve the registry certificate which serves as the verification
    # public key (JWK) for the JWS later
    reg_cert_response = requests.get(cert_url)
    if reg_cert_response.status_code != 200:
        raise Exception(
            f"Unable to retrieve verification certificate "
            f"from: {cert_url}"
        )
    verification_cert_pem = reg_cert_response.text.encode('UTF-8')
    verification_key = jwk.JWK.from_pem(verification_cert_pem)

    # The proof object is part of the credential response, however
    # it resembles JWS data applicable to the response without the
    # proof object. Hence, we need to strip the proof object from
    # the response.
    proof = verifiable_credential.pop("proof")

    # The remaining structure is the actual credential data that
    # JWS was created for. The signature was applied to its
    # normalized and hashed form, which we need to recreate here
    # in order to verify the signature.
    normalized_credential = normalize(verifiable_credential)
    hashed_credential = sha256_normalized_vc(normalized_credential)

    # Instantiate a JWS object based on the jws attribute of the
    # proof object, which contains a base64 representation of the JWS.
    received_jws_token = jws.JWS()
    received_jws_token.deserialize(proof["jws"])

    # Finally, use the verification key (Gaia X registry public cert)
    # and the hashed credential (which is the JWS' detached payload)
    # in conjunction with the JWS token to verify the credential.
    # This method will throw an exception if verification fails.
    received_jws_token.verify(
        verification_key,
        detached_payload=hashed_credential.hexdigest()
    )

[mbuechse]

LGTM. Markus has a better grasp of the functionality, so he should approve as well.

[github-actions]

Package	Line Rate	Health
.	93%	➖
common	99%	✔
discovery	99%	✔
discovery.openstack	99%	✔
Summary	98% (783 / 800)	✔

Minimum allowed line rate is 90%

[markus-hentsch]

Thanks for the adjustments! I tested the OpenStack part again. LGTM now.