Upload pipeline from URL - PermissionDenied

[Barteus]

When creating the pipeline using the UI - import by URL, there is an error message:

{"error":"Failed to authorize with API resource references: PermissionDenied: User 'admin' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:,Verb:create,Group:,Version:,Resource:pipelines,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to authorize with API resource references: PermissionDenied: User 'admin' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:,Verb:create,Group:,Version:,Resource:pipelines,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/api.Error","error_message":"User 'admin' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:,Verb:create,Group:,Version:,Resource:pipelines,Subresource:,Name:,})","error_details":"Failed to authorize with API resource references: PermissionDenied: User 'admin' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:,Verb:create,Group:,Version:,Resource:pipelines,Subresource:,Name:,}): Unauthorized access"}]}

Expected
The pipeline was successfully created.

Reproduce
Go to Pipelines in Kubeflow UI, fill in the name and description.
Import from URL - https://raw.githubusercontent.com/canonical/bundle-kubeflow/master/docs/examples/mnist.yaml
Press Create

[Barteus]

Creating a pipeline using the file (downloaded from the same link) works.

[VariableDeclared]

I was unable to replicate this with the https://pastebin.canonical.com/p/wY7WwSBbGw/, maybe im missing something?

I had to remove description elements from the YAML, I was then able to upload the YAML, I was unable to use the YAML URL method due to issues with parsing YAML

[Barteus]

Maybe the problem is with the link - I will try with different one

[VariableDeclared]

thanks @Barteus I forgot to caveat that I am running this on microk8s. I will try again with AKS

[DomFleischmann]

@Barteus did a different work? If not, is it possible this is related to AKS?

[Barteus]

@VariableDeclared did you have RBAC enabled in microk8s? I have it enabled in the AKS cluster.

[VariableDeclared]

indeed, Microk8s has it disabled by default. Enabled it now, restarting

[VariableDeclared]

okay, I am now able to replicate the issue after ensuring RBAC is enabled on Microk8s

[DnPlas]

I was able to reproduce this issue with our latest/edge versions. FYI, there is an upstream bug (kubeflow/pipelines#7196), and we'll follow up with upstream to see if this is fixed in Kubeflow 1.6.

[i-chvets]

Jira

[NohaIhab]

A similar error message came up in 1.7, when creating a pipeline from file

Experiment creation failed
{"error":"Failed to authorize the request: Failed to authorize with API resource references: PermissionDenied: User 'test' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:test,Verb:create,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:testexp,}): Unauthorized access","code":7,"message":"Failed to authorize the request: Failed to authorize with API resource references: PermissionDenied: User 'test' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:test,Verb:create,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:testexp,}): Unauthorized access","details":[{"@type":"type.googleapis.com/api.Error","error_message":"User 'test' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:test,Verb:create,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:testexp,})","error_details":"Failed to authorize the request: Failed to authorize with API resource references: PermissionDenied: User 'test' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:test,Verb:create,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:testexp,}): Unauthorized access"}]}

[ca-scribner]

to clarify @NohaIhab's message, we see this in a pre-release version of 1.7 that we're currently testing.

@NohaIhab your issue may be related to this issue, but it might also be that we're failed to deploy some RBAC. Can you show what permissions test user has? I wonder if we've somehow lost the kfp roles that are bound to each user

[NohaIhab]

@ca-scribner running kubectl get ClusterRole -n test | grep pipeline I got:

aggregate-to-kubeflow-pipelines-edit                                   2023-02-27T11:35:27Z
aggregate-to-kubeflow-pipelines-view                                   2023-02-27T11:35:27Z

when describing aggregate-to-kubeflow-pipelines-edit:

kubectl describe ClusterRole aggregate-to-kubeflow-pipelines-edit
Name:         aggregate-to-kubeflow-pipelines-edit
Labels:       app.juju.is/created-by=kubeflow-roles
              app.kubernetes.io/component=ml-pipeline
              app.kubernetes.io/name=kubeflow-pipelines
              application-crd-id=kubeflow-pipelines
              rbac.authorization.kubeflow.org/aggregate-to-kubeflow-pipelines-edit=true
...

I think we've missed the kubeflow-pipelines-edit ClusterRole during upgrades of kubeflow-roles, same goes for kubeflow-pipelines-view

[NohaIhab]

Concluding that the issue discussed above is not the same as this issue. Filed #156 and moving conversation there.

[DnPlas]

Hi @Barteus, I checked the status of this issue in our latest version (1.7, soon to be released), and I could not reproduce. Please note I had to use a different Workflow, as the one you are linking in the description cannot be applied anymore due to pipelines being unable to recognize certain fields (maybe too old?).

I will update the issue once 1.7 is released, in the meantime, would you be able to provide another URL that we can use to ensure this error is not present anymore?

[Barteus]

I have used this URL: https://raw.githubusercontent.com/canonical/bundle-kubeflow/master/docs/examples/mnist.yaml

[NohaIhab]

Findings:

• Namespace attribute is empty in the request: Namespace:,Verb:create,Group:,Version:,Resource:pipelines,Subresource:,Name:,, meanwhile it's correctly filled when creating pipeline from file.

• upload from url works when adding to an existing pipeline i.e. Create a new pipeline version under an existing pipeline

[NohaIhab]

I just testing with Charmed Kubeflow 1.8/stable and I no longer see this issue.
Using the mnist-workflow that @Barteus was using, the pipeline was created successfully from URL, but the run failed at the Load task step with:

URL fetch failure on https://github.com/canonical/bundle-kubeflow/raw/test-artifacts/tests/pipelines/artifacts/train-images-idx3-ubyte.gz: 404 -- Not Found

this is just due to changes done in the bundle-kubeflow repository so the link to the data is now broken. I'll file an issue in bundle-kubeflow repo to handle the mnist.yaml example.
Using the map-reduce workflow created the pipeline and I was able to create a successful run.

[NohaIhab]

closing this as it was resolved in ckf 1.8 and we cannot fix it in 1.7 because it's due to an upstream bug