Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List bundle dependencies #204

Open
ashvinmoro opened this issue Jul 28, 2021 · 15 comments
Open

List bundle dependencies #204

ashvinmoro opened this issue Jul 28, 2021 · 15 comments
Labels
enhancement This issue is a feature request priority/important-soon Must be staffed and worked on currently or soon

Comments

@ashvinmoro
Copy link

ashvinmoro commented Jul 28, 2021

Describe the problem/challenge you have
List dependencies:
An imgpkg bundle can either be a standalone bundle or a package repository, with them referencing dependent artifacts which can either be OCI artifacts or other imgpkg bundles. The imgpkg CLI today does not provide an option to query a bundle to list all dependent artifacts (regardless of the level of nesting) without doing a pull and parsing through the contents of the bundle.

This requires knowing the internals of the imgpkg bundle layout which is an implementation detail and causes too tight a coupling for clients consuming the CLI.

This feature is very helpful to inspect contents of the bundle metadata without requiring to download it, similar to how artifact manifest can be inspected using the crane command.

Describe the solution you'd like
Provide a CLI option and an API to allow clients consuming the bundles to query for all dependent artifacts and their types (bundle, repo, image) with their references locked to the relocated repository, similar to how “imgpkg pull” would fix the images lock.

Example:
imgpkg -b <bundle-ref> --list-deps
returns images referenced after the images are locked
Images referenced in the ImagesLock of the bundle:
index.docker.io/test/package-repo@sha256:b5e1cfb8731bf5ba6a0fb7060c37bbca03e9fb12be56dbec4a1fe2bbdfff2bd3
index.docker.io/test/package-repo@sha256:42030f63b66e25a03c7d9ef28fa5369d230df5697b4ab2d94860e3f69026d18a

Anything else you would like to add:
None


Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

@ashvinmoro ashvinmoro added carvel triage This issue has not yet been reviewed for validity enhancement This issue is a feature request labels Jul 28, 2021
@ashvinmoro
Copy link
Author

cc: @ncdc

@DennisDenuto
Copy link
Contributor

DennisDenuto commented Aug 16, 2021

Thanks for creating this issue.

Provide a CLI option and an API... to query for... types

The CLI option can be added to the pull command. If this flag is present, the last section of output can be a flat list of all the dependent images that were pulled (also whether the dependent image is a bundle or an OCI image) Assuming we display whether the dependent image is a bundle or OCI image, would you still expect an API to 'query' the type?

Are you able to expand on the use case driving this request? or, why do you want a full list of dependent images per bundle?

I am trying to understand whether this is purely for information (which in that case the pull command outputting additional information seems like a fine idea), or whether this is being used to maybe verify / validate something?

In which case, I think discussing a new command to 'inspect' a bundle, and output pertinent information such as dependencies, ImagesLocation, signatures and tags might be useful. As well as any additional options such as filtering too.

@DennisDenuto DennisDenuto added triage/needs-more-information priority/unprioritized-backlog Higher priority than priority/awaiting-more-evidence but not planned. Contributions are welcome. and removed carvel triage This issue has not yet been reviewed for validity labels Aug 16, 2021
@ashvinmoro
Copy link
Author

ashvinmoro commented Aug 26, 2021

The CLI option can be added to the pull command. If this flag is present, the last section of output can be a flat list of all the dependent images that were pulled (also whether the dependent image is a bundle or an OCI image) Assuming we display whether the dependent image is a bundle or OCI image, would you still expect an API to 'query' the type?

To keep the semantic separation a separate query command is better. Under the hood, this command may likely do a pull command but that will then be an implementation detail.

Are you able to expand on the use case driving this request? or, why do you want a full list of dependent images per bundle?

I am trying to understand whether this is purely for information (which in that case the pull command outputting additional information seems like a fine idea), or whether this is being used to maybe verify / validate something?

The use case is to track the dependent images and establish policy based access controls for users downloading the bundles and its images. Right now we are having to unpack the bundle and identify the dependents manually which is too dependent on the bundle implementation.

In which case, I think discussing a new command to 'inspect' a bundle, and output pertinent information such as dependencies, ImagesLocation, signatures and tags might be useful. As well as any additional options such as filtering too.

+1

@DennisDenuto
Copy link
Contributor

Thanks @ashvinmoro

I think given the specific use case for this feature, a new command to 'inspect' the bundle should be done.

@pivotaljohn
Copy link
Contributor

We've gotten information that this is a higher priority:

  • there was an assumption that this could achieved through inspecting images.yml by pull recursive. However, we get an incomplete set of images; that approach is not viable.

@pivotaljohn pivotaljohn added priority/important-soon Must be staffed and worked on currently or soon and removed priority/unprioritized-backlog Higher priority than priority/awaiting-more-evidence but not planned. Contributions are welcome. labels Sep 2, 2021
@DennisDenuto
Copy link
Contributor

DennisDenuto commented Sep 2, 2021

TODO:
Prior to picking this up / working on this

  • detail what the command options/flags will be, and the format of the output and get 'early' feedback from @ashvinmoro

@pivotaljohn
Copy link
Contributor

Review #124 — which may be requesting the same functionality.

  • note the sample output: Create a describe command for a bundle #124 (comment)
  • that other story was describing this feature, in general. What context from this story can help us prioritize what aspects of this feature to implement now (and what we might want to defer).

@joaopapereira
Copy link
Member

As @pivotaljohn mentioned we have discussed the creation of a describe command in issue #124, @ashvinmoro do you think that the proposed output can solve your needs?

@joaopapereira
Copy link
Member

@ashvinmoro any comments about the output of the describe command proposed in #124 ?

@ashvinmoro
Copy link
Author

@pivotaljohn @joaopapereira I went through the issue #124. There are a couple of aspects I had suggestions about

  • List all of the image references that are part of the bundle. This appears to be covered in issue Create a describe command for a bundle #124.
  • If the bundle is a nested bundle, list the image references of nested bundles as well as the image references of each nested bundle. This could either be provided as a recursive listing or as a separate option by passing in the reference of the nested bundle.
$ imgpkg describe -b new.registry.io/simple-app-install-package

Bundle SHA: aaaaad700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
Tags: latest, v1.0.0

Authors:
  Carvel Team <[email protected]>
Websites:
  carvel.dev/imgpkg
Metadata:
  - Some Version: 1.0.0
  - Other Information: Some text here

Images:
  - new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
     **Type: Bundle**
    Origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    Images:
      - new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        **Type: Image**
        Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        Annotations:
          kbld.carvel.dev/id: my.registry.io/simple-application

      - new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
        **Type: Image**
        Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
  - new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

@joaopapereira
Copy link
Member

@ashvinmoro Take a look at the following example to see if they look good for your use case:

Examples

Example 1: Describe command without any argument

$ imgpkg describe -b new.registry.io/simple-app-install-package 

Bundle SHA: aaaaad700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
Tags: latest, v1.0.0

Authors:
  Carvel Team <[email protected]>
Websites:
  carvel.dev/imgpkg
Metadata:
  - Some Version: 1.0.0
  - Other Information: Some text here

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    Type: Bundle
    Origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    Images:
      - Image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        Type: Image
        Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        Annotations:
          kbld.carvel.dev/id: my.registry.io/simple-application

      - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
        Type: Image
        Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Type: Image
    Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 2: Describe command listing all images

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location

Bundles:
  - Image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 3: Describe command listing all images in the current bundle without searching inner bundles

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location --no-recursion

Bundles:
  - Image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 4: Describe command listing all images displaying the origin

$ imgpkg describe -b new.registry.io/simple-app-install-package --images

Bundles:
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 5: Describe command listing all images when Bundle was not copied

$ imgpkg describe -b new.registry.io/simple-app-install-package --images

Bundles:
  - Origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4

Images:
  - Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Previous examples but using a yaml output

Example 1: Describe command with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --output-type yaml

sha: aaaaad700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
tags: 
 - latest
 - v1.0.0
authors:
  - name: Carvel Team
    email: [email protected]
websites:
  - carvel.dev/imgpkg
metadata:
  - Some Version: 1.0.0
  - Other Information: Some text here

images:
  - image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    type: Bundle
    origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    images:
      - image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        type: Image
        origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        annotations:
          kbld.carvel.dev/id: my.registry.io/simple-application
      - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
        type: Image
        origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    type: Image
    origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 2: Describe command listing all images with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location --output-type yaml

bundles:
  - image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
images:
  - image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 3: Describe command listing all images in the current bundle without searching inner bundles with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location --no-recursion --output-type yaml

bundles:
  - image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
images:
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 4: Describe command listing all images displaying the origin with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --output-type yaml

bundles:
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0

images:
  - image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

@jorgemoralespou
Copy link

Nice, but as mentioned before in other discussions, it'll be nice to somehow have the original image tags annotated so that describe can provide some information that a human can understand :-D

@joaopapereira
Copy link
Member

In some sense, the original tag will be present if you use kbld to generate your ImagesLock and that tag is used in the files read by kbld. Look at this example:

$  kbld -f basic-step-2/config.yml --imgpkg-lock-output basic-step-2/.imgpkg/images.yml
resolve | final: index.docker.io/dkalinin/k8s-simple-app:latest -> index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
---
apiVersion: v1
kind: Service
metadata:
  name: simple-app
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    simple-app: ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kbld.k14s.io/images: |
      - origins:
        - resolved:
            tag: latest
            url: index.docker.io/dkalinin/k8s-simple-app:latest
        url: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  name: simple-app
  namespace: default
spec:
  selector:
    matchLabels:
      simple-app: ""
  template:
    metadata:
      labels:
        simple-app: ""
    spec:
      containers:
      - env:
        - name: HELLO_MSG
          value: stranger
        image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        name: simple-app

Succeeded

$ imgpkg push -b localhost:5000/test -f basic-step-2
dir: .
dir: .imgpkg
file: .imgpkg/bundle.yml
file: .imgpkg/images.yml
file: config.yml
Pushed 'localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291'
Succeeded

$ imgpkg describe -b localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291 -o yaml
sha: sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291
content:
  images:
    sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0:
      annotations:
        kbld.carvel.dev/id: index.docker.io/dkalinin/k8s-simple-app:latest
        kbld.carvel.dev/origins: |
          - resolved:
              tag: latest
              url: index.docker.io/dkalinin/k8s-simple-app:latest
      image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
      imageType: Image
      origin: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
image: localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291
metadata: {}
origin: localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291

Succeeded

Another thing we need to keep in mind is that imgpkg only deal with Full Described Images(uses digest only), so unless the tool that creates the ImagesLock provides this information, imgpkg as no way to know tag information

@jorgemoralespou
Copy link

Nice thanks for clarification. Is this also true for imgpkg copy, which is how most of the times images get to local clusters?

@joaopapereira
Copy link
Member

Not following 100% what you mean, but the information we are displaying is the information that is part of the ImagesLock, so if the information is in the ImagesLock it will be displayed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This issue is a feature request priority/important-soon Must be staffed and worked on currently or soon
Projects
Status: To Triage
Development

No branches or pull requests

5 participants