Load metadata even when the image is locally available

[ttc0419]

Hi, even if images in the Dockerfile are available in local storage, buildit still load metadata from docker.io/library, which takes a lot of time:

[internal] load metadata for docker.io/library/xxx:latest

Is there a way to disable it?

[tonistiigi]

The only request that needs to happen if you have already accessed the image once is a HEAD request that checks if the digest where the latest tag points to has changed or not. It does not actually need to pull in image manifest/config (nor layers) if these have already been accessed.

If you are using buildkit in dockerd or with a containerd worker you can override it by pulling image with the same name to the image store (and not setting --pull on build), but note that your builds can become stale then and it is up to you to frequently pull newer versions of the image to the image store.

[thaJeztah]

I guess related;

• [proposal] implement pull-policies (docker build --pull=missing|never|always) docker/buildx#1889

Would implementing that be an option for this?

[tonistiigi]

@thaJeztah The version described there that is missing is --pull=never that does not seem to be related to this issue. BuildKit never performs a "docker pull", --pull defines the behavior of BuildKit resolving an image from a registry or some image store.

[thaJeztah]

Perhaps from a pure technical perspective, but I'd consider "contacting the registry" to be a "pull" from a user perspective; --pull=never to be a "don't contact registry; use offline" mode

[tonistiigi]

to be a "don't contact registry; use offline" mode

That's not what your definition is in the proposal. There is means "error if there is no local image". "don't contact registry; use offline" is the default (but you need to have offline version of the image).

[thaJeztah]

But BuildKit pulls the image and checks the registry if it's up to date??

I understand the build-cache with BuildKit may not store it in the list of images. But (really!) from the user's perspective it's pulling an image.

If we don't consider that an image, then we must update all our docs, because they're wrong; https://docs.docker.com/reference/dockerfile/#from

[tonistiigi]

But BuildKit pulls the image and checks the registry if it's up to date??

No, it does not if you have an image with that name already.

If we don't consider that an image, then we must update all our docs, because they're wrong;

🤷‍♂️ BuildKit 100% deals with images and takes image names as input. Images are in the registry or in image stores.

[ttc0419]

to be a "don't contact registry; use offline" mode

That's not what your definition is in the proposal. There is means "error if there is no local image". "don't contact registry; use offline" is the default (but you need to have offline version of the image).

I meant do not contact the registry in any way, because the reason I want to use the local image only is because the registry might not be reachable. Even only loading the metadata makes it pointless. Is it possible?

[tonistiigi]

@ttc0419 It is possible with docker or containerd that can keep local snapshots of the images for you. See #5340 (comment)

It is also possible if you use immutable image references with digest checksums.

[ttc0419]

@ttc0419 It is possible with docker or containerd that can keep local snapshots of the images for you. See #5340 (comment)

It is also possible if you use immutable image references with digest checksums.

I know it can store the image snaphosts and not to "pull" it, I meant is it possible not to send the HEAD request? Because some of my images are built by myself, checking them with the registry does not make any sense.

[tonistiigi]

Storing the image snapshot is how you avoid sending the HEAD request.

[ttc0419]

Storing the image snapshot is how you avoid sending the HEAD request.

But my problem was it still sends the HEAD request when the image is already stored in the local registry. What I want is disable version checking for local images that have latest tags.

[tonistiigi]

If you have a local registry that you are pointing to the HEAD goes against that registry.

What I was saying is that if you do docker pull alpine:latest you now have "alpine:latest" in the docker image store (docker image ls, not registry) and then FROM alpine:latest against the same instance will not do and HEAD request against any registry but just use that local image directly.

[ttc0419]

If you have a local registry that you are pointing to the HEAD goes against that registry.

What I was saying is that if you do docker pull alpine:latest you now have "alpine:latest" in the docker image store (docker image ls, not registry) and then FROM alpine:latest against the same instance will not do and HEAD request against any registry but just use that local image directly.

Wait, did you mean it only works with images that pulled from a registry? My base images were built from another Dockerfile, is that the reason why it's loading metadata every time because it cannot find any on docker hub?

[tonistiigi]

Wait, did you mean it only works with images that pulled from a registry? My base images were built from another Dockerfile, is that the reason why it's loading metadata every time because it cannot find any on docker hub?

No, the images from docker images are used if you are using builder in dockerd daemon. It doesn't matter how you get the image to docker images (pull, load, build).

[ttc0419]

Wait, did you mean it only works with images that pulled from a registry? My base images were built from another Dockerfile, is that the reason why it's loading metadata every time because it cannot find any on docker hub?

No, the images from docker images are used if you are using builder in dockerd daemon. It doesn't matter how you get the image to docker images (pull, load, build).

I'm using nerdctl with buildkit, will that be a problem?