Fix Clean Up Prebuilt SDK Images job

[Amar3tto]

Fixes #30514

Successful run: https://github.com/Amar3tto/beam/actions/runs/13266503777/job/37035192931

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[Amar3tto]

R: @liferoad

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[Abacn]

It's taking long because some images are failed to delete:

Command: gcloud container images delete gcr.io/apache-beam-testing/beam-sdk/beam_java21_sdk_distroless@sha256:b9dc6a3b2ca3cb7453267d658ef28b682d6b0ca9a7c7e947ed9d76bcf998446f --force-delete-tags -q
Digests:
- gcr.io/apache-beam-testing/beam-sdk/beam_java21_sdk_distroless@sha256:b9dc6a3b2ca3cb7453267d658ef28b682d6b0ca9a7c7e947ed9d76bcf998446f
ERROR: gcloud crashed (V2DiagnosticException): response: {'content-type': 'application/json; charset=utf-8', 'docker-distribution-api-version': 'registry/2.0', 'x-content-type-options': 'nosniff', 'x-frame-options': 'SAMEORIGIN', 'x-xss-protection': '0', 'date': 'Tue, 11 Feb 2025 01:29:53 GMT', 'x-gcr-using-artifact-registry': 'true', 'content-encoding': 'gzip', 'server': 'Docker Registry', 'transfer-encoding': 'chunked', 'status': 400}
Manifest is still referenced by one or more parent images: failed precondition manifest has referenced parents: beam-sdk/beam_java21_sdk_distroless/manifests/sha256:c4a4425e5fa26c2d269785474917de0c04a83be1962741c7b6c17049f9331308: None

checked that actually it shouldn't be deleted because it is created less than 60 days.

Likely the logic in

beam/.test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

Line 95 in 6fcf161

# Check to see if this image is built on top of earlier images. This is the case for multiarch images,

has issues after migrated from GCR -> AR

[Abacn]

I think what currently happens is this check is no longer working:

beam/.test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

Lines 100 to 101 in 6fcf161

	if [ -z "$MANIFEST" ]; then
	# Sometimes "no such manifest" seen. Skip current if command hit error

and always added to FAILED_IMAGES

[Abacn]

changing docker manifest inspect ${image_name}@"${current}" -> docker buildx imagetools inspect ${image_name}@"${current}" --raw should help.

[Abacn]

ah sorry, it's because some old java distroless (before Jan 13) image was built on different manifest.

However switching docker manifest inspect to docker buildx imagetools inspect indeed accelerates the query, and the job would no longer timeout.

[Amar3tto]

docker buildx imagetools inspect ${image_name}@"${current}" --raw

Changed. Looks like it helped. (21 minutes) https://github.com/Amar3tto/beam/actions/runs/13280158151/job/37076735510

[Abacn]

sorry forgot to mention already integrated in #33957 , we can close this for now. Thanks!