[SURE-7399] Deploy new bundle versions when chart URL is unreachable on part of the managed clusters

[kkaempf]

SURE-7399

Issue description:

In a fleet.yaml file, the client adds their Helm charts via URLs to .tgz files for each of their cluster by using targetCustomizations. This is to allow them to use a specific registry per cluster.

They found out that when adding a URL to a registry that should only be used in their DR environment into the DR cluster, Fleet seems to try to access the URL on all controllers, which will not work since the DR chart URL will not be accessible in other environments (such as Dev and Prod).

This results in new bundle versions not being deployed when using one fleet.yaml for multiple Fleet controllers, even if the unreachable target URL is not in-scope of the targetCustomization.

Business impact:

Client cannot use a single fleet.yaml file to deploy charts to isolated clusters via targetCustomizations since this will stop the other charts (for other targetCustomizations) in the same fleet.yaml from deploying correctly.

Repro steps:

Create three downstream clusters in a Rancher 2.7.6 instance (with Fleet 0.7.1). Each cluster must have a label to identify it, which we'll use in the next steps.
Create a Fleet Git repo. It must have a fleet.yaml file with targetCustomizations and a chart URL for each downstream cluster (see attachments for an example, make sure to comment out the 'dr' block to remove the invalid hostname for now)
Once the initial deployment is done, uncomment the 'dr' block and replace the URL to a value that is only reachable by that downstream cluster

Workaround:

Is workaround available and implemented? yes (assumed)

What is the workaround: The client did not specify what workaround they used, but I think this can be worked around by deploying a different dedicated fleet.yaml per isolated registry.

Actual behavior:

Fleet tests all URLs in the fleet.yaml file and fails to deploy new versions of a bundle if one of the chart URLs is unreachable in any downstream cluster.

Expected behavior:

Fleet should only try to reach out to the registry of each targetCustomization that is in-scope for each cluster.

[manno]

The title is misleading, the JIRA issue is now about continuing to create bundles when a download fails.

Keeping this around till the fix in #3141 is done.

[manno]

See also #3114 for template errors, which do stop bundles from deploying.

[manno]

Closing in favor of #3141