[Bug]: ProviderConfig not taking into account the spec.endpoint.url.dynamic.host for bucket creation in OVHcloud s3 compatible service

[zwindler]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

• aws.upbound.io/v1beta1 - ProviderConfig
• s3.aws.upbound.io/v1beta1 - Bucket

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

OVHcloud is a cloud provider that has a s3 compatible service. They support the S3 bucket creation using the terraform hashicorp/aws default provider, as can be seen in this example

https://github.com/ovh/public-cloud-examples/blob/main/storage/s3-with-tf/provider.tf

I'm trying to reproduce this with crossplane by I have issues with the ProviderConfig object, which doesn't seem to respect the spec.endpoint.url.dynamic.host value I give it

Here is the ProviderConfig

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: s3-for-ovhcloud
spec:
  credentials:
    secretRef:
      key: creds
      name: aws-secret
      namespace: crossplane-system
    source: Secret
  endpoint:
    source: Custom
    url:
      dynamic:
        host: io.cloud.ovh.net
        protocol: https
      type: Dynamic
  # OVHcloud regions aren't known to AWS hence skipping is needed.
  skip_region_validation: true
  #OVH implementation has no STS service
  skip_requesting_account_id: true
  skip_credentials_validation: true

Here is the bucket

apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  generateName: crossplane-bucket-
spec:
  forProvider:
    region: eu-west-par
  providerConfigRef:
    name: s3-for-ovhcloud

What happened?

When I create the bucket, I get an error showing that the host is not correctly set to io.cloud.ovh.net

│   Warning  CannotObserveExternalResource  94s (x4 over 4m34s)   managed/s3.aws.upbound.io/v1beta1, kind=bucket  failed to observe the resource: [{0 reading S3 Bucket (crossplane-bucket-c45vw): operation error S3: HeadBucket, https respo │
│ nse error StatusCode: 0, RequestID: , HostID: , request send failed, Head "https://crossplane-bucket-c45vw.s3.eu-west-par.amazonaws.com/": dial tcp: lookup crossplane-bucket-c45vw.s3.eu-west-par.amazonaws.com on 10.255.40.10:53: no such │
│  host  []}]    

Url should be https://crossplane-bucket-c45vw.s3.eu-west-par.io.cloud.ovh.net, not https://crossplane-bucket-c45vw.s3.eu-west-par.amazonaws.com

If I use the localstack trick (https://docs.localstack.cloud/user-guide/integrations/crossplane/), to use static url instead of dynamic one, the Bucket creation fails as well because the ".s3." prefix before the region canno't be appended (this is documented and normal, but I just wanted to point out it's not a solution)

  endpoint:
    hostnameImmutable: true
    services: [s3]
    url:
      type: Static
      static: https://io.cloud.ovh.net

Relevant Error Output Snippet

│   Warning  CannotObserveExternalResource  94s (x4 over 4m34s)   managed/s3.aws.upbound.io/v1beta1, kind=bucket  failed to observe the resource: [{0 reading S3 Bucket (crossplane-bucket-c45vw): operation error S3: HeadBucket, https respo │
│ nse error StatusCode: 0, RequestID: , HostID: , request send failed, Head "https://crossplane-bucket-c45vw.s3.eu-west-par.amazonaws.com/": dial tcp: lookup crossplane-bucket-c45vw.s3.eu-west-par.amazonaws.com on 10.255.40.10:53: no such │
│  host  []}]

Crossplane Version

1.18.2

Provider Version

1.19.0

Kubernetes Version

1.29.4

Kubernetes Distribution

Talos Linux

Additional Info

No response

[zwindler]

The code to change the URL to a static one with the given host is here

provider-upjet-aws/internal/clients/aws.go

Line 205 in 6e0f4f2

if pc.Spec.Endpoint != nil {

But I don't see anything for the dynamic part (it may be in a previous call)