Sync-Catalog across kubernetes clusters to same external Consul server with upsert strategy

[webmutation]

Question

Two Kubernetes clusters that represent a hybrid environment, writing concurrently to the same consul external server. What happens is that each kubernetes cluster then invalidates the existing services from the other cluster, concurrently updatign the catalog with only the services they see on their own cluster.

What we need is a way to upsert services from the two clusters and not have the two sync catalog agents rewrite each other. Similar to how externalDNS works for DNS records.

Is there a way to define the merge or update strategy of the consul -sync catalog agent?

Or do we always have to run multiple consul servers and federate them to have a proper catalog for the same environment (hybrid)?

Helm Configuration

# Contains values that affect multiple components of the chart.
global:
  logLevel: debug
  # The main enabled/disabled setting.
  # If true, servers, clients, Consul DNS and the Consul UI will be enabled.
  enabled: true
  # The prefix used for all resources created in the Helm chart.
  name: consul
  # The consul image version.
  image: hashicorp/consul:1.15.2
  # The name of the datacenter that the agents should register as.
  datacenter: dc1
  # Enables TLS across the cluster to verify authenticity of the Consul servers and clients.
  tls:
    enabled: false
  # Enables ACLs across the cluster to secure access to data and APIs.
  acls:
    # If true, automatically manage ACL tokens and policies for all Consul components.
    manageSystemACLs: false
# Configures values that configure the Consul server cluster.
server:
  enabled: false
  # The number of server agents to run. This determines the fault tolerance of the cluster.
  #replicas: 3
externalServers:
  enabled: true
  hosts: ["1.1.1.1"]
  https: false
  httpsPort: 8500
  k8sAuthMethodHost: https://IlikeConsul:8500
# Contains values that configure the Consul UI.
#ui:
#  enabled: true
  # Registers a Kubernetes Service for the Consul UI as a LoadBalancer.
#  service:
#    type: LoadBalancer
# Configures and installs the automatic Consul Connect sidecar injector.
connectInject:
  enabled: false
# Enable catalog sync on cc-dev namespace only
syncCatalog:
  enabled: true
  default: true
  k8sAllowNamespaces: ['dev']

Current understanding and Expected behavior

Each of the Sync-catalog agents only sees its services, it will invalidate the services the other agent has written and this will happens concurrently.

Expected behavior is that services are only deregistered if they are unhealthy. So the merge strategy should be upsert, similar to how externalDNS can be configured.