Fix IP Assignment to Gateway in On-Prem Environment

[sinkog]

Issue Title: Fix IP Assignment to Gateway in On-Prem Environment

Description:

Our goal is to introduce Envoy Gateway in an on-prem Kubernetes environment using MetalLB and Cilium. Below is a summary of the issues and observations, particularly when trying to assign a fixed access IP to a service.

Observations:

1. Static IP Assignment to Gateway:

   • When assigning a static IP to the Gateway as per the documentation, it creates a LoadBalancer type service with the assigned IP appearing as the externalIP.

2. Kubernetes and externalIP Handling:

   • In Kubernetes, the externalIP is primarily handled by internal routing. If an external provider sets up a route to the IP, the routing works well, allowing external traffic to reach the service.

3. MetalLB and LoadBalancerIP:

   • The CNI (Container Network Interface) and MetalLB operate in a way that they manage services of LoadBalancer type, and if a LoadBalancerIP is assigned, it will pick an IP from the pool if the IP is available and assign it.

4. Pool Settings and IP Assignment:

   • If the MetalLB pool is configured to assign only dedicated IPs, a standard LoadBalancer service will not receive an IP. This results in the service staying in a pending state until a suitable IP is found.
   • If the externalIP is assigned, the pending state is overridden, but no automatic advertisement takes place, requiring manual route configuration to reach the desired IP.

5. Multiple externalIP Issue:

   • If there is no enforced pool, a service may end up with two externalIPs, which could be identical. This can create confusion regarding the service's availability, as multiple IPs are associated with the same service.

Topic and Suggested Solution:

In an on-prem environment where fixed IP addresses are required, Kubernetes' externalIP and LoadBalancerIP behave differently. The externalIP often requires manual configuration in routers and the network infrastructure, whereas the LoadBalancerIP can be dynamically managed by Kubernetes and MetalLB.

• Public Cloud Services: The externalIP can be appropriately used if the IP address is advertised by the provider. In such configurations, IPs are usually managed and advertised by the cloud provider’s infrastructure.
• On-prem Environments: The externalIP may not be sufficient, as IP advertisement does not happen automatically, and manual routing configuration is necessary to direct traffic to the desired service IP.
• The LoadBalancerIP is more standardized in Kubernetes, and both MetalLB and Cilium rely on this for automatic advertisement. Using LoadBalancerIP ensures proper traffic routing and allows Kubernetes to handle scaling automatically.

Request:

• We ask for an investigation into how the integration between externalIP and LoadBalancerIP can be improved in on-prem Kubernetes environments to ensure fixed IP handling. Additionally, we request that MetalLB and Cilium properly handle the advertisement and routing of IPs, especially in scenarios where the externalIP is set.

[arkodg]

hey @sinkog

1. You should be able to set the loadBalancerIP field using EnvoyProxy resource
   https://gateway.envoyproxy.io/docs/api/extension_types/#kubernetesservicespec

2. a. You should also be able to set the externalIP field by using Gateway.Spec.Addresses which internally will set the value in the generated service's externalIP or you could
   b. directly edit the generated service and modify it using the patch field https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/#patching-service-for-envoyproxy

[tekulvw]

Specifically for metallb, you can also specify the metallb.io/loadBalancerIPs annotation which is applied to the envoy service:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: proxy
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          metallb.io/loadBalancerIPs: x.x.x.x
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: proxy

[sinkog]

Thank you for the suggestions and detailed responses! The proposed solution works for configuring the Envoy Gateway, especially in handling the externalIP and loadBalancerIP. However, from a code maintainability perspective, we have a few concerns:

1. Missing Validation for Annotations: Using annotations, particularly those not validated at the API level, increases the risk of incorrect configurations. This complicates maintaining system stability and makes debugging more challenging.

2. API Extension Possibilities: It raises the question of why these functionalities couldn't be directly integrated into the Envoy Gateway API itself, such as extending the handling of externalIP and loadBalancerIP. This approach would improve system transparency and reduce the need for manual interventions.

3. Simplifying Annotations: If annotations are necessary, we suggest that these settings should be directly included in the Gateway descriptor. This would not only reduce complexity but also establish a direct connection with future implementations, simplifying the work for developers and maintenance tasks.

Overall, while the current solution is functional, we recommend considering extending the API to better handle externalIP and loadBalancerIP. Additionally, if annotations are used, integrating them into the Gateway descriptor would enhance maintainability and transparency.

[arkodg]

@sinkog this the annotation usage is directed by metalLB https://metallb.universe.tf/usage/, you can use Envoy Gateway's https://gateway.envoyproxy.io/docs/api/extension_types/#kubernetesservicespec to also set loadBalancerIP which should work for MetalLB based on the doc

[sinkog]

You're absolutely right. On the other hand, I step out of the Gateway object and need to create a child object whose definition, moreover, is not part of the Gateway API. The solution is good, but in your case, it doubles the necessary lines of code. <-> Okay, for an actual Gateway object, it's much less, but it still represents a log that needs maintenance.

By the way, this issue has come up on your side multiple times in one way or another. It’s a fact that if I had found this in the documentation, the change request wouldn't have been made. But from this point on, there’s practically no solution that isn’t in a gray area. From a DevOps perspective: logically, this information is part of the Gateway because I want to assign an IP to it (and an externalIP can be assigned), not to a sub-object. Apologies for the term, but I have no involvement in your internal logic and connection points. From here, I’m not building the EnvoyProxy with your mechanisms, and I can only hope it works. Additionally, when building the infrastructure, I’m not combining just a few components, and if possible, I avoid complexity. If and when I need to use EnvoyProxy functionality, I’ll familiarize myself with its API and use it.

So, the only problem with your proposal is reading through and understanding that x00 lines of API description, which shouldn’t even be necessary for a colleague who isn’t working on-prem.

By the way, do you have annotations for Cilium, Calico, and other CNI implementations that use loadBalancerIP or other Layer 2 implementations? You have a few, and those are what I need to use.