Route from loopback being invalidated #14529

ianbarrere · 2023-10-03T19:08:03Z

Hello,

We have a health check which relies on exabgp conditionally advertising an anycast route to FRR which should then forward the route into our network infrastructure. FRR and exabgp are running on the same box. The route itself is an IP from a directly connected loopback interface. FRR considers the route invalid for some reason and thus does not advertise it.

To Reproduce

Loopback configuration from host:

[root@cacp]~# ifconfig lo_bgp
lo_bgp: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 10.255.255.252  netmask 255.255.255.255  broadcast 0.0.0.0
        ether ae:e7:eb:b6:79:d5  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 126 (126.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@cacp]~#

FRR config:

Current configuration:
!
frr version 9.0.1
frr defaults traditional
hostname cacp
allow-reserved-ranges
no ipv6 forwarding
!
router bgp 64512
 bgp router-id 10.6.4.47
 bgp log-neighbor-changes
 no bgp network import-check
 neighbor 10.6.4.2 remote-as 64560
 neighbor 10.6.4.3 remote-as 64560
 neighbor 127.0.0.2 remote-as 64512
 neighbor 127.0.0.2 description local-exabgp-daemon
 neighbor 127.0.0.2 passive
 bgp allow-martian-nexthop
 !
 address-family ipv4 unicast
  neighbor 10.6.4.2 prefix-list DEFAULT in
  neighbor 10.6.4.2 prefix-list squid_cast out
  neighbor 10.6.4.2 route-map squid_cast out
  neighbor 10.6.4.3 prefix-list DEFAULT in
  neighbor 10.6.4.3 prefix-list squid_cast out
  neighbor 10.6.4.3 route-map squid_cast out
  neighbor 127.0.0.2 soft-reconfiguration inbound
  neighbor 127.0.0.2 prefix-list squid_cast in
  neighbor 127.0.0.2 prefix-list NONE out
 exit-address-family
exit
!
ip prefix-list squid_cast seq 5 permit 10.255.255.252/32
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0
ip prefix-list DEFAULT seq 10 deny 0.0.0.0/0 le 32
ip prefix-list NONE seq 5 deny 0.0.0.0/0 le 32
!
route-map squid_cast permit 10
 match ip address prefix-list squid_cast
exit
!
route-map squid_cast deny 99
exit
!
end

The peering with the local exabgp daemon is working fine, and FRR is receiving the route, but it considers it inaccessible/invalid and it's not advertising it:

cacp# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, A - Babel, F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B   0.0.0.0/0 [20/0] via 10.6.4.2, ens192, weight 1, 00:22:49
                     via 10.6.4.3, ens192, weight 1, 00:22:49
K>* 0.0.0.0/0 [0/100] via 10.6.4.1, ens192, 00:22:52
C>* 10.6.4.0/23 is directly connected, ens192, 00:22:52
C>* 10.255.255.252/32 is directly connected, lo_bgp, 00:22:52
cacp# show ip bgp 10.255.255.252
BGP routing table entry for 10.255.255.252/32, version 0
Paths: (1 available, no best path)
  Not advertised to any peer
  Local
    10.255.255.252 (inaccessible) from 127.0.0.2 (127.0.0.2)
      Origin IGP, localpref 100, invalid, internal
      Last update: Tue Oct  3 14:41:48 2023
cacp#

Expected behavior
I would think that FRR should advertise the route as it is originated from a directly connected interface.

Versions

OS Version: Red Hat Enterprise Linux release 8.8 (Ootpa)
Kernel: Linux 4.18.0
FRR Version: 9.0.1 (frr-9.0.1-01.el8.x86_64)

The text was updated successfully, but these errors were encountered:

jlixfeld · 2023-10-04T09:41:47Z

Might redistribute connected in router bgp be helpful?

cgucker · 2023-10-04T13:50:08Z

the route is in the routing table, so it's not a redistribute issue.

C>* 10.255.255.252/32 is directly connected, lo_bgp, 00:22:52

ianbarrere · 2023-10-04T14:46:16Z

Might redistribute connected in router bgp be helpful?

We need it conditionally announced though, redistribute connected would advertise it at all times. What we are trying to achieve is a sort of dead man switch, where the route is only advertised when the health check is functional and passing. If the health check conditions fail, if the health check code itself fails, i.e. if anything related to the health check fails we want to withdraw the route, which is why we're doing it through exa.

ton31337 · 2023-10-04T18:45:09Z

Can you show what are you injecting from the ExaBGP side? What you actually have to do is to set a proper next-hop from the ExaBGP side to a valid next-hop from a shared network (most likely 10.6.4.0/23 in your case).

ianbarrere · 2023-10-04T19:49:11Z

Can you show what are you injecting from the ExaBGP side? What you actually have to do is to set a proper next-hop from the ExaBGP side to a valid next-hop from a shared network (most likely 10.6.4.0/23 in your case).

Hello, ExaBGP is injecting 10.255.255.252/32 with a next-hop of 10.255.255.252. We assumed that that would work since the loopback interface has that same IP. ExaBGP and FRR are peering over another loopback interface with the address 127.0.0.2, and we tried setting that as the next hop as well which also didn't work.

The route is being originated from inside the same device though, so advertising a next-hop of something off the box doesn't really make sense. Can you explain what you mean by "proper" next hop though? Does it need to be an IP on a physical interface or something?

donaldsharp · 2023-10-04T19:58:30Z

FRR does not allow routes to be recursive through themselves.

ianbarrere · 2023-10-04T20:00:52Z

That possibility crossed my mind, but then I was wondering why it was also invalid when the next hop was 127.0.0.2.

cgucker · 2023-10-04T22:18:53Z

That possibility crossed my mind, but then I was wondering why it was also invalid when the next hop was 127.0.0.2.

I haven't tried this, but would 0.0.0.0 be considered a valid next-hop? Many devices will use default as the next hop for covering routes being injected into BGP with underpinning static routes to null0, where covering or more specific routes exist for the actual next-hop forwarding

From a Cisco IOS-XR device:
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> X.X.X.0/23 0.0.0.0 0 32768 i

X.X.X.0/23 is being injected via a network statement under BGP, this would be the same thing except its being introduced by exaBGP and then more specifically forwarded to the local loopback interface.

donaldsharp · 2023-10-05T12:17:04Z

I say give it a try. What could go wrong?

ton31337 · 2023-10-05T14:33:01Z

Setting 0.0.0.0 as the next-hop would mean (self), but I can't say 100%. In any case, this shouldn't work, because from the ExaBGP side, you are setting the wrong next-hop anyway (127.0.0.2). Can you show the config of ExaBGP?

ianbarrere · 2023-10-05T14:59:34Z

Using 0.0.0.0 indeed not not work, I just tried it. New error, but roughly the same:

cacp# show ip bgp 10.255.255.252
BGP routing table entry for 10.255.255.252/32, version 0
Paths: (1 available, no best path)
  Not advertised to any peer
  Local
    0.0.0.0 (inaccessible) from 127.0.0.2 (127.0.0.2)
      Origin IGP, localpref 100, invalid, internal
      Last update: Thu Oct  5 10:47:32 2023
cacp#

ExaBGP config is very minimal:

process announce-routes {
   run /root/bin/health_check.sh;
   encoder json;
}

neighbor 127.0.0.1 {
        description "Local BGP daemon";
        router-id 127.0.0.2;
        local-address 127.0.0.2;
        local-as 64512;
        peer-as 64512;
        family {
           ipv4 unicast;
        }
        api {
           processes [announce-routes];
        }
}

The health check code just runs announce route 10.255.255.252/32 next-hop 0.0.0.0 (or whatever other next hop we have been testing with) when things are in good shape.

ton31337 · 2023-10-05T15:21:01Z

announce route 10.255.255.252/32 next-hop self also does not work? I've just tested quickly:

neighbor 127.0.0.2 {
  description "ToR";
  router-id 192.168.10.21;
  local-address 127.0.0.1;
  local-as 65001;
  peer-as 65001;
  hold-time 90;
  family {
    ipv4 unicast;
  }
  static {
    route 10.255.255.252/32 next-hop self;
  }
}

and

allow-reserved-ranges
!
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp network import-check
 neighbor 127.0.0.1 remote-as internal
exit
!

Result is:

donatas-pc# show ip bgp 10.255.255.252/32
BGP routing table entry for 10.255.255.252/32, version 3
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  Local
    127.0.0.1 (metric 100) from 127.0.0.1 (192.168.10.21)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      Last update: Thu Oct  5 18:20:16 2023
donatas-pc#

ianbarrere · 2023-10-06T20:40:07Z

It looks like this was actually because of an unexpected setup on our side. The next hop address of the route was not explicitly in the interface table on the linux host (although ExaBGP was successfully operating on it). After adding it the route shows up properly. I'm assuming that FRR does some verification to ensure that the next-hop is reachable via one of the interfaces listed in ifconfig or the routing table?

ton31337 · 2023-10-08T18:28:56Z

For sure, the next hop address should be in the RIB.

ianbarrere added the triage Needs further investigation label Oct 3, 2023

ton31337 self-assigned this Oct 4, 2023

ton31337 added the bgp label Oct 4, 2023

ton31337 removed the triage Needs further investigation label Oct 4, 2023

ton31337 closed this as completed Oct 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Route from loopback being invalidated #14529

Route from loopback being invalidated #14529

ianbarrere commented Oct 3, 2023

jlixfeld commented Oct 4, 2023

cgucker commented Oct 4, 2023

ianbarrere commented Oct 4, 2023

ton31337 commented Oct 4, 2023

ianbarrere commented Oct 4, 2023

donaldsharp commented Oct 4, 2023

ianbarrere commented Oct 4, 2023

cgucker commented Oct 4, 2023 •

edited

Loading

donaldsharp commented Oct 5, 2023

ton31337 commented Oct 5, 2023

ianbarrere commented Oct 5, 2023

ton31337 commented Oct 5, 2023

ianbarrere commented Oct 6, 2023

ton31337 commented Oct 8, 2023

Route from loopback being invalidated #14529

Route from loopback being invalidated #14529

Comments

ianbarrere commented Oct 3, 2023

jlixfeld commented Oct 4, 2023

cgucker commented Oct 4, 2023

ianbarrere commented Oct 4, 2023

ton31337 commented Oct 4, 2023

ianbarrere commented Oct 4, 2023

donaldsharp commented Oct 4, 2023

ianbarrere commented Oct 4, 2023

cgucker commented Oct 4, 2023 • edited Loading

donaldsharp commented Oct 5, 2023

ton31337 commented Oct 5, 2023

ianbarrere commented Oct 5, 2023

ton31337 commented Oct 5, 2023

ianbarrere commented Oct 6, 2023

ton31337 commented Oct 8, 2023

cgucker commented Oct 4, 2023 •

edited

Loading