The BGP conditional advertisement feature causes existing routes being marked as unreachable #14598

mehrdadrad · 2023-10-13T21:29:00Z

Version : version 8.5.2
OS: Fedora 38
Kernel: 6.4.10-200.fc38.x86_64

Describe the bug:
We're using BGP conditional advertisement under network namespace (VRF) per peer and we noticed local configuration changes on peers cause the existing route(s) become unreachable and withdrawn. We've configured redistribute static routes to BGP and this bug appears when we add one new static route with appropriate prefix-list or when the BGP conditional adv. detects a change. The issue can be recovered by soft reset of BGP neighbors in out direction also when we remove the BGP conditional everything works properly. Please let me know if you need more information.

To Reproduce

Step 1: Configure FRR to advertise 172.16.16.0/24 to peer 169.254.100.15 when 10.0.0.0/8 exist (condition) and advertise 100.64.15.1/32 to peer 169.254.110.15 when 0.0.0.0 exist (condition).

config terminal
vrf vrf1
 netns /run/netns/vrf1
 ip route 172.16.16.0/24 eno2
 ip route 100.64.15.1/32 eno1
exit-vrf

router bgp 65000 vrf vrf1
 bgp router-id 169.254.1.14
 bgp conditional-advertisement timer 5
 neighbor 169.254.100.15 remote-as 65000
 neighbor 169.254.110.15 remote-as 65000
 !
 address-family ipv4 unicast
  redistribute static
  neighbor 169.254.100.15 soft-reconfiguration inbound
  neighbor 169.254.100.15 route-map IMPORT_RED_V4 in
  neighbor 169.254.100.15 route-map DENY_ALL out
  neighbor 169.254.100.15 advertise-map EXPORT_RED_V4 exist-map IMPORT_GREEN_V4
  neighbor 169.254.110.15 soft-reconfiguration inbound
  neighbor 169.254.110.15 route-map IMPORT_GREEN_V4 in
  neighbor 169.254.110.15 route-map DENY_ALL out
  neighbor 169.254.110.15 advertise-map EXPORT_GREEN_V4 exist-map IMPORT_RED_V4
 exit-address-family
exit

ip prefix-list IMPORT_GREEN_V4 seq 5 permit 10.0.0.0/8
ip prefix-list IMPORT_RED_V4 seq 5 permit 0.0.0.0/0
ip prefix-list EXPORT_GREEN_V4 seq 5 permit 100.64.15.1/32
ip prefix-list EXPORT_RED_V4 seq 5 permit 172.16.16.0/24

route-map EXPORT_RED_V4 permit 10
 match ip address prefix-list EXPORT_RED_V4
exit
!
route-map EXPORT_GREEN_V4 permit 10
 match ip address prefix-list EXPORT_GREEN_V4
exit
!
route-map IMPORT_RED_V4 permit 10
 match ip address prefix-list IMPORT_RED_V4
exit
!
route-map IMPORT_RED_V4 deny 20
exit
!
route-map IMPORT_GREEN_V4 permit 10
 match ip address prefix-list IMPORT_GREEN_V4
exit
!
route-map IMPORT_GREEN_V4 deny 20
exit
!
route-map DENY_ALL deny 10
exit


end

Log:

18:38:11 BGP: [Z5K42-C2S56] bgp_conditional_adv_routes: Advertise routes to/from 169.254.110.15 for IPv4 Unicast
18:38:11 BGP: [TN0HX-6G1RR] u30:s16 send UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path 
18:38:11 BGP: [HVRWP-5R9NQ] u30:s16 send UPDATE 100.64.15.1/32 IPv4 unicast <<<<<<<
18:38:11 BGP: [WEV7K-2GAQ5] u30:s16 send UPDATE len 57 (max message len: 4096) numpfx 1
18:38:11 BGP: [MBFVT-8GSC6] u30:s16 169.254.110.15 send UPDATE w/ nexthop 169.254.1.14

18:38:16 BGP: [Z5K42-C2S56] bgp_conditional_adv_routes: Advertise routes to/from 169.254.110.15 for IPv4 Unicast
18:38:16 BGP: [MAXTH-W5WMN] 169.254.110.15 suppress UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path 
18:38:16 BGP: [TN0HX-6G1RR] u32:s15 send UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path 
18:38:16 BGP: [HVRWP-5R9NQ] u32:s15 send UPDATE 172.16.16.0/24 IPv4 unicast
18:38:16 BGP: [WEV7K-2GAQ5] u32:s15 send UPDATE len 56 (max message len: 4096) numpfx 1
18:38:16 BGP: [MBFVT-8GSC6] u32:s15 169.254.100.15 send UPDATE w/ nexthop 169.254.0.14
18:38:16 BGP: [WHJ94-YXNE1] u30:s16 send UPDATE 100.64.15.1/32 IPv4 unicast -- unreachable <<<<<<<
18:38:16 BGP: [G259R-NCBEJ] u30:s16 send UPDATE (withdraw) len 28 numpfx 1

In the above log, you can see 100.64.15.1/32 was advertised and when BGP conditional adv. detected a change then 100.64.15.1/32 is marked as unreachable but the static route (ip route 100.64.15.1/32 eno1) was in place.

# show bgp vrf all sum

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.100.15    4       6507         7         7        0    0    0 00:03:16            1        1 N/A
169.254.110.15    4       6507         7         8        0    0    0 00:03:16            1        0 N/A

# clear ip bgp vrf vrf1 * soft out
# show bgp vrf all sum

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.100.15    4       65000        23        25        0    0    0 00:19:17            1        1 N/A
169.254.110.15    4       65000        23        25        0    0    0 00:19:17            1        1 N/A

Step 2 : Adding new static route 100.64.15.2 with appropriate prefix-list. It needs to advertise to peer 169.254.110.15

config terminal
vrf vrf1
ip route 100.64.15.2/32 eno1
exit-vrf
ip prefix-list EXPORT_GREEN_V4 seq 10 permit 100.64.15.2/32
end

Log:

17:49:00 BGP: [Z5K42-C2S56] bgp_conditional_adv_routes: Advertise routes to/from 169.254.110.15 for IPv4 Unicast
17:49:00 BGP: [MAXTH-W5WMN] 169.254.110.15 suppress UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path 
17:49:00 BGP: [WHJ94-YXNE1] u8:s3 send UPDATE 172.16.16.0/24 IPv4 unicast -- unreachable <<<<<<<
17:49:00 BGP: [G259R-NCBEJ] u8:s3 send UPDATE (withdraw) len 27 numpfx 1
17:49:00 BGP: [WHJ94-YXNE1] u6:s4 send UPDATE 100.64.15.1/32 IPv4 unicast -- unreachable <<<<<<<
17:49:00 BGP: [G259R-NCBEJ] u6:s4 send UPDATE (withdraw) len 28 numpfx 1
17:49:00 BGP: [TN0HX-6G1RR] u6:s4 send UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path 
17:49:00 BGP: [HVRWP-5R9NQ] u6:s4 send UPDATE 100.64.15.2/32 IPv4 unicast
17:49:00 BGP: [WEV7K-2GAQ5] u6:s4 send UPDATE len 57 (max message len: 4096) numpfx 1
17:49:00 BGP: [MBFVT-8GSC6] u6:s4 169.254.110.15 send UPDATE w/ nexthop 169.254.1.14

In the above log shows, when the static route (100.64.15.2/32) has been added, the two existed static routes (172.16.16.0/24 and 100.64.15.1/32) became unreachable and withdrawn.

# show bgp vrf all sum

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.100.15    4       65000        10        12        0    0    0 00:07:01            1        0 N/A
169.254.110.15    4       65000        10        13        0    0    0 00:07:01            1        1 N/A

# show ip bgp vrf vrf1 neighbors 169.254.110.15 advertised-routes

    Network          Next Hop            Metric LocPrf Weight Path
 *> 100.64.15.2/32   0.0.0.0                  0    100  32768 ?

The soft BGP reset recovers this issue.
# clear ip bgp vrf vrf1 * soft out
# show bgp vrf all sum

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.100.15    4       65000        33        35        0    0    0 00:29:37            1        1 N/A
169.254.110.15    4       65000        33        37        0    0    0 00:29:37            1        2 N/A

# show ip bgp vrf vrf1 neighbors 169.254.110.15 advertised-routes

    Network          Next Hop            Metric LocPrf Weight Path
 *> 100.64.15.1/32   0.0.0.0                  0    100  32768 ?
 *> 100.64.15.2/32   0.0.0.0                  0    100  32768 ?

# show ip bgp vrf vrf1 neighbors 169.254.100.15 advertised-routes

    Network          Next Hop            Metric LocPrf Weight Path
 *> 172.16.16.0/24  0.0.0.0                  0    100  32768 ?

The text was updated successfully, but these errors were encountered:

ton31337 · 2023-10-14T20:10:11Z

Can you try and confirm if this is happening also for 9.0.1 and/or master?

mehrdadrad · 2023-10-17T23:35:27Z

Can you try and confirm if this is happening also for 9.0.1 and/or master?

Same issue with master. the existed static routes become unreachable (BGP withdraw) and new static route advertises through BGP.

BGP: [Z5K42-C2S56] bgp_conditional_adv_routes: Advertise routes to/from 169.254.110.15 for IPv4 Unicast
BGP: [MAXTH-W5WMN] 169.254.110.15 suppress UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path
BGP: [WHJ94-YXNE1] u8:s3 send UPDATE 172.16.16.0/24 IPv4 unicast -- unreachable <<<<<<<<< "existed static route"
BGP: [G259R-NCBEJ] u8:s3 send UPDATE (withdraw) len 27 numpfx 1
BGP: [WHJ94-YXNE1] u4:s4 send UPDATE 100.64.15.1/32 IPv4 unicast -- unreachable <<<<<<<< "existed static route"
BGP: [G259R-NCBEJ] u4:s4 send UPDATE (withdraw) len 28 numpfx 1
BGP: [TN0HX-6G1RR] u4:s4 send UPDATE w/ attr: nexthop 0.0.0.0, localpref 100, metric 0, path 
BGP: [HVRWP-5R9NQ] u4:s4 send UPDATE 100.64.15.2/32 IPv4 unicast  <<<<<<<<< "new static route"
BGP: [WEV7K-2GAQ5] u4:s4 send UPDATE len 57 (max message len: 4096) numpfx 1
BGP: [MBFVT-8GSC6] u4:s4 169.254.110.15 send UPDATE w/ nexthop 169.254.1.14

ton31337 · 2023-10-18T04:03:43Z

Seems I managed to replicate, I'll dig into this issue more.

ton31337 · 2023-10-19T17:52:03Z

I found the issue, gonna fix it this week.

mehrdadrad · 2023-10-19T18:43:42Z

I found the issue, gonna fix it this week.

Great! Thanks @ton31337
Is this related to suppress UPDATE? I checked the code and noticed because this return here causes the existed routes become unreachable and once I force subgrp to update at here before bgp_adj_out_set_subgroup through below set_flag, everything works properly except it sends extra update which it doesn't need because that force update.

SET_FLAG(subgrp->sflags, SUBGRP_STATUS_FORCE_UPDATES);

ton31337 · 2023-10-20T10:04:19Z

@mehrdadrad yes, this is due to the suppression mechanism. There are some specific paths (e.g. route-refresh) where this is necessary (to force), and conditional advertisement is one of them, but was missed.

mehrdadrad · 2023-10-24T18:21:04Z

@ton31337 Can we have this fix for Stable8.5/8.5.x? also when are you going to cut new release? any estimate?

mehrdadrad · 2023-11-09T16:50:06Z

@ton31337 Any ETA for next release?

ton31337 · 2023-11-09T16:53:34Z

next week most likely ✌️

mehrdadrad added the triage Needs further investigation label Oct 13, 2023

ton31337 added the bgp label Oct 14, 2023

ton31337 self-assigned this Oct 14, 2023

idryzhov assigned idryzhov and unassigned idryzhov Oct 14, 2023

ton31337 added bug and removed triage Needs further investigation labels Oct 19, 2023

ton31337 mentioned this issue Oct 20, 2023

bgpd: Do not suppress conditional advertisement updates if triggered #14628

Merged

donaldsharp closed this as completed in #14628 Oct 23, 2023

github-actions bot added the backport label Oct 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The BGP conditional advertisement feature causes existing routes being marked as unreachable #14598

The BGP conditional advertisement feature causes existing routes being marked as unreachable #14598

mehrdadrad commented Oct 13, 2023 •

edited

Loading

ton31337 commented Oct 14, 2023

mehrdadrad commented Oct 17, 2023

ton31337 commented Oct 18, 2023 •

edited

Loading

ton31337 commented Oct 19, 2023

mehrdadrad commented Oct 19, 2023

ton31337 commented Oct 20, 2023

mehrdadrad commented Oct 24, 2023

mehrdadrad commented Nov 9, 2023

ton31337 commented Nov 9, 2023

The BGP conditional advertisement feature causes existing routes being marked as unreachable #14598

The BGP conditional advertisement feature causes existing routes being marked as unreachable #14598

Comments

mehrdadrad commented Oct 13, 2023 • edited Loading

ton31337 commented Oct 14, 2023

mehrdadrad commented Oct 17, 2023

ton31337 commented Oct 18, 2023 • edited Loading

ton31337 commented Oct 19, 2023

mehrdadrad commented Oct 19, 2023

ton31337 commented Oct 20, 2023

mehrdadrad commented Oct 24, 2023

mehrdadrad commented Nov 9, 2023

ton31337 commented Nov 9, 2023

mehrdadrad commented Oct 13, 2023 •

edited

Loading

ton31337 commented Oct 18, 2023 •

edited

Loading