getnetworkinfo shows ~45k channels in lnd 14+ vs. ~85k in lnd 13- and other implementations

[kingonly]

I'm trying to understand why newer versions of lnd show significantly less channels in getnetworkinfo.
An example of getnetworkinfo in lnd 15:
{
"graph_diameter": 12,
"avg_out_degree": 5.586546546546547,
"max_out_degree": 135,
"num_nodes": 16650,
"num_channels": 46508,
"total_network_capacity": "142767459423",
"avg_channel_size": 3069739.8173002494,
"min_channel_size": "1050",
"max_channel_size": "1000000000",
"median_channel_size_sat": "1000000",
"num_zombie_chans": "162479"
}
An example of getnetworkinfo in lnd 13:
{
"graph_diameter": 0,
"avg_out_degree": 9.716581276199348,
"max_out_degree": 2944,
"num_nodes": 17176,
"num_channels": 83446,
"total_network_capacity": "391523691254",
"avg_channel_size": 4691940.7910984345,
"min_channel_size": "1050",
"max_channel_size": "1400000000",
"median_channel_size_sat": "1000000",
"num_zombie_chans": "113658"
}
An example from a cln node:
lightning-cli listchannels | grep short_channel_id | sort -u | wc -l
80493

There can be ofc variation in the numbers between nodes and implementations, but I'm trying to understand the ~40k difference. I think there are two options here:

1. There are actually ~45k active channels. If that's the case, we need to discuss with cln and acinq and understand why they are showing 80k channels that probably lead to payment failures.
2. There are missing channels in lnd.

I hope you guys can help me make sense of these numbers.

[Roasbeef]

AFAICT, it's due to pruning, here's my node doing a historical sync:

2022-10-15 17:48:34.938 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): requesting new chans from height=0 and 758807 blocks after
2022-10-15 17:48:35.326 [INF] DISC: GossipSyncer(031a714070931a35552bd42244d0f85b94b1b641416fc1da1399ce7b7b7f16c9e2): filtering chan range: start_height=625085, num_blocks=10000
2022-10-15 17:48:35.329 [INF] DISC: GossipSyncer(031a714070931a35552bd42244d0f85b94b1b641416fc1da1399ce7b7b7f16c9e2): sending final chan range chunk, size=252
2022-10-15 17:48:35.352 [INF] DISC: GossipSyncer(031a714070931a35552bd42244d0f85b94b1b641416fc1da1399ce7b7b7f16c9e2): filtering chan range: start_height=605085, num_blocks=20000
2022-10-15 17:48:35.354 [INF] DISC: GossipSyncer(031a714070931a35552bd42244d0f85b94b1b641416fc1da1399ce7b7b7f16c9e2): sending final chan range chunk, size=340
2022-10-15 17:48:36.153 [INF] RPCS: elapsed time for diameter (13) calculation: 53.313712ms
2022-10-15 17:48:39.786 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=8000
2022-10-15 17:48:40.155 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=8000
2022-10-15 17:48:41.170 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=8000
2022-10-15 17:48:42.846 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=7998
2022-10-15 17:48:43.365 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=7999
2022-10-15 17:48:43.976 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=7997
2022-10-15 17:48:44.466 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=8000
2022-10-15 17:48:45.193 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=8000
2022-10-15 17:48:46.055 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=7972
2022-10-15 17:48:46.675 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=7990
2022-10-15 17:48:47.015 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): buffering chan range reply of size=3003
2022-10-15 17:48:47.015 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): filtering through 82959 chans
2022-10-15 17:48:47.130 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): remote peer has no new chans
2022-10-15 17:48:51.233 [INF] CRTR: Pruning channel graph using block 00000000000000000005a010eebf4e7270eea268355dfa6e391afaacb95b7b1b (height=758808)

It gets all the channels, but then it already either has them or considers them zombies:

2022-10-15 17:48:47.015 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): filtering through 82959 chans
2022-10-15 17:48:47.130 [INF] DISC: GossipSyncer(03e72fb9b743546a81df807f0b9f8664beed1b27f8ba768a119a3e5a9b40ba23d2): remote peer has no new chans

[kingonly]

Another piece of interesting information from @t-bast: there are ~13k channels that were excluded due to bidirectional check. So it seems hard to believe that the ~40k drop in getnetworkinfo can be explained by this extra check.

[Roasbeef]

there are ~13k channels that were excluded due to bidirectional check

What's this mean exactly? That one or both of the edges are disabled?

[kingonly]

13k channels were excluded by the bidirectional check and were included with only a unidirectional check

[t-bast]

@t-bast I think one difference is that y'alls node is extremely well connected (1k + connections). Our current logic depends on us actually getting the new updates to resurrect a channel. However if that never comes across, then that won't happen. Though nodes should be regularly broadcasting some sort of update, so given enough time the number should converge.

Right, if you never get the update then there's nothing you can do about it. But that's still surprising, because there's a big difference between 40k channels and 66k channels, and I don't think the difference between those is only because of pruned channels. There may be a more general issue about channel update propagation, or a check that makes lnd reject channel updates that eclair accepts?

[kingonly]

This is from WoS, maybe the most connected lnd node:
{
"graph_diameter": 12,
"avg_out_degree": 5.535581395348837,
"max_out_degree": 192,
"num_nodes": 17200,
"num_channels": 47606,
"total_network_capacity": "146814622984",
"avg_channel_size": 3083952.085535437,
"min_channel_size": "1050",
"max_channel_size": "1000000000",
"median_channel_size_sat": "1000000",
"num_zombie_chans": "183458"
}

[kingonly]

@Roasbeef you are referring to this change, correct?
#5138

[guggero]

Yes, that's the change. But that was already present in 0.13.x. Did you add the --routing.strictgraphpruning flag when upgrading? Or is this a Neutrino node?

[kingonly]

I'm aware, that's the reason I'm not sure the drop stems from this change (+ the large number of 40k diff). These stats are not just from our nodes. I got dozens of getnetworkinfo from other nodes in the network (big and small), all showing roughly the same numbers, none explicitly added strictgraphpruning.

[Roasbeef]

Ok, so I got a dump from a CLN node and did a diff (Ran The Numbers) against an lnd node. FWIW the lnd node this is running against is running w/ --routing.strictgraphpruning.

Here're two examples of channels that lnd has pruned 777299745360642049, and 604563170066497536:

• https://1ml.com/channel/777299745360642049
• https://1ml.com/channel/604563170066497536

Just an isolated example, but in both cases, the node has a single channel to 1 ML, and hasn't had an update in over a year.

Not all of them fit this mold though, here's one that still looks somewhat live: https://1ml.com/channel/803890334602166273

Here's a full dump of all 22k cids that cln has, but lnd doesn't: https://0bin.net/paste/NI-WE9hB#aNxlY1a3zENj+nXoOeVVNaAFSEhf6Zc98-VB9JN6QaD

[kingonly]

Fyi @cdecker

[Roasbeef]

Ok so I noticed that in my script I get these channel counts for cln vs lnd:

num cln chans:  80239
num lnd chans:  62213
chan diff:  22638

Note the higher number there than we've expected for lnd. For this I used lncli describegraph, then parsed the JSON. This is a lower number than reported by lncli getnetworkinfo, but reflects the actual graph that lnd uses to do path finding. The two calls use a different source: one uses the graph cache (networkinfo) while the other reads the graph directly.

In other words, I think we might be ok here, but the graph cache excludes items if we're missing a node policy (?).

One way to test the above hypothesis would be to get a node that isn't running the channel graph cache --db.no-graph-cache and see fi the numbers match up.