IRRExplorer view of our data is unexpectedly out of date

[tangledhelix]

I'm one of the operators of NTTCOM (rr.ntt.net). We have had a complaint about old data still showing up in IRRexplorer.

https://irrexplorer.nlnog.net/prefix/116.199.202.0/23

At this moment it returns the result seen in the below screenshot:

Here are the entries it complains about:

route:          116.199.202.0/23
descr:          route-object for NAPINFO
origin:         AS38548
remarks:        Transit via AS10217
notify:         [email protected]
mnt-by:         MAINT-AS10217
changed:        [email protected] 20071001
source:         NTTCOM

route:          116.199.202.0/24
descr:          route-object for NAPINFO
origin:         AS38548
remarks:        Transit via AS10217
notify:         [email protected]
mnt-by:         MAINT-AS10217
changed:        [email protected] 20071001
source:         NTTCOM

route:          116.199.203.0/24
descr:          route-object for NAPINFO
origin:         AS38548
remarks:        Transit via AS10217
notify:         [email protected]
mnt-by:         MAINT-AS10217
changed:        [email protected] 20071001
source:         NTTCOM

If you query those prefixes from rr.ntt.net, even with !fno-rpki-filter active, they are not present. The only matching entry is this valid route mirrored from APNIC:

route:          116.199.202.0/23
origin:         AS152094
descr:          indigo internet
mnt-by:         MAINT-ATHPL-AU
last-modified:  2023-11-08T01:06:03Z
source:         APNIC
rpki-ov-state:  valid

route:          116.199.202.0/23
descr:          RPKI ROA for 116.199.202.0/23 / AS152094
remarks:        This AS152094 route object represents routing data retrieved
                from the RPKI. This route object is the result of an automated
                RPKI-to-IRR conversion process performed by IRRd.
max-length:     23
origin:         AS152094
source:         RPKI  # Trust Anchor: apnic

The invalid AS38548 entries were deleted from our IRR yesterday (14 Nov.) The IRRexplorer status page currently says NTTCOM was refreshed at 2023-11-15 02:34.

[troy2914]

Still showing the old NTTCOM data.

[teunvink]

Looking at our logs I see something odd with the NTT database updates in irrd:

2023-11-14 00:00:16,471 irrd[3128609]: [irrd.mirroring.mirror_runners_import#INFO] Retrieving NRTM updates for NTTCOM from serial 1596175 on rr.ntt.net:43
...
2023-11-19 21:49:35,213 irrd[1408468]: [irrd.mirroring.mirror_runners_import#INFO] Retrieving NRTM updates for NTTCOM from serial 1596175 on rr.ntt.net:43
...
2023-11-21 07:53:07,901 irrd[1974266]: [irrd.mirroring.mirror_runners_import#INFO] Retrieving NRTM updates for NTTCOM from serial 1596175 on rr.ntt.net:43

For over a week time, we're seeing the same serial for NTT's data. So we're pulling data, but since the serial is still the same, we assuming there are no updates. The serial is taken from ftp://rr1.ntt.net/nttcomRR/NTTCOM.CURRENTSERIAL. However, when I check the serial on that URL manually, I see a newer serial:

$ cat NTTCOM.CURRENTSERIAL
1676408

Forcing a reload of the mirror data (using irrd_mirror_force_reload) resulted in fresh data:

2023-11-21 21:02:24,512 irrd[2191732]: [irrd.storage.database_handler#INFO] force_reload flag set for NTTCOM, serial synchronisation will be False for current settings, actual reload process wll take place in next scheduled importer run
2023-11-21 21:06:37,288 irrd[2192999]: [irrd.mirroring.mirror_runners_import#INFO] Running full RPSL import of NTTCOM from ['ftp://rr1.ntt.net/nttcomRR/nttcom.db.gz'], serial from ftp://rr1.ntt.net/nttcomRR/NTTCOM.CURRENTSERIAL
2023-11-21 21:06:39,773 irrd[2192999]: [irrd.mirroring.mirror_runners_import#INFO] Downloaded ftp://rr1.ntt.net/nttcomRR/NTTCOM.CURRENTSERIAL, contained 1676408
2023-11-21 21:21:23,925 irrd[2192999]: [irrd.mirroring.parsers#INFO] File import for NTTCOM: 392695 objects read, 392695 objects inserted, ignored 0 due to errors, ignored 0 due to object_class_filter, source /tmp/tmp36zeaf7j

So it seems irrd is incorrectly not updating data because it fails to see the updated serial. What makes it extra weird, is that I see the same issues with serial numbers not updating on my test instance as well, but it was stuck on different serial numbers: 1480799. A forced refresh of the NTT database solved it there as well.

@tangledhelix are you aware of any issues with the serial number of the NTT database? I guess not, since I see different serial numbers on those two instances. So possibly this is a bug in irrd we're seeing.

One last remark: we use this repository to track issues with the codebase. If you experience issues with the hosted IRR Explorer data freshness, contacting us on the email address on the website is better, since that will get more people's attention (and faster).

[teunvink]

This seems to be related to irrdnet/irrd#877