Note extension table

[jb55]

I want a way to extend nostrdb notes arbitrarily without splitting the note layout too drastically

This would fix a few things which are a bit janky:

• Remove separate note block db and just store note blocks in the node
• Store a minimal perfect hashmap of note works for optimized note mute checking.

Plan

Because I'm an idiot and didn't add any reserved fields, I will use the padding after the version number as an offset into the extension table. The note version will be 2 so we know to look after the strings table for the extension table.

Extension table

The extension table will be an offset array like so:

{elements: u16, capacity: u16} [{offset: u32},...] <padded to 8 byte boundary>

Where the index of the elements represents the extension type, and the elements count so we don't need to increase the note version number for each extension. We have fixed extensions to have O(1) lookup.

The offsets are the position of the data relative to the extension table. Each extension block MUST start with its own length (u32), so we don't need to store it here.

Since this data would have to be at the end of the strings table, it is already aligned because padding is added after the strings table for alignment.

The extensions will be nice for cache performance since all note related data will be local to the note, unlike right now with note blocks being somewhere else.

The extension table should be padded to 8 byte alignment

Extension block

{size: u32} {data: [u8]}

Proposed Extensions

• NOTE_EXT_BLOCKS - replaces the note block table
• NOTE_EXT_WORDS - for optimized mute words

[jb55]

how much space should be reserved here so that we don't have to migrate chunks of data whenever an extension is added. perhaps we can do like 8 slots to start before needing a migration? 7 slots would avoid needing any padding

this would total 32 extra bytes for the extension table on every note, an extra 3MB every 100,000 notes. not that bad