Combining indexes

[jphastings]

If I had a pagefind index for site A, and a separate pagefind index for site B, would it be possible to configure the JS client to query both indexes to collect results (or N different indexes, if there are more)?

Would this be (vaguely) efficient for large (10+) sites? (Is the index structure set up in a way that would allow combining indexes after the fact?)

Finally, would you consider standardising on <link rel="search" type="application/pagefind" href="/search" /> (or similar), as a way to declare that the current HTML page (and site) has a pagefind index available at the given href? Combined with the above I could see a browser extension that lets you search all the sites you've visited that use pagefind. (I'd love to build it!)

---

As an aside: I'm using pagefind to index my blog, hosted via IPFS and it's perfect! The best part being that (for people that retrieve it over IPFS, instead of HTTP; or who pay attention to cache headers), parts of the index that remain the same from old versions of my blog can be pulled from peers that haven't retrieved the latest version yet (content addressed hashing — so this works even if the index hash & filenames have changed!). It's awesome, thank you!

[bglw]

👋 Hey @jphastings

If I had a pagefind index for site A, and a separate pagefind index for site B, would it be possible to configure the JS client to query both indexes to collect results (or N different indexes, if there are more)?

Yes! This is available with the multisite feature. Let me know if that's what you're after :)

Would this be (vaguely) efficient for large (10+) sites?

Vaguely efficient is probably a good way to describe it 😅 It does incur some of the cost of searching each site independently, but it can share the heavy load of the assets.

Actually, conveniently, I have a description of the bandwidth characteristics at the 4:00 mark in this video!

(Is the index structure set up in a way that would allow combining indexes after the fact?)

I haven't given much thought to this, but at the moment it would be in the too hard basket.

I could see a path forward — at some point we would like to give Pagefind something akin to an incremental mode, where it can re-parse its index and add files to it without reparsing the source files. If this existed, it would be within reason to re-parse multiple indexes and write them out as one. I don't think it'd be too hard, it just hasen't been prioritized yet.

For now, if the scale reaches this level I tend to look at mounting the static files of every site in one folder, and indexing the entire unit with Pagefind. (then fixing the URLs in the search JS to point at the correct site based on the folder).

Finally, would you consider standardising on a way to declare that the current HTML page (and site) has a pagefind index available at the given href?

I would definitely consider it! That extension sounds fun 😄

As an aside: I'm using pagefind to index my blog, hosted via IPFS

oooh! Very nice, and awesome stuff around the caching! 👀

[jphastings]

Fantastic! Thank you for the rapid and detailed reply! I'll see what I can whip up over the weekend — that your multisite search features already exist should make this very simple 👌