Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support efficient filtering rules for switching to exact KNN with nmslib #1341

Closed
tstadel opened this issue Dec 11, 2023 · 1 comment

Comments

@tstadel
Copy link

tstadel commented Dec 11, 2023

Is your feature request related to a problem?
Efficient filtering rules of the FAISS implementation for switching to exact KNN search are library-agnostic and improve search results independent of the native FAISS filtering capabilities (especially for highly selective filters). For nmslib these rules would be beneficial as well and make the overall behavior of KNN plugin's search-filter-logic more consistent.

What solution would you like?
Use the same logic as depicted on https://opensearch.org/docs/latest/images/faiss-algorithm.jpg for nmslib when using efficient filtering. The issue that nmslib does not support applying filters natively can be solved by applying simple postfiltering in JNIService.queryIndex:

if (KNNEngine.NMSLIB.getName().equals(engineName)) {
return NmslibService.queryIndex(indexPointer, queryVector, k);
}

What alternatives have you considered?

  • Postfiltering could be implemented in nmslib's JNI-layer, but I don't see too many benefits of this as we would need another method/interface here
  • Applying the rules to any ANN queries with filters (not just with efficient filtering syntax): I guess that's harder to implement as the usual postfiltering happens in a later stage of the search (where we do not have access to KNNWeight?). Probably we don't want that for lucene ANN queries, so at least we would need some param to switch it off, which makes it way more complex.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

@tstadel
Copy link
Author

tstadel commented Dec 11, 2023

duplicate of #1340

@tstadel tstadel closed this as completed Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant