[Doc] Lucene inbuilt scalar quantization #7797
Conversation
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
hdhalter
added
2 - In progress
naveentatikonda
force-pushed
the
lucene_inbuilt_sq
branch
4 times, most recently
from
332da9b to
75415ce
Compare
naveentatikonda
requested review from
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
AMoo-Miki,
natebower,
dlvenable,
stephen-crawford and
epugh
as code owners
Signed-off-by: Naveen Tatikonda <[email protected]>
Signed-off-by: Naveen Tatikonda <[email protected]>
naveentatikonda
force-pushed
the
lucene_inbuilt_sq
branch
from
2651652 to
709428a
Compare
|
@kolchfa-aws Can you pls review this PR. Thanks! |
|
@naveentatikonda Sure, will do! |
hdhalter
added
4 - Doc review
|
|
||
| Optionally, you can specify the parameters in `method.parameters.encoder` shown below: | ||
| * `confidence_interval` - used to compute the `minQuantile` and `maxQuantile` parameters which are used to quantize the vectors. The accepted values are: | ||
| - It can be any value between and including `0.9` to `1.0`. For example, if we set it to `0.9` then it will consider the middle 90% of the vector values for computing the min and max Quantiles excluding the minimum and maximum 5% of the values. |
There was a problem hiding this comment.
So, to make sure, valid values are 0.9--1.0, inclusive (for static computation), or 0 (for dynamic computation)?
There was a problem hiding this comment.
yes, that's correct. Also, users can skip this parameter which will be computed as shown under the default case
|
|
||
| #### HNSW memory estimation | ||
|
|
||
| The memory required for Hierarchical Navigable Small Worlds (HNSW) is estimated to be `1.1 * (dimension + 8 * M)` bytes/vector. |
There was a problem hiding this comment.
M is max number of connections. This isn't a new parameter.
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: kolchfa-aws <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: kolchfa-aws <[email protected]>
There was a problem hiding this comment.
@kolchfa-aws @naveentatikonda Please see my comments and changes and let me know if you have any questions. Thanks!
|
|
||
| Optionally, you can specify the `confidence_interval` parameter in the `method.parameters.encoder` object. | ||
| The `confidence_interval` is used to compute the minimum and maximum quantiles in order to quantize the vectors: | ||
| - If you set the `confidence_interval` to a value in `0.9` to `1.0` range, inclusive, then the quantiles are calculated statically. For example, setting the `confidence_interval` to `0.9` specifies to compute the minimum and maximum quantiles based on the middle 90% of the vector values, excluding the minimum 5% and maximum 5% of the values. |
There was a problem hiding this comment.
"middle" => "median"?, "minimum" => "lowest"?, "maximum" => "highest"?
There was a problem hiding this comment.
Definitely "middle". I would keep as is. @naveentatikonda WDYT?
There was a problem hiding this comment.
This is definitely not my area of expertise, so that's fine with me :-)
There was a problem hiding this comment.
Yes, we should use middle
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
hdhalter
added
3 - Done
Description
Add documentation for Lucene inbuilt scalar quantization in k-NN plugin.
Issues Resolved
Closes #6496
Version
2.16
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.