Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update training validation to be handled per algo type #2462

Open
wants to merge 4 commits into
base: 2.x
Choose a base branch
from
Open

Update training validation to be handled per algo type #2462

wants to merge 4 commits into from

Conversation

anntians
Copy link
Contributor

Description

This PR is a follow up to another PR #2378. In the other PR we added more detailed error messages to validate training parameters, and this PR builds on those changes by updating the validation to be handled per algo type.

Related Issues

Resolves #2268

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@anntians
Copy link
Contributor Author

This PR is a follow up to another PR #2378. Can I get help with adding the skip-changelog and backport main label to this PR please. Thanks!

fix bwc tests
9887948 
Signed-off-by: AnnTian Shao <[email protected]>
jmazanec15
jmazanec15 reviewed Jan 29, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/AbstractMethodResolver.java Outdated
@@ -182,4 +182,30 @@ protected void validateCompressionConflicts(CompressionLevel originalCompression
throw validationException;
}
}

protected void validateMDivisibleByVectorDimension(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we generalize this? M is specific to PQ so shouldnt go in abstract class.

jmazanec15
jmazanec15 reviewed Jan 29, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/AbstractFaissMethod.java Outdated
.getParameters()
.get(METHOD_ENCODER_PARAMETER);

if (knnMethodContext.getMethodComponentContext().getParameters().containsKey(METHOD_PARAMETER_NLIST)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to move specifics into the HNSW and IVF method classes?

jmazanec15
jmazanec15 reviewed Jan 29, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/FaissFlatEncoder.java Outdated

TrainingConfigValidationOutput.TrainingConfigValidationOutputBuilder builder = TrainingConfigValidationOutput.builder();

// validate ENCODER_PARAMETER_PQ_M is divisible by vector dimension
Copy link
Member

@jmazanec15 jmazanec15 Jan 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flat encoder doesnt have M parameter. Only PQ does. So you can just noop on this

jmazanec15
jmazanec15 reviewed Jan 29, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/FaissSQEncoder.java Outdated

TrainingConfigValidationOutput.TrainingConfigValidationOutputBuilder builder = TrainingConfigValidationOutput.builder();

// validate ENCODER_PARAMETER_PQ_M is divisible by vector dimension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PQ_M is only valid for PQ encoders. You can noop on this.

jmazanec15
jmazanec15 reviewed Feb 3, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/FaissHNSWMethod.java Outdated
.getParameters()
.get(METHOD_ENCODER_PARAMETER);

if (knnMethodContext.getMethodComponentContext().getParameters().containsKey(METHOD_PARAMETER_NLIST)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we delegate this to the encoder?

jmazanec15
jmazanec15 reviewed Feb 3, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/FaissIVFMethod.java Outdated
.getParameters()
.get(METHOD_ENCODER_PARAMETER);

if (knnMethodContext.getMethodComponentContext().getParameters().containsKey(METHOD_PARAMETER_NLIST)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

jmazanec15
jmazanec15 reviewed Feb 3, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/FaissSQEncoder.java Outdated
@@ -61,4 +63,10 @@ public CompressionLevel calculateCompressionLevel(
// TODO: Hard code for now
return CompressionLevel.x2;
}

@Override
public TrainingConfigValidationOutput validateEncoderConfig(TrainingConfigValidationInput trainingConfigValidationInput) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this to default in Encoder.java? Like https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/engine/Encoder.java#L20

jmazanec15
jmazanec15 reviewed Feb 3, 2025
View reviewed changes
src/main/java/org/opensearch/knn/index/engine/faiss/QFrameBitEncoder.java Outdated

TrainingConfigValidationOutput.TrainingConfigValidationOutputBuilder builder = TrainingConfigValidationOutput.builder();

// validate ENCODER_PARAMETER_PQ_M is divisible by vector dimension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We dont need this for QFrameEncoder - this should just noop

@anntians anntians requested a review from Vikasht34 as a code owner February 5, 2025 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jmazanec15 jmazanec15 jmazanec15 left review comments

@heemin32 heemin32 Awaiting requested review from heemin32 heemin32 is a code owner

@navneet1v navneet1v Awaiting requested review from navneet1v navneet1v is a code owner

@VijayanB VijayanB Awaiting requested review from VijayanB VijayanB is a code owner

@vamshin vamshin Awaiting requested review from vamshin vamshin is a code owner

@naveentatikonda naveentatikonda Awaiting requested review from naveentatikonda naveentatikonda is a code owner

@junqiu-lei junqiu-lei Awaiting requested review from junqiu-lei junqiu-lei is a code owner

@martin-gaievski martin-gaievski Awaiting requested review from martin-gaievski martin-gaievski is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan ryanbogan is a code owner

@luyuncheng luyuncheng Awaiting requested review from luyuncheng luyuncheng is a code owner

@shatejas shatejas Awaiting requested review from shatejas shatejas is a code owner

@0ctopus13prime 0ctopus13prime Awaiting requested review from 0ctopus13prime 0ctopus13prime is a code owner

@Vikasht34 Vikasht34 Awaiting requested review from Vikasht34 Vikasht34 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@anntians @jmazanec15