Fix long pending issue when deleting model #1882

zane-neo · 2024-01-17T04:17:56Z

Description

This fixes the occasional pending forever issue of deprovision API in flow framework. When delete model failed, e.g. model status is note allowed to delete, the delete model request is not been responded and flow framework will pending waiting.

Issues Resolved

NA

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ylwu-amzn · 2024-01-17T18:51:20Z

Github CI failed

[ant:jacocoReport] Rule violated for class org.opensearch.ml.action.models.DeleteModelTransportAction.1: branches covered ratio is 0.5, but expected minimum is 0.7
> Task :opensearch-ml-plugin:jacocoTestCoverageVerification FAILED

Signed-off-by: zane-neo <[email protected]>

ylwu-amzn · 2024-02-07T10:24:17Z

plugin/src/main/java/org/opensearch/ml/action/undeploy/TransportUndeployModelAction.java

@@ -174,7 +175,7 @@ protected MLUndeployModelNodesResponse newResponse(
                            deployToAllNodes.put(modelId, false);
                        }
                        updateRequest.index(ML_MODEL_INDEX).id(modelId).doc(updateDocument);
-                        bulkRequest.add(updateRequest);
+                        bulkRequest.add(updateRequest).setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);


Do you see anywhere else missed this?

* Fix long pending issue when deleting model Signed-off-by: zane-neo <[email protected]> * Refine the delete model code Signed-off-by: zane-neo <[email protected]> * refactor delete model flow to make sure all dependent resources are deleted together with model metadata Signed-off-by: zane-neo <[email protected]> * fix minor issue to make sure only non-remote model will deelete chunks Signed-off-by: zane-neo <[email protected]> * format code Signed-off-by: zane-neo <[email protected]> * fix failure UTs Signed-off-by: zane-neo <[email protected]> * Change to delete model metadata first Signed-off-by: zane-neo <[email protected]> * format code Signed-off-by: zane-neo <[email protected]> * Remove remote function check Signed-off-by: zane-neo <[email protected]> * Fix failure UTs Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: zane-neo <[email protected]> (cherry picked from commit f18eaf3)

* Fix long pending issue when deleting model Signed-off-by: zane-neo <[email protected]> * Refine the delete model code Signed-off-by: zane-neo <[email protected]> * refactor delete model flow to make sure all dependent resources are deleted together with model metadata Signed-off-by: zane-neo <[email protected]> * fix minor issue to make sure only non-remote model will deelete chunks Signed-off-by: zane-neo <[email protected]> * format code Signed-off-by: zane-neo <[email protected]> * fix failure UTs Signed-off-by: zane-neo <[email protected]> * Change to delete model metadata first Signed-off-by: zane-neo <[email protected]> * format code Signed-off-by: zane-neo <[email protected]> * Remove remote function check Signed-off-by: zane-neo <[email protected]> * Fix failure UTs Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: zane-neo <[email protected]> (cherry picked from commit f18eaf3) Co-authored-by: zane-neo <[email protected]>

* Fix long pending issue when deleting model Signed-off-by: zane-neo <[email protected]> * Refine the delete model code Signed-off-by: zane-neo <[email protected]> * refactor delete model flow to make sure all dependent resources are deleted together with model metadata Signed-off-by: zane-neo <[email protected]> * fix minor issue to make sure only non-remote model will deelete chunks Signed-off-by: zane-neo <[email protected]> * format code Signed-off-by: zane-neo <[email protected]> * fix failure UTs Signed-off-by: zane-neo <[email protected]> * Change to delete model metadata first Signed-off-by: zane-neo <[email protected]> * format code Signed-off-by: zane-neo <[email protected]> * Remove remote function check Signed-off-by: zane-neo <[email protected]> * Fix failure UTs Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: zane-neo <[email protected]>

zane-neo requested review from b4sjoo, dhrubo-os, jngz-es, model-collapse, rbhavna, ylwu-amzn, Zhangxunmt, austintlee and HenryL27 as code owners January 17, 2024 04:17

zane-neo temporarily deployed to ml-commons-cicd-env January 17, 2024 04:18 — with GitHub Actions Inactive

zane-neo temporarily deployed to ml-commons-cicd-env January 17, 2024 04:41 — with GitHub Actions Inactive

zane-neo had a problem deploying to ml-commons-cicd-env January 17, 2024 06:02 — with GitHub Actions Error

zane-neo had a problem deploying to ml-commons-cicd-env January 17, 2024 06:02 — with GitHub Actions Failure

zane-neo had a problem deploying to ml-commons-cicd-env January 17, 2024 06:02 — with GitHub Actions Error

zane-neo had a problem deploying to ml-commons-cicd-env January 17, 2024 06:02 — with GitHub Actions Failure

zane-neo had a problem deploying to ml-commons-cicd-env January 17, 2024 06:02 — with GitHub Actions Error

zane-neo force-pushed the delete-model-hang-fix branch from 3725de3 to 337f1d0 Compare January 26, 2024 07:44

zane-neo had a problem deploying to ml-commons-cicd-env January 26, 2024 07:44 — with GitHub Actions Error

zane-neo had a problem deploying to ml-commons-cicd-env January 26, 2024 07:44 — with GitHub Actions Failure

zane-neo had a problem deploying to ml-commons-cicd-env January 26, 2024 07:44 — with GitHub Actions Error

Remove remote function check

6d60e95

Signed-off-by: zane-neo <[email protected]>

zane-neo had a problem deploying to ml-commons-cicd-env February 7, 2024 09:27 — with GitHub Actions Error

zane-neo had a problem deploying to ml-commons-cicd-env February 7, 2024 09:27 — with GitHub Actions Failure

zane-neo had a problem deploying to ml-commons-cicd-env February 7, 2024 09:27 — with GitHub Actions Error

zane-neo had a problem deploying to ml-commons-cicd-env February 7, 2024 09:27 — with GitHub Actions Failure

Fix failure UTs

21afabf

Signed-off-by: zane-neo <[email protected]>

zane-neo temporarily deployed to ml-commons-cicd-env February 7, 2024 09:53 — with GitHub Actions Inactive

zane-neo temporarily deployed to ml-commons-cicd-env February 7, 2024 10:22 — with GitHub Actions Inactive

ylwu-amzn reviewed Feb 7, 2024

View reviewed changes

ylwu-amzn approved these changes Feb 7, 2024

View reviewed changes

model-collapse approved these changes Feb 7, 2024

View reviewed changes

zane-neo merged commit f18eaf3 into opensearch-project:main Feb 7, 2024
14 checks passed

zane-neo added backport 2.x backport 2.12 labels Feb 7, 2024

opensearch-trigger-bot bot mentioned this pull request Feb 7, 2024

[Backport 2.12] Fix long pending issue when deleting model #2046

Merged

opensearch-trigger-bot bot mentioned this pull request Feb 7, 2024

[Backport 2.x] Fix long pending issue when deleting model #2047

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix long pending issue when deleting model #1882

Fix long pending issue when deleting model #1882

zane-neo commented Jan 17, 2024 •

edited

Loading

ylwu-amzn commented Jan 17, 2024

ylwu-amzn Feb 7, 2024

Fix long pending issue when deleting model #1882

Fix long pending issue when deleting model #1882

Conversation

zane-neo commented Jan 17, 2024 • edited Loading

Description

Issues Resolved

Check List

ylwu-amzn commented Jan 17, 2024

ylwu-amzn Feb 7, 2024

Choose a reason for hiding this comment

zane-neo commented Jan 17, 2024 •

edited

Loading