Fix model not deploy issue under intensive prediction tasks

[zane-neo]

Description

Under intensive prediction tasks, the model not deployed error can happen occasionally like below:

[ERROR] Cannot execute-test. Error in load generator [0]
        Cannot run task [bulk]: Request returned an error. Error type: bulk, Description: HTTP status: 400, message: Model not ready yet. Please run this first: POST /_plugins/_ml/models/Cc36M40Bxy1iWleaz3Bl/_deploy

The reason is the SyncUpJob cleans timed out deploying model tasks from all tasks not filtering prediction tasks, and a prediction task can be removed from cache when prediction is done. And the corresponding taskCache is cleared, the syncUpJob will encounter NPE and return error response for gatherInfoRequest, and in this case, SyncUpJob sends clearRoutingTable request and clear all routingTable info, the next prediction task will see no model in the cache and throw model needs deploy exception.

Issues Resolved

NA

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (54c788a) 82.61% compared to head (9d7b9dc) 82.63%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
.../java/org/opensearch/ml/utils/RestActionUtils.java	75.00%	4 Missing ⚠️
.../ml/action/syncup/TransportSyncUpOnNodeAction.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1903      +/-   ##
============================================
+ Coverage     82.61%   82.63%   +0.01%     
- Complexity     5383     5388       +5     
============================================
  Files           521      521              
  Lines         21715    21727      +12     
  Branches       2210     2212       +2     
============================================
+ Hits          17940    17954      +14     
+ Misses         2878     2872       -6     
- Partials        897      901       +4     

Flag	Coverage Δ
ml-commons	82.63% <68.42%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.