[BUG]-(flaky tests) ITs involving models have race conditions

[brianf-aws]

What is the bug?
There are failures on ITs that depend on a creating a model. This usually involves creating a model (Failed to find model) and when the model tries to be deleted but it can't because the model is in deploying or deployed state (Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete).

How can one reproduce the bug?
Steps to reproduce the behavior:

./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound" -Dtests.seed=4E5188680CFAD2E9 -Dtests.security.manager=false -Dtests.locale=fr-ML -Dtests.timezone=Africa/Mbabane -Druntime.java=21

VisualizationsToolIT > testVisualizationFound FAILED
    org.opensearch.client.ResponseException: method [DELETE], host [http://[::1]:35467], URI [/_plugins/_ml/models/p7RlYZMB7jpbSjye2g3j], status line [HTTP/1.1 400 Bad Request]
    {"error":{"root_cause":[{"type":"status_exception","reason":"Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete"}],"type":"status_exception","reason":"Model cannot be deleted in deploying or deployed state. Try undeploy model first then delete"},"status":400}
        at __randomizedtesting.SeedInfo.seed([4E5188680CFAD2E9:84740AFD7158DDDD]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
        at app//org.opensearch.ml.rest.MLCommonsRestTestCase.deleteModel(MLCommonsRestTestCase.java:654)
        at app//org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:75)
        at [email protected]/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)

This is likely found here

ml-commons/plugin/src/test/java/org/opensearch/ml/tools/ToolIntegrationWithLLMTest.java

Lines 63 to 65 in c84c947

	public void deleteModel() throws IOException {
	undeployModel(modelId);
	deleteModel(client(), modelId, null);

./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLGuardrailsIT.testPredictRemoteModelFailedWithModelGuardrail" -Dtests.seed=4E5188680CFAD2E9 -Dtests.security.manager=false -Dtests.locale=mdf-RU -Dtests.timezone=America/Goose_Bay -Druntime.java=21

RestMLGuardrailsIT > testPredictRemoteModelFailedWithModelGuardrail FAILED
    java.lang.AssertionError: 
    Expected: (an instance of org.opensearch.client.ResponseException and exception with message a string containing "guardrails triggered for user input")
         but: exception with message a string containing "guardrails triggered for user input" message was "method [POST], host [http://[::1]:45733], URI [/_plugins/_ml/models/null/_deploy], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"status_exception","reason":"Failed to find model"}],"type":"status_exception","reason":"Failed to find model"},"status":404}"
    Stacktrace was: org.opensearch.client.ResponseException: method [POST], host [http://[::1]:45733], URI [/_plugins/_ml/models/null/_deploy], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"status_exception","reason":"Failed to find model"}],"type":"status_exception","reason":"Failed to find model"},"status":404}
    	at org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
    	at org.opensearch.client.RestClient.performRequest(RestClient.java:384)
    	at org.opensearch.client.RestClient.performRequest(RestClient.java:359)
    	at org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
    	at org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
    	at org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
    	at org.opensearch.ml.rest.RestMLGuardrailsIT.deployRemoteModel(RestMLGuardrailsIT.java:582)
    	at org.opensearch.ml.rest.RestMLGuardrailsIT.testPredictRemoteModelFailedWithModelGuardrail(RestMLGuardrailsIT.java:318)
    	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)

Which is found here

ml-commons/plugin/src/test/java/org/opensearch/ml/rest/RestMLGuardrailsIT.java

Lines 581 to 583 in ab2fae3

	protected Response deployRemoteModel(String modelId) throws IOException {
	return TestHelper.makeRequest(client(), "POST", "/_plugins/_ml/models/" + modelId + "/_deploy", null, "", null);
	}

Version

OS:2.18

I suspect there is some sort of race condition. I am noticing there is no retry involved nor is there a waitFor involved for creating this resource.

This is also similar to #3228

[brianf-aws]

Can I get assigned this? I noticed that this PR #3244 which has a dependency with this issue. I was trying to understand it this morning.

[brianf-aws]

There is a backport (2.x) PR for this specific issue #3263