[BUG] Tag Mismatch error on VisualizationsToolIT.testVisualizationFound Windows Test

[brianf-aws]

What is the bug?

ml-commons/plugin/src/test/java/org/opensearch/ml/tools/VisualizationsToolIT.java

Lines 59 to 66 in 85d0c9e

	public void testVisualizationNotFound() throws IOException, ParseException {
	String requestBody = "{\"parameters\":{\"question\":\"can you show me RAM info with visualization?\"}}";
	Response response = TestHelper
	.makeRequest(client(), "POST", "/_plugins/_ml/agents/" + agentId + "/_execute", null, requestBody, null);
	String responseStr = TestHelper.httpEntityToString(response.getEntity());
	String toolOutput = extractAdditionalInfo(responseStr);
	Assert.assertEquals("No Visualization found", toolOutput);
	}

There is a retry enabled on the VisualizationsToolIT.testVisualizationFound test but it seems that retry has a bit of a flaw if the underlying problem is different I am seeing that the problem here is a encryption issue. This might be the source of all of our flaky tests

VisualizationsToolIT > testVisualizationFound FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:54529/], URI [/_plugins/_ml/agents/bLjaRJQB515KRnslfdUv/_execute], status line [HTTP/1.1 500 Internal Server Error]
    {"status":500,"error":{"type":"AEADBadTagException","reason":"System Error","details":"Tag mismatch"}}
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
        at app//org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound(VisualizationsToolIT.java:74)

    java.lang.AssertionError: The response failed to meet condition after 5 attempts. Attempted to perform GET : /_plugins/_ml/models/arjaRJQB515KRnsleNWv
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.waitResponseMeetingCondition(ToolIntegrationWithLLMTest.java:103)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.checkForModelUndeployedStatus(ToolIntegrationWithLLMTest.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:74)
        at

... 

2> REPRODUCE WITH: gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound" -Dtests.seed=AD7A0603B7C68274 -Dtests.security.manager=false -Dtests.locale=fr-GN -Dtests.timezone=America/Argentina/Buenos_Aires -Druntime.java=21
  2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:54529/], URI [/_plugins/_ml/agents/bLjaRJQB515KRnslfdUv/_execute], status line [HTTP/1.1 500 Internal Server Error]
    {"status":500,"error":{"type":"AEADBadTagException","reason":"System Error","details":"Tag mismatch"}}
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
        at app//org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound(VisualizationsToolIT.java:74)

    java.lang.AssertionError: The response failed to meet condition after 5 attempts. Attempted to perform GET : /_plugins/_ml/models/arjaRJQB515KRnsleNWv
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.waitResponseMeetingCondition(ToolIntegrationWithLLMTest.java:103)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.checkForModelUndeployedStatus(ToolIntegrationWithLLMTest.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:74)
        at

How can one reproduce the bug?
This was discovered in a build failure.

What is the expected behavior?
This test should pass or timeout but not have this encryption issue.

[mingshl]

seeing VisualizationIT failing again but with different error. #3353

[brianf-aws]

seeing VisualizationIT failing again but with different error. #3353

Hmm This is confusing I would think that the retry would help but like you said here it didnt help. Its clearly failing even when the retries are according to how many nodes there are. If only there was some way to dump all possible info and configuration when this happens

[brianf-aws]

Hey @Hailong-am do you mind taking a look? Thanks

[krisfreedain]

Catch All Triage - 1, 2, 3

[Hailong-am]

Hey @Hailong-am do you mind taking a look? Thanks

do you the link or the logs for this failure?

[brianf-aws]

Hey Hailong, we are trying to get to paste the stack traces with reproduction line too. Thankfully this build failure log didn't expire. Can you take a look?

[brianf-aws]

Adding the log here in txt format so it doesn't expire

6_Build and Test MLCommons Plugin on linux (21).txt

Here is another example of another build failure.
Linking the txt file here as well to make sure it does not expire.

7_Build and Test MLCommons Plugin on linux (21).txt