Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FlakyTest] org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow #1035

Open
heemin32 opened this issue Dec 18, 2024 · 3 comments · Fixed by #1063
Open

[FlakyTest] org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow #1035

heemin32 opened this issue Dec 18, 2024 · 3 comments · Fixed by #1063
Assignees
@will-hwang
Labels
bug Something isn't working

Comments

@heemin32
Copy link
Collaborator

heemin32 commented Dec 18, 2024 •
edited
Loading 

REPRODUCE WITH: gradlew ':qa:rolling-upgrade:testAgainstTwoThirdsUpgradedCluster' --tests "org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow" -Dtests.seed=3D4E586DC98C0331 -Dtests.security.manager=false -Dtests.bwc.version=2.12.0 -Dtests.locale=ja-JP -Dtests.timezone=Australia/South -Druntime.java=17
org.opensearch.neuralsearch.bwc.SemanticSearchIT > testSemanticSearch_E2EFlow FAILED
    org.opensearch.client.ResponseException: method [GET], host [http://[::1]:61211], URI [/_cluster/health?wait_for_nodes=3&timeout=60s&wait_for_status=green&cluster_manager_timeout=60s], status line [HTTP/1.1 408 Request Timeout]


Suite: Test class org.opensearch.neuralsearch.bwc.SemanticSearchIT
  2> REPRODUCE WITH: gradlew ':qa:rolling-upgrade:testAgainstTwoThirdsUpgradedCluster' --tests "org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow" -Dtests.seed=3D4E586DC98C0331 -Dtests.security.manager=false -Dtests.bwc.version=2.12.0 -Dtests.locale=ja-JP -Dtests.timezone=Australia/South -Druntime.java=17
  2> org.opensearch.client.ResponseException: method [GET], host [http://[::1]:61211], URI [/_cluster/health?wait_for_nodes=3&timeout=60s&wait_for_status=green&cluster_manager_timeout=60s], status line [HTTP/1.1 408 Request Timeout]
    {"cluster_name":"neuralSearchBwcCluster-rolling","status":"yellow","timed_out":true,"number_of_nodes":3,"number_of_data_nodes":3,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":19,"active_shards":38,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":97.43589743589743}
        at __randomizedtesting.SeedInfo.seed([3D4E586DC98C0331:8B5CF7F56E9F9CF6]:0)
    {"cluster_name":"neuralSearchBwcCluster-rolling","status":"yellow","timed_out":true,"number_of_nodes":3,"number_of_data_nodes":3,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":19,"active_shards":38,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":97.43589743589743}
        at __randomizedtesting.SeedInfo.seed([3D4E586DC98C0331:8B5CF7F56E9F9CF6]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.waitForClusterHealthGreen(BaseNeuralSearchIT.java:1286)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow(SemanticSearchIT.java:28)
  1> [2024-12-19T07:00:37,819][INFO ][o.o.n.b.SemanticSearchIT ] [testSemanticSearch_E2EFlow] before test
  1> [2024-12-19T07:00:37,819][INFO ][o.o.n.b.SemanticSearchIT ] [testSemanticSearch_E2EFlow] initializing REST clients against [http://127.0.0.1:60984, http://[::1]:60985, http://127.0.0.1:61210, http://[::1]:61211, http://[::1]:60760, http://127.0.0.1:60761]
  1> [2024-12-19T07:01:38,211][INFO ][o.o.n.b.SemanticSearchIT ] [testSemanticSearch_E2EFlow] after test
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.waitForClusterHealthGreen(BaseNeuralSearchIT.java:1286)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow(SemanticSearchIT.java:28)
  2> NOTE: leaving temporary files on disk at: D:\a\neural-search\neural-search\qa\rolling-upgrade\build\testrun\testAgainstTwoThirdsUpgradedCluster\temp\org.opensearch.neuralsearch.bwc.SemanticSearchIT_3D4E586DC98C0331-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=1042, maxMBSortInHeap=5.5[425](https://github.com/opensearch-project/neural-search/actions/runs/12398942653/job/34618623713?pr=1028#step:4:426)43137940153, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ja-JP, timezone=Australia/South
  2> NOTE: Windows Server 2022 10.0 amd64/Azul Systems, Inc. 17.0.13 (64-bit)/cpus=4,threads=2,free=269216192,total=536870912
  2> NOTE: All tests run in this JVM: [BatchIngestionIT, HybridSearchIT, HybridSearchWithRescoreIT, KnnRadialSearchIT, MultiModalSearchIT, NeuralQueryEnricherProcessorIT, NeuralSparseSearchIT, NeuralSparseTwoPhaseProcessorIT, SemanticSearchIT]

Case 1

ERROR][o.o.m.a.MLModelAutoReDeployer] [neuralSearchBwcCluster-rolling-1] Exception occurred when auto redeploying the model, model id is: UF3OSJQBUk6b2XcQnjx2, exception is: no eligible node found, skipping current model auto redeploy and starting next model redeploy!
»  java.lang.IllegalArgumentException: no eligible node found
»  	at org.opensearch.ml.action.deploy.TransportDeployModelAction.deployModel(TransportDeployModelAction.java:248) [opensearch-ml-2.14.0.0.jar:2.14.0.0]
»  	at org.opensearch.ml.action.deploy.TransportDeployModelAction.lambda$doExecute$4(TransportDeployModelAction.java:152) [opensearch-ml-2.14.0.0.jar:2.14.0.0]
»  	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.14.0.jar:2.14.0]
»  	at org.opensearch.ml.model.MLModelManager.lambda$getModel$69(MLModelManager.java:1572) [opensearch-ml-2.14.0.0.jar:2.14.0.0]
»  	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:420) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:412) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:167) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:119) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:108) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:100) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:784) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.forwardFragments(NativeInboundBytesHandler.java:157) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.doHandleBytes(NativeInboundBytesHandler.java:94) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:143) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:119) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.14.0.jar:2.14.0]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.109.Final.jar:4.1.109.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
»   ↑ repeated 2 times ↑
» ERROR][o.o.m.a.MLModelAutoReDeployer] [neuralSearchBwcCluster-rolling-1] Exception occurred when auto redeploying the model, model id is: hkrOSJQBLic8uyMcsizY, exception is: no eligible node found, skipping current model auto redeploy and starting next model redeploy!
»  java.lang.IllegalArgumentException: no eligible node found
»  	at org.opensearch.ml.action.deploy.TransportDeployModelAction.deployModel(TransportDeployModelAction.java:248) [opensearch-ml-2.14.0.0.jar:2.14.0.0]
»  	at org.opensearch.ml.action.deploy.TransportDeployModelAction.lambda$doExecute$4(TransportDeployModelAction.java:152) [opensearch-ml-2.14.0.0.jar:2.14.0.0]
»  	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.14.0.jar:2.14.0]
»  	at org.opensearch.ml.model.MLModelManager.lambda$getModel$69(MLModelManager.java:1572) [opensearch-ml-2.14.0.0.jar:2.14.0.0]
»  	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:420) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:412) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:167) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:119) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:108) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:100) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:784) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.forwardFragments(NativeInboundBytesHandler.java:157) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.doHandleBytes(NativeInboundBytesHandler.java:94) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:143) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:119) [opensearch-2.14.0.jar:2.14.0]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.14.0.jar:2.14.0]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:[919](https://github.com/opensearch-project/neural-search/actions/runs/12682096283/job/35347320319?pr=1081#step:4:920)) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.109.Final.jar:4.1.109.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.109.Final.jar:4.1.109.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
»   ↑ repeated 2 times ↑

Case 2

Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.task.MLTaskCache.getMlTask()" because "taskCache" is null
»  	at org.opensearch.ml.task.MLTaskManager.remove(MLTaskManager.java:141) ~[?:?]
»  	at org.opensearch.ml.task.MLTaskManager.updateMLTask(MLTaskManager.java:296) ~[?:?]
»  	at org.opensearch.ml.task.MLTaskManager.updateMLTask(MLTaskManager.java:276) ~[?:?]
»  	at org.opensearch.ml.action.forward.TransportForwardAction.doExecute(TransportForwardAction.java:150) ~[?:?]
»  	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) ~[opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:133) ~[opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:129) ~[opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.17.0.jar:2.17.0]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.17.0.jar:2.17.0]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) [netty-common-4.1.112.Final.jar:4.1.112.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.112.Final.jar:4.1.112.Final]
»  	at java.lang.Thread.run(Thread.java:1583) [?:?]
» ERROR][o.o.m.m.MLModelManager   ] [neuralSearchBwcCluster-rolling-0] No controller is deployed because the model 6oMrSZQB_veoRncfWHHV is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected.
@heemin32 heemin32 added bug Something isn't working untriaged labels Dec 18, 2024
@martin-gaievski
Copy link
Member

Is this for main or 2.x? Earlier I opened the issue for 2.x #981, BWC in there are especially unstable, need to restart CIO action 3-4 times for all tests to pass.

@heemin32
Copy link
Collaborator Author

2.x. Seem like duplicate ticket then..

@martin-gaievski martin-gaievski mentioned this issue Dec 24, 2024
Open
@heemin32 heemin32 moved this to Backlog in Neural Search RoadMap Dec 26, 2024
@heemin32 heemin32 mentioned this issue Dec 30, 2024
Merged
5 tasks
@will-hwang
Copy link
Contributor

assign this bug to me please

@heemin32 heemin32 moved this from Backlog to Backlog(Hot) in Neural Search RoadMap Jan 3, 2025
@heemin32 heemin32 moved this from Backlog(Hot) to 2.19 in Neural Search RoadMap Jan 3, 2025
This was referenced Jan 5, 2025
Closed
Merged
@heemin32 heemin32 mentioned this issue Jan 6, 2025
Closed
@github-project-automation github-project-automation bot moved this from 2.19 to Done in Neural Search RoadMap Jan 6, 2025
@heemin32 heemin32 reopened this Jan 6, 2025
@heemin32 heemin32 moved this from Done to 2.19 in Neural Search RoadMap Jan 7, 2025
@minalsha minalsha removed the untriaged label Jan 8, 2025
@heemin32 heemin32 mentioned this issue Jan 9, 2025
Open
@heemin32 heemin32 moved this from 2.19 to Backlog(Hot) in Neural Search RoadMap Jan 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@will-hwang will-hwang

Labels
bug Something isn't working
Projects
Neural Search RoadMap
Status: Backlog(Hot)
Milestone
No milestone
4 participants
@heemin32 @will-hwang @minalsha @martin-gaievski