DriverTask 20230709_155907_18022_3.1.0.1 is aborted by timeout

[y1065w]

我在centos7里kill -9 tomcat的pid，iotdb-v1.1.0就无法查询。重现几次情况和日志都一样。

2023-07-10 00:00:07,634 [pool-38-IoTDB-ClientRPC-Processor-3] WARN o.a.i.d.u.ErrorHandlingUtils:89 - Status code: 301, Query Statement: "select last r_1633352521974095877 from root.iot.p_e9ca23d68d199095138994094080.d_1633351571632889858 where time < 2023-07-09 23:59:07". executeStatement failed
org.apache.iotdb.commons.exception.IoTDBException: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: DriverTask 20230709_155907_18022_3.1.0.1 is aborted by timeout
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.dealWithException(QueryExecution.java:479)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:462)
at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getByteBufferBatchResult(QueryExecution.java:497)
at org.apache.iotdb.db.utils.QueryDataSetUtils.convertQueryResultByFetchSize(QueryDataSetUtils.java:254)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.lambda$static$0(ClientRPCServiceImpl.java:166)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementInternal(ClientRPCServiceImpl.java:239)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatementV2(ClientRPCServiceImpl.java:480)
at org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeQueryStatementV2(ClientRPCServiceImpl.java:470)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeQueryStatementV2.getResult(IClientRPCService.java:3579)
at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeQueryStatementV2.getResult(IClientRPCService.java:3559)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.iotdb.db.mpp.execution.fragment.FragmentInstanceFailureInfo$FailureException: DriverTask 20230709_155907_18022_3.1.0.1 is aborted by timeout
at org.apache.iotdb.db.mpp.execution.schedule.DriverScheduler.clearDriverTask(DriverScheduler.java:339)
at org.apache.iotdb.db.mpp.execution.schedule.DriverScheduler.access$1000(DriverScheduler.java:63)
at org.apache.iotdb.db.mpp.execution.schedule.DriverScheduler$Scheduler.toAborted(DriverScheduler.java:512)
at org.apache.iotdb.db.mpp.execution.schedule.DriverTaskTimeoutSentinelThread.execute(DriverTaskTimeoutSentinelThread.java:77)
at org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:73)
2023-07-10 00:00:07,642 [pool-38-IoTDB-ClientRPC-Processor-3$20230709_155907_18022_3] INFO o.a.i.d.m.p.Coordinator:221 - Cost: 60511 ms, sql is select last r_1633352521974095877 from root.iot.p_e9ca23d68d199095138994094080.d_1633351571632889858 where time < 2023-07-09 23:59:07
2023-07-10 00:00:08,225 [AsyncDataNodeMPPDataExchangeServiceClientPool-selector-80] INFO o.a.i.c.i.l.IndexController:121 - version file updated, previous: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/5-1528870, current: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/5-1528871
2023-07-10 00:00:08,230 [AsyncDataNodeMPPDataExchangeServiceClientPool-selector-80] INFO o.a.i.c.i.l.IndexController:121 - version file updated, previous: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/4-1528870, current: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/4-1528871
2023-07-10 00:00:09,586 [AsyncDataNodeMPPDataExchangeServiceClientPool-selector-80] INFO o.a.i.c.i.l.IndexController:121 - version file updated, previous: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/5-1528871, current: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/5-1528872
2023-07-10 00:00:10,144 [AsyncDataNodeMPPDataExchangeServiceClientPool-selector-80] INFO o.a.i.c.i.l.IndexController:121 - version file updated, previous: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/4-1528871, current: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/4-1528872
2023-07-10 00:00:11,285 [AsyncDataNodeMPPDataExchangeServiceClientPool-selector-80] INFO o.a.i.c.i.l.IndexController:121 - version file updated, previous: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/4-1528872, current: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/4-1528873
2023-07-10 00:00:11,285 [AsyncDataNodeMPPDataExchangeServiceClientPool-selector-80] INFO o.a.i.c.i.l.IndexController:121 - version file updated, previous: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/5-1528872, current: /work/iotdb-cluster/apache-iotdb-1.1.0-node1/sbin/../data/datanode/consensus/data_region/1_1/5-1528873

log_confignode_all.log
log_datanode_all1.log

[jixuan1989]

@JackieTien97

[JackieTien97]

1. kill tomcat应该跟跟IoTDB是无关的，最多会造成正在执行的查询需要等到超时才会被终止，你的意思是，kill了tomcat，IoTDB就无法查询？在无法查询的时候，能jstack -l DataNode进程的pid > stack.log，然后贴一下stack.log

[JackieTien97]

You can also try to upgrade to IoTDB V1.1.2, that version has fixed lots of query resource leak which may cause all the following queries never finish successfully.

[Gitboy-litter]

发现再1.1.2版本也会出现这个问题，而且再数据量不大的情况下写入和读取之后就偶发性的出现这个问题了

[stefanie6]

光正常查询都老出这个错误