Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestDiscoverIntentAcrossLeaseTransferAwayAndBack failed #122102

Closed
cockroach-teamcity opened this issue Apr 10, 2024 · 3 comments
Closed

kv/kvserver: TestDiscoverIntentAcrossLeaseTransferAwayAndBack failed #122102

cockroach-teamcity opened this issue Apr 10, 2024 · 3 comments
Labels
A-testing Testing tools and infrastructure branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-kv KV Team X-stale
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 10, 2024 •
edited by cockroach-jira-scripts
Loading

kv/kvserver.TestDiscoverIntentAcrossLeaseTransferAwayAndBack failed on release-24.1 @ 87740bd080658e93b40637556c9dfa27a91f8cf5:

Fatal error:

panic: test timed out after 1h16m37s
running tests:
	TestDiscoverIntentAcrossLeaseTransferAwayAndBack (1h16m14s)

Stack:

goroutine 390566 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45
Log preceding fatal error 

=== RUN   TestDiscoverIntentAcrossLeaseTransferAwayAndBack
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDiscoverIntentAcrossLeaseTransferAwayAndBack4234901910
    test_log_scope.go:81: use -show-logs to present logs inline

Parameters:

  • attempt=1
  • race=true
  • run=3
  • shard=16
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-37704
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Apr 10, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Apr 10, 2024
@nvanbenschoten
Copy link
Member

The test is stuck here on the <-unblockCh, waiting for the test's main goroutine to close the txn2UnblockC channel:

goroutine 27269 [chan receive, 75 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDiscoverIntentAcrossLeaseTransferAwayAndBack.func2.1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:4580 +0xdd
sync.(*Once).doSlow(0xc0074556e4, 0xc006395f18)
	GOROOT/src/sync/once.go:74 +0xf1
sync.(*Once).Do(0xc0074556e4, 0xc006395f18)
	GOROOT/src/sync/once.go:65 +0x45
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDiscoverIntentAcrossLeaseTransferAwayAndBack.func2({0xc006396198?, 0x1?}, 0xc017b7ca20, 0xc01540b7c0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:4573 +0x15e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc001e1c888, {0xe3efa78, 0xc00c0402d0}, 0xc017b7ca20, 0xa9838f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:540 +0x7d0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc001e1c888, {0xe3efa78, 0xc00c040270}, 0xc017b7ca20)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x565
...
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run(0xc00158ac60, {0xe3ef1f8, 0x13ca5980}, 0xc017b50608)
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:800 +0xa8
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Get(0xc00158ac60, {0xe3ef1f8, 0x13ca5980}, {0x98048a0, 0xc005ead938})
	github.com/cockroachdb/cockroach/pkg/kv/txn.go:508 +0x19c
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDiscoverIntentAcrossLeaseTransferAwayAndBack.func3()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:4613 +0x93
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDiscoverIntentAcrossLeaseTransferAwayAndBack in goroutine 13534
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:4612 +0xbfe

The test's main goroutine who should eventually close the channel is blocked on latches:

goroutine 13534 [select, 75 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).waitForSignal(0xc002f5b980, {0xe3efa78, 0xc005ac5290}, 0xc018867088, 0x1, 0x0, 0x1, 0xc008b92858, 0xc015e6b6f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:588 +0x505
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).iterAndWait(0xc002f5b980, {0xe3efa78, 0xc005ac5290}, 0xc018867088, 0xc018867160, 0x1, 0x0, 0x1, 0xc008b92858, 0xa989a30)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:563 +0x27f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).wait(0xc002f5b980, {0xe3efa78, 0xc005ac5290}, 0xc008b927e0, {{{{0x0, 0x0}, {0xc004e76b40, 0x1}}, {{0x0, 0x0}, ...}}})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:512 +0x53a
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).Acquire(0xc002f5b980, {0xe3efa78, 0xc005ac5290}, 0xc008a64770, 0x1, {0xe39c320, 0xc0086c7c20})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:251 +0x205
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*latchManagerImpl).Acquire(0xc002f5b980, {0xe3efa78, 0xc005ac5290}, {0x0, {0x17c4e6096f6704fb, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/latch_manager.go:29 +0x72
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc0030a92c0, {0xe3efa78, 0xc005ac5290}, 0xc008604410, 0x1)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:296 +0x709
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc0030a92c0, {0xe3efa78, 0xc005ac5290}, 0x0, {0x0, {0x17c4e6096f6704fb, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x332
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc001e1c888, {0xe3efa78, 0xc005ac5290}, 0xc0086c7c20, 0xa9838f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x517
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc001e1c888, {0xe3efa78, 0xc005ac5260}, 0xc0086c7c20)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x565
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc005676c08, {0xe3efa78, 0xc005ac5230}, 0xc0086c7c20)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd12
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc004d33ef0, {0xe3efa78, 0xc005ac5230}, 0xc0086c7c20)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0xdf
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc004730008, {0xe3efa78, 0xc005ac5170}, {0xc00b901980?}, 0xc0086c7c20)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1431 +0x931
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc004730008, {0xe3efa78, 0xc005ac5080}, 0xc0086c7c20)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1631 +0x4a5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20}, 0xc00baf16e0, 0xc005bfc240)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x72c
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20}, 0xc00baf16e0, 0xc00baf1700)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf1
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc002926000?, {{0xc009b89068?}, {0xe42eea0?, 0xc00701c930?}}}, {0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20}, 0xc00baf16e0, 0xc00baf1720)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x2b7
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xe3efa78, 0xc005ac5080})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc009b89040, {0xe3efa78, 0xc005ac5080}, {0x1?, 0x4c6360?}, 0xc01886a468)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0x186
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20}, 0xc00baf16e0, 0xc00baf1740)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x151
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0xe3efa78, 0xc005ac5080}, {0xa477560, 0xc0086c7c20})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x89
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0xe3efa78, 0xc005ac5080}, {0x1?, 0x1?}, {0xa477560, 0xc0086c7c20}, {0xa2f1aa0, 0xc008cb3580}, 0xc01886a608?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0xe3efa78, 0xc005ac5080}, {0xa5cb747, 0x21}, {0xa477560, 0xc0086c7c20}, {0xa2f1aa0, 0xc008cb3580}, 0x0, 0xc0089a71c0, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x5ee
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0xe3efa78, 0xc005ac5080}, {0xa5cb747, 0x21}, {0xa477560, 0xc0086c7c20}, {0xa2f1aa0, 0xc008cb3580}, 0x0, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x19a
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0xe3efa40, 0xc00bfa2af0}, 0xc0086c79e0, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x463
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc005e04c00, {0xe3efa40, 0xc00bfa2af0}, 0x1352938?, {0xe3d00b0, 0xc002e52cc0}, 0xc0086c79e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214 +0x283
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc005e04c00, {0xe3efa40, 0xc00bfa2af0}, 0xc0086c79e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192 +0x13a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xe43c890?, 0xc005e04c00?}}, {0xe3efa40, 0xc00bfa2af0}, 0xc0086c79e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x2ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc00eb58c08, {0xe3efa78, 0xc005ac5020}, 0xc0086c7560, {0xc00bc7ec00, 0xc003150270}, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2739 +0x36a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc00eb58c08, {0xe3efa78, 0xc005ac5020}, 0xc0086c7560, {{0xc00977cb48, 0x1, 0x8}, {0xc00977cb48, 0x2, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2080 +0x9c5
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc00eb58c08, {0xe3efa78, 0xc005ac5020}, 0xc0086c7560, {{0xc00977cb48, 0x1, 0x8}, {0xc00977cb48, 0x2, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1615 +0x1031
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc00eb58c08, {0xe3ef1f8, 0x13ca5980}, 0xc0086c7560)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1231 +0xa3a
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc008f7e2e8, {0xe3ef1f8, 0x13ca5980}, 0xc0086c7560)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:229 +0xf3
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc008f7e280, {0xe3ef1f8, 0x13ca5980}, 0xc0086c7560, {0xe39c7a0, 0xc008f7e2e8})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1124 +0x182
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1107
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0xe3ef1f8, 0x13ca5980}, 0xc01886cc80, 0xc005169808)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:956 +0x27e
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc008f7e280, {0xe3ef1f8, 0x13ca5980}, 0xc005169808)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:979 +0xa8
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Get(0xc008f7e280, {0xe3ef1f8, 0x13ca5980}, {0x9873c00, 0xe382fe0})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:354 +0xea
github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDiscoverIntentAcrossLeaseTransferAwayAndBack(0xc00b9729c0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_replica_test.go:4632 +0xdfe

The latches are likely held by the first goroutine, which would create a deadlock and explain the timeout. But why doesn't this usually deadlock? These are two Get requests to different keys, so their latches should not conflict. Perhaps there's an intermediate latch that's creating the transitive dependency.

We do see a lease request (must be a TransferLease) that is also waiting on latches, which would explain this:

goroutine 27554 [select, 75 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).waitForSignal(0xc002f5b980, {0xe3efa78, 0xc001c27920}, 0xc009ece3f0, 0x1, 0x1, 0x0, 0xc015e6b6f8, 0xc017b4e438)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:588 +0x505
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).iterAndWait(0xc002f5b980, {0xe3efa78, 0xc001c27920}, 0xc009ece3f0, 0xc009ece440, 0x1, 0x1, 0x0, 0xc015e6b6f8, 0xa989a28)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:563 +0x27f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).wait(0xc002f5b980, {0xe3efa78, 0xc001c27920}, 0xc015e6b680, {{{{0xc0047ff680, 0x1}, {0x0, 0x0}}, {{0xc00247c360, 0x1}, ...}}})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:530 +0x75c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).Acquire(0xc002f5b980, {0xe3efa78, 0xc001c27920}, 0xc005db1d50, 0x1, {0xe39c320, 0xc0052419e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:251 +0x205
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*latchManagerImpl).Acquire(0xc002f5b980, {0xe3efa78, 0xc001c27920}, {0x0, {0x17c4e6096dcf3bfc, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/latch_manager.go:29 +0x72
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc0030a92c0, {0xe3efa78, 0xc001c27920}, 0xc01235be10, 0x1)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:296 +0x709
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc0030a92c0, {0xe3efa78, 0xc001c27920}, 0x0, {0x0, {0x17c4e6096dcf3bfc, 0x0}, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x332
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc001e1c888, {0xe3efa78, 0xc001c27920}, 0xc0052419e0, 0xa983900)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x517
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc001e1c888, {0xe3efa78, 0xc001c27920}, 0xc0052419e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:186 +0x5d0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc001e1c888, {0xe3efa78, 0xc001c27920}, 0xc0052419e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:121 +0x45
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*pendingLeaseRequest).requestLease(0xc001e1cfe0, {0xe3efa78, 0xc001c27920}, {0x9b89040?, 0xc0?, 0xe3efa40?, 0x0?}, {{{0x0, 0x0}, 0x0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_range_lease.go:614 +0x1025
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*pendingLeaseRequest).requestLeaseAsync.func1({0xe3efa78, 0xc001c27920})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_range_lease.go:459 +0x1be
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 27358
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d

@nvanbenschoten nvanbenschoten added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure P-2 Issues/test failures with a fix SLA of 3 months and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 10, 2024
@andrewbaptist
Copy link
Collaborator

This is interesting that it is failing. I had a problem understanding why this was failing and and removed it from the new proxy behavior in my PR here: #121273 because it was hitting the same issue and I couldn't figure out how to address it. So there is at least a way to make this failure happen much more easily, but I'm not sure I understand exactly why this is failing or if this has become more common in 24.1 due to some changes made.

I'm not going to have time this week to look at this failure, but it would be good to understand it better and to remove the TODO from the linked PR once this is fixed.

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 18, 2024
Closed
@github-actions GitHub Actions
Copy link

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

@github-project-automation github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024
@stevendanna stevendanna mentioned this issue Jan 9, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-testing Testing tools and infrastructure branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). no-test-failure-activity O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-kv KV Team X-stale
Projects
No open projects
KV
Status: roachtest/unit test backlog
Milestone
24.1
Development

No branches or pull requests
3 participants
@nvanbenschoten @cockroach-teamcity @andrewbaptist