Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: panic.go:770: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | runtime.p... #138701

Open
cockroach-sentry opened this issue Jan 8, 2025 · 2 comments
Open

Sentry: panic.go:770: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | runtime.p... #138701

cockroach-sentry opened this issue Jan 8, 2025 · 2 comments
Labels
A-disaster-recovery branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-disaster-recovery X-blathers-triaged blathers was able to find an owner

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Jan 8, 2025 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6206299620/?referrer=webhooks_plugin

Panic Message:

panic.go:770: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:261
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:881
  | golang.org/x/oauth2/authhandler.authHandlerSource.Token
  | 	golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go:80
  | golang.org/x/oauth2.(*reuseTokenSource).Token
  | 	golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go:307
  | golang.org/x/oauth2.(*Transport).RoundTrip
  | 	golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go:45
  | net/http.send
  | 	GOROOT/src/net/http/client.go:259
  | net/http.(*Client).send
  | 	GOROOT/src/net/http/client.go:180
  | net/http.(*Client).do
  | 	GOROOT/src/net/http/client.go:724
  | net/http.(*Client).Do
  | 	GOROOT/src/net/http/client.go:590
  | cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:834
  | cloud.google.com/go/storage.run.func1
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:67
  | cloud.google.com/go/internal.retry
  | 	cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:40
  | cloud.google.com/go/internal.Retry
  | 	cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:33
  | cloud.google.com/go/storage.run
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:65
  | cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:833
  | cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:887
  | cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go:118
  | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile.func1
  | 	github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:305
  | github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open.func1
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:273
  | github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry.func1
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:143
  | github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:190
  | github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:167
  | github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:189
  | github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:142
  | github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:271
  | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile
  | 	github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:319
  | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).ReadFile
  | 	github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:395
  | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.containsManifest
  | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:68
  | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.ResolveDest
  | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:166
  | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*backupResumer).Resume
  | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_job.go:631
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
  | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1627
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:770: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/jobs/registry.go

Lines 1626 to 1628 in beddec1

}()
err = resumer.Resume(resumeCtx, execCtx)
}()

cockroach/pkg/ccl/backupccl/backup_job.go

Lines 630 to 632 in beddec1

var err error
backupDest, err = backupdest.ResolveDest(ctx, p.User(), details.Destination, details.EndTime,
details.IncrementalFrom, p.ExecCfg())

cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go

Lines 165 to 167 in beddec1

defer defaultStore.Close()
exists, err := containsManifest(ctx, defaultStore)
if err != nil {

cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go

Lines 67 to 69 in beddec1

func containsManifest(ctx context.Context, exportStore cloud.ExternalStorage) (bool, error) {
r, _, err := exportStore.ReadFile(ctx, backupbase.BackupManifestName, cloud.ReadOptions{NoFileSize: true})
if err != nil {

cockroach/pkg/cloud/impl_registry.go

Lines 394 to 396 in beddec1

}
r, s, err := e.ExternalStorage.ReadFile(ctx, basename, opts)
if err != nil {

cockroach/pkg/cloud/gcp/gcs_storage.go

Lines 318 to 320 in beddec1

if err := r.Open(ctx); err != nil {
if errors.Is(err, gcs.ErrObjectNotExist) {

cockroach/pkg/cloud/cloud_io.go

Lines 270 to 272 in beddec1

return DelayedRetry(ctx, "Open", r.ErrFn, func() error {
var readErr error

cockroach/pkg/cloud/cloud_io.go

Lines 141 to 143 in beddec1

return retry.WithMaxAttempts(ctx, base.DefaultRetryOptions(), MaxDelayedRetryAttempts, func() error {
err := fn()

cockroach/pkg/util/retry/retry.go

Lines 188 to 190 in beddec1

opts.MaxRetries = n - 1 // >= 1
return opts.Do(ctx, func(ctx context.Context) error {
return fn()

cockroach/pkg/util/retry/retry.go

Lines 166 to 168 in beddec1

for r := StartWithCtx(ctx, opts); r.Next(); {
err = fn(ctx)
if err == nil {

cockroach/pkg/util/retry/retry.go

Lines 189 to 191 in beddec1

return opts.Do(ctx, func(ctx context.Context) error {
return fn()
})

cockroach/pkg/cloud/cloud_io.go

Lines 142 to 144 in beddec1

return retry.WithMaxAttempts(ctx, base.DefaultRetryOptions(), MaxDelayedRetryAttempts, func() error {
err := fn()
if err == nil {

cockroach/pkg/cloud/cloud_io.go

Lines 272 to 274 in beddec1

var readErr error
r.Reader, r.Size, readErr = r.Opener(ctx, r.Pos)
if readErr != nil {

cockroach/pkg/cloud/gcp/gcs_storage.go

Lines 304 to 306 in beddec1

}
r, err := g.bucket.Object(object).NewRangeReader(ctx, pos, length)
if err != nil {

cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go#L117-L119
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L886-L888
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L832-L834
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L64-L66
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L32-L34
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L39-L41
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L66-L68
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L833-L835
GOROOT/src/net/http/client.go#L589-L591
GOROOT/src/net/http/client.go#L723-L725
GOROOT/src/net/http/client.go#L179-L181
GOROOT/src/net/http/client.go#L258-L260
golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go#L44-L46
golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go#L306-L308
golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81
GOROOT/src/runtime/signal_unix.go#L880-L882
GOROOT/src/runtime/panic.go#L260-L262
GOROOT/src/runtime/panic.go#L769-L771

pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1627
pkg/ccl/backupccl/backup_job.go in pkg/ccl/backupccl.(*backupResumer).Resume at line 631
pkg/ccl/backupccl/backupdest/backup_destination.go in pkg/ccl/backupccl/backupdest.ResolveDest at line 166
pkg/ccl/backupccl/backupdest/backup_destination.go in pkg/ccl/backupccl/backupdest.containsManifest at line 68
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).ReadFile at line 395
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile at line 319
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open at line 271
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry at line 142
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts at line 189
pkg/util/retry/retry.go in pkg/util/retry.Options.Do at line 167
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts.func1 at line 190
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry.func1 at line 143
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open.func1 at line 273
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile.func1 at line 305
cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go in cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader at line 118
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader at line 887
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2 at line 833
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run at line 65
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.Retry at line 33
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.retry at line 40
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run.func1 at line 67
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1 at line 834
GOROOT/src/net/http/client.go in net/http.(*Client).Do at line 590
GOROOT/src/net/http/client.go in net/http.(*Client).do at line 724
GOROOT/src/net/http/client.go in net/http.(*Client).send at line 180
GOROOT/src/net/http/client.go in net/http.send at line 259
golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go in golang.org/x/oauth2.(*Transport).RoundTrip at line 45
golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go in golang.org/x/oauth2.(*reuseTokenSource).Token at line 307
golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go in golang.org/x/oauth2/authhandler.authHandlerSource.Token at line 80
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 881
GOROOT/src/runtime/panic.go in runtime.panicmem at line 261
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770

Tags

Tag Value
Command mt start-sql
Environment v24.2.5
Go Version go1.22.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.2.5
Cockroach SHA beddec1
# of CPUs 16
# of Goroutines 301

Jira issue: CRDB-46304
@cockroach-sentry cockroach-sentry added branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jan 8, 2025
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Jan 8, 2025

CC'ing via the CODEOWNERS-based sentry heuristic:

  • @cockroachdb/disaster-recovery

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added T-disaster-recovery X-blathers-triaged blathers was able to find an owner labels Jan 8, 2025
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Jan 8, 2025

cc @cockroachdb/disaster-recovery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-disaster-recovery branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-disaster-recovery X-blathers-triaged blathers was able to find an owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-sentry