Package removal in pdata protogen breaks upstream modules

[eliottness]

Describe the bug

This issue happens on modules containing dependencies on the module go.opentelemetry.io/collector/pdata and but not yet on the module go.opentelemetry.io/collector/pdata/pprofile. Then by running go get -u ./..., module go.opentelemetry.io/collector/pdata/pprofile ends up in the new dependency closure.

Effectively, what this setup does is upgrading go.opentelemetry.io/collector/pdata to latest and not upgrading go.opentelemetry.io/collector/pdata/pprofile. But since go.opentelemetry.io/collector/pdata/pprofile depends on the package go.opentelemetry.io/collector/pdata/internal/protogen/profiles/v1experimental, which was deleted in go.opentelemetry.io/collector/[email protected], running go mod tidy creates this error:

go: finding module for package go.opentelemetry.io/collector/pdata/internal/data/protogen/profiles/v1experimental
go: test imports
	github.com/DataDog/datadog-agent/pkg/trace/stats imports
	github.com/DataDog/datadog-agent/pkg/trace/transform imports
	github.com/DataDog/datadog-agent/pkg/trace/sampler tested by
	github.com/DataDog/datadog-agent/pkg/trace/sampler.test imports
	go.opentelemetry.io/collector/processor/processortest imports
	go.opentelemetry.io/collector/pdata/testdata imports
	go.opentelemetry.io/collector/pdata/pprofile imports
	go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/profiles/v1experimental: module go.opentelemetry.io/collector/pdata@latest found (v1.22.0), but does not contain package go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/profiles/v1experimental

Steps to reproduce

It is difficult to reproduce independently from our code because it involves multiple modules. In the end I believe it is both an issue from the collector and also an issue from the go package version resolver not upgrading pdata/pprofile alongside pdata. Anyhow, here is a small reproducer:

Here is a go.mod

module github.com/DataDog/orchestrion.testing

go 1.23.1

require github.com/DataDog/datadog-agent/pkg/trace v0.58.0

here is a sample main.go:

package main

import _ "github.com/DataDog/datadog-agent/pkg/trace/stats"

func main() {}

Running go get -u . and go mod tidy will create the error mentioned above.

What did you expect to see?

As this is a minor release of go.opentelemetry.io/collector/pdata (v1.21.0 -> v1.22.0) I would expect to be no breaking change in it. I understand that the deleted package is an internal package but maybe the current setup between pdata and pdata/pprofile is not adequate because one module should not depends on internal packages of another module.

What did you see instead?

I see the error message described above ☝️. Running go get -u is very common in the Go ecosystem because semver is widely known and respected. This bug is very insidious because it causes issues in transitive upstream code until a module breaks the chain. Especially since fixing seems non-trivial for libraries in the dependency chain that dont' want to do a go get -u themselves to upgrade their own dependencies. (cf. DataDog/dd-trace-go#3059)

Workaround

Since pdata/pprofile is not in the initial dependency closure running go get [...]/pdata/pprofile@latest does nothing. So we haved to add it manually like this PR does by adding an blank import to it. This will make go get -u upgrade both modules past their respective breaking point for upstream modules.

[dmathieu]

It looks like this would only break when using non-matching versions of pdata/testdata and/or pdata/pprofile, which are both unstable.
When using only pdata, there's no reason for this to break, since pdata/internal/data/protogen/collector/profiles/v1experimental isn't used.

So while I agree this isn't a great experience, it actually seems like an issue in unstable packages, not the stable pdata.
You need to keep both packages in version sync.