cmd/go: go get -u for package with submodule vendor: package <VENDOR PACKAGE>: <GIT DIR> exists but is not a directory

[tsaikd]

What version of Go are you using (go version)?

go version devel +9a97c3b Mon Oct 17 17:05:39 2016 +0000 linux/amd64

• In fact, after 9a97c3b will cause this issue, even in the latest ef8e85e [Tue Oct 18 16:38:54 2016 -0700]

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/go-src"
GOTOOLDIR="/go-src/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build210141904=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

go get -u -v github.com/tsaikd/go-get-vendor-submodule-test

What did you expect to see?

response without error message

What did you see instead?

github.com/tsaikd/go-get-vendor-submodule-test (download)
github.com/tsaikd/go-get-vendor-submodule-test/vendor/github.com/golang/example (download)
package github.com/tsaikd/go-get-vendor-submodule-test/vendor/github.com/golang/example/stringutil: /go/src/github.com/tsaikd/go-get-vendor-submodule-test/vendor/github.com/golang/example/.git exists but is not a directory

---

No problem with 9a97c3b patch it self, but not completely. In src/cmd/go/get.go

go/src/cmd/go/get.go

Lines 419 to 423 in 9a97c3b

	meta := filepath.Join(root, "."+vcs.cmd)
	st, err := os.Stat(meta)
	if err == nil && !st.IsDir() {
	return fmt.Errorf("%s exists but is not a directory", meta)
	}

func downloadPackage(p *Package) error {
    ...

    meta := filepath.Join(root, "."+vcs.cmd)
    st, err := os.Stat(meta)
    if err == nil && !st.IsDir() {
        return fmt.Errorf("%s exists but is not a directory", meta)
    }

    ...

Should it must to be a directory for meta ?

• Maybe related to cmd/go: go get -u: panic: runtime error: slice bounds out of range #16167

[quentinmit]

@rsc since you just fixed the other half of this

[akerl]

Just ran into this. Is there a workaround, since it looks like 1.8 was released w/o a fix?

[evantbyrne]

@akerl, Ended up getting around this for one of my repositories that stopped working on upgrading to 1.8 by using git submodules to vendor dependencies and avoiding go get entirely. Installing from source is then a matter of cloning with the --recursive flag and then running go install. Ideally it could be done through go get like before, but using git should work in the meantime.

[gopherbot]

CL https://golang.org/cl/49391 mentions this issue.

[ldemailly]

I'm seeing something very similar too - not sure I quite understand:

$ docker run -ti golang:1.8
root@ea4f09bd961f:/go# go get -u istio.io/fortio
package istio.io/fortio/vendor/golang.org/x/net/context: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc/health: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc/health/grpc_health_v1: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc/reflection: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/github.com/golang/protobuf/proto: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git

note it works fine without -u:

root@ea4f09bd961f:/go# go get istio.io/fortio
root@ea4f09bd961f:/go# 

istio.io/fortio maps to github.com/istio/fortio
vendor in fortio is a submodule

ps: same with go 1.10rc2:

Status: Downloaded newer image for golang:1.10rc2
root@d7edc8d7cfda:/go# go get -u istio.io/fortio
package istio.io/fortio/vendor/golang.org/x/net/context: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc/health: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc/health/grpc_health_v1: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/google.golang.org/grpc/reflection: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git
package istio.io/fortio/vendor/github.com/golang/protobuf/proto: istio.io/fortio is a custom import path for https://github.com/istio/fortio, but /go/src/istio.io/fortio/vendor is checked out from https://github.com/istio-releases/fortio-vendor.git

[ldemailly]

after fixing the vanity url to add something for vendor (which shouldn't be needed... imo):

     <meta name="go-import" content="istio.io/fortio/vendor git https://github.com/istio-releases/fortio-vendor">
     <meta name="go-source" content="istio.io/fortio/vendor     https://github.com/istio-releases/fortio-vendor https://github.com/istio-releases/fortio-vendor/tree/master{/dir} https://github.com/istio-releases/fortio-vendor/blob/master{/dir}/{file}#L{line}">

I now get

$ go get -u istio.io/fortio
package istio.io/fortio/vendor/golang.org/x/net/context: /Users/ldemailly/go/src/istio.io/fortio/vendor/.git exists but is not a directory

[ldemailly]

and with 1.10rc2, different error:

$ docker run -ti golang:1.10rc2
root@453b293b15df:/go# go get -u istio.io/fortio
# cd /go/src/istio.io/fortio/vendor; git pull --ff-only
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

package istio.io/fortio/vendor/golang.org/x/net/context: exit status 1
root@453b293b15df:/go# 

[ldemailly]

I tried adding a branch to my submodule but exact same error

root@5ecfa2cf12a8:/go/src/istio.io/fortio# cat .gitmodules 
[submodule "vendor"]
	path = vendor
	url = https://github.com/istio-releases/fortio-vendor.git
	branch = master

[ldemailly]

cc @bradfitz any chance this can be fixed before 1.10 goes final, or at least in 1.11 ?

[bradfitz]

@ldemailly, it's way too late for Go 1.10. Go 1.10 froze over 3 months ago. It's marked for Go 1.11, but it was also marked for Go 1.8, so.... we'll see.

[Nicolas-Peiffer]

Hi all, this issue seems to be stalled. But let me add a comment.

On the cyclonedx-go, I created an issue about using a git submodule inside a GoLang project on Github to "fetch" specification files and sample tests files from a specification repository.

On this particular context, my goal was to propose that the CycloneDX project create git submodules to fetch the content of the CycloneDX/specification repository in every CycloneDX reference implementations repositories (x3: Java, python and Go).

                    ┌────────────────────────────────────────────────────┐
                    │  github.com/CycloneDX/specification commit 59e7d88 │
                    │      └── tools/src/test/resources (local)          │
                    └─────┬─────────────────┬───────────────────┬────────┘
                          │                 │                   │
                 fetch submodule    fetch submodule      fetch submodule
                          │                 │                   │
 ┌────────────────────────▼───────────────┐ │                   │
 │ github.com/CycloneDX/cyclonedx-go      │ │                   │
 │ └── specification @59e7d88 (submodule) │ │                   │
 └────────────────────────────────────────┘ │                   │
                                            │                   │
                  ┌─────────────────────────▼────────────────┐  │
                  │ github.com/CycloneDX/cyclonedx-core-java │  │
                  │  └── specification @59e7d88 (submodule)  │  │
                  └──────────────────────────────────────────┘  │
                                                                │
                                ┌───────────────────────────────▼──────────┐
                                │github.com/CycloneDX/cyclonedx-python-lib │
                                │   └── specification @59e7d88 (submodule) │
                                └──────────────────────────────────────────┘

I though it was a good idea that prevent from copying / duplicating content from one repo to another. However, I was unaware that go get and go install do not support having a git submodule inside a Go project.

And I find no satifactory workarrounds, like using git clone --recurse-submodules https://github|lab.com... and then install localy, instead of using go get github|lab.com... and go install github|lab.com....

I have the feeling adding such support for git submodules would not break current Go features and should not be that complicated because it is already supported by Git... But I don't have the big picture.

[rcalixte]

Greetings, this functionality is critical for projects/modules I am working on but have yet to publish. The crux of it is that I need to include a submodule pointing to a Git tag of an upstream C++ toolkit's repository that contains the header files that will be required for downstream developers using the module I am cultivating in order to build their application(s). Using a submodule in my projects/modules is preferred to manage this structure since there are thousands of upstream header files that change between releases and checking out a version tag significantly reduces the future maintenance burden. I believe that this file/directory structure is not quite supported by vendoring but I would love to be corrected.

My two cents:
There could be a -r flag added to go get (short for the git-clone flag --recurse-submodules so that it would hopefully be easy to remember) that would recursively parse and checkout the contents of the .gitmodules file when found. To keep things simple, this could be an all or nothing recursive operation. (While a configurable max depth flag could be nice, it seems like overkill for now.) There could be a non-configurable hard limit for the maximum recursive depth as well. For now, the only supported VCS would also be Git.

Would a proposal like this be accepted if a pull request were opened?

[Nicolas-Peiffer]

On paper, I like your idea @rcalixte: using either go get -r or go get -R short for go get --recurse-submodules.

go get -R github|lab.com/<project_organisation>/<project_name>

But where would this modification propagate in the go CLI/API ? I think go mod vendor or go mod download or go install are probably affected.

How should go mod vendor and go mod download behave ? Should they be also recursive by default ? Or should they use a flag go mod vendor|download --recurse-submodules?

I assumed (but I do not know) that go mod vendor|download somehow use something similar than go get, so implementing the feature for go get would beneficiate go mod vendor|download?

Same remark for go install: does it make sense to have a go install --recurse-submodules recursive flag? I don't think so.

I do not think that go mod tidy should be impacted by this modification. But maybe I miss something?

Should go get be recursive by default (i.e. no need for --recurse-submodules because it would be the default behaviour?)

Should we not use --recurse-submodules to fetch a Go module/project, since we can add this Go module the natural way with go mod tidy etc..? Should we dedicate --recurse-submodules for non-Go-related artifacts?

In any case, I am probably missing some implications, and I am not capable of implementing this feature. But for my use case of XSD/JSON schemas, it would be awesome and cleaner that duplicating files.

[seankhliao]

Given the new module system, I don't think this issue is relevant anymore.
Note that go primarily operates on modules, not git repos.

[Nicolas-Peiffer]

@seankhliao hey, I do not know what is the "new module system". Can you share a link?

[seankhliao]

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

[Nicolas-Peiffer]

@seankhliao thank you for taking the time to share the link about where I am suppose to ask questions. I apology for trying to understand the comment that closed the issue.

If I go to the question repository and ask "what is the new golang module system that was the reason this particular issue is closed", this feels unnessary. But I will comply.

Good day and sorry for inconvenience.