Skip to content

vendor: use go mod #3046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dperny merged 7 commits into from
Dec 14, 2021
Merged

vendor: use go mod #3046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
420befe
vendor: use go mod
errordeveloper Dec 9, 2021
9f89a07
Makefile: add `go-mod-vendor` helper, update `dep-validate`
errordeveloper Dec 9, 2021
2de2dab
vendor: rewrite all deps with `make go-mod-vendor`
errordeveloper Dec 9, 2021
47bd817
vendor: remove references to vndr
errordeveloper Dec 9, 2021
e7932f6
vendor: remove `vendor.conf`
errordeveloper Dec 9, 2021
5e2152f
tools: manage protobuild using modules
errordeveloper Dec 13, 2021
e5eca60
vendor: add protobuf specs back
errordeveloper Dec 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,10 @@ jobs:

- run:
name: Validate dependency vendoring
environment:
# ensure GOPATH is set to single directory so the make target doesn't need to handle a column-separated list
GOPATH: /home/circleci/.go_workspace
GO111MODULE: "on"
command: |
git fetch origin
if test -n "`git diff --stat=1000 origin/master | grep -E \"^[[:space:]]*vendor\"`"; then
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,6 @@ bin/swarmkitstate

# dev sync, if used
/.docker-sync/

# top-level go.mod is not meant to be checked in yet
/go.mod
17 changes: 2 additions & 15 deletions BUILDING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,22 +80,9 @@ for details.

### Update vendored dependencies

To update dependency you need just change `vendor.conf` file and run `vndr` tool:
To update dependency you need just change `vendor.mod` file and run:
```
go get github.com/LK4D4/vndr
vndr
```

It's possible to update only one dependency:
```
vndr github.com/coreos/etcd v3.0.9
```

but it should be in sync with `vendor.conf`.

Also, you can update dependency from fork for testing:
```
vndr github.com/coreos/etcd <revision> https://github.com/LK4D4/etcd.git
make go-mod-vendor
```

### Regenerating protobuf bindings
Expand Down
1 change: 1 addition & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ WORKDIR /go/src/github.com/docker/swarmkit/
# install the dependencies from `make setup`
# we only copy `direct.mk` to avoid busting the cache too easily
COPY direct.mk .
COPY tools .
RUN make --file=direct.mk setup

# now we can copy the rest
Expand Down
37 changes: 19 additions & 18 deletions direct.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,9 @@ INTEGRATION_PACKAGE=${PROJECT_ROOT}/integration
COMMANDS=swarmd swarmctl swarm-bench swarm-rafttool protoc-gen-gogoswarm
BINARIES=$(addprefix bin/,$(COMMANDS))

VNDR=$(shell which vndr || echo '')

GO_LDFLAGS=-ldflags "-X `go list ./version`.Version=$(VERSION)"

GOBIN=$(shell go env GOPATH)/bin

.DEFAULT_GOAL = all
.PHONY: all
Expand All @@ -41,20 +40,17 @@ setup: ## install dependencies
# TODO(stevvooe): Install these from the vendor directory
# install golangci-lint version 1.17.1 to ./bin/golangci-lint
@curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.17.1
@go get -u github.com/lk4d4/vndr
# use GO111MODULE=on to get protobuild with the appropriate versions of its
# dependencies
@GO111MODULE=on go get github.com/stevvooe/protobuild
@(cd tools ; GO111MODULE=on go install github.com/containerd/protobuild)

.PHONY: generate
generate: protos
@echo "🐳 $@"
@PATH=${ROOTDIR}/bin:${PATH} go generate -x ${PACKAGES}
@PATH=${ROOTDIR}/bin:${GOBIN}:${PATH} go generate -x ${PACKAGES}

.PHONY: protos
protos: bin/protoc-gen-gogoswarm ## generate protobuf
@echo "🐳 $@"
@PATH=${ROOTDIR}/bin:${PATH} protobuild ${PACKAGES}
@PATH=${ROOTDIR}/bin:${GOBIN}:${PATH} protobuild ${PACKAGES}

.PHONY: checkprotos
checkprotos: generate ## check if protobufs needs to be generated again
Expand Down Expand Up @@ -139,14 +135,19 @@ help: ## this help
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) | sort

.PHONY: dep-validate
dep-validate:
dep-validate: go-mod-vendor
@echo "+ $@"
$(if $(VNDR), , \
$(error Please install vndr: go get github.com/lk4d4/vndr))
@rm -Rf .vendor.bak
@mv vendor .vendor.bak
@$(VNDR)
@test -z "$$(diff -r vendor .vendor.bak 2>&1 | tee /dev/stderr)" || \
(echo >&2 "+ inconsistent dependencies! what you have in vendor.conf does not match with what you have in vendor" && false)
@rm -Rf vendor
@mv .vendor.bak vendor
@go mod tidy -modfile vendor.mod
Copy link
Copy Markdown
Contributor Author

@errordeveloper errordeveloper Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thaJeztah @crazy-max 👁️

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙈 should this one be under the go-mod-vendor: target?

Copy link
Copy Markdown
Member

@crazy-max crazy-max Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah We should only mod tidy on validation. See docker/cli#3387 (comment)

Copy link
Copy Markdown
Contributor Author

@errordeveloper errordeveloper Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, as @crazy-max said - currently the idea is to detect untidy cases in validation stage, tidying as such should be up to the contributor. We could change this in the future, just that initially it seems reasonable to assume that go mod tidy has too much of a side-effect as it can rewrite vendor.sum entirely.

@git diff --exit-code vendor.mod vendor.sum vendor || \
(echo >&2 "+ inconsistent dependencies! what you have in vendor.mod does not match with what you have in vendor" && false)

go.mod:
@printf "module github.com/docker/swarmkit\n\ngo 1.17\n" > $@

GOGO_PROTOBUF_PATH=$(shell go env GOPATH)/pkg/mod/github.com/gogo/protobuf@$(shell go list -modfile vendor.mod -m -f '{{.Version}}' github.com/gogo/protobuf)/protobuf

.PHONY: go-mod-vendor
go-mod-vendor: go.mod
@go mod vendor -modfile vendor.mod
@# ensure that key protobuf spec files are in vendor dir
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mentioned these files were intentionally not vendored by go mod vendor - out of curiosity; is that intentionally by the google/protobuf maintainers, or go mod ?

Copy link
Copy Markdown
Contributor Author

@errordeveloper errordeveloper Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Go team believes that files that don't belong to a Go package should not belong in vendor...

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, so like .appveyor.yml and .circleci.yml and random .json,.yaml and Makefiles, because they remove those .....right? 😉

I guess it's their way of telling "we don't like vendor, so we break things if you use vendor, but not if you use module cache" 🤷‍♂️

Copy link
Copy Markdown
Contributor Author

@errordeveloper errordeveloper Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my understanding those random files are kept as they are contained in top-level dirs, which do contain Go code...

Copy link
Copy Markdown
Contributor Author

@errordeveloper errordeveloper Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we break things if you use vendor

The thing is that different vendoring tools offered different behaviour, some had prune exclusion/inclusion parameters with defaults etc, so Go team chose to basically avoid having parameters... I think a sensible thing to do will be to add parameters, cause this is clearly messy as is.

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are contained in top-level dirs, which do contain Go code...

Ah, right. If there's no .go files at all in there, then it makes somewhat sense (it's odd that they remove README.md files in some cases though; I recall vndr explicitly kept those as they sometimes mention copyright etc).

So, one solution for such cases would be to add a doc.go, but it likely still would need to be imported somewhere to prevent being pruned.

Copy link
Copy Markdown
Contributor Author

@errordeveloper errordeveloper Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, one solution for such cases would be to add a doc.go, but it likely still would need to be imported somewhere to prevent being pruned.

I don't think it prunes unimproved packages, but cannot say for sure...

@cp -a $(GOGO_PROTOBUF_PATH) vendor/github.com/gogo/protobuf/
13 changes: 13 additions & 0 deletions tools/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
module github.com/moby/moby/tools

go 1.17

require github.com/containerd/protobuild v0.1.1-0.20211025221430-7e5ee24bc1f7

require (
github.com/golang/protobuf v1.5.0 // indirect
Comment thread
errordeveloper marked this conversation as resolved.
github.com/pelletier/go-toml v1.8.1 // indirect
)

// match ../vendor.mod
replace github.com/golang/protobuf => github.com/golang/protobuf v1.3.5
61 changes: 61 additions & 0 deletions tools/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/containerd/protobuild v0.1.1-0.20211025221430-7e5ee24bc1f7 h1:MndHrq2xWkyjig8bS8dCiOtpqRgz7WiLYLbaAxAfLHM=
github.com/containerd/protobuild v0.1.1-0.20211025221430-7e5ee24bc1f7/go.mod h1:Uj5zHB+2pb+Zs/+/u1Bv0M/VLgBow+kBdzxKZE1HJ9E=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/pelletier/go-toml v1.8.1 h1:1Nf83orprkJyknT6h7zbuEGUEjcyVlCxSUGTENmNCRM=
github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.28.1/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
7 changes: 7 additions & 0 deletions tools/tools.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
//go:build tools

package main

import (
_ "github.com/containerd/protobuild"
)
92 changes: 0 additions & 92 deletions vendor.conf
View file
Open in desktop

This file was deleted.

Loading