Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: kube support namespace #118

Closed
wants to merge 110 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
ecf98ee
fix: kube deployment char limit
spy16 Mar 2, 2023
d48ba77
feat: add deployment_id feature
spy16 Mar 3, 2023
5d3d7ad
Merge pull request #1 from goto/fix/bugs
spy16 Mar 13, 2023
4867aa2
refactor: rename to goto (#2)
ishanarya0 Mar 14, 2023
3b338aa
fix: golanci config (#3)
ishanarya0 Mar 14, 2023
4d909e6
fix: kube config handling (#4)
spy16 Mar 15, 2023
e96dcf9
fix: deployment id name (#5)
spy16 Mar 15, 2023
202e4ad
Fix/generic (#6)
spy16 Mar 16, 2023
e758108
fix: change odpf reference to goto (#7)
ishanarya0 Mar 16, 2023
26d5226
generic refactors (#8)
spy16 Mar 17, 2023
ab5b2dc
fix: firehose image name (#9)
spy16 Mar 17, 2023
022f7a4
refactor: re-implement firehose module (#10)
spy16 Mar 21, 2023
4b6a238
fix: use previous telegraf conf (#11)
spy16 Mar 21, 2023
152a191
fix: length validation issue (#12)
spy16 Mar 21, 2023
f66d8bf
fix: goreleaser commit author (#14)
ishanarya0 Mar 24, 2023
fade8c7
feat: add stop flag support (#13)
spy16 Mar 24, 2023
226fd7f
feat: add stop flag support (#13) (#15)
spy16 Mar 25, 2023
cffb11f
fix: revision listing db query (#16)
spy16 Mar 27, 2023
b93cbb8
fix: expose chart values type (#17)
spy16 Mar 27, 2023
b063bba
feat: strip trailing colon in image tag, add telegraf config template
spy16 Mar 30, 2023
fb2202d
fix: telegraf config format (#19)
spy16 Mar 30, 2023
eacb30f
feat: add support for dynamic telegraf labels (#20)
spy16 Mar 30, 2023
9a37814
fix: use go map for telegraf image spec (#21)
spy16 Mar 30, 2023
52671a6
fix: use map to fix yaml marshal (#22)
spy16 Mar 30, 2023
2b799d7
fix: set deployment-id length limit as 63 (#23)
spy16 Mar 30, 2023
3a368d2
fix: revert 63 limit to 53 due helm limitation (#24)
spy16 Mar 30, 2023
d3e42c0
feat: refactor release name to be consistent with old format (#25)
spy16 Mar 30, 2023
cbc5e88
fix: helm release name limit (#26)
spy16 Mar 30, 2023
dc4b7ca
feat: implement deployment label templating (#27)
spy16 Apr 3, 2023
199def5
fix: expose name in firehose module (#28)
spy16 Apr 4, 2023
00fc0d6
refactor: simplify helm package (#29)
spy16 Apr 5, 2023
a354d34
feat: add labels to list-resource api (#30)
spy16 Apr 5, 2023
84ec3cc
feat: merge action and existing labels (#31)
spy16 Apr 6, 2023
4538cb7
fix: remove name label to prevent length issues (#32)
spy16 Apr 6, 2023
71766cb
feat: add jdbc sink type (#33)
ishanarya0 Apr 11, 2023
4e03452
feat: allow custom usage for each firehose (#34)
ishanarya0 Apr 12, 2023
b013688
feat: add tolerations feature (#35)
spy16 Apr 18, 2023
a5f3898
fix: move volume values outside firehose key (#36)
spy16 Apr 21, 2023
d3193e7
fix: add secret key (#37)
ishanarya0 Apr 21, 2023
0919cc5
feat: add toleration to kube module config (#38)
ishanarya0 Apr 27, 2023
74d17e4
feat: add support for sink specific requests & limits (#39)
spy16 May 9, 2023
7e4caa7
feat: add firehose base config (#40)
ishanarya0 May 17, 2023
57a30e8
feat: offset reset history (#42)
ishanarya0 Jul 17, 2023
0128a81
feat: add statsd variables to envVariables (#41)
ishanarya0 Jul 17, 2023
e54c6f9
fix: goreleaser action upgrade (#43)
ishanarya0 Jul 17, 2023
272eb86
fix: change rm-dist flag to clean (#44)
ishanarya0 Jul 17, 2023
1538c02
fix: goreleaser (#45)
ishanarya0 Jul 17, 2023
f3642c1
feat: stoptime in firehose start (#46)
ishanarya0 Jul 19, 2023
cadc134
refactor: public ScaleParams & StartParams (#47)
ishanarya0 Jul 25, 2023
2217ff1
feat: add urn to deployment labels (#48)
ishanarya0 Jul 26, 2023
178f6a8
feat: createdBy and updatedBy in resource & revision (#49)
ishanarya0 Jul 27, 2023
178e82c
feat: broaden AdditionalGlobalTags scope (#50)
ishanarya0 Jul 27, 2023
aac1e7f
fix: initialize as empty arrays (#51)
ishanarya0 Jul 28, 2023
0cb3d28
feat: firehose update to take the latest driver configs (#52)
ishanarya0 Jul 31, 2023
b66407d
fix: update stopped status (#53)
ishanarya0 Aug 7, 2023
01dc9fe
feat: add logging request body (#54)
ishanarya0 Aug 16, 2023
b58f67f
feat: mounts as secrets (#55)
ishanarya0 Aug 21, 2023
43d5983
fix: mount secrets (#56)
ishanarya0 Aug 21, 2023
63bc82b
fix: key name (#57)
ishanarya0 Aug 21, 2023
198c661
fix: add bigtable cred path (#58)
ishanarya0 Aug 24, 2023
e0da338
fix: remove skipping empty values for env variables (#59)
ishanarya0 Aug 24, 2023
bc5e389
fix: mount path to be independent of dep id (#60)
ishanarya0 Aug 28, 2023
4397eb2
feat: return only running & non-terminating pods (#61)
ishanarya0 Aug 30, 2023
305fb47
feat: get resource to persist output (#62)
ishanarya0 Aug 30, 2023
f82209f
feat: add name as extra label (#64)
ishanarya0 Aug 31, 2023
32547ef
feat: support google service account for gke usage (#63)
mabdh Sep 4, 2023
851bee4
test: payloads sent to LOG, BQ and GCS/BLOB sinks (#66)
ishanarya0 Sep 7, 2023
4137fee
fix: add delay before executing reset (#69)
ishanarya0 Sep 11, 2023
9773fcd
fix: job name should be within 63 char limit (#70)
ishanarya0 Sep 14, 2023
ee9a105
fix: remove -firehose from job name (#71)
ishanarya0 Sep 14, 2023
2c4ef1f
feat: old reset strategy (#72)
ishanarya0 Sep 20, 2023
4fdfd50
fix: empty body while logging (#67)
ishanarya0 Sep 21, 2023
876f246
feat: add global logger (#75)
lavkesh Sep 27, 2023
e6b56cc
feat: add global logger (#76)
lavkesh Sep 27, 2023
f0f1a8c
feat: adding job module (#74)
lavkesh Sep 29, 2023
44b26ce
feat: add shared volume (#77)
lavkesh Oct 3, 2023
0afa18d
fead: add sequence if group id does not have it (#73)
ishanarya0 Oct 4, 2023
11a3627
feat: output refresh and logs for job module (#78)
lavkesh Oct 4, 2023
b0bff0f
fix: init container issue (#79)
ishanarya0 Oct 6, 2023
8222c63
feat: on delete cascade in tags table (#80)
ishanarya0 Oct 9, 2023
ae35b4b
feat: add joins in list query (#82)
ishanarya0 Oct 12, 2023
76eba64
feat: optional spec configs (#83)
ishanarya0 Oct 13, 2023
4d0e906
fix: allow overiding user passed resource & limits (#84)
ishanarya0 Oct 20, 2023
2a0d67d
feat: add sink based kube namespace selection (#86)
ishanarya0 Nov 2, 2023
8579741
feat: templatize telegraf config output (#87)
ishanarya0 Nov 2, 2023
d0f20ea
fix: remove validation on resource name (#89)
ishanarya0 Nov 10, 2023
f2387ca
chore: if namespace is empty take from default (#85)
lavkesh Nov 15, 2023
16ffdac
Namespace conf (#90)
lavkesh Nov 22, 2023
3089ca9
Namespace should be taken from resource conf (#91)
lavkesh Nov 22, 2023
168a45d
feat: make kube deployment timeout, extendInterval etc configurable (…
ishanarya0 Dec 13, 2023
69f16e6
fix: avoid typecheck and upgrade linter version (#94)
manishdangi98 Feb 23, 2024
f77ad4f
feat: upgrade lint action (#95)
ishanarya0 Feb 23, 2024
bd86c45
Update .golangci.yml (#96)
ishanarya0 Feb 23, 2024
db0fd9a
feat: exclude rules for typecheck linter (#97)
ishanarya0 Feb 23, 2024
d4ec916
feat(firehose): add HTTPV2 sink type (#93)
abhishekv24 Feb 23, 2024
b1a2680
feat: inc writeTimeout (#98)
ishanarya0 Mar 7, 2024
35322f1
feat: helm release config validation during plan (#99)
ishanarya0 Apr 22, 2024
e252bc6
feat: list resources pagination (#100)
FemiNoviaLina May 8, 2024
ef1cee9
fix: take pagination default value from config (#101)
FemiNoviaLina May 14, 2024
1778229
feat: module cli (#102)
FemiNoviaLina Jul 11, 2024
136b628
feat: overridable firehose image repository (#103)
mabdh Jul 30, 2024
9077875
test: e2e-test (#104)
FemiNoviaLina Aug 13, 2024
3fcd8c4
feat: modular worker (#105)
FemiNoviaLina Aug 23, 2024
d947041
fix: change worker config to map (#106)
FemiNoviaLina Aug 27, 2024
417f10e
feat: enable opentelemetry for worker (#108)
FemiNoviaLina Aug 28, 2024
80d5637
feat: add kafka resource type (#109)
FemiNoviaLina Sep 9, 2024
eb750e2
feat: kafka module config merge (#111)
FemiNoviaLina Sep 10, 2024
bac23e0
feat: add dry run (#116)
ishanarya0 Oct 16, 2024
46a524e
feat: allow maxcompute sink (#118)
FemiNoviaLina Nov 14, 2024
40b199a
feat: kube resource support namespace
mabdh Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ jobs:
steps:
- uses: actions/setup-go@v2
with:
go-version: "1.18"
go-version: "1.21"
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
uses: golangci/golangci-lint-action@v3
with:
skip-go-installation: true
version: v1.46.2
args: --timeout=10m
version: v1.52.0
args: --timeout=10m
8 changes: 4 additions & 4 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,17 @@ jobs:
fetch-depth: 0
- uses: actions/setup-go@v2
with:
go-version: "1.18"
go-version: "1.21"
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v2.6.1
uses: goreleaser/goreleaser-action@v4.3.0
with:
distribution: goreleaser
version: latest
args: --rm-dist
version: v1.18.2
args: --clean
env:
GITHUB_TOKEN: ${{ secrets.GO_RELEASER_TOKEN }}
11 changes: 10 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@ jobs:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
with:
go-version: "1.18"
go-version: "1.21"
- name: run tests
run: make test
e2e-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
with:
go-version: "1.21"
- name: run e2e-tests
run: make e2e-test
9 changes: 6 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ tmp/

expt/

entropy.dev.yaml

requests.http
entropy*.dev.yaml
entropy*.dev.yml
test_*.yml
test_*.json
requests.http
entropy.yaml
73 changes: 39 additions & 34 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
run:
go: '1.17'
go: '1.21'
timeout: 10m
skip-files:
- expt/main.go
Expand All @@ -8,44 +8,46 @@ output:
format: colored-line-number

linters:
enable-all: true
disable:
- exhaustruct
- cyclop
- exhaustive
- exhaustivestruct
- exhaustruct
- funlen
- gochecknoglobals
disable-all: true
enable-all: false
enable:
- bodyclose
- depguard
- dogsled
- dupl
- errcheck
- exportloopref
- gci
- gochecknoinits
- goconst
- gocritic
- gocyclo
- godox
- gofumpt
- golint
- interfacer
- lll
- maintidx
- ireturn
- maligned
- nlreturn
- gofmt
- goimports
- gomnd
- goprintffuncname
- gosec
- gosimple
- govet
- decorder
- ineffassign
- misspell
- nakedret
- noctx
- nolintlint
- prealloc
- promlinter
- scopelint
- tagliatelle
- testpackage
- paralleltest
- tparallel
- containedctx
- varnamelen
- wrapcheck
- wsl

- revive
- staticcheck
- stylecheck
- unconvert
- unparam
- unused
- whitespace
linters-settings:
decorder:
dec-order:
- type
- const
- var
- type
- func
disable-dec-order-check: false
disable-init-func-first-check: false
Expand Down Expand Up @@ -112,12 +114,12 @@ linters-settings:
checks:
- "all"
goimports:
local-prefixes: github.com/odpf/entropy
local-prefixes: github.com/goto/entropy
gci:
sections:
- standard # Captures all standard packages if they do not match another section.
- default # Contains all imports that could not be matched to another section type.
- prefix(github.com/odpf/entropy) # Groups all imports with the specified Prefix.
- prefix(github.com/goto/entropy) # Groups all imports with the specified Prefix.
gocritic:
disabled-checks:
- ifElseChain
Expand All @@ -143,5 +145,8 @@ issues:
linters:
- forbidigo
- contextcheck
- path: /
linters:
- typecheck
severity:
default-severity: error
20 changes: 10 additions & 10 deletions .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ builds:
binary: entropy
flags: [-a]
ldflags:
- -X github.com/odpf/entropy/pkg/version.Version={{.Tag}}
- -X github.com/odpf/entropy/pkg/version.Commit={{.FullCommit}}
- -X github.com/odpf/entropy/pkg/version.BuildTime={{.Date}}
- -X github.com/goto/entropy/pkg/version.Version={{.Tag}}
- -X github.com/goto/entropy/pkg/version.Commit={{.FullCommit}}
- -X github.com/goto/entropy/pkg/version.BuildTime={{.Date}}
goos:
- darwin
- linux
Expand Down Expand Up @@ -58,16 +58,16 @@ dockers:
- entropy
dockerfile: Dockerfile
image_templates:
- 'docker.io/odpf/{{.ProjectName}}:latest'
- 'docker.io/odpf/{{.ProjectName}}:{{ .Version }}'
- 'docker.io/odpf/{{.ProjectName}}:{{ .Version }}-amd64'
- 'docker.io/gotocompany/{{.ProjectName}}:latest'
- 'docker.io/gotocompany/{{.ProjectName}}:{{ .Version }}'
- 'docker.io/gotocompany/{{.ProjectName}}:{{ .Version }}-amd64'

brews:
- name: entropy
homepage: "https://github.com/odpf/entropy"
homepage: "https://github.com/goto/entropy"
description: "Infrastructure orchestration tool."
tap:
owner: odpf
owner: goto
name: homebrew-tap
license: "Apache 2.0"
folder: Formula
Expand All @@ -76,5 +76,5 @@ brews:
install: |-
bin.install "entropy"
commit_author:
name: Rohil Surana
email: rohilsurana96@gmail.com
name: github-actions[bot]
email: 41898282+github-actions[bot]@users.noreply.github.com
34 changes: 28 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,22 +1,31 @@
NAME=github.com/odpf/entropy
NAME=github.com/goto/entropy
VERSION=$(shell git describe --tags --always --first-parent 2>/dev/null)
COMMIT=$(shell git rev-parse --short HEAD)
PROTON_COMMIT="6c5bc2b621abe2812cc8288a5f6363570bab911a"
BUILD_TIME=$(shell date)
COVERAGE_DIR=coverage
BUILD_DIR=dist
EXE=entropy

.PHONY: all build clean tidy format test test-coverage
.PHONY: all build clean tidy format test test-coverage proto

all: clean test build format lint
all: format clean test build

tidy:
@echo "Tidy up go.mod..."
@go mod tidy -v

install:
@echo "Installing Entropy to ${GOBIN}..."
@go install
install: ## install required dependencies
@echo "> installing dependencies"
go install github.com/vektra/mockery/v2@latest
go install google.golang.org/protobuf/cmd/[email protected]
go get -d google.golang.org/protobuf/[email protected]
go get -d google.golang.org/[email protected]
go install google.golang.org/grpc/cmd/[email protected]
go install github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
go install github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
go install github.com/bufbuild/buf/cmd/[email protected]
go install github.com/envoyproxy/[email protected]

format:
@echo "Running gofumpt..."
Expand All @@ -29,6 +38,8 @@ lint:
clean: tidy
@echo "Cleaning up build directories..."
@rm -rf ${COVERAGE_DIR} ${BUILD_DIR}

generate:
@echo "Running go-generate..."
@go generate ./...

Expand All @@ -37,6 +48,10 @@ test: tidy
@echo "Running unit tests..."
@go test ./... -coverprofile=${COVERAGE_DIR}/coverage.out

e2e-test: tidy
@echo "Running e2e-test tests..."
@go test -v ./test/e2e_test/... -timeout 20m

test-coverage: test
@echo "Generating coverage report..."
@go tool cover -html=${COVERAGE_DIR}/coverage.out
Expand All @@ -46,6 +61,13 @@ build: clean
@echo "Running build for '${VERSION}' in '${BUILD_DIR}/'..."
@CGO_ENABLED=0 go build -ldflags '-X "${NAME}/pkg/version.Version=${VERSION}" -X "${NAME}/pkg/version.Commit=${COMMIT}" -X "${NAME}/pkg/version.BuildTime=${BUILD_TIME}"' -o ${BUILD_DIR}/${EXE}

proto: ## Generate the protobuf files
@echo " > generating protobuf from goto/proton"
@echo " > [info] make sure correct version of dependencies are installed using 'make install'"
@rm -rf ./proto
@buf generate https://github.com/goto/proton/archive/${PROTON_COMMIT}.zip#strip_components=1 --template buf.gen.yaml --path gotocompany/entropy --path gotocompany/common
@echo " > protobuf compilation finished"

download:
@go mod download

Expand Down
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Entropy

![test workflow](https://github.com/odpf/entropy/actions/workflows/test.yml/badge.svg)
[![Go Report Card](https://goreportcard.com/badge/github.com/odpf/entropy)](https://goreportcard.com/report/github.com/odpf/entropy)
[![Version](https://img.shields.io/github/v/release/odpf/entropy?logo=semantic-release)](Version)
![test workflow](https://github.com/goto/entropy/actions/workflows/test.yml/badge.svg)
[![Go Report Card](https://goreportcard.com/badge/github.com/goto/entropy)](https://goreportcard.com/report/github.com/goto/entropy)
[![Version](https://img.shields.io/github/v/release/goto/entropy?logo=semantic-release)](Version)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg?logo=apache)](LICENSE)

Entropy is an extensible infrastructure orchestration and application deployment tool. Entropy provides features
Expand All @@ -23,7 +23,7 @@ Install Entropy on macOS, Windows, Linux, OpenBSD, FreeBSD, and on any machine.

#### Binary (Cross-platform)

Download the appropriate version for your platform from [releases](https://github.com/odpf/entropy/releases) page. Once
Download the appropriate version for your platform from [releases](https://github.com/goto/entropy/releases) page. Once
downloaded, the binary can be run from anywhere. You don’t need to install it into a global location. This works well
for shared hosts and other systems where you don’t have a privileged account. Ideally, you should install it somewhere
in your PATH for easy use. `/usr/local/bin` is the most probable location.
Expand All @@ -32,7 +32,7 @@ in your PATH for easy use. `/usr/local/bin` is the most probable location.

```sh
# Install entropy (requires homebrew installed)
$ brew install odpf/tap/entropy
$ brew install goto/tap/entropy

# Check for installed entropy version
$ entropy version
Expand All @@ -58,7 +58,7 @@ $ entropy serve --config ./my_config.yaml

```sh
# Clone the repo
$ git clone https://github.com/odpf/entropy.git
$ git clone https://github.com/goto/entropy.git

# Build entropy binary file
$ make build
Expand All @@ -83,14 +83,14 @@ $ make test
Development of Entropy happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and
improvements. Read below to learn how you can take part in improving Entropy.

Read our [contributing guide](https://odpf.github.io/entropy/docs/contribute/contributing) to learn about our
Read our [contributing guide](https://goto.github.io/entropy/docs/contribute/contributing) to learn about our
development process, how to propose bugfixes and improvements, and how to build and test your changes to Entropy.

To help you get your feet wet and get you familiar with our contribution process, we have a list
of [good first issues](https://github.com/odpf/entropy/labels/good%20first%20issue) that contain bugs which have a
of [good first issues](https://github.com/goto/entropy/labels/good%20first%20issue) that contain bugs which have a
relatively limited scope. This is a great place to get started.

This project exists thanks to all the [contributors](https://github.com/odpf/entropy/graphs/contributors).
This project exists thanks to all the [contributors](https://github.com/goto/entropy/graphs/contributors).

## License

Expand Down
26 changes: 26 additions & 0 deletions buf.gen.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
version: v1
plugins:
- plugin: buf.build/protocolbuffers/go:v1.30.0
out: proto
opt: paths=source_relative
- plugin: buf.build/grpc/go:v1.3.0
out: proto
opt: paths=source_relative,require_unimplemented_servers=true
- plugin: buf.build/bufbuild/validate-go:v1.0.1
out: proto
opt:
- paths=source_relative
- plugin: buf.build/grpc-ecosystem/gateway:v2.15.2
out: proto
opt:
- paths=source_relative
- allow_repeated_fields_in_body=true
- plugin: buf.build/grpc-ecosystem/openapiv2:v2.15.2
out: proto
opt:
- allow_repeated_fields_in_body=true
- output_format=yaml
- allow_merge=true
- merge_file_name=entropy
- openapi_naming_strategy=simple
- json_names_for_fields=false
Loading
Loading