Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Explicit Commit, Dynamo Equivalence Check, and Temporary and Permanent Error Distinctions #179

Merged
merged 100 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
a002379
fixed valid to from and issuer signing
clD11 Jul 29, 2022
d533976
update so buffer can be one
clD11 Jul 29, 2022
4bdeae7
commented test
clD11 Jul 29, 2022
e86b39c
added blinded creds to schema
clD11 Aug 2, 2022
2368408
Add temporary error handling
Sneagan Feb 10, 2022
352cfd4
Merge branch 'master' into feature/temporary-permanent-error-distinct…
Sneagan Aug 11, 2022
b011345
Resolve conflicts
Sneagan Aug 11, 2022
07e4333
Resolve conflicts
Sneagan Aug 11, 2022
e6cdbd8
Remove leading caps and trailing punctuation from error message strings
Sneagan Aug 11, 2022
56c540e
Make error messages match master
Sneagan Aug 11, 2022
c9a7a9d
Resolve conflicts
Sneagan Feb 9, 2022
5710210
WIP
Sneagan Feb 9, 2022
6a89b66
Merge branch 'feature/dynamo-duplicate-distinction' of github.com:bra…
Sneagan Aug 12, 2022
180fb26
WIP
Sneagan Aug 12, 2022
b877ea4
Fix some linting
Sneagan Aug 12, 2022
dd7be0f
Fix const reference
Sneagan Aug 12, 2022
a1d3e8b
Remove tolerable equivalence reference
Sneagan Aug 12, 2022
0aa857d
Manage equivalence
Sneagan Aug 12, 2022
23b2b8b
Remove unneeded function
Sneagan Aug 12, 2022
54a8ba0
Make duplicate all the new integer instead of remapping integers
Sneagan Aug 12, 2022
bdb1b69
Remove unrelated offset change
Sneagan Aug 12, 2022
8736e97
Merge pull request #99 from brave-intl/feature/improve-batching
Sneagan Feb 9, 2022
fdc2421
WIP Restore explicit commit
Sneagan Aug 16, 2022
5c4020a
WIP
Sneagan Aug 16, 2022
829d935
WIP
Sneagan Aug 16, 2022
2d7e165
Add simple backoff for temporary errors
Sneagan Feb 11, 2022
0a83e2e
Default offset for unrelated use case
Sneagan Aug 16, 2022
d4cd246
Move brace
Sneagan Aug 16, 2022
613230b
Normalize strings
Sneagan Aug 16, 2022
94db0c0
Restore old logging variable names
Sneagan Aug 16, 2022
ffc2db0
Restore old logging variable names
Sneagan Aug 16, 2022
9eddf8c
Defer context cancel
Sneagan Aug 17, 2022
6275e9e
Improve comments
Sneagan Aug 17, 2022
687f2dd
Only commit after result emission
Sneagan Aug 17, 2022
a2b82ee
Improve variable name
Sneagan Aug 17, 2022
051870b
Remove unneeded kafka emit.
Sneagan Aug 17, 2022
85392d4
Emit a response for permanent errors, but not temporary
Sneagan Aug 18, 2022
e406890
Use logger with request id
Sneagan Aug 18, 2022
678c938
Return errors as avro structures
Sneagan Aug 19, 2022
57135aa
Emit permanent errors in the goroutines instead of after the channel …
Sneagan Aug 19, 2022
681b08a
Make emission code more DRY and readable
Sneagan Aug 19, 2022
7fb12e5
Remove unneeded EOF handling
Sneagan Aug 23, 2022
d16cab9
Remove unneeded continue
Sneagan Aug 23, 2022
ab466f9
Move context outside of loop
Sneagan Aug 23, 2022
d8acc46
Map kafka message in error handling
Sneagan Aug 23, 2022
5e5bb58
Omit timestamp from equivalence check
Sneagan Aug 24, 2022
4949399
Only check payload equivalence and use BindingEquivalence for the ass…
Sneagan Aug 25, 2022
f41fc19
Add idempotent redemption and use avro result status instead of enum
Sneagan Aug 25, 2022
7972f22
Resolve conflicts
Sneagan Aug 26, 2022
0071619
Merge pull request #192 from brave-intl/feature/explicit-commit
Sneagan Sep 1, 2022
c119200
Merge pull request #187 from brave-intl/feature/dynamo-duplicate-dist…
Sneagan Sep 1, 2022
490c3b5
WIP
Sneagan Sep 7, 2022
60827d4
Implement channel-based commit management
Sneagan Sep 13, 2022
7fd264a
Break out behavior into functions
Sneagan Sep 13, 2022
ce8f2a7
Merge branch 'feature/explicit-commit' into feature/temporary-permane…
Sneagan Sep 13, 2022
baa45dc
Resolve conflicts
Sneagan Sep 13, 2022
bab4dc8
Temporarily revert linter workflow go version
Sneagan Sep 13, 2022
91061c6
Undo temporary revert of linter workflow go version
Sneagan Sep 13, 2022
3904bad
Try latest version of golang-lint in CI action
Sneagan Sep 13, 2022
f0103fa
Revert version of golang-lint in CI action
Sneagan Sep 13, 2022
44ec8e8
WIP Conform to linting
Sneagan Sep 13, 2022
b02237e
Improve linting results slightly
Sneagan Sep 13, 2022
c3227cd
Fix go.mod
Sneagan Sep 14, 2022
3e67d68
Use version 1.49 of golangci-lint
Sneagan Sep 14, 2022
7807ecf
Skip cache for golangci-lint
Sneagan Sep 14, 2022
5a12575
Test go mod tidy in lint script
Sneagan Sep 14, 2022
a1b9da2
Try golangci-lint with go 1.17
Sneagan Sep 14, 2022
3d59362
Return to go 1.18 in golangci-lint
Sneagan Sep 14, 2022
bcee8d2
Add clean to golangci-lint
Sneagan Sep 14, 2022
3f6ddb1
Return golangci-lint configuration to match master
Sneagan Sep 14, 2022
4c0e5e0
up golint version to 1.49 in github action
husobee Sep 14, 2022
0d53807
Add changes relating to PR feedback
Sneagan Sep 15, 2022
bbf7e62
Resolve minor conflict
Sneagan Sep 15, 2022
d08711b
Revert challenge-bypass-ristretto-ffi
Sneagan Sep 16, 2022
cd1306e
Change from temporary-based logic to nil indicating temporary failure…
Sneagan Sep 19, 2022
b706a4d
Fix minor comment typo
Sneagan Sep 24, 2022
c2e1249
Re-add error that is needed for results
Sneagan Sep 27, 2022
a9a90cb
Merge branch 'feature/temporary-permanent-error-distinctions' of gith…
Sneagan Sep 28, 2022
d637417
Improve error handling readability
Sneagan Oct 6, 2022
0835261
Refactor Issuer Fetch from Postgres
Sneagan Oct 10, 2022
f79498c
Address PR comments
Sneagan Oct 11, 2022
cfb8c9a
Use error instead of ProcessingError for MessageContext
Sneagan Oct 12, 2022
4f9ad6b
Refactor errors and panics
Sneagan Oct 15, 2022
b8678a4
Break from inner loop
Sneagan Oct 15, 2022
20396d4
Update comments
Sneagan Oct 15, 2022
a234a4b
Make small idiomatic change
Sneagan Oct 15, 2022
03fbc21
Resolve conflicts
Sneagan Nov 5, 2022
521852d
Fix go.mod and go.sum from conflict resolution
Sneagan Nov 11, 2022
67e4368
Fix string formatting mistake.
Sneagan Nov 11, 2022
0c9f202
Move linting step into CI github workflow.
husobee Nov 14, 2022
0904d27
merge master
husobee Nov 15, 2022
f4ca678
Make batchPipeline a buffered channel
Sneagan Nov 15, 2022
617c4f4
Merge branch 'feature/temporary-permanent-error-distinctions' of gith…
Sneagan Nov 15, 2022
9f3f78e
Use generic and errors.As for dynamo failures
Sneagan Nov 15, 2022
bf67ca6
Use generic error instead of ProcessingError for exposed function
Sneagan Nov 15, 2022
4bdf169
Use errors.Is instead of string comparison
Sneagan Nov 15, 2022
e054fde
Resolve conflicts
Sneagan Nov 16, 2022
6977dae
fixing linting issues
husobee Nov 16, 2022
8cddd72
fixing testing issues
husobee Nov 16, 2022
27fe455
error type issue for get issuer fix
husobee Nov 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/challenge-bypass-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,7 @@ jobs:
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: run lint
run: make lint
- name: run tests
run: make docker-test
26 changes: 0 additions & 26 deletions .github/workflows/golangci-lint.yaml

This file was deleted.

18 changes: 9 additions & 9 deletions .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ run:
timeout: 3m

linters-settings:
cyclop:
#cyclop:
# The maximal code complexity to report.
# Default: 10
max-complexity: 10
# max-complexity: 10
# The maximal average package complexity.
# If it's higher than 0.0 (float) the check is enabled
# Default: 0.0
package-average: 10.0
#package-average: 10.0

errcheck:
# Report about not checking of errors in type assertions: `a := b.(MyStruct)`.
Expand Down Expand Up @@ -41,23 +41,23 @@ linters:
disable-all: true
enable:
## enabled by default
- deadcode # Finds unused code
#- deadcode # Finds unused code
- errcheck # Errcheck is a program for checking for unchecked errors in go programs. These unchecked errors can be critical bugs in some cases
- gosimple # Linter for Go source code that specializes in simplifying a code
- govet # Vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string
- ineffassign # Detects when assignments to existing variables are not used
- staticcheck # Staticcheck is a go vet on steroids, applying a ton of static analysis checks
- structcheck # Finds unused struct fields
#- structcheck # Finds unused struct fields
- typecheck # Like the front-end of a Go compiler, parses and type-checks Go code
- unused # Checks Go code for unused constants, variables, functions and types
- varcheck # Finds unused global variables and constants
#- varcheck # Finds unused global variables and constants
# ## disabled by default
- contextcheck # check the function whether use a non-inherited context
- cyclop # checks function and package cyclomatic complexity
#- cyclop # checks function and package cyclomatic complexity
- errname # Checks that sentinel errors are prefixed with the Err and error types are suffixed with the Error.
- gocritic # Provides diagnostics that check for bugs, performance and style issues.
- gocyclo # Computes and checks the cyclomatic complexity of functions
- nestif # Reports deeply nested if statements
#- gocyclo # Computes and checks the cyclomatic complexity of functions
#- nestif # Reports deeply nested if statements
- revive # Fast, configurable, extensible, flexible, and beautiful linter for Go. Drop-in replacement of golint.
- sqlclosecheck # Checks that sql.Rows and sql.Stmt are closed
- stylecheck # Stylecheck is a replacement for golint
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ generate-avro:
sed -i 's/"public_key/"issuer_public_key/g' ./avro/generated/signing_result*.go

lint:
docker run --rm -v "$$(pwd):/app" --workdir /app golangci/golangci-lint:v1.46.2 go get ./... && golangci-lint run -v ./...
docker run --rm -v "$$(pwd):/app" --workdir /app golangci/golangci-lint:v1.49.0 golangci-lint run -v ./...
4 changes: 2 additions & 2 deletions avro/generated/redeem_result.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions avro/generated/redeem_result_set.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 9 additions & 4 deletions avro/generated/redeem_result_status.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion avro/schemas/redeem_result.avsc
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"type": {
"name": "RedeemResultStatus",
"type": "enum",
"symbols": ["ok", "duplicate_redemption", "unverified", "error"]
"symbols": ["ok", "duplicate_redemption", "unverified", "error", "idempotent_redemption"]
}
},
{"name": "associated_data", "type": "bytes", "doc": "contains METADATA"}
Expand Down
8 changes: 5 additions & 3 deletions btd/issuer.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ import (
)

var (
ErrInvalidMAC = errors.New("binding MAC didn't match derived MAC")
ErrInvalidBatchProof = errors.New("New batch proof for signed tokens is invalid")
// ErrInvalidMAC - the mac was invalid
ErrInvalidMAC = errors.New("binding MAC didn't match derived MAC")
// ErrInvalidBatchProof - the batch proof was invalid
ErrInvalidBatchProof = errors.New("new batch proof for signed tokens is invalid")

latencyBuckets = []float64{.25, .5, 1, 2.5, 5, 10}

Expand Down Expand Up @@ -74,7 +76,7 @@ func init() {
func ApproveTokens(blindedTokens []*crypto.BlindedToken, key *crypto.SigningKey) ([]*crypto.SignedToken, *crypto.BatchDLEQProof, error) {
var err error
if len(blindedTokens) < 1 {
err = errors.New("Provided blindedTokens array was empty.")
err = errors.New("provided blindedTokens array was empty")
return []*crypto.SignedToken{}, nil, err
}

Expand Down
4 changes: 1 addition & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ require (
github.com/actgardner/gogen-avro/v10 v10.2.1
github.com/aws/aws-sdk-go v1.44.136
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.17.4
github.com/brave-intl/bat-go/libs v0.0.0-20220823005459-d3a4d8ccf976
github.com/brave-intl/bat-go/libs v0.0.0-20220913154833-730f36b772de
github.com/brave-intl/challenge-bypass-ristretto-ffi v0.0.0-20190717223301-f88d942ddfaf
github.com/getsentry/raven-go v0.2.0
github.com/go-chi/chi v4.1.2+incompatible
Expand All @@ -32,7 +32,6 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/containerd/containerd v1.6.6 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/getsentry/sentry-go v0.13.0 // indirect
github.com/go-chi/chi/v5 v5.0.7 // indirect
Expand All @@ -48,7 +47,6 @@ require (
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
Expand Down
6 changes: 2 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,8 @@ github.com/blang/semver v3.1.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnweb
github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/brave-intl/bat-go/libs v0.0.0-20220823005459-d3a4d8ccf976 h1:kz83/D17IsaIVrDSqYvNgyLgMRvgSExcHvFOsRAOPEM=
github.com/brave-intl/bat-go/libs v0.0.0-20220823005459-d3a4d8ccf976/go.mod h1:bIOgpByIK7sC11XzdMZlM1Ri17g0eYqLFs5sd/D1wF8=
github.com/brave-intl/bat-go/libs v0.0.0-20220913154833-730f36b772de h1:A7l6jiuZW6ED7SuDK331LhkCqQNUYNv0RclciTwvIZU=
github.com/brave-intl/bat-go/libs v0.0.0-20220913154833-730f36b772de/go.mod h1:Hdx1PUXLp4TevCH6X7hzfCBcjaQnuechLVUWqD2I3aQ=
github.com/brave-intl/challenge-bypass-ristretto-ffi v0.0.0-20190717223301-f88d942ddfaf h1:ZAsT/fM7Kxipf3wtoY7xa2bpFmAxzYPhVJ3hUcSdTRI=
github.com/brave-intl/challenge-bypass-ristretto-ffi v0.0.0-20190717223301-f88d942ddfaf/go.mod h1:I9sAUIQc7AvvUU0Ustl5WMTdqmlNjXsX6dRLnDNxXiE=
github.com/bshuster-repo/logrus-logstash-hook v0.4.1/go.mod h1:zsTqEiSzDgAa/8GZR7E1qaXrhYNDKBYy5/dWPTIflbk=
Expand Down Expand Up @@ -274,7 +274,6 @@ github.com/containerd/containerd v1.5.7/go.mod h1:gyvv6+ugqY25TiXxcZC3L5yOeYgEw0
github.com/containerd/containerd v1.5.8/go.mod h1:YdFSv5bTFLpG2HIYmfqDpSYYTDX+mc5qtSuYx1YUb/s=
github.com/containerd/containerd v1.6.1/go.mod h1:1nJz5xCZPusx6jJU8Frfct988y0NpumIq9ODB0kLtoE=
github.com/containerd/containerd v1.6.6 h1:xJNPhbrmz8xAMDNoVjHy9YHtWwEQNS+CDkcIRh7t8Y0=
husobee marked this conversation as resolved.
Show resolved Hide resolved
github.com/containerd/containerd v1.6.6/go.mod h1:ZoP1geJldzCVY3Tonoz7b1IXk8rIX0Nltt5QE4OMNk0=
github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20190815185530-f2a389ac0a02/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20191127005431-f65d91d395eb/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
Expand Down Expand Up @@ -935,7 +934,6 @@ github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zM
github.com/opencontainers/image-spec v1.0.2-0.20211117181255-693428a734f5/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/opencontainers/image-spec v1.0.2/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799 h1:rc3tiVYb5z54aKaDfakKn0dDjIyPpTtszkjuMzyt7ec=
github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/opencontainers/runc v0.0.0-20190115041553-12f6a991201f/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opencontainers/runc v0.1.1/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opencontainers/runc v1.0.0-rc8.0.20190926000215-3e425f80a8c9/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
Expand Down
12 changes: 6 additions & 6 deletions kafka/avro_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@ package kafka

import (
"bytes"
"testing"
"time"

avroSchema "github.com/brave-intl/challenge-bypass-server/avro/generated"
"github.com/brave-intl/challenge-bypass-server/utils/test"
"github.com/stretchr/testify/assert"
"testing"
"time"
)

// Tests v2 adds new fields validTo, validFrom and BlindedTokens.
Expand Down Expand Up @@ -37,8 +38,8 @@ func TestSchemaCompatability_SigningResult_V2ToV1(t *testing.T) {
assert.Equal(t, v2.Status.String(), v1.Status.String())
}

//// Tests v2 consumers reading v1 messages.
//func TestSchemaCompatability_SigningResult_V1ToV2(t *testing.T) {
// Tests v2 consumers reading v1 messages.
evq marked this conversation as resolved.
Show resolved Hide resolved
// func TestSchemaCompatability_SigningResult_V1ToV2(t *testing.T) {
// v1 := &avroSchema.SigningResultV1{
// Signed_tokens: []string{test.RandomString()},
// Issuer_public_key: test.RandomString(),
Expand All @@ -50,7 +51,6 @@ func TestSchemaCompatability_SigningResult_V2ToV1(t *testing.T) {
// var buf bytes.Buffer
// err := v1.Serialize(&buf)
// assert.NoError(t, err)
//
// v2, err := avroSchema.DeserializeSigningResultV2(&buf)
// assert.NoError(t, err)
//
Expand All @@ -61,4 +61,4 @@ func TestSchemaCompatability_SigningResult_V2ToV1(t *testing.T) {
// //assert.Nil(t, v2.Valid_to)
// //assert.Nil(t, v2.Valid_from)
// assert.Empty(t, v2.Blinded_tokens)
//}
// }
Loading