Metrics (Accounts): State changes (#916)

* Darren/logdb remove leading zeros (#865)

* Add LogIndex and TxIndex into logs/event response body (#862)

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Show all issues on lint (#869)

* Show all issues on lint

* fix lint

* fix(docker): using AWS docker repo for trivy (#872)

* fix(docker): using AWS docker repo for trivy

* fix(docker): using AWS docker repo for trivy

* Darren/feat/add subscription cache (#866)

* ehancement: create a cache for block based subscriptions

* minor: change function names for subscriptions

* test: add unit test for message cache

* chore: add license headers

* refactor: fix up error handling

* fix: remove bad test

* fix: PR comments

* fix: PR comments - remove block cache

* refactor(subscriptions): store structs in cache, not bytes

* fix(license): add license header

* chore(subscriptions): revert unit test changes

* enhancement: resolve pr comments to use simplelru

* enhancement: resolve pr comments - use id as key

* Add additional block tests (#863)

* enhancement(logging): leverage trace level (#873)

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* chore(docs): update spec for validator nodes (#875)

* chore(docs): update spec for validator nodes

* chore(docs): update cores

* chore(docs): remove public node stuff

* Darren/logdb remove leading zeros (#865)

* feat: add new txIndex column to event meta response

* test: add convert event test

* feat: make txLog and txIndex as optional return params

* chore: update swagger with new event optional data

* feat: save logIndex in sequence

* feat: tweaked bits in sequence

* refactor: rename optional log meta field

* refactor: comments, yaml and txIndex counts

* rebase to master

* fix: remove stale struct

* add txIndex to returned logdb query

* reset to 0 eventCount and transferCount each receipt and write blockId only once

* fix lint

* rephrase logIndex description in yaml file

* refactor: use filter.Option instead of eventFilter.Option

* move includeIndexes to api

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Makis Christou <[email protected]>

* fix: set range.To to max logDB block (#880)

* Update Convert Filter to match BlockMask bit space (#881)

* Update Convert Filter to match BlockMask bit space

* use exported BlockNumMask

* return error in newSequence (#885)

* return error in newSequence

* revert type change of sequence

* adjust sequence to 63bit

* fix test

---------

Co-authored-by: otherview <[email protected]>

* add safety guard to the sequence (#887)

* add safety guard to the sequence

* move bit distribution to tests

* chore: update version for mainDB v4

* maindb v4 (#868)

* fix(documentation): use absolute links in markdown (#889)

* Add benchmark test to node block process (#892)

* Add benchmark test to node block process

* added file-based storage

* use tempdir

* update dependency go-ethereum (#895)

* chore: update API metrics bucket and endpoint names (#893)

* chore: update API metrics bucket and endpoint names

* fix: typo & tests

* fix: lint

* chore: add websocket total counter

* fix: txs endpoints names & ws subject

* fix: unit tests

* chore: standardise naming convention

* chore: add websocke duration & http code

* chore: add websocke duration & http code

* fix: lint issues

* fix: sync issues with metrics

* chore: update websocket durations bucket

* fix: PR comments - use sync.Once

* chore: update builtin generation (#896)

* chore: update builtin generation

* fix: update GHA

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* improve cache stats log and metric

* totally removed SetBestBlockID

* Maindb v4 Transaction benchmark plus cache (#894)

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* Adding transactions benchmark + repository cache

* improve cache stats log and metric

* totally removed SetBestBlockID

* removed unused tests

* update bench tests

* getreceipts metrics + lint

---------

Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>

* reduce clauses() allocations

* bug: fix logs endpoints query (#900)

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* improve cache stats log and metric

* totally removed SetBestBlockID

* fix: logs API not returning results when to=0,from=omitted

* minor update

* fix: PR comments + lint

* improve convert range logic

* chore: remove debug log

---------

Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>

* chore(chain): add repo cache metrics (#910)

* chore(chain): add repo cache metrics

* refactor(chain): cache hit miss

---------

Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Pedro Gomes <[email protected]>
Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>

* Pedro/merge/feat/db (#914)

* fix(documentation): use absolute links in markdown (#889)

* Add benchmark test to node block process (#892)

* Add benchmark test to node block process

* added file-based storage

* use tempdir

* update dependency go-ethereum (#895)

* chore: update API metrics bucket and endpoint names (#893)

* chore: update API metrics bucket and endpoint names

* fix: typo & tests

* fix: lint

* chore: add websocket total counter

* fix: txs endpoints names & ws subject

* fix: unit tests

* chore: standardise naming convention

* chore: add websocke duration & http code

* chore: add websocke duration & http code

* fix: lint issues

* fix: sync issues with metrics

* chore: update websocket durations bucket

* fix: PR comments - use sync.Once

* chore: update builtin generation (#896)

* chore: update builtin generation

* fix: update GHA

* getreceipts metrics + lint (#902)

* chore: add flag to enable/disable deprecated APIs (#897)

* chore: add flag to enable/disable deprecated APIs

* chore: update for PR comments

* chore: update for PR comments

* fix: update e2e commit sha

* fix: update e2e commit sha

* fix: update flag name

* fix: solo start flags (#906)

* chore: make thorclient configurable + fix type error (#908)

* chore: make thorclient configurable

* fix: subscriptions block type

* fix: compile errors

* fix: remove test with lint error

* add 'raw' query parameter to the blocks (#899)

* add 'raw' query parameter to the blocks

* summary -> summary.Header

Co-authored-by: libotony <[email protected]>

* change variable name

* make expanded and raw mutually exclusive

* add unit tests

* fix linting

---------

Co-authored-by: libotony <[email protected]>

* Adding Health endpoint (#836)

* Adding Health endpoint

* pr comments + 503 if not healthy

* refactored admin server and api + health endpoint tests

* fix health condition

* fix admin routing

* added comments + changed from ChainSync to ChainBootstrapStatus

* Adding healthcheck for solo mode

* adding solo + tests

* fix log_level handler funcs

* refactor health package + add p2p count

* remove solo methods

* moving health service to api pkg

* added defaults + api health query

* pr comments

* pr comments

* pr comments

* Update cmd/thor/main.go

* Darren/admin api log toggler (#877)

* Adding Health endpoint

* pr comments + 503 if not healthy

* refactored admin server and api + health endpoint tests

* fix health condition

* fix admin routing

* added comments + changed from ChainSync to ChainBootstrapStatus

* Adding healthcheck for solo mode

* adding solo + tests

* fix log_level handler funcs

* feat(admin): toggle api logs via admin API

* feat(admin): add license headers

* refactor health package + add p2p count

* remove solo methods

* moving health service to api pkg

* added defaults + api health query

* pr comments

* pr comments

---------

Co-authored-by: otherview <[email protected]>

* Darren/chore/backport metrics (#909)

* chore(muxdb): backport muxdb cache metrics

* chore(muxdb): backport muxdb cache metrics

* chore(metrics): backport disk IO

* chore(metrics): fix lint

* chore(chain): add repo cache metrics

* fix(chain): fix cache return value

* refactor(chain): cache hit miss

* chore(thor): update version (#912)

* chore(thor): update version

* chore(openapi): version

* feat(api/debug): support debug trace without blockId (#905)

* api/debug: support debug with txhash

Signed-off-by: jsvisa <[email protected]>

api/debug: blockId should use tx's instead

Signed-off-by: jsvisa <[email protected]>

fix tests

Signed-off-by: jsvisa <[email protected]>

* debug: add test

Signed-off-by: jsvisa <[email protected]>

* improve parseTarget

Signed-off-by: jsvisa <[email protected]>

* update doc

Signed-off-by: jsvisa <[email protected]>

* fix tests

Signed-off-by: jsvisa <[email protected]>

---------

Signed-off-by: jsvisa <[email protected]>
Co-authored-by: tony <[email protected]>

* version

---------

Signed-off-by: jsvisa <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: YeahNotSewerSide <[email protected]>
Co-authored-by: Delweng <[email protected]>

* chore: fixed vulnerability

* chore: fixed vulnerability

* first commit

* second commit

* added receipt

* added receipt

* state commit

* fix: Get Devnet ID after the Thor flags are set (#915)

* first commit

* using bytes32 method

* added more dashboard

* added target

* fix: compile errors

* linter

* chore(maindbv4): add a migration guide" (#944)

* Update metrics/prometheus.go

Co-authored-by: Pedro Gomes <[email protected]>

* modifications after call with pedro

* just getting state changes in metrics

---------

Signed-off-by: jsvisa <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: Makis Christou <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: YeahNotSewerSide <[email protected]>
Co-authored-by: Delweng <[email protected]>

state/metrics.go

@@ -0,0 +1,10 @@
+// Copyright (c) 2024 The VeChainThor developers
+//
+// Distributed under the GNU Lesser General Public License v3.0 software license, see the accompanying
+// file LICENSE or <https://www.gnu.org/licenses/lgpl-3.0.html>
+
+package state
+
+import "github.com/vechain/thor/v2/metrics"
+
+var metricAccountChanges = metrics.LazyLoadCounter("account_state_changes_count")

state/state.go

@@ -538,6 +538,8 @@ func (s *State) Stage(newVer trie.Version) (*Stage, error) {
 					return err
 				}
 			}
+			// Just once for the account trie.
+			metricAccountChanges().Add(int64(len(changes)))
 			return nil
 		},
 	}, nil