feat: add metrics to proxy (#1017)

* feat: add metrics to proxy

feat: add request latency and count metrics

feat: add grafana dashboard

feat: add metrics endpoint to proxy config

chore: update changelog

docs: update README

chore: add prometheus.yml

* review: update RequestContext documentation

* review: replace lazy_static with LazyLock

* review: use always localhost for metrics host

* review: improve separators in metrics definition

* review: re-add metrics host configuration and default to localhost

* review: update prometheus.yml

* review: add information about grafana dashboard creation and export to readme

* review: add documentation on local installation of prometheus and grafana

* review: add units to histograms

* review: rename the tag in prometheus.yml to tx_prover

* review: fix WORKER_COUNT metric update

* review: fix WORKER_UTILIZATION updates

* review: update RequestQueue docs

* review: move WORKER_UNHEALTHY update logic

* review: fix WORKER_UTILIZATION desc

* review: rename WORKER_UTILIZATION to WORKER_BUSY

* review: add metrics host to readme

* review: mention config file by name

* review: update doc about prometheus and grafana

* review: add missing config field to readme example

* docs: improve prometheus.yml docs

* chore: add safety comment to metrics

* chore: initialize metrics on load balancer instance

* chore: update dashboard

* review: change unhealthy workers metric description

* review: add missing field to README example

* review: rewrite Grafana introduction on readme

* review: change WORKER_COUNT metric description

* review: do not count update worker requests for metrics

* review: move busy workers metric update

* review: update grafana dashboard

* review: update dashboard

* review: move metric update to create response function

* chore: address lint errors

* review: rename grafana dashboard file

* review: update dashboard

---------

Co-authored-by: Bobbin Threadbare <[email protected]>

CHANGELOG.md

@@ -4,8 +4,6 @@
 
 ### Changes
 
-- Implemented `to_hex` for `AccountIdPrefix` and `epoch_block_num` for `BlockHeader` (#1039).
-- Added tracing to the `miden-tx-prover` CLI (#1014).
 - Added health check endpoints to the prover service (#1006).
 - Implemented serialization for `AccountHeader` (#996).
 - Updated Pingora crates to 0.4 and added polling time to the configuration file (#997).
@@ -21,6 +19,9 @@
 - [BREAKING] Refactor error messages in `miden-lib` and `miden-tx` and use `thiserror` 2.0 (#1005).
 - [BREAKING] Extend `AccountId` to two `Felt`s and require block hash in derivation (#982).
 - Removed workers list from the proxy configuration file (#1018).
+- Added tracing to the `miden-tx-prover` CLI (#1014).
+- Added metrics to the `miden-tx-prover` proxy (#1017).
+- Implemented `to_hex` for `AccountIdPrefix` and `epoch_block_num` for `BlockHeader` (#1039).
 
 ## 0.6.2 (2024-11-20)
 

bin/tx-prover/Cargo.toml

@@ -54,6 +54,7 @@ figment = { version = "0.10", features = ["toml", "env"] }
 miden-lib = { workspace = true, default-features = false }
 miden-objects = { workspace = true, default-features = false }
 miden-tx = { workspace = true, default-features = false }
+prometheus = "0.13"
 prost = { version = "0.13", default-features = false, features = ["derive"] }
 reqwest = { version = "0.11" }
 serde = { version = "1.0", features = ["derive"] }

bin/tx-prover/README.md

@@ -59,8 +59,14 @@ max_queue_items = 10
 max_retries_per_request = 1
 # Maximum amount of requests that a given IP address can make per second
 max_req_per_sec = 5
+# Time to wait before checking the availability of workers
+available_workers_polling_time_ms = 20
 # Interval to check the health of the workers
 health_check_interval_secs = 1
+# Host of the metrics server
+prometheus_host = "127.0.0.1"
+# Port of the metrics server
+prometheus_port = 6192
 ```
 
 Then, to start the proxy service, you will need to run:
@@ -120,6 +126,30 @@ Then access the Jaeger UI at `http://localhost:16686/`.
 
 If Docker is not an option, Jaeger can also be set up directly on your machine or hosted in the cloud. See the [Jaeger documentation](https://www.jaegertracing.io/docs/) for alternative installation methods.
 
+## Metrics
+
+The proxy includes a service that exposes metrics to be consumed by [Prometheus](https://prometheus.io/docs/introduction/overview/). This service is always enabled and uses the host and port defined in the `miden-tx-prover.toml` file.
+
+The metrics architecture works by having the proxy expose metrics at an endpoint (`/metrics`) in a format Prometheus can read. Prometheus periodically scrapes this endpoint, adds timestamps to the metrics, and stores them in its time-series database. Then, we can use tools like Grafana to query Prometheus and visualize these metrics in configurable dashboards.
+
+The simplest way to install Prometheus and Grafana is by using Docker containers. To do so, run:
+
+```bash
+docker run \
+    -d \
+    -p 9090:9090 \
+    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
+    prom/prometheus
+
+docker run -d -p 3000:3000 --name grafana grafana/grafana-enterprise:latest
+```
+
+In case that Docker is not an option, Prometheus and Grafana can also be set up directly on your machine or hosted in the cloud. See the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/getting_started/) and [Grafana documentation](https://grafana.com/docs/grafana/latest/setup-grafana/) for alternative installation methods.
+
+A prometheus configuration file is provided in this repository, you will need to modify the `scrape_configs` section to include the host and port of the proxy service.
+
+Then, to add the new Prometheus collector as a datasource for Grafana, you can [follow this tutorial](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/existing-datasource/). A Grafana dashboard under the name `proxy_grafana_dashboard.json` is provided, see this [link](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) to import it. Otherwise, you can [create your own dashboard](https://grafana.com/docs/grafana/latest/getting-started/build-first-dashboard/) using the metrics provided by the proxy and export it by following this [link](https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/#export-a-dashboard-as-json).
+
 ## Features
 
 Description of this crate's feature: