implement more useful alerts on chainhook node health #466
Assignees
Milestone
Comments
|
|
|
@lgalabru to indicate which alarms are needed on this repo. |
|
Just adding a quick follow up here - I'm taking on this issue. We'll be adding alerts to Grafana that will notify us if Chainhook block ingestion falls behind block production for either the stacks or bitcoin nodes. This is requiring us to update Chainhook to emit Prometheus metrics. PR will be incoming soon! |
MicaiahReid
changed the title
2 tasks
MicaiahReid
added a commit
that referenced
this issue
Feb 8, 2024
### Description To enable improved alerts on downtime for Hiro's hosted Chainhook service, we need Chainhook to provide metrics that can be ingested by Prometheus. This PR changes some how we track our metrics (that are served over the `/ping` endpoint of the observer) to enable Prometheus compatibility, and adds a flag to optionally start a server to supply metrics to a Prometheus client. ### Example Starting chainhook with the `--prometheus-port XXXX` flag now enables a service that can supply Prometheus metrics at `localhost:XXXX/metrics`. If using a config file, this option can be specified via: ```yaml [monitoring] prometheus_monitoring_port = XXXX ``` Chainhook will behave as usual with this flag ommitted - metrics can still be retrieved via the observer's `/ping` endpoint, but they will not be formatted for ingestion by a Prometheus client. --- ### Checklist - [X] All tests pass - [X] Tests added in this PR (if applicable) Fixes #474, addresses #466
MicaiahReid
added a commit
that referenced
this issue
Feb 8, 2024
### Description To enable improved alerts on downtime for Hiro's hosted Chainhook service, we need Chainhook to provide metrics that can be ingested by Prometheus. This PR changes some how we track our metrics (that are served over the `/ping` endpoint of the observer) to enable Prometheus compatibility, and adds a flag to optionally start a server to supply metrics to a Prometheus client. ### Example Starting chainhook with the `--prometheus-port XXXX` flag now enables a service that can supply Prometheus metrics at `localhost:XXXX/metrics`. If using a config file, this option can be specified via: ```yaml [monitoring] prometheus_monitoring_port = XXXX ``` Chainhook will behave as usual with this flag ommitted - metrics can still be retrieved via the observer's `/ping` endpoint, but they will not be formatted for ingestion by a Prometheus client. --- ### Checklist - [X] All tests pass - [X] Tests added in this PR (if applicable) Fixes #474, addresses #466
|
Closing, because everything is complete on the chainhook side with PR #473. The devops side is being tracked by https://github.com/hirosystems/devops/issues/1543 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Now that the ping endpoint is reporting the bitcoin / stacks block heights, the alarming should be revisited, taking this into account.
The text was updated successfully, but these errors were encountered: