Skip to content

Commit

Permalink
feat(cosmic-swingset): Split inbound queue length metrics by queue na…
Browse files Browse the repository at this point in the history
…me (#10904)

Fixes #10900

## Description
The first commit is pure refactoring. After that, kernel-stat.js is updated to document QUEUE_METRICS ({length,add,remove} triples sharing a common "cosmic_swingset_inbound_queue" prefix) and `makeInboundQueueMetrics` is updated to be an internal detail of that file (i.e., rather than constructing `inboundQueueMetrics`, launch-chain.js passes in initial queue lengths to `exportKernelStats` and extracts it from the response).

This PR changes Prometheus `GET /metrics` output to add "queue" dimensional labels corresponding with the containing ~"forced"/"high-priority"/"queued" [CrankerPhase](https://github.com/Agoric/agoric-sdk/blob/f047c93df3ba5871cd9f576782f133e6d2bd21ff/packages/cosmic-swingset/src/launch-chain.js#L91)~ ("forced"/"priority"/"inbound" as of this work):
```diff
 # HELP cosmic_swingset_inbound_queue_length inbound queue length
 # TYPE cosmic_swingset_inbound_queue_length gauge
-cosmic_swingset_inbound_queue_length 0 1738018661949
+cosmic_swingset_inbound_queue_length{queue="forced"} 0 1738018661949
+cosmic_swingset_inbound_queue_length{queue="priority"} 0 1738018661949
+cosmic_swingset_inbound_queue_length{queue="inbound"} 0 1738018661949
 # HELP cosmic_swingset_inbound_queue_add inbound queue increments
 # TYPE cosmic_swingset_inbound_queue_add counter
-cosmic_swingset_inbound_queue_add 0 1738018661949
+cosmic_swingset_inbound_queue_add{queue="forced"} 0 1738018661949
+cosmic_swingset_inbound_queue_add{queue="priority"} 0 1738018661949
+cosmic_swingset_inbound_queue_add{queue="inbound"} 0 1738018661949
 # HELP cosmic_swingset_inbound_queue_remove inbound queue decrements
 # TYPE cosmic_swingset_inbound_queue_remove counter
-cosmic_swingset_inbound_queue_remove 0 1738018661949
+cosmic_swingset_inbound_queue_remove{queue="forced"} 0 1738018661949
+cosmic_swingset_inbound_queue_remove{queue="priority"} 0 1738018661949
+cosmic_swingset_inbound_queue_remove{queue="inbound"} 0 1738018661949
```

And slog entries to capture those same dimensions in object property name suffixes:
```diff
 {
   "type": "cosmic-swingset-end-block-finish",
   "blockHeight": 21,
   "blockTime": 1738022131,
   "inboundQueueStats": {
+    "cosmic_swingset_inbound_queue_length_forced": 0,
+    "cosmic_swingset_inbound_queue_length_priority": 0,
+    "cosmic_swingset_inbound_queue_length_inbound": 0,
     "cosmic_swingset_inbound_queue_length": 0,
+    "cosmic_swingset_inbound_queue_add_forced": 4,
+    "cosmic_swingset_inbound_queue_add_priority": 0,
+    "cosmic_swingset_inbound_queue_add_inbound": 0,
     "cosmic_swingset_inbound_queue_add": 4,
+    "cosmic_swingset_inbound_queue_remove_forced": 4,
+    "cosmic_swingset_inbound_queue_remove_priority": 0,
+    "cosmic_swingset_inbound_queue_remove_inbound": 0,
     "cosmic_swingset_inbound_queue_remove": 4
   },
   "time": 1738022137.072313,
   "monotime": 733.9035001497268
 }
```

### Security Considerations
None known.

### Scaling Considerations
This does increase the size of cosmic-swingset-begin-block and cosmic-swingset-end-block-finish slog entries, but I don't think to an extent that should dissuade us from including the detail.

### Documentation Considerations
This introduces implicit documentation of cosmic-swingset metrics.

### Testing Considerations
Expected output was confirmed manually (see the diffs above), but automated verification of expected instrument names, but automated verification of expected metric names would be better.

### Upgrade Considerations
I believe that existing consumers of cosmic_swingset_inbound_queue_{length,add,remove} data should tolerate the new dimensionality and naïvely sum over all labels, giving the same data as before. Verification in live networks should cover that in addition to the above slogfile/Prometheus scrape[^1] checks.

[^1]: as configured by environment variable `OTEL_EXPORTER_PROMETHEUS_PORT`.
  • Loading branch information
mergify[bot] authored Jan 29, 2025
2 parents 1d8b7d2 + 529d2f6 commit 74fc45b
Show file tree
Hide file tree
Showing 2 changed files with 253 additions and 115 deletions.
Loading

0 comments on commit 74fc45b

Please sign in to comment.