admission: insufficient tokens under high write load and WAL failover

[sumeerbhola]

Observations from https://cockroachlabs.slack.com/archives/C06UFBJ743F/p1736348529278319?thread_ts=1736262662.853989&cid=C06UFBJ743F:

A single flush adds a huge amount of data (~1.6GiB and 800 files) to L0. AC's ioLoadListener notices this at the same time as Pebble compactions (out of L0) start ramping up. Within 30s the backlog is cleared. During this time period, AC tokens are insufficient for regular work. The memtable backlog did not cause a write stall since WAL failover was active (the primary disk was not actually stalled, but slower because of high bandwidth usage -- this test was not using AC's bandwidth limiting).

The ioLoadListener can use a low value of compaction bandwidth for deciding on tokens for various reasons:

• [AC deficiency] Compactions haven't yet ramped up enough. The ioLoadListener samples at 15s intervals. The assumption has been that if it sees L0 overload, Pebble has been seeing it for a few seconds. This doesn't apply when all the L0 bytes were added via a single flush. In the above experiment, there probably wasn't a ramp up issue since 800 files results in an uncompensated score of 1.6 and 2 sublevels (prior to the flush) result in a score of 2.0. It was running with compaction concurrency of 3, and competing with L3 and L4 which also had high scores.
• [AC/Pebble inconsistency] Pebble uses L0CompactionThreshold=2, L0CompactionFileThreshold=500. 1 sublevel results in a score of 1. 500 files result in a score of 1. So 1 sublevel==500 files.
  AC uses 20 sublevels == 1000 files in its scoring. This is wildly inconsistent. It the latter had a higher file count threshold, AC would give out more tokens.
• [AC deficiency] During WAL failover ioLoadListener stops changing its estimates of the compacted bytes. This means that if WAL failover is only due to slight slowness and the compactions out of L0 are successfully ramping up, ioLoadListener will ignore this ramp up and give out fewer tokens. It should update its estimates if the observed compaction bytes are increasing.

Jira issue: CRDB-46185