[#22537] YSQL: Benchmark restart read requests metric.

[pao214]

Motivation

YugabyteDB provides read-after-commit-visibility guarantee as described by the read restart error section in Yugabyte Docs. This guarantee prevents the stale read anomaly and we need a mechanism to detect stale reads. Moreover, several optimizations are introduced into the database to reduce the number of read restart errors, and it is desirable to have benchmarks that demonstrate these improvements.

Changes

1. A stale read detector app.
2. A mechanism to throttle read and write throughput.
3. A mechanism to execute read and write operations in lock step.
4. Collector for read restart prom metrics.
5. Cosmetic changes: use console style in the README, use YSQL where I could in comments.

Stale read detection example

Cluster Setup

To reproduce stale reads,

1. Limit hybrid time propagation.
   a. Use replication_factor=1
   b. Use a single zone.
   c. heartbeat_interval_ms=10000
2. Disable clock skew related checks.
   T-Server GFlags
   a. max_clock_skew_usec=0
   b. fail_on_out_of_range_clock_skew=false
   c. clock_skew_force_crash_bound_usec=0
3. Simulate clock skew.
   a. sudo systemctl stop chronyd
   b. sudo timedatectl set-ntp false
   c. sudo date -s "$(date -d '+0.4 seconds' '+%Y-%m-%d %H:%M:%S')"

As a final step, verify the clock skew between the cluster nodes using the /tablet-server-clocks page of the WebUI.

Command

$ java -jar target/yb-sample-apps.jar --workload SqlStaleReadDetector --nodes x.x.x.x:5433,y.y.y.y:5433,z.z.z.z:5433 >& ../temp.log

Sample log output

... Stale read detected! ...
...
312689 [Thread-0] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 1.07 ops/sec (215.67 ms/op), 300 total ops  |  Write: 55.06 ops/sec (18.09 ms/op), 17651 total
ops  |  Stale reads: x total ops | Num restart read requests: y | Uptime: 312638 ms |