[sophora-server] Adjust alert "SophoraServerAPISlow" (#141)

* [sophora-server] Adjust alert "SophoraServerAPISlow" * [sophora-server] Use variable in runbook for alert "SophoraServerAPISlow"
subshell · Jan 23, 2025 · f1db591 · f1db591
1 parent e67f23f
commit f1db591
Show file tree

Hide file tree

Showing 3 changed files with 5 additions and 5 deletions.
diff --git a/charts/sophora-server/Chart.yaml b/charts/sophora-server/Chart.yaml
@@ -15,7 +15,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 2.5.2
+version: 2.6.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

diff --git a/charts/sophora-server/alerting-runbook.md b/charts/sophora-server/alerting-runbook.md
@@ -21,7 +21,7 @@ This document is a reference to the alerts this Helm chart can fire.
 
 **Severity:** high
 
-**Summary:** The API of the server exhibits a response time exceeding 300ms for more than 15 minutes at the 95th percentile.
+**Summary:** The API of the server exhibits a response time exceeding ${threshold} for more than 15 minutes at the 95th percentile.
 
 **Remediation steps:**
 
@@ -105,4 +105,4 @@ This document is a reference to the alerts this Helm chart can fire.
 * Check if the primary server is running
 * Check the logs of the server
 * Check the logs of the primary server
-* Check whether there are any network issues
+* Check whether there are any network issues
diff --git a/charts/sophora-server/templates/prometheusrule.yaml b/charts/sophora-server/templates/prometheusrule.yaml
@@ -21,12 +21,12 @@ spec:
             runbook_url: 'https://github.com/subshell/helm-charts/blob/main/charts/sophora-server/alerting-runbook.md'
         - alert: SophoraServerAPISlow
           for: 15m
-          expr: 'histogram_quantile(0.95, sum(rate(sophora_server_contentmanager_call_duration_seconds_bucket{job="{{ include "sophora-server.fullname" . }}"}[1m])) by (pod, le)) > 0.3'
+          expr: 'histogram_quantile(0.95, sum(rate(sophora_server_contentmanager_call_duration_seconds_bucket{job="{{ include "sophora-server.fullname" . }}"}[1m])) by (pod, le)) > 0.5'
           labels:
             severity: high
           annotations:
             summary: Sophora Server API is slow
-            description: The API of the server "{{`{{ $labels.pod }}`}}" exhibits a response time exceeding 300ms for more than 15 minutes at the 95th percentile.
+            description: The API of the server "{{`{{ $labels.pod }}`}}" exhibits a response time exceeding 500ms for more than 15 minutes at the 95th percentile.
             runbook_url: 'https://github.com/subshell/helm-charts/blob/main/charts/sophora-server/alerting-runbook.md'
         - alert: SophoraServerAsyncEventQueueBlocked
           for: 10m