diff --git a/_posts/2025-01-23-ad-rule-cx-success.md b/_posts/2025-01-23-ad-rule-cx-success.md index 94a99599d..b73f6e38d 100644 --- a/_posts/2025-01-23-ad-rule-cx-success.md +++ b/_posts/2025-01-23-ad-rule-cx-success.md @@ -34,6 +34,8 @@ To address these challenges, we've implemented significant improvements in OpenS The suppression rules feature is particularly powerful for handling common scenarios that previously generated false positives. For example, in monitoring transaction volumes, EBSCO could configure rules to suppress alerts when variations stay within expected thresholds (such as ignoring drops less than 25% and increases greater than 50% above normal levels), while still detecting significant drops that might indicate real issues. +Additionally, in OpenSearch 2.19 and above, we have added the capability for users to add more general criteria for what a detector will consider as an anomaly. Each feature in a detector can now be configured so that we only look at either spikes or dips in the data for that feature. For example, if a user is configuring a feature which looks at CPU data across their fleet, they can configure their features so anomalies are only detected if the actual values are higher than the expected values because of a spike in CPU. A dip in CPU in this case would not trigger anomalies. + ## Implementation and Results After implementing these improvements, EBSCO saw significant enhancements in their anomaly detection capabilities: @@ -54,7 +56,7 @@ Initially without rule-based AD, the system reported multiple anomalies—even f By configuring a rule to only flag anomalies that deviate at least 10% (either above or below) from the expected value, we filtered out the noise. This ensures that only the meaningful spikes or dips are detected. Note: The user interface for setting rules may change. Refer to the [latest documentation page](https://opensearch.org/docs/latest/observing-your-data/ad/index/) for details. -![rule](/assets/media/blog-images/2025-01-23-ad-rule-cx-success/rule.png){:class="img-centered"} +![rule](/assets/media/blog-images/2025-01-23-ad-rule-cx-success/new_rule.png){:class="img-centered"} Anomaly result with rule: @@ -62,8 +64,6 @@ Anomaly result with rule: Note, if no custom rule is specified, AD defaults to a built-in filter that suppresses anomalies with less than a 20% deviation from the expected value for each enabled feature. -Additionally, in OpenSearch 2.19 and above, we have added the capability for users to add more general criteria for what a detector will consider as an anomaly. Each feature in a detector can now be configured so that we only look at either spikes or dips in the data for that feature. For example, if a user is configuring a feature which looks at CPU data across their fleet, they can configure their features so anomalies are only detected if the actual values are higher than the expected values because of a spike in CPU. A dip in CPU in this case would not trigger anomalies. - ## Technical Details The core of suppression rules is expected normal values. To build an expected value for an anomalous point, Random Cut Forest (RCF), the algorithm behind OpenSearch Anomaly Detection, replaces the anomalous point’s most suspicious dimension with a typical value learned from historical data. Here, each dimension corresponds to a metric value within a context or recent-history window, and the overall anomaly score depends on these dimensions. RCF partitions the data by selecting a dimension $i$ with probability proportional to its range, then choosing a random cut value within that range. If $l_i$ is the difference between the maximum and minimum values observed in dimension $i$, the probability of cutting along $i$ is $\frac{l_i}{\sum_j l_j}$. These random cuts isolate anomalous points, and RCF sums the anomaly scores across all dimensions to compute the final anomaly score. To identify which dimension contributes most to an anomaly, RCF tracks each dimension’s portion of the total anomaly score. The dimension with the highest contribution is deemed the most suspicious. diff --git a/assets/media/blog-images/2025-01-23-ad-rule-cx-success/new_rule.png b/assets/media/blog-images/2025-01-23-ad-rule-cx-success/new_rule.png new file mode 100644 index 000000000..9d69f659b Binary files /dev/null and b/assets/media/blog-images/2025-01-23-ad-rule-cx-success/new_rule.png differ diff --git a/assets/media/blog-images/2025-01-23-ad-rule-cx-success/rule.png b/assets/media/blog-images/2025-01-23-ad-rule-cx-success/rule.png deleted file mode 100644 index d12edd84a..000000000 Binary files a/assets/media/blog-images/2025-01-23-ad-rule-cx-success/rule.png and /dev/null differ