Skip to content

Commit

Permalink
address Amit's comments
Browse files Browse the repository at this point in the history
Signed-off-by: Kaituo Li <[email protected]>
  • Loading branch information
kaituo committed Jan 31, 2025
1 parent 6c5ef37 commit 4d442f1
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions _posts/2025-01-23-ad-rule-cx-success.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ To address these challenges, we've implemented significant improvements in OpenS

The suppression rules feature is particularly powerful for handling common scenarios that previously generated false positives. For example, in monitoring transaction volumes, EBSCO could configure rules to suppress alerts when variations stay within expected thresholds (such as ignoring drops less than 25% and increases greater than 50% above normal levels), while still detecting significant drops that might indicate real issues.

Additionally, in OpenSearch 2.19 and above, we have added the capability for users to add more general criteria for what a detector will consider as an anomaly. Each feature in a detector can now be configured so that we only look at either spikes or dips in the data for that feature. For example, if a user is configuring a feature which looks at CPU data across their fleet, they can configure their features so anomalies are only detected if the actual values are higher than the expected values because of a spike in CPU. A dip in CPU in this case would not trigger anomalies.

## Implementation and Results
After implementing these improvements, EBSCO saw significant enhancements in their anomaly detection capabilities:

Expand All @@ -54,16 +56,14 @@ Initially without rule-based AD, the system reported multiple anomalies—even f
By configuring a rule to only flag anomalies that deviate at least 10% (either above or below) from the expected value, we filtered out the noise. This ensures that only the meaningful spikes or dips are detected.
Note: The user interface for setting rules may change. Refer to the [latest documentation page](https://opensearch.org/docs/latest/observing-your-data/ad/index/) for details.

![rule](/assets/media/blog-images/2025-01-23-ad-rule-cx-success/rule.png){:class="img-centered"}
![rule](/assets/media/blog-images/2025-01-23-ad-rule-cx-success/new_rule.png){:class="img-centered"}

Anomaly result with rule:

![result-after](/assets/media/blog-images/2025-01-23-ad-rule-cx-success/result-after.png){:class="img-centered"}

Note, if no custom rule is specified, AD defaults to a built-in filter that suppresses anomalies with less than a 20% deviation from the expected value for each enabled feature.

Additionally, in OpenSearch 2.19 and above, we have added the capability for users to add more general criteria for what a detector will consider as an anomaly. Each feature in a detector can now be configured so that we only look at either spikes or dips in the data for that feature. For example, if a user is configuring a feature which looks at CPU data across their fleet, they can configure their features so anomalies are only detected if the actual values are higher than the expected values because of a spike in CPU. A dip in CPU in this case would not trigger anomalies.

## Technical Details
The core of suppression rules is expected normal values. To build an expected value for an anomalous point, Random Cut Forest (RCF), the algorithm behind OpenSearch Anomaly Detection, replaces the anomalous point’s most suspicious dimension with a typical value learned from historical data. Here, each dimension corresponds to a metric value within a context or recent-history window, and the overall anomaly score depends on these dimensions.
RCF partitions the data by selecting a dimension $i$ with probability proportional to its range, then choosing a random cut value within that range. If $l_i$ is the difference between the maximum and minimum values observed in dimension $i$, the probability of cutting along $i$ is $\frac{l_i}{\sum_j l_j}$. These random cuts isolate anomalous points, and RCF sums the anomaly scores across all dimensions to compute the final anomaly score. To identify which dimension contributes most to an anomaly, RCF tracks each dimension’s portion of the total anomaly score. The dimension with the highest contribution is deemed the most suspicious.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.

0 comments on commit 4d442f1

Please sign in to comment.