Skip to content

Commit

Permalink
adding images and moving suppression rules
Browse files Browse the repository at this point in the history
Signed-off-by: Amit Galitzky <[email protected]>
  • Loading branch information
amitgalitz committed Jan 30, 2025
1 parent ef1a34c commit 9b89af6
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 24 deletions.
57 changes: 33 additions & 24 deletions _observing-your-data/ad/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,39 @@ For each configured feature, you can also select the criteria for considering da

For example, when creating a detector for the cpu_utilization field, users may choose to register anomalies only when the value spikes to reduce alert fatigue.


#### Suppressing anomalies with threshold-based rules

In the Feature selection pane, you can suppress anomalies by setting rules that define acceptable differences between the expected and actual values, either as an absolute value or a relative percentage. This helps reduce false anomalies caused by minor fluctuations, allowing you to focus on significant deviations.

Suppose you want to detect substantial changes in log volume while ignoring small variations that are not meaningful. Without customized settings, the system might generate false alerts for minor changes, making it difficult to identify true anomalies. By setting suppression rules, you can ignore minor deviations and focus on real anomalous patterns.

To suppress anomalies for deviations of less than 30% from the expected value, you can set the following rules:

```
Ignore anomalies for feature logVolume when the actual value is no more than 30% above the expected value.
Ignore anomalies for feature logVolume when the actual value is no more than 30% below the expected value.
```

Example UI:

<img src="{{site.url}}{{site.baseurl}}/images/anomaly-detection/add-feature-with-relative-rules.png" alt="Interface of adding a feature with suppression rules" width="800" height="800">


If you expect that the log volume should differ by at least 10,000 from the expected value before being considered an anomaly, you can set absolute thresholds:

```
Ignore anomalies for feature logVolume when the actual value is no more than 10000 above the expected value.
Ignore anomalies for feature logVolume when the actual value is no more than 10000 below the expected value.
```

Example UI:

<img src="{{site.url}}{{site.baseurl}}/images/anomaly-detection/add-suppression-rules-absolute.png" alt="Interface of adding suppression rules with absolute rules" width="800" height="800">

If no custom suppression rules are set, then the system defaults to a filter that ignores anomalies with deviations of less than 20% from the expected value for each enabled feature.


A multi-feature model correlates anomalies across all its features. The [curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality) makes it less likely that multi-feature models will identify smaller anomalies as compared to a single-feature model. Adding more features can negatively impact the [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall) of a model. A higher proportion of noise in your data can further amplify this negative impact. Selecting the optimal feature set is usually an iterative process. By default, the maximum number of features for a detector is `5`. You can adjust this limit using the `plugins.anomaly_detection.max_anomaly_features` setting.
{: .note}

Expand Down Expand Up @@ -203,30 +236,6 @@ Using these options can improve recall in anomaly detection. For instance, if yo
Be cautious when imputing extensively missing data, as excessive gaps can compromise model accuracy. Quality input is critical---poor data quality leads to poor model performance. The confidence score also decreases when imputations occur. You can check whether a feature value has been imputed using the `feature_imputed` field in the anomaly results index. See [Anomaly result mapping]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/result-mapping/) for more information.
{: note}

### Suppressing anomalies with threshold-based rules

In the Feature selection pane, you can suppress anomalies by setting rules that define acceptable differences between the expected and actual values, either as an absolute value or a relative percentage. This helps reduce false anomalies caused by minor fluctuations, allowing you to focus on significant deviations.

Suppose you want to detect substantial changes in log volume while ignoring small variations that are not meaningful. Without customized settings, the system might generate false alerts for minor changes, making it difficult to identify true anomalies. By setting suppression rules, you can ignore minor deviations and focus on real anomalous patterns.

To suppress anomalies for deviations of less than 30% from the expected value, you can set the following rules:

```
Ignore anomalies for feature logVolume when the actual value is no more than 30% above the expected value.
Ignore anomalies for feature logVolume when the actual value is no more than 30% below the expected value.
```

Ensure that a feature, for example, `logVolume`, is properly defined in your model. Suppression rules are tied to specific features.
{: .note}

If you expect that the log volume should differ by at least 10,000 from the expected value before being considered an anomaly, you can set absolute thresholds:

```
Ignore anomalies for feature logVolume when the actual value is no more than 10000 above the expected value.
Ignore anomalies for feature logVolume when the actual value is no more than 10000 below the expected value.
```

If no custom suppression rules are set, then the system defaults to a filter that ignores anomalies with deviations of less than 20% from the expected value for each enabled feature.

### Previewing sample anomalies

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 9b89af6

Please sign in to comment.