Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update expression and anomaly detector documentation #8041

Merged
merged 14 commits into from
Dec 3, 2024
Merged

Update expression and anomaly detector documentation #8041

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
2178906
Update expression and anomaly detector documentation
kkondaka Aug 16, 2024
12ea363
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
60b4977
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
3fbd722
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
d442ceb
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
b0c9fc4
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
3e86e12
Merge branch 'main' into expr-doc-fix
vagimeli Aug 19, 2024
8730369
Merge branch 'main' into expr-doc-fix
vagimeli Aug 22, 2024
2d870c9
Merge branch 'main' into expr-doc-fix
vagimeli Oct 9, 2024
10ee243
Merge branch 'main' into expr-doc-fix
vagimeli Oct 16, 2024
578a6a2
Merge branch 'main' into expr-doc-fix
vagimeli Nov 1, 2024
9b37a75
Merge branch 'main' into expr-doc-fix
vagimeli Nov 13, 2024
abebfd2
Merge branch 'main' into expr-doc-fix
Naarcha-AWS Dec 3, 2024
d1a346b
Apply suggestions from code review
Naarcha-AWS Dec 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _data-prepper/pipelines/configuration/processors/anomaly-detector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
| `sample_size` | `256` | 100--2500 | The sample size used in the ML algorithm. |
| `time_decay` | `0.1` | 0--1.0 | The time decay value used in the ML algorithm. Used as the mathematical expression `timeDecay` divided by `SampleSize` in the ML algorithm. |
| `type` | `metrics` | N/A | The type of data sent to the algorithm. |
| `output_after` | 32 | N/A | Indicates the number of events to consume before outputting anamolies |

Check failure on line 56 in _data-prepper/pipelines/configuration/processors/anomaly-detector.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: anamolies. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: anamolies. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/processors/anomaly-detector.md", "range": {"start": {"line": 56, "column": 91}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
| `version` | `1.0` | N/A | The algorithm version number. |

## Usage
Expand Down
64 changes: 62 additions & 2 deletions _data-prepper/pipelines/expression-syntax.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
|----------------------|-------------------------------------------------------|---------------|
| `()` | Priority expression | Left to right |
| `not`<br> `+`<br> `-`| Unary logical NOT<br>Unary positive<br>Unary negative | Right to left |
| `*`, `/` | Multiplication and division operators | Left to right |
| `+`, `-` | Addition and subtraction operators | Left to right |
| `+` | String concatenation operator | Left to right |
| `<`, `<=`, `>`, `>=` | Relational operators | Left to right |
| `==`, `!=` | Equality operators | Left to right |
| `and`, `or` | Conditional expression | Left to right |
Expand Down Expand Up @@ -78,7 +81,6 @@
<Any> or <Any>
not <Any>
```
{% include copy-curl.html %}

The following are some example conditional expressions:

Expand All @@ -91,9 +93,64 @@
```
{% include copy-curl.html %}

### Arithmetic expressions

Arithmetic expressions allow you to do few basic arithmetic operations. Arithmetic expressions may be combined with conditional expressions to make more complex conditional expressions. The available arithmetic operators are `+`, `-`, `*`, and `/` for doing addition, subtraction, multiplication and division respectively. The syntax for using the arithmetic operators is as follows:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```
<Any> + <Any>
<Any> - <Any>
<Any> * <Any>
<Any> / <Any>
```

The following are some example arithmetic expressions:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```
/value + length(/message)
/bytes / 1024
/value1 - /value2
/TimeInSeconds * 1000
```
{% include copy-curl.html %}

The following are some example arithmetic expressions used in conditional expressions :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following are some example arithmetic expressions used in conditional expressions :
The following are example arithmetic expressions that can be used in conditional expressions:


```
/value + length(/message) > 200
/bytes / 1024 < 10
/value1 - /value2 != /value3 + /value4
```
{% include copy-curl.html %}

### String concatenation expressions

String concatenation expressions allow you to concatenate strings to generate new strings. The concatenated strings can be used in conditional expressions too. The syntax for using string concatenation is as follows:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```
<String Variable or String Literal> + <String Variable or String Literal>
```

The following are some example string concatenation expressions:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```
/name + "suffix"
"prefix" + /name
"time of " + /timeInMs + " ms"
```
{% include copy-curl.html %}

The following are some example string concatenation expressions used in conditional expressions :
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```
/service + ".com" == /url
"www." + /service != /url
```
{% include copy-curl.html %}

### Reserved symbols

Reserved symbols are symbols that are not currently used in the expression syntax but are reserved for possible future functionality or extensions. Reserved symbols include `^`, `*`, `/`, `%`, `+`, `-`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.
Reserved symbols are symbols that are not currently used in the expression syntax but are reserved for possible future functionality or extensions. Reserved symbols include `^`, `%`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Syntax components

Expand Down Expand Up @@ -170,6 +227,9 @@
| `()` | Priority expression | Yes | `/a==(/b==200)`<br>`/a in ({200})` | `/status in({200})` |
| `in`, `not in` | Set operators | Yes | `/a in {200}`<br>`/a not in {400}` | `/a in{200, 202}`<br>`/a not in{400}` |
| `<`, `<=`, `>`, `>=` | Relational operators | No | `/status < 300`<br>`/status>=300` | |
| `+` | String concatenation operator | No | `/status_code + /message + "suffix"`

Check failure on line 230 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 230, "column": 92}}}, "severity": "ERROR"}
| `+`, `-` | Arithmetic addition and subtraction operators | No | `/status_code + length(/message) - 2`

Check failure on line 231 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 231, "column": 93}}}, "severity": "ERROR"}
| `*`, `/` | Multiplication and division operators | No | `/status_code * length(/message) / 3`

Check failure on line 232 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 232, "column": 93}}}, "severity": "ERROR"}
| `=~`, `!~` | Regex equality operators | No | `/msg =~ "^\w*$"`<br>`/msg=~"^\w*$"` | |
| `==`, `!=` | Equality operators | No | `/status == 200`<br>`/status_code==200` | |
| `and`, `or`, `not` | Conditional operators | Yes | `/a<300 and /b>200` | `/b<300and/b>200` |
Expand Down
Loading