Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Expose lua filter type_array_key parameter #1323

Merged
merged 2 commits into from
Aug 26, 2024
Merged

feat: Expose lua filter type_array_key parameter #1323

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
649aeb5
feat: Expose lua filter type_array_key parameter
reegnz Aug 23, 2024
c4d1cfb
fix var name
reegnz Aug 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions apis/fluentbit/v1alpha2/plugins/filter/lua_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ type Lua struct {
// Note that starting from Fluent Bit v1.6 integer data types are preserved
// and not converted to double as in previous versions.
TypeIntKey []string `json:"typeIntKey,omitempty"`
// If these keys are matched, the fields are handled as array. If more than
// one key, delimit by space. It is useful the array can be empty.
TypeArrayKey []string `json:"typeArrayKey,omitempty"`
// If enabled, Lua script will be executed in protected mode.
// It prevents to crash when invalid Lua script is executed. Default is true.
ProtectedMode *bool `json:"protectedMode,omitempty"`
Expand Down Expand Up @@ -76,12 +79,17 @@ func (l *Lua) Params(_ plugins.SecretLoader) (*params.KVs, error) {
kvs.Insert("type_int_key", strings.Join(l.TypeIntKey, " "))
}

if l.TypeArrayKey != nil && len(l.TypeArrayKey) > 0 {
kvs.Insert("type_array_key", strings.Join(l.TypeIntKey, " "))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be kvs.Insert("type_array_key", strings.Join(l.TypeArrayKey, " ")) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, right, my mistake. fixing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjaminhuo please check again, now that should be fixed.

}

if l.ProtectedMode != nil {
kvs.Insert("protected_mode", strconv.FormatBool(*l.ProtectedMode))
}

if l.TimeAsTable {
kvs.Insert("time_as_table", "true")
}

return kvs, nil
}
5 changes: 5 additions & 0 deletions apis/fluentbit/v1alpha2/plugins/filter/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_clusterfilters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down
7 changes: 7 additions & 0 deletions charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_filters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down
7 changes: 7 additions & 0 deletions config/crd/bases/fluentbit.fluent.io_clusterfilters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down
7 changes: 7 additions & 0 deletions config/crd/bases/fluentbit.fluent.io_filters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down
1 change: 1 addition & 0 deletions docs/plugins/fluentbit/filter/lua.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,6 @@ The Lua Filter allows you to modify the incoming records using custom Lua Script
| call | Lua function name that will be triggered to do filtering. It's assumed that the function is declared inside the Script defined above. | string |
| code | Inline LUA code instead of loading from a path via script. | string |
| typeIntKey | If these keys are matched, the fields are converted to integer. If more than one key, delimit by space. Note that starting from Fluent Bit v1.6 integer data types are preserved and not converted to double as in previous versions. | []string |
| typeArrayKey | If these keys are matched, the fields are handled as array. If more than one key, delimit by space. It is useful the array can be empty. | []string |
| protectedMode | If enabled, Lua script will be executed in protected mode. It prevents to crash when invalid Lua script is executed. Default is true. | *bool |
| timeAsTable | By default when the Lua script is invoked, the record timestamp is passed as a Floating number which might lead to loss precision when the data is converted back. If you desire timestamp precision enabling this option will pass the timestamp as a Lua table with keys sec for seconds since epoch and nsec for nanoseconds. | bool |
2 changes: 1 addition & 1 deletion docs/plugins/fluentbit/output/open_telemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The OpenTelemetry plugin allows you to take logs, metrics, and traces from Fluen
| tracesUri | Specify an optional HTTP URI for the target web server listening for traces, e.g: /v1/traces | string |
| header | Add a HTTP header key/value pair. Multiple headers can be set. | map[string]string |
| logResponsePayload | Log the response payload within the Fluent Bit log. | *bool |
| logsBodyKeyAttributes | Adds unmatched keys as attributes when true. | *bool |
| addLabel | This allows you to add custom labels to all metrics exposed through the OpenTelemetry exporter. You may have multiple of these fields. | map[string]string |
| logsBodyKeyAttributes | If true, remaining unmatched keys are added as attributes. | *bool |
| tls | | *[plugins.TLS](../tls.md) |
| networking | Include fluentbit networking options for this output-plugin | *plugins.Networking |
14 changes: 14 additions & 0 deletions manifests/setup/fluent-operator-crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down Expand Up @@ -14959,6 +14966,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down
14 changes: 14 additions & 0 deletions manifests/setup/setup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down Expand Up @@ -14959,6 +14966,13 @@ spec:
If you desire timestamp precision enabling this option will pass the timestamp as
a Lua table with keys sec for seconds since epoch and nsec for nanoseconds.
type: boolean
typeArrayKey:
description: |-
If these keys are matched, the fields are handled as array. If more than
one key, delimit by space. It is useful the array can be empty.
items:
type: string
type: array
typeIntKey:
description: |-
If these keys are matched, the fields are converted to integer.
Expand Down
Loading