Skip to content

Commit

Permalink
Update _data-prepper/pipelines/configuration/sinks/s3.md
Browse files Browse the repository at this point in the history 
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Melissa Vagi <[email protected]>
  • Loading branch information
@vagimeli @natebower
vagimeli and natebower authored Jun 25, 2024
1 parent 11a2612 commit cb19c6b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion _data-prepper/pipelines/configuration/sinks/s3.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ The `codec` determines how the `s3` source formats data written to each S3 objec

The `avro` codec writes an event as an [Apache Avro](https://avro.apache.org/) document. Because Avro requires a schema, you may either define the schema or have Data Prepper automatically generate it. Defining your own schema is recommended because this will allow it to be tailored to your particular use case.

Check failure on line 170 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 170, "column": 48}}}, "severity": "ERROR"}

Check failure on line 170 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 170, "column": 98}}}, "severity": "ERROR"}

When you provide your own Avro schema, that schema defines the final structure of your data. Any extra values inside any incoming events that are not mapped in the Arvo schema will not be included in the final destination. Data Prepper does not allow the use of the `include_keys` or `exclude_keys` with a custom schema. This is to avoid confusion between a custom Arvo schema and the `include_keys` or `exclude_keys` sink configurations.
When you provide your own Avro schema, that schema defines the final structure of your data. Any extra values in any incoming events that are not mapped in the Avro schema will not be included in the final destination. Data Prepper does not allow the use of `include_keys` or `exclude_keys` with a custom schema so as to avoid confusion between a custom Avro schema and the `include_keys` or `exclude_keys` sink configurations.

Check failure on line 172 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 172, "column": 27}}}, "severity": "ERROR"}

Check failure on line 172 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 172, "column": 161}}}, "severity": "ERROR"}

Check failure on line 172 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Avro. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 172, "column": 355}}}, "severity": "ERROR"}

In cases where your data is uniform, you may be able to automatically generate a schema. Auto-generated schemas are based on the first event that the codec receives. The schema will only contain keys from this event, and you must have all keys present in all events in order for the auto-generated schema to produce a working schema. Auto-generated schemas make all fields nullable. Use the `include_keys` and `exclude_keys` sink configurations to control what data is included in the auto-generated schema.

Check failure on line 174 in _data-prepper/pipelines/configuration/sinks/s3.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: nullable. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: nullable. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/sinks/s3.md", "range": {"start": {"line": 174, "column": 374}}}, "severity": "ERROR"}

Expand Down

0 comments on commit cb19c6b

Please sign in to comment.