Skip to content

2.8.0
Compare
Choose a tag to compare
@github-actions github-actions released this 16 May 20:36
· 380 commits to main since this release
2.8.0
10b94f1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

2024-05-16 Version 2.8.0

Features

  • Support Full load and CDC from AWS DocumentDB [#4534] (#4534)
  • Support conditional expression to evaluate based on the data type for a given field (#4478 #4523, #4500))
  • Allow using event fields in s3 sink object_key [#3310] (#3310)
  • Support ndjson with a codec [#2700] (#2700)
  • Support S3 bucket ownership validation on the S3 sink (#4468)
  • Support encoding JSON (#832 #4514)
  • Support for Event Json input and output codecs (#4436)
  • Add support for dynamic bucket and default bucket in S3 sink (#4402)
  • Add support to export/full load MongoDB/DocumentDB collection with _id field of different data type (#4503)

Enhancements

  • HTTP data chunking support for kafka buffer (#4475)
  • ENH: automatic credential refresh in kafka source (#4258)
  • Add creation and aggregation of dynamic S3 groups based on events (#4346)
  • Truncate Processor: Add support to truncate all fields in an event (#4317)
  • Provide validations of AWS accountIds (#4398)
  • Better metrics on OpenSearch document errors (#4344)
  • Better metrics for OpenSearch duplicate documents (#4343)
  • Address route and subpipeline for pipeline tranformation (#4528)
  • Add support for BigDecimal in ConvertType processor (#4316)
  • Checkpoint records at an interval for TPS case when AckSet is enabled (#4526)
  • Write stream events that timeout to write to internal buffer in separate thread (#4524)
  • Key value processor enhancements (#4521)
  • Add bucket owner support to s3 sink (#4504)
  • Initial work to support core data types in Data Prepper (#4496)
  • Changing logging level for config transformation and fixing rule (#4466)
  • Add folder-based partitioning for s3 scan source (#4455)
  • Pipeline Configuration Transformation (#4446)
  • Added support for multiple workers in S3 Scan Source (#4439)
  • Bootstrap the RuleEngine package (#4442)
  • Make s3 partition size configurable and add unit test for S3 partition creator classes (#4437)
  • Remove creating S3 prefix path partition upfront (#4432)
  • Change s3 sink client to async client (#4425)
  • Create new codec for each s3 group in s3 sink (#4410)
  • Validate the AWS account Id in the S3 source using a new annotation (#4400)
  • Add server connections metric to http and otel sources (#4393)
  • Log the User-Agent when Data Prepper shuts down from POST /shutdown (#4390)
  • Add aggregate_threshold with maximum_size to s3 sink (#4385)
  • Refactor PipelinesDataFlowModelParser to take in an InputStream instead of a file path (#4289)
  • Add support to use old ddb stream image for REMOVE events (#4275)

Bug Fixes

  • Fix count aggregation exemplar data (#4341)
  • Revert HTTP data chunking changes for kafka buffer done in PR 4266 (#4329)
  • Fix Router performance issue (#4327)
  • Do not require field_split_characters to not be empty for key_value processor (#4358)
  • Do not write empty lists of DlqObject to the DLQ (#4403)
  • Fix transient test failure for subpipelines (#4479)
  • Fix JacksonEvent to propagate ExternalOriginalTime if its set at the time of construction (#4489)
  • FIX: null certificate value should be valid in opensearch connection (#4494)
  • [BUG]Incorrect Behavior of Obfuscate Processor with Predefined Pattern "%{CREDIT_CARD_NUMBER}" (#4340)
  • [BUG] Empty DLQ entries when version conflicts occur (#4301)
  • [BUG] otel sources should show a more clear exception when receiving data that cannot be processed based on the configured compression type (#4022)
  • [BUG] : unable to set field_delimiter_regex (#2946)
  • Fix aggregate processor local mode (#4529)
  • Add long as a target type for convert_entry_type processor (#4120)
  • Fix write json basic test (#4527)
  • Fix depth field in template (#4509)
  • Fix for S3PartitionCreatorScheduler ConcurrentModification Exception (#4473)
  • Fix acknowledgements in DynamoDB (#4419)
  • Fix DocumentDB source S3PathPrefix null or empty (#4472)
  • Fix an issue that exception messages are masked (#4416)
  • Fix bug where using upsert or update without routing parameter caused… (#4397)
  • Fix bug in s3 sink dynamic bucket and catch invalid bucket message (#4413)
  • Fix flaky PipelineConfigurationFileReaderTest (#4386)
  • Aggregate Processor: local mode should work when there is no when condition (#4380)

Security

Maintenance

  • Gradle 8.7 (#4417)
  • Adds a Gradle convention plugin for Maven publication (#4421)
  • MAINT: allow latest schema version if not specified in confluent schema (#4453)
  • Publish expression and logstash-configuration to Maven (#4474)
  • Create unit test report as html (#4384)
  • Update Stream Ack Manager unit test and code refactor (#4383)
  • Grpc exception handler: Modified to return BADREQUEST for some internal errors (#4387)
  • Remove unexpected event handle message (#4388)
  • Bump parquet version to 1.14.0. (#4520)
  • Clear system property to disable s3 scan when stream worker exits, set s3 sink threshold to 15 seconds for docdb streams (#4522)
  • ExportPartitionWorkerTest testProcessPartitionSuccess(String) failure (#4298)
  • MAINT: inject external origination timestamp (#4507)
  • Updates Ameria to 1.28.2 (#4440)
  • MAINT: use authentication block in opensearch sink (#4438)
  • MAINT: use authentication for basic credentials in opensearch source (#4435)
  • MAINT: deprecate certificate_content with certificate_key (#4434)
  • MAINT: deprecate plaintext with plain under sasl in kafka (#4433)
  • MAINT: deprecate pipeline_configurations with extension (#4428)
  • Maint/renaming kafka source plugin setting (#4429)
Assets 3
Loading