Skip to content

Commit

Permalink
in_sample: support to renew record functionality
Browse files Browse the repository at this point in the history
When in_sample plugin is used with filter parser which uses
remove_key_name_field, it raises the following error repeatedly.

  #0 dump an error event: error_class=ArgumentError error="message
  does not exist"

This kind of error occurs when key_name and remove_key_name_field
removes key from record with destructive change in filter parser.
It affects generated sample data.

To fix this issue, it is simple to just dup every record, but it has a
significant performance penalty.

Considering keeping compatibility and providing way to a workaround,
added option to control whether it always re-generate a new record or
re-use existing record - renew_record.

ref. #4575

Signed-off-by: Kentaro Hayashi <[email protected]>
  • Loading branch information
kenhys committed Aug 16, 2024
1 parent a6978e5 commit 7777f30
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions lib/fluent/plugin/in_sample.rb
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ class SampleInput < Input
config_param :auto_increment_key, :string, default: nil
desc "The boolean to suspend-and-resume incremental value after restart"
config_param :suspend, :bool, default: false,deprecated: 'This parameters is ignored'
desc "It always force to generate a new record. It has performance penalty"
config_param :renew_record, :bool, default: false
desc "The sample data to be generated. An array of JSON hashes or a single JSON hash."
config_param :sample, alias: :dummy, default: [{"message" => "sample"}] do |val|
begin
Expand Down Expand Up @@ -118,14 +120,22 @@ def emit(num)
end

def generate
d = @sample[@sample_index]
if @renew_record
d = @sample[@sample_index].dup
else
d = @sample[@sample_index]
end
unless d
@sample_index = 0
d = @sample[@sample_index]
if @renew_record
d = @sample[@sample_index].dup
else
d = @sample[@sample_index]
end
end
@sample_index += 1
if @auto_increment_key
d = d.dup
d = d.dup unless @renew_record
d[@auto_increment_key] = @storage.update(:auto_increment_value){|v| v + 1 }
end
d
Expand Down

0 comments on commit 7777f30

Please sign in to comment.