Skip to content

Commit

Permalink
in_tail: fix slice length to reduce memory usage (#4760)
Browse files Browse the repository at this point in the history
**Which issue(s) this PR fixes**: 
Fixes #

**What this PR does / why we need it**: 

`String#slice` will allocate redundant heap area internally if an
incorrect length is specified.
This PR might reduce the memory usage with `in_tail` plugin.


* verify code
```ruby
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'benchmark-memory'
end

str = "a" * 10_000_000
split_point = 9_999_900

Benchmark.memory do |x|
  x.report("1") {
    str.slice(split_point, str.length)
  }

  x.report("2") {
    str.slice(split_point, str.length - split_point)
  }

  x.report("3") {
    str.slice(split_point..str.length)
  }
end
```

* result
```
Calculating -------------------------------------
                   1    10.000M memsize (    10.000M retained)
                         2.000  objects (     1.000  retained)
                         2.000  strings (     1.000  retained)
                   2    40.000  memsize (     0.000  retained)
                         1.000  objects (     0.000  retained)
                         1.000  strings (     0.000  retained)
                   3    80.000  memsize (     0.000  retained)
                         2.000  objects (     0.000  retained)
                         1.000  strings (     0.000  retained)
```

The all cases will generate same String object. However, `1` case will
use 10 MB heap area with above code.

**Docs Changes**:

**Release Note**:

Signed-off-by: Shizuo Fujita <[email protected]>
  • Loading branch information
Watson1978 authored Jan 8, 2025
1 parent abe335a commit 5080f81
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions lib/fluent/plugin/in_tail.rb
Original file line number Diff line number Diff line change
Expand Up @@ -1056,8 +1056,9 @@ def read_lines(lines)
# Using freeze and slice is faster than slice!
# See https://github.com/fluent/fluentd/pull/2527
@buffer.freeze
rbuf = @buffer.slice(0, idx + 1)
@buffer = @buffer.slice(idx + 1, @buffer.size)
slice_position = idx + 1
rbuf = @buffer.slice(0, slice_position)
@buffer = @buffer.slice(slice_position, @buffer.size - slice_position)
idx = @buffer.index(@eol)

is_long_line = @max_line_size && (
Expand Down

0 comments on commit 5080f81

Please sign in to comment.