Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add 2 features in s3 output plugin #14

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

add 2 features in s3 output plugin #14

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c5f2829
make codec configurable
cnhappier Apr 17, 2015
3da02e3
Merge branch 'master' of https://github.com/logstash-plugins/logstash…
cnhappier Jun 2, 2015
3429972
remove duplicate codec in base
cnhappier Jun 2, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions lib/logstash/outputs/s3.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@
# bucket => "boss_please_open_your_bucket" (required)
# size_file => 2048 (optional)
# time_file => 5 (optional)
# format => "plain" (optional)
# canned_acl => "private" (optional. Options are "private", "public_read", "public_read_write", "authenticated_read". Defaults to "private" )
# }
#
Expand Down Expand Up @@ -110,6 +109,9 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
# Specify how many workers to use to upload the files to S3
config :upload_workers_count, :validate => :number, :default => 1

# Flags for enable move file to s3 in tear down. It's useful if you need to automatic teminate some EC2 instances in aws.
config :move_file_in_tear_down, :validate => :boolean, :default => false

# Exposed attributes for testing purpose.
attr_accessor :tempfile
attr_reader :page_counter
Expand All @@ -123,7 +125,7 @@ def aws_s3_config
def aws_service_endpoint(region)
# Make the deprecated endpoint_region work
# TODO: (ph) Remove this after deprecation.

if @endpoint_region
region_to_use = @endpoint_region
else
Expand Down Expand Up @@ -230,7 +232,7 @@ def test_s3_write
File.delete(test_filename)
end
end

public
def restore_from_crashes
@logger.debug("S3: is attempting to verify previous crashes...")
Expand Down Expand Up @@ -308,10 +310,17 @@ def write_to_tempfile(event)

public
def teardown
#close temp file
@tempfile.close

if @move_file_in_tear_down == true
# upload current temporary file if needed
move_file_to_bucket(@tempfile.path)
end

shutdown_upload_workers
@periodic_rotation_thread.stop! if @periodic_rotation_thread

@tempfile.close
finished
end

Expand All @@ -333,7 +342,7 @@ def handle_event(encoded_event)
else
@logger.debug("S3: tempfile file size report.", :tempfile_size => @tempfile.size, :size_file => @size_file)
end
end
end

write_to_tempfile(encoded_event)
end
Expand Down