sanitize_null_bytes should add the unicode replacement character

[collimarco]

Thanks for this useful gem. Currently if you use the default :replace strategy, invalid characters are replaced with �, but the null byte is replace with nothing.

This behavior seems unexpected and inconsistent.

Expected: "Hello \x00 world" => "Hello � world"
Actual: "Hello \x00 world" => "Hello world"

[collimarco]

The fix would be straightforward, we just need to change this line:

rack-utf8_sanitizer/lib/rack/utf8_sanitizer.rb

Line 43 in 7dcc1e0

input = input.gsub(NULL_BYTE_REGEX, "")

Even the test looks strange (it suggests a replacement, but it actually removes null bytes):

rack-utf8_sanitizer/test/test_utf8_sanitizer.rb

Line 352 in 7dcc1e0

it "optionally sanitizes null bytes with the replace strategy" do

If this choice is by design (is it good? is it bad?), it should be clarified in the documentation in any case, because this is definitely not what you expect from reading the docs.

[geoffharcourt]

Hi @collimarco this is achievable with a custom strategy without too much code:

# config/application.rb

replace_null_byte = lambda do |input, sanitize_null_bytes: true|
   input.
     force_encoding(Encoding::ASCII_8BIT).
     encode!(Encoding::UTF_8,
             invalid: :replace,
             undef:   :replace)

  if sanitize_null_bytes && input =~ Rack::UTF8Sanitizer::NULL_BYTE_REGEX
    input = input.gsub(Rack::UTF8Sanitizer::NULL_BYTE_REGEX, "")
  end

  input
end

config.middleware.insert 0, Rack::UTF8Sanitizer, sanitize_null_bytes: true, strategy: replace_null_byte