Server errors when using large JSON payloads

[jaxonadams-lp]

Metro2 Version: latest

What were you trying to do?
We were trying to validate and convert large JSON files to the Metro2 format using the HTTP Docker server.

What did you expect to see?
We ran tests with JSON files consisting of 10k, 25k, 50k, and ~250k records. We expected to see each file validate and, if valid, have the server return a Metro2 file.

What did you see?
The JSON payload holding 10k records validated and was converted with no issues.

The 25k record payload validated, but when hitting the /convert endpoint we received a 502 Bad Gateway response. Our cloud architects don't think this is an issue with server resources or how we've set up the server. At times we have also seen a 504 Gateway Time-out response and an SSL "end of file" error.

On the two larger files, we received a 400 status code on validation with the message "is not a metro file". We are unsure if this is from the server not receiving the whole file or otherwise having trouble parsing it, or if we're just sending bad payloads.

Here's an example of how we've used curl to test these payloads:

curl --form "file=@./25k_record.json" <URL>/validator
curl --form "file=@./25k_record.json" --form "format=metro" <URL>/convert

How can we reproduce the problem?
I've attached the payloads we've been testing with. The problem can be reproduced by sending these payloads to the /validator and /convert endpoints of the HTTP server. If you'd like to use our instance, feel free to test with this url:

https://workato-container-service.ou23u2ki0eaiq.us-west-2.cs.amazonlightsail.com

Payloads:
10k_20k_50k.zip

[adamdecaf]

Thanks for reporting this issue. We defiantly need to better support larger requests. I've created #174 to reproduce this and can see some issues and solutions.

1. Streaming req.Body instead of loading the request in memory should help. We've seen issues when the network is flakey though. net/http.Request doesn't always buffer, but it should be better for you.
2. I'm seeing the 25k file work every time locally. The 50k file looks to have an error which is why /valdiator fails.
3. The 50k file fails to convert locally. I get an EOF every time.

=== RUN   TestServer__LargeRequests/validate_testdata/50k_record.json.gz
    large_requests_test.go:61: Response: total_social_numbers_all_segments in trailer record has an invalid value
    large_requests_test.go:63: 
        	Error Trace:	/Users/adam/code/src/github.com/moov-io/metro2/pkg/server/large_requests_test.go:63
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 501
        	Test:       	TestServer__LargeRequests/validate_testdata/50k_record.json.gz
=== RUN   TestServer__LargeRequests/convert_testdata/50k_record.json.gz

=== RUN   TestServer__LargeRequests/convert_testdata/50k_record.json.gz
    large_requests_test.go:90: 
        	Error Trace:	/Users/adam/code/src/github.com/moov-io/metro2/pkg/server/large_requests_test.go:90
        	Error:      	Received unexpected error:
        	            	Post "http://localhost:15551/convert": EOF
        	Test:       	TestServer__LargeRequests/convert_testdata/50k_record.json.gz
--- FAIL: TestServer__LargeRequests (100.56s)
    --- FAIL: TestServer__LargeRequests/convert_testdata/50k_record.json.gz (100.56s)

[integrations-loanpro]

Thank you @adamdecaf Any sort of Eta on when this will be available for us to deploy and test?

[adamdecaf]

I'm asking about some test failures which seem odd to me, but we'll get it released as soon as we can.

#174 (comment)

[adamdecaf]

Figured out the testing issue. v0.8.1 has been released with much better support for large files. Thank you again for reporting this and let us know if you need anything else!

[jaxonadams-lp]

Thank you very much!