Download data gives error HTTP code: 500 for large files.

[ashwinikalantri]

We have a questionnaire with 4 versions. Ver 1 and 2 have 99 an 177 interviews respectively, while version 3 and version 4 have large no of interviews (~14000 and ~20000 respectively).

While downloading data using download_data() or download_matching() functions, the export process breaks for questionnaire version that have large no of interviews. The function downloads files for the first 2 versions but I get HTTP code: 500 for version 3 and 4. The size of the files generated is ~12MB for ver 1 and 2, while ~30MB for ver 3 and 4.

...
Export status: Running. Percent progress: 55
Export status: Running. Percent progress: 55
Export status: Running. Percent progress: 61
Export status: Running. Percent progress: 100
Downloading file
Unexpected result for job 17030. HTTP code: 500.

Pls get back if you need additional help troubleshooting.

[ashwinikalantri]

While using the download_* functions produces the above error, the get_export_file() from the susoapi package works.

...
Export status: Running. Percent progress: 55
Export status: Running. Percent progress: 55
Export status: Running. Percent progress: 61
Export status: Running. Percent progress: 100
Downloading file
Unexpected result for job 17032. HTTP code: 500.
> susoapi::get_export_file(
+   job_id = 17032,
+   path = "test"
+ )
File successfully downloaded to:
test/w2_update_3_STATA_All.zip

[arthur-shaw]

Thanks so much for the helpful bug report!🙏

The function downloads files for the first 2 versions but I get HTTP code: 500 for version 3 and 4. The size of the files generated is ~12MB for ver 1 and 2, while ~30MB for ver 3 and 4.

On the surface of things, the 500 code would point to an unspecified problem with the server rather than the client (i.e., your script). With that in mind, I wonder if it would be possible to investigate the resource usage of the server at or around the time of the 500 code. On that same theme, do you experience any issues downloading the files through the UI?

Given that this is likely a server load/capacity/performance issue, I'm wondering how best to handle it. Perhaps we could introduce a user-specified wait period between requests (e.g., wait 20 seconds between requesting v1 and v2) or segments of the download process (e.g., wait N seconds between the file being complete and an attempt to download it). Perhaps there's also a way to query whether the file has been placed in the export folder for download. With large file sizes, perhaps it takes some time to copy or write large files even after the export process is signalled as complete, and download requests in that period will result in a 500 code.

Ideally, we'd find a way for this to just work. And I can do some research on how to improve the process.

But if there's no way to improve on this situation, what would be your preferred outcome:

• Retry on failure some number of times (possible with httr2 but, I believe, not with httr, which the suso* packages currently use). See here.
• Fail with an error when the server issues a code in the 400s or 500s
• Succeed with a warning describing failed downloads

In practice, I think it will be the first bullet plus the second or third bullet. Even if we retry, there's no guarantee that the request will succeed.

[ashwinikalantri]

I tried using the susoapi functions seperately without this error.
I didn't experience any issues from the UI.
Downloading even a single version caused this error.

I tried to simplify the function. This function worked as expected.

download_data_new <- function(qnr_id,
                              export_type,
                              interview_status = "All",
                              from = "",
                              to = "",
                              access_token = "",
                              refresh_token = "",
                              storage_type = "",
                              translation_id = "",
                              include_meta = TRUE,
                              path,
                              server = Sys.getenv("SUSO_SERVER"),
                              workspace = Sys.getenv("SUSO_WORKSPACE"),
                              user = Sys.getenv("SUSO_USER"),
                              password = Sys.getenv("SUSO_PASSWORD")) {
  job_id <- susoapi::start_export(
    qnr_id = qnr_id,
    export_type = export_type,
    interview_status = interview_status
  )
  
  status <- TRUE
  
  while (status == TRUE) {
    status_x <- susoapi::get_export_job_details(job_id = job_id)
    
    status <-
      if_else(status_x$ExportStatus == "Running", TRUE, FALSE)
    
    print(paste0(status_x$ExportStatus, ". ", status_x$Progress, "%"))
  }
  
  susoapi::get_export_file(job_id = job_id,
                           path = path)
}

wait N seconds between the file being complete and an attempt to download it

Then, I added a two second Sys.sleep(2) before downloading the file (line 270). This seems to have eliminated the issue for me. But I don't understand why the above function above works. I tried removing various sections of the download_data function hoping to find another culprit. But only adding some time before downloading worked.

[ashwinikalantri]

The survey solutions logs has this when the I get the 500 error.

2024-02-05 22:32:57.111 +05:30 [ERR] Connection id "0HN16BVNS1Q7F", Request id "0HN16BVNS1Q7F:00000003": An unhandled exception was thrown by the application.
System.IO.IOException: The process cannot access the file 'C:\Survey Solutions\Data_Site\export\0e5dfb3b-0502-4a85-87c1-8b4036c63caa\17054\55481de8fdea4c21bb4e552a48842b3c$3_STATA_All.zip' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
   at WB.Services.Export.Jobs.ExportArchiveHandleService.DownloadArchiveAsync(ExportSettings settings, String questionnaireNamePrefixOverride)
   at WB.Services.Export.Host.Controllers.JobController.DownloadArchive(Int64 id, String archiveName)
   at lambda_method1669(Closure, Object)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)