Random could not find file errors when downloading packages

[credfeto]

Describe the bug

deployment: docker
version: bagetter/bagetter:latest
Image Id: 45b25a6b21d6
Database: Postgres

Configuration:

  devpkg-nuget:
    image: bagetter/bagetter:latest
    container_name: devpkg-nuget
    hostname: nuget
    restart: always
    stop_grace_period: 500s
    stop_signal: SIGINT
    environment:
      - AllowPackageOverwrites=true
      - ApiKey=Upload
      - Database__ConnectionString=Server=nuget-postgresql;Port=5432;Database=nuget;User Id=nuget;Password=nuget;
      - Database__Type=PostgreSql
      - Logging__Console__LogLevel__Default=Warning
      - MaxPackageSizeGiB=10
      - MaxVersionsPerPackage=5
      - Mirror__Enabled=true
      - Mirror__PackageDownloadTimeoutSeconds=60000
      - Mirror__PackageSource=https://api.nuget.org/v3/index.json
      - PackageDeletionBehavior=HardDelete
      - Search__Type=Database
      - Storage__Path=/data/packages
      - Storage__Type=FileSystem
    volumes:
      - /cache/nuget/security/nuget:/root/.aspnet
      - /cache/nuget/packages/nuget:/data
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
      interval: 5s
      timeout: 30s
      retries: 5
      start_period: 120s
    depends_on:
      devpkg-nuget-postgresql:
        condition: service_healthy
    networks:
      backend:

To Reproduce

Not 100% reproducible -> sometimes it works; sometimes it doesn't - its not obvious when/why this happens - I don't see anything in the logs saying its failed to download & save, just the error saying that the file isn't there.

Quite often when running (various versions) bagetter fails to download the package from nuget, yet publishes it as downloaded in the feed - so when something comes to restore the package I get this in logs:

devpkg-nuget               | fail: Microsoft.AspNetCore.Server.Kestrel[13]
devpkg-nuget               |       Connection id "0HN9M3F2VM43H", Request id "0HN9M3F2VM43H:00000001": An unhandled exception was thrown by the application.
devpkg-nuget               |       System.IO.FileNotFoundException: Could not find file '/data/packages/packages/microsoft.netcore.platforms/1.0.1/microsoft.netcore.platforms.1.0.1.nupkg'.
devpkg-nuget               |       File name: '/data/packages/packages/microsoft.netcore.platforms/1.0.1/microsoft.netcore.platforms.1.0.1.nupkg'
devpkg-nuget               |          at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
devpkg-nuget               |          at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
devpkg-nuget               |          at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
devpkg-nuget               |          at BaGetter.Core.PackageStorageService.GetStreamAsync(String id, NuGetVersion version, Func`3 pathFunc, CancellationToken cancellationToken)
devpkg-nuget               |          at BaGetter.Core.PackageStorageService.GetPackageStreamAsync(String id, NuGetVersion version, CancellationToken cancellationToken)
devpkg-nuget               |          at BaGetter.Core.DefaultPackageContentService.GetPackageContentStreamOrNullAsync(String id, NuGetVersion version, CancellationToken cancellationToken)
devpkg-nuget               |          at BaGetter.Web.PackageContentController.DownloadPackageAsync(String id, String version, CancellationToken cancellationToken)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
devpkg-nuget               |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
devpkg-nuget               |          at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
devpkg-nuget               |          at BaGetter.Web.OperationCancelledMiddleware.Invoke(HttpContext context)
devpkg-nuget               |          at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
devpkg-nuget               |          at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
devpkg-nuget               |          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Dropping the docker image, clearing database and restarting the docker image most of the time allows builds again. This can be with any package

Expected behavior

A clear and concise description of what you expected to happen.

Packages to only be published in the index when they've successfully downloaded.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.