No space left on device with lambda s3

[AnthonyGWeb]

Description of the problem

Hi,

To begin thanks a lot for your work. I use AWS Lambda to aggregate a collection of files on one zip.
Lambda has a limitation of 512Mo in /tmp so I would like use a stream to zip my files collection. And store it on S3.
I have tried many manipulation without success, because I have an error with PHP Notice: fwrite(): Write of 269 bytes failed with errno=28 No space left on device in /var/task/vendor/guzzlehttp/psr7/src/Stream.php.

I have seen on the S3Wrapper (register by registerStreamWrapper()) it's use php://temp, but I have try to open my zipFile with a (append) but I have the same error.

For information with less data it's work perfectly.

Have you got an idea just for append files into my zip ?

Thanks,

Example code

$urlCollection : List of S3 url files (~2Mo by content). (Zip attempted ~2Go)

$this->client->registerStreamWrapper();

$outputContext = stream_context_create([
    's3' => ['ACL' => 'public-read'],
]);

$zipFile = fopen("s3://{my_s3_bucket}/tmp/test.zip", 'w', false, $outputContext); // I have tried to replace 'w' by 'a'

$options = new Archive();
$options->setEnableZip64(false);
$options->setZeroHeader(true);
$options->setOutputStream($zipFile);

$zip = new ZipStream(null, $options);

foreach ($urlCollection as $url) {
  if ($streamRead = fopen($url, 'r')) {
      $zip->addFileFromStream(basename($url), $streamRead);
  } else {
      var_dump('fopen error');
  }
}

$zip->finish();

fclose($zipFile);

Informations

• ZipStream-PHP version: 2.1.0
• PHP version: 8.0

[korridor]

I am also using AWS Lambda (with bref) and I am experiencing the exact same problem.
I also tried this, but it did not help:

$options->setFlushOutput(true);

@AnthonyGWeb You haven't by any chance already found a solution for it or debugged it further?

I am using the following versions:
ZipStream-PHP version: 2.1.0
PHP version: 7.4

Stack trace:

[2021-04-21 01:53:42] staging.ERROR: fwrite(): write of 8052 bytes failed with errno=28 No space left on device {"exception":"[object] (ErrorException(code: 0): fwrite(): write of 8052 bytes failed with errno=28 No space left on device at /var/task/vendor/guzzlehttp/psr7/src/Stream.php:247)
[stacktrace]
#0 [internal function]: Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /var/task/vendor/guzzlehttp/psr7/src/Stream.php(247): fwrite()
#2 /var/task/vendor/aws/aws-sdk-php/src/S3/StreamWrapper.php(213): GuzzleHttp\\Psr7\\Stream->write()
#3 [internal function]: Aws\\S3\\StreamWrapper->stream_write()
#4 /var/task/vendor/maennchen/zipstream-php/src/ZipStream.php(462): fwrite()
#5 /var/task/vendor/maennchen/zipstream-php/src/File.php(361): ZipStream\\ZipStream->send()
#6 /var/task/vendor/maennchen/zipstream-php/src/File.php(424): ZipStream\\File->readStream()
#7 /var/task/vendor/maennchen/zipstream-php/src/File.php(336): ZipStream\\File->processStreamWithComputedHeader()
#8 /var/task/vendor/maennchen/zipstream-php/src/ZipStream.php(296): ZipStream\\File->processStream()
#9 /var/task/app/Console/Commands/Export/PlatformExport.php(84): ZipStream\\ZipStream->addFileFromStream()
#10 /var/task/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php(44): App\\Console\\Commands\\Export\\PlatformExport->App\\Console\\Commands\\Export\\{closure}()
#11 /var/task/app/Console/Commands/Export/PlatformExport.php(87): Illuminate\\Database\\Eloquent\\Builder->chunk()
#12 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Console\\Commands\\Export\\PlatformExport->handle()
#13 /var/task/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#14 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#15 /var/task/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#16 /var/task/vendor/laravel/framework/src/Illuminate/Container/Container.php(611): Illuminate\\Container\\BoundMethod::call()
#17 /var/task/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call()
#18 /var/task/vendor/symfony/console/Command/Command.php(256): Illuminate\\Console\\Command->execute()
#19 /var/task/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run()
#20 /var/task/vendor/symfony/console/Application.php(971): Illuminate\\Console\\Command->run()
#21 /var/task/vendor/symfony/console/Application.php(290): Symfony\\Component\\Console\\Application->doRunCommand()
#22 /var/task/vendor/symfony/console/Application.php(166): Symfony\\Component\\Console\\Application->doRun()
#23 /var/task/vendor/laravel/framework/src/Illuminate/Console/Application.php(92): Symfony\\Component\\Console\\Application->run()
#24 /var/task/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run()
#25 /var/task/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle()
#26 {main}
"}

In Stream.php line 247:

  fwrite(): write of 8052 bytes failed with errno=28 No space left on device

[ralphmorris]

@AnthonyGWeb @korridor Did either of you figure this out at all? Thanks

[akki-io]

I am guessing that they didn't. I posted similarish discussions here - #277

[maennchen]

See answer: #277 (reply in thread)

[scottlaurent]

if anyone spends 2 days killing themselves over this... I had no idea this was an added feature. It fixed the issue we had, which was the same as the one above. Took 30 seconds and took effect instantly.

https://aws.amazon.com/blogs/compute/using-larger-ephemeral-storage-for-aws-lambda/