Kubernetes POD shutdown due to memory issues

[vroola03]

We use ImageMagick to create smaller previews of files. However, every now and then, our process fails due memory issues and the Kubernetes (Windows)POD we run our code on just shuts down.
I try to understand why it fails on a certain TIF file that is 1.7GB
Our Windows POD has 6GB of memory and about 15GB of free disk space. As soon as I increase the amount of memory to 8GB, the process succeeds.

We reference Magick.NET-Q8-AnyCPU 13.3.0 and as far as I can see we run 64bit.

When I limit the amount of memory by setting ImageMagick.ResourceLimits.Memory and ImageMagick.ResourceLimits.MaxMemoryRequest to about 2.5GB (and other higher and lower values) there is no positive effect.
I read that when there is not enough memory available ImageMagick would fall back on mapping memory to disk. Setting MAGICK_TEMPORARY_PATH and MAGICK_OPENCL_CACHE_DIR to a larger share does not make a difference either.
Tried it too with the methods LimitMemory and SetTempDirectory.
Would ImageMagick fall back on disk mapping for all internal processes or are there still actions that do not respect the ImageMagick.ResourceLimits.Memory and ImageMagick.ResourceLimits.MaxMemoryRequest?

The following message is logged in the POD console:
C:\s\g284\Temp\701822549_22\SPLC\1909271298\D850471FE7A56A55D0751B85BBBD3842.tif TIFF 5354x5586 5354x5586+0+0 16-bit TrueColor sRGB 1.6885GiB 10.660u 0:10.660
Tools.FileProcessor.exe: Memory allocation failed `Not enough space' @ fatal/string.c/AcquireStringInfo/163.
waiting 0 milliseconds

When I enable verbose logging, the last log lines of a failed run are:

2024-04-12T14:36:51+00:00 0:21.056 21.056u 7.1.1 Trace Tools.FileProcessor.exe[23368]: nt-base.c/NTResourceToBlob/1958/Trace locale.xml
2024-04-12T14:36:46+00:00 0:16.132 16.132u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/AcquireMagickResource/385/Resource Map: 1.6885GiB/1.6885GiB/32GiB
2024-04-12T14:36:46+00:00 0:16.093 16.093u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/RelinquishMagickResource/1069/Resource Map: 1.6885GiB/0B/32GiB
2024-04-12T14:36:46+00:00 0:16.090 16.090u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/AcquireMagickResource/385/Resource Map: 1.6885GiB/1.6885GiB/32GiB
2024-04-12T14:36:46+00:00 0:15.925 15.925u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/RelinquishMagickResource/1069/Resource Map: 1.6885GiB/0B/32GiB
2024-04-12T14:36:42+00:00 0:12.422 12.422u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/AcquireMagickResource/385/Resource Memory: 85.566MiB/85.566MiB/2.5GiB
2024-04-12T14:36:42+00:00 0:12.420 12.420u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/AcquireMagickResource/385/Resource Area: 28.522MiB/0B/63.999GiB
2024-04-12T14:36:40+00:00 0:09.969 9.969u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/RelinquishUniqueFileResource/1108/Resource magick-1wSBk_P0VygJ9i4ago2dZZ-rUqOaZKNf
2024-04-12T14:36:40+00:00 0:09.960 9.960u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/AcquireUniqueFileResource/636/Resource magick-1wSBk_P0VygJ9i4ago2dZZ-rUqOaZKNf
2024-04-12T14:36:40+00:00 0:09.957 9.957u 7.1.1 Trace Tools.FileProcessor.exe[23368]: policy.c/GetPolicyValue/563/Trace resource:temporary-path
2024-04-12T14:36:40+00:00 0:09.810 9.810u 7.1.1 Resource Tools.FileProcessor.exe[23368]: resource.c/AcquireUniqueFileResource/579/Resource ...

The logs show me that the 2.5GiB is applied on the Resource Memory. However, I also see 32GB on Resource Map and 63.999GB on Resource Area. Not sure what that means and if that is about to memory or disk space? Perhaps memory is not read from the POD but Node.

Are there more handles that I can adjust to tweak the outcome? Again, by increasing to 8GB the file gets processed correctly. I am only trying to understand why it might fail because I do not know if 8GB is enough for the next file that might be a bit larger.
Any advice would be appreciated.

[dlemstra]

The error message Not enough space is from a call to strerror with errno and that looks like it is running out of disk space. But Not enough space is the output when errno is set to ENOMEM on Windows.

ImageMagick will only fallback to disk for pixels. The error message includes fatal/string.c/AcquireStringInfo/163 and that is used to create image profiles so I suspect that your input image contains a large profile. But there is not much you can do about that than increasing the available memory for your pod. I do wonder if we could add a new resource limit for this that will cause large profiles to be ignored and not added to the image. I will need discuss with the team if we would want to add a feature like that.

[dlemstra]

Good news! In the next version of Magick.NET you will be able to set a maximum profile size that will probably help you in this situation. Is it possible to share that file somewhere so I can give you a better answer?

[vroola03]

Thanks for looking into this so quickly. I will get back to you on sending the file.

[vroola03]

It took some time but I have send you an email with the download link. (WeTransfer)

Some additional note on the memory usage. When I run a sample console app, calling the IMagickImage.Thumbnail method increases the memory usage for that console app with ~4GB.

[dlemstra]

Setting ResourceLimits.MaxProfileSize seems to reduce the memory usage for this image. This option was added in the latest release. This should probably help with the memory issues that you have.

[vroola03]

Looks like this makes the difference. If I set the MaxProfileSize to a lower value than the profile size (1.6GB) I see a warning in the logs and apparently it skips it. Memory consumption stays low also when I call the IMagickImage.Thumbnail method.

One more question, I use this to generate a smaller JPG from the TIF file. What would be the impact on not loading a profile for the preview creation process? We are not manipulating the original file.

[dlemstra]

You should allow for small profiles to load because the image might contain color profiles. But not loading these kind of large profiles has no effect on the output file.

[pinkfloydx33]

Somewhat related #1744