Any idea why we run into error "Insufficient memory (case 4)" on IMagickImage.Write() ?

[amytant]

We are running into this error when attempting to save an edited image:
"Failed to save image to the disk. All errors: Insufficient memory (case 4) ... file path ... @ error/jpeg.c/JPEGErrorHandler/348"

This occurs with files of varying size, from <1 MB to >1 GB. The only common denominator between these failures seems to be that the output filetype is JPEG.

Stack trace:

0
{"level":0,"method":"ImageMagick.MagickExceptionHelper.Check","assembly":"Magick.NET-Q8-AnyCPU, Version=13.1.0.0, Culture=neutral, PublicKeyToken=2004825badfa91ec","fileName":"/_/src/Magick.NET/Exceptions/MagickExceptionHelper.cs","line":18}
1
{"level":1,"method":"ImageMagick.MagickImage+NativeMagickImage.WriteFile","assembly":"Magick.NET-Q8-AnyCPU, Version=13.1.0.0, Culture=neutral, PublicKeyToken=2004825badfa91ec","fileName":"/_/src/Magick.NET/Native/MagickImage.cs","line":8653}
2
... Where IMagickImage.Write() is called in client code with file path string ...

[dlemstra]

The user seems to be running out of memory. I noticed that you are using the AnyCPU version of this library. I would recommend you to switch to the x64 version when you are running out of memory.

[amytant]

I'm trying to investigate why the Q8-AnyCPU version would be running in a 32-bit mode in our use case (in a Kubernetes pod which should be running in a 64-bit context), but so far no answers... Found the following info about it:

AnyCPU?
The AnyCPU version of Magick.NET is designed to allow your application to be used in a 32 bit or a 64 bit environment. When one of the classes in the library is used it will detect if the application pool is 32 or 64 bit. It will then read the x86 or the x64 version of the dll from an embedded resource. This resource is written to a temporary directory to improve the start up time the next it is used. You can change the directory that is used with the CacheDirectory property of the MagickAnyCPU class when the default directory is causing issues in your production environment. You should install the x86 and the x64 version of the Visual C++ Redistributable on the machine you will be using because you don't know if the x86 or the x64 will be used. If you do you are probably better of using the x86 or the x64 version of Magick.NET.

Since it talks about an application pool, I assume this relates to running Magick.NET in IIS, but that is not our use case. So how would the AnyCPU version decide to run in 32-bit mode? We can try switching to the x64 version, but are hoping to get some clarity on why it's running out of memory, which is presumably due to not running in 64-bit mode when that should be supported?

I also saw that 13.1.1 has an optimization to reduce memory usage, so I assume we should try updating the version to see whether that has an effect.

[dlemstra]

When you are running in Kubernetes you are probably on Linux so that means you are already running in 64-bit mode. With Kubernetes you are probably hitting memory limits of the pod quicker. And this is happening inside the jpeg coder so I don't think that setting ResourceLimits.Memory will help you in this case but you could give that a try. But you are probably better off having a pod with a high memory request and keep that alive for multiple requests.

[amytant]

@dlemstra We're still seeing this exception pop up regularly, though less often than we did earlier this year, possibly because of changes we've made to the memory management of our Kubernetes deployments. We've also updated MagickNET from 8.5.0 to 13.1.0 earlier this year, so that could be related as well. Seeing that between 13.1.0 and 13.3.0 the release notes specify a few changes related to memory, so will update the version we use to latest.

Using the Q8-AnyCPU version I've checked MagickNET.ImageMagickVersion on both my local Windows 10 Enterprise 64-bit machine and on Kubernetes (running in a Windows Server 2019 64-bit VM). For MagickNET 13.1.0 it's ImageMagick 7.1.1-8 Q8 x64 920f79206:20230422. For 13.3.0 it's ImageMagick 7.1.1-17 Q8 x64 44a26b167:20230919. So we're definitely running it in 64-bit mode.

Playing around with ResourceLimits.Memory, the following findings are interesting:

• Locally (64 GB RAM installed, 38 GB RAM free) with either 13.1.0 or 13.3.0, the default value of ResourceLimits.Memory (when not set to a specific percentage, so 100%) is 34187276288 bytes ~= 31.84 GB. I don't see any documentation clarifying what this value is based on.
• After setting ImageMagick.ResourceLimits.LimitMemory(new Percentage(1));, the value is 683745280 bytes (652.07 MB) or a multiple of that when the percentage is increased, as expected, until it reaches the default value of 34187276288 bytes ~= 31.84 GB at about 50%, and then the value calculated by ImageMagick doesn't seem to go any higher than the default value when you increase the percentage further. It's unclear to me why 652 MB is being considered 1% of 31.84 GB, I would assume that in this case, 1 percent of the maximum memory usable by the pixel cache should be closer to 341872762.88 bytes ~= 326 MB.
• In a development cloud environment running Kubernetes (pod is limited to being able to use 6 GB max) the default value of ResourceLimits.Memory is 17179631616 bytes ~= 16 GB. Setting it to 1% leads to a usage of 343588864 bytes ~~= 327.67 MB, which is also not 1% of the default max (100%) value.

I've set ResourceLimits.Memory to a percentage for our Kubernetes deployments that should mean it stays below the 6 GB max RAM limit for each pod. Additionally I've also set ResourceLimits.MaxMemoryRequest to a value below that limit (which is far more straightforward than setting the Memory limit, since MaxMemoryRequest just takes a specific number of bytes instead of a percentage value, and the percentage value isn't even being taken into account 100% correctly as far as I can tell...).

As far as I understand how ImageMagick works, this should mean that when IM hits the known memory limit, it will start memory-mapping to disk, which did not seem to happen beforehand because of the unrealistically high default memory limit values it was assigning itself, it would seemingly just run out of memory to allocate itself and throw that exception. So hopefully this will resolve this issue for us, but ImageMagick's default memory allocation behavior seems strange and arbitrary...

[dlemstra]

The default value of ResourceLimits.Memory is not 100% of the memory. This is 50% of the available memory in 64-bit mode. But you are allowed to set it to a higher value. When you set the value to 1% it will use 1% of the available memory. I will update the documentation of the Memory property to make it more clear what default value is used.

I suspect that your local development environment only use the available memory on the system? I am not a Kubernetes expert so I don't know why this is happening.

And you are also allowed to set ResourceLimits.Memory, the LimitMemory method is only there to make it more convenient for you to limit the memory. I hope this makes things more clear for you?

[Raildex]

Speaking of ResourceLimits is there a verbose/debug output so I know when ImageMagick memory-maps?

Is ResourceLimits.Area used in conjunction with ResourceLimits.Memory? what if Memory is lower than what Area can contain?

[amytant]

My apologies for my previous message, I got the bytes -> gigabytes conversion wrong which muddled the findings. Just updated that post to fix the numbers, and it does indeed look like the default value is being set to 50% of available memory in 64-bit mode (at least on my local dev machine, Kubernetes might be a different story as explained below). Couldn't find anything about this in docs earlier, so thank you for clarifying that and updating documentation.

I had another chat with a Kubernetes expert in our team, and it seems like ImageMagick is basing its default memory value in a Kubernetes pod on the amount of memory the node has access to (32GB), not the maximum set for the pod itself (6 GB). Hence why the default value for ResourceLimits.Memory is being set to 16 GB instead of 3 GB. However, he had a look and it seems the Kubernetes pod's container OS actually isn't reporting 32 GB available memory, but 16 GB memory. So in that case it would make more sense if IM in our Kubernetes deployment determined a default value of 8 GB, but that's not the case?

OsTotalVisibleMemorySize - 16776756 - 16GB
OsFreePhysicalMemory - 1574424 - 1.5GB
OsTotalVirtualMemorySize - 25067060 - 25GB
OsInUseVirtualMemory - 17202308 - 17GB

In any case, it would be nice if we had a better idea of what IM is looking at to retrieve the memory stats. Is it a particular performance counter? That might help us track down how the default value is determined.

I had not found any documentation saying that we can directly set ResourceLimits.Memory without having to use ResourceLimits.LimitMemory(), being able to do so helps our use case since now we can be sure both ResourceLimits.Memory and ResourceLimits.MaxMemoryRequest are locked to a max limit of 5 GB. Thank you for clarifying that, though it would be helpful if that was clearer in documentation as well.

Update: Since Raildex asked this, I too would love to have insight into when IM is memory-mapping, verbose/debug output would be helpful for that. And I'm also not sure how ResourceLimits.Memory relates to ResourceLimits.Area.

[dlemstra]

Once ResourceLimits.Memory has been reached the pixels will be swapped to disk and ResourceLimits.Area will be used. And for the technical details, we are using sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGE_SIZE) to get the available memory.

And if you want some verbose output the please read the DetailedDebugInformation docs on how to get some more information.