Memory Issues / Configuring For Kubernetes #1744
Replies: 2 comments 17 replies
-
|
I saw that you opened an issue about this with some more information but let me first answer your questions here:
The native library uses
The
The size of an image doesn't tell you anything about how much memory it needs. An image of
Below is the document for both methods: /// <summary>
/// Gets or sets the max memory request in bytes. ImageMagick maintains a separate memory pool for large
/// resource requests. If the limit is exceeded when allocating pixels, the allocation is instead memory-mapped
/// on disk.
/// </summary>
ulong MaxMemoryRequest { get; set; }
/// <summary>
/// Gets or sets the pixel cache limit in bytes. Once this memory limit is exceeded, all subsequent pixels cache
/// operations are to/from disk. The default value of this is 50% of the available memory on the machine in 64-bit mode.
/// When running in 32-bit mode this is 50% of the limit of the operating system.
/// </summary>
ulong Memory { get; set; }What could I do to make it more clear the
I do think that you should want to set the memory limit with |
Beta Was this translation helpful? Give feedback.
-
|
We're using 14.2.0 but if I pass 20% to Limits.LimitMemory(new Percentage(20));
Console.WriteLine("{0}bytes", Limits.Memory); // 2930716672bytes |
Beta Was this translation helpful? Give feedback.
-
|
I've been able to slam my app concurrently with pretty much any Jpegs and some PNGs/Gifs (of all sizes) without issue. I can seem to replicate it with certain PNGs, HEIC and animated gifs, but what they have in common (if anything) I cannot figure out. I added in the logging like you suggested and while I don't fully understand them, I do see it acquiring and releasing. For the most part the request/current/limits look to be about what I'd expect. I have noticed a few things that seem to be true when I get OOM Killed (for PNG/HEIC files anyways):
I do get some log messages that look wrong to me. These I also get these I think that's just |
Beta Was this translation helpful? Give feedback.
-
|
The Map resource limit maps a memory request to a file on disk instead. And Disk is used when memory cannot be allocated and a file is created to contain the memory. They work similar but have a different api. What happens when you set |
Beta Was this translation helpful? Give feedback.
-
|
First, I wanted to thank you for your continued support here. This is really baffling us and I do appreciate you taking the time to respond. I couldn't find a way to set If I'm on 14.2.0 shouldn't |
Beta Was this translation helpful? Give feedback.
-
|
It looks like I never added Map to the api, not sure why I didn't. Might be because there was no support for memory mapped files at an earlier point in time. Will need to investigate this and check why I did not add it. |
Beta Was this translation helpful? Give feedback.
-
|
You mentioned to make sure I'm disposing correctly. I see the "release" in the logs, and I believe my code looks correct, but I'm wondering if we are perhaps doing something just plain wrong w/r/t this library. I would expect as many relinquish messages in the logs as acquire messages, but that's not what I see, so perhaps we are missing something. We generate thumbnails only. We resize the target image, decide on what format we want to convert it to and then export the bytes and try to compress. If the produced image is over some threshold and was a PNG with transparency, we convert it to a JPG instead. Here's the code we are running (error handling/our logging removed and everything moved into a single method). csharp codeinternal static byte[] GetThumbnail(ReadOnlyMemory<byte> input)
{
const uint maxSizeBytes = 65536;
const uint maxDim = 256;
// load image
using var image = new MagickImage(input.Span);
// resize to our target thumbnail size
var size = new MagickGeometry(maxDim)
{
IgnoreAspectRatio = false,
Greater = true,
Less = false
};
image.Resize(size);
// set target format and quality
// GIF->GIF, Anything With Transparency->Png, Jpeg for everything else
(image.Format, image.Quality) = image.Format switch
{
MagickFormat.Gif or MagickFormat.Gif87 => (MagickFormat.Gif, 100),
_ when image.HasAlpha => (MagickFormat.Png, 97),
_ => (MagickFormat.Jpeg, 70)
};
image.Strip();
// get bytes compressing if possible
// if the size is less than some max we define, then return it
var bytes = GetBytesAndCompress(image);
if (bytes.Length <= maxSizeBytes)
return bytes;
// if we've got a GIF/JPEG over the limits, return empty
if (image.Format != MagickFormat.Png)
return [ ];
// now try and convert a PNG (with transparency) to a JPEG
// setting the background color and removing the Alpha
image.BackgroundColor = MagickColors.White;
image.Alpha(AlphaOption.Remove);
image.Format = MagickFormat.Jpg;
image.Quality = 70;
image.Strip();
// get bytes and once again try and compress
bytes = GetBytesAndCompress(image);
// return bytes if less than max, otherwise empty
return bytes.Length <= maxSizeBytes ? bytes : [ ];
}
static readonly ImageOptimizer Optimizer = new() { IgnoreUnsupportedFormats = true, };
static byte[] GetBytesAndCompress(MagickImage image)
{
var bytes = image.ToByteArray(image.Format);
using var ms = new MemoryStream(bytes);
if (Optimizer.Compress(ms))
return ms.ToArray();
return bytes;
}Now when I run a Jpeg through these are the logs I get (slightly redacted and the timestamps removed to reduce clutter): logsNotice that there's 5 "relinquish" messages to the 9 "acquire" messages. I'm not sure if that's expected or not. Does the above show us doing anything horribly wrong that would cause a native memory leak? |
Beta Was this translation helpful? Give feedback.
-
|
First, I really want to thank you again for your assistance and putting up with me on this. I realize it's not very scientific, but I connected to my pod and output the contents of It's all dependent on the image of course. JPEGs seem well behaved. The memory use goes up and then comes down and I can send a big batch of them (50 concurrently) without worrying about the process being killed. Take this JPEG for example. 6000x4000 / 7,805KB / JPG
It produces this memory usage output: It does make a jump there consistent with the file size, but as you can see it recovers. Even if I send 50 concurrent requests of that same image I don't get OOM Killed. The process comes close though, at one point jumping up to 648mb but it seems to cap out and throttle the requests before relinquishing the memory and eventually getting GC'd, ending up back where I started. Now take this HEIC file which seems to be my real problem child. (GitHub wouldn't let me upload a .heic so I had to .zip it) The memory usage makes an increase equivalent to the file's size, but then never drops back down It then stays there forever. If I send the file 3-4 more times (or try to send 4 concurrently) the memory use climbs considerably, never recovers and I get OOM killed. But that said, the logs look fine: logsI even tried setting I also get real weird behavior with a specific GIF file: 300x300 / 110KB / Animated GIF (flashes!)
If I set It seems so far that that I am better off keeping the memory limit low, around 10% which I guess just means most things will end up getting sent to disk. It has solved most of the issues I am having, for example that reduction has allowed me to send 30 concurrent uploads of a 20mb PNG file when otherwise I couldn't send more than 2. The only thing it's yet to help is the issue with the HEIC. For whatever it's worth, here's how I'm setting the current Limits:
Unfortunately I have no control over what my users upload. I need to support "common" formats, but also iPhone formats (hence the HEIC). Other than setting limits, is there anything I can do to optimize these flows or even to simply stop processing in its tracks if the file has a good chance of causing issues? |
Beta Was this translation helpful? Give feedback.
-
|
It looks like you are running out of memory because of managed code instead. You are allocating multiple arrays and .NET will not directly return memory to the operating system. And then you also have unmanaged memory being allocated. Combining that with the managed memory could result in out of memory issues. Maybe you should also introduce some limits on the managed side yourself? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure what else I can do on the managed side other than limiting files sizes like we already are, but you've already pointed out that that doesn't really matter. We're already using Span/Memory and avoiding allocations as much as we can. Setting the memory limit low seems to solve most of my problems since I guess that just means it's doing more work via the disk. That's absolutely fine for our use case. We aren't dealing with high load here so if it takes a performance hit that's no big deal to us. The problem really is that over time something is happening that eventually pushes us over that native + managed limit. Kubernetes is indiscriminate in it's OOM kill. If there were some way to gracefully terminate in this case then I'd just deal with it, but there's not and we end up with some form of data corruption. I've gotten it to the point where I can process 50+ 20mb PNG files concurrently without any issue (the memory spikes, but the GC collects)... yet I can't process five 2mb HEIC files over the lifetime of my application no matter how I tune the limits. Just one request causes the memory usage of the app to increase by ~40mb forever, whereas without image processing it only increases temporarily by 2mb. I realize that the logs look fine and I'm not going to convince you otherwise, but something just feels wrong about that. And unless there's something egregious in my code sample above, there's really nothing left that I can do.
That's what I had originally thought might be the issue. GC heap dumps and runtime metrics aren't showing that that's the case though. The GC is definitely collecting as it gets pressure. Even with tons of requests the usage spikes (approximately by the file sizes) but then gets collected in short order if I'm sending lots of requests. The arrays don't show up in the memory dump outside of what the array pool caches per TLS and if I take Magick.NET out of the picture everything is fine no matter how hard I hammer. I've forced GC, including the LOH and it helps a little. Setting the GC heap hard limit to something lowish forces collections more frequently which helps a bit as well. I've also tried Workstation GC since that doesn't pre-allocate as much, but that only prolongs the inevitable. I apologize for the spam, and once again thank you for your time. I've been trying to resolve this for a few weeks now. It's a huge PITA to try and troubleshoot since I can't do it on my local machine. I am grasping at straws and growing increasingly frustrated. If I upload many more files or review just one more memory dump I might lose it soon 😛 |
Beta Was this translation helpful? Give feedback.
-
|
This weekend I ran your heic image through a memory analyzer and no leaks were reported. I also tried to run your code to see memory increments and I did not notice anything strange. The only thing that I noticed with local debugging was that I had to disable OpenCL because OpenCL acceleration would require extra host memory. But this is disabled by default and you did not mention that you turned it on. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I don't even know what that is 😄 I'm sure you're right that it's something else in our code, but I've been unable to figure out what. I was wondering if it could be Linux distro specific or something but that's the only thought I have left. That said, we've decided to go a different route for now given that I've spent so much time on this. I've added a liveness probe to our app that will cause Kubernetes to gracefully restart the container if it's using more than X percent of overall memory for Y duration. At least then the app recovers and we have enough replicas for that to hopefully not be an issue. In my load testing it's performed quite well. It's definitely a hammer, but I've spent enough time trying to figure it out unfortunately. We've also switched to DATAS (default in net9) which gives workstation-GC-like memory consumption, and we've enabled conservativeGC and lowered the GC high-memory percent. All of which at least prolong the overall issue a good deal as well. Thanks again for your assistance. I really do appreciate it. |
Beta Was this translation helpful? Give feedback.
-
The Problem
We are using Magick.NET as part of an ASP.NET application that runs as a Linux Docker containers in Kubernetes. The application creates thumbnails for any images our users might upload. The maximum upload size is 30Mb. Everything has been working generally fine. However once we get a handful of user uploads (usually in short succession or triggered by QA testing) the pod's memory consumption rises and Kubernetes OOM kills the pod.
We run our containers with a memory limit of
350Mi. Because of this, before the OOM's I originally setResourceLimits.LimitMemoryto 35% out of precaution. Once we started getting the OOM kills I lowered it even further to 20% which didn't seem to help. I even increased the container max memory to700Miand despite this it eventually gets OOM killed.I had read some other threads here that suggested making sure you were running as x64 (since we're using the AnyCPU build) and that
ResourceLimitswere configured properly. So I added some basic console logging to check whatResouceLimits.Memorywas actually defaulting to, and this was the output:Well,
14653595648bytes equates to more than 13GB which is well over the memory limits set on the pods. It appears that when running in Kubernetes as a Linux container, the defaults are being taken from theMemTotalentry in/proc/meminfo(or it's equivalent) which equate to the node's resources, and not the pod's.Here's the relevant portion of
/proc/meminfo:Instead, the defaults should be taken from the cgroups (v2) file at
/sys/fs/cgroup/memory.max, which in this instance containscorrelating to the limit of
350Mithat this pod currently has.Questions
Now that I realize what's happening, I can make changes to both the pod configuration and our application so that I'm able to set
ResourceLimitsbased on the pod's actual memory availability. That leaves me with a couple of questions:ResourceLimits.Memoryin this constrained environment (350mi), where the largest images we work with are 30mb?ResourceLimits.Memorytune the same knobs asResourceLimits.LimitMemory? I can't use the latter since passing a percentage won't workResourceLimits.MaxMemoryRequestas well?MemoryandMaxMemoryRequest. The documentation doesn't really explain it to meIn case it matters:
MagickNET.SetTempDirectoryas we were expecting to spill to disk.Magick.NET-Q8-AnyCPUversion14.0.0Beta Was this translation helpful? Give feedback.
All reactions