@steven-murray what do you think is the best option for doing this? I've been thinking about it and have a couple of options.

1. A stand-alone Python function which contains all possible combinations of FlagOptions etc., to estimate the amount of memory use. Pros, simple interface and easy to implement. Cons, must be manually maintained and updated each time memory usage is added to any function
2. Pass a flag option through all functions and wrappers that calculates the amount of memory used. Pros, easier to maintain its proper usage as just need to add an option where ever memory is allocated. Cons, requires passing through extra flags and that the return structures will probably need to change depending on if the option is on or off (or just NULL it if off). Also, would require adding/modifying all existing functions to handle the options (i.e. don't need to perform all checks for existing data etc., as nothing is to be calculated).

Unfortunately, I don't see a clean and robust way to do it. I'd prefer number 2) however, I'm concerned it may get a little messy within the function structures. 1) is clean and easy, but does require manual updating which is far from ideal.

Any thoughts?

Hey @BradGreig , thanks for thinking through this. Yeah, I don't think there's going to be a perfectly elegant solution. I think option (2) is the way to go, messy as it is, because there's less chance of it becoming outdated.

I wonder if we could kind of get the best of both worlds by creating a python function such as get_memory_usage, to which you pass a function and arguments etc, and it goes off and fills out the relevant flag options etc., and ensures that nothing is passed back, just memory printed to screen (or perhaps the memory usage is passed back).

It also can't be a FlagOption to set it to do this, because init and perturb don't receive flag options (though perhaps they should?!). It could be a UserParam, but not sure if that fits either. It also could be an extra argument to each function.

Actually, this is supposed to just record memory usage, and not do any computations, right? I don't know if it's even going to be possible to do that in a robust way throughout the functions, because the functions are going to error.

Maybe in the end it's just easier to write a standalone function, and try to ensure that anyone who changes any C code adjusts that function if they add/remove any allocations?

@steven-murray yeah, the idea here is to have a function that can be called that'll output just the amount of memory such a setup will require. Basically, to know whether or not it'll fit within memory on the specific machine, before trying to run it and it failing because it runs out of resources.

I agree, definitely would prefer a robust solution, but cannot think of a clean way to do it. I'll go ahead and construct a standalone function which should be easy enough to modify/update.

Perhaps in its implementation, or over time we'll come up with a more robust way of doing things.