Maybe the context (or the process stack) should only hold a weak reference to the process? Or is there a use case where the process needs to be kept "in memory" by the context?

It doesn't completely eliminate the possibility of someone keeping the process alive - the weak reference can be evaluated into a "regular" reference. But probably it's less likely to happen in parts of the code that don't need the process in the first place.

In [1]: import weakref

In [2]: class TestObject:
   ...:     """needed because built-in types do not support weak references"""
   ...:

In [3]: f = TestObject()

In [4]: g = weakref.ref(f)

In [5]: g
Out[5]: <weakref at 0x7f219514d048; to 'TestObject' at 0x7f21928fd048>

In [6]: del f

In [7]: g  # dead after f has been deleted
Out[7]: <weakref at 0x7f219514d048; dead>

In [8]: f = TestObject()

In [9]: g = weakref.ref(f)

In [10]: h = g() # turn back into a normal reference

In [11]: del f

In [12]: g
Out[12]: <weakref at 0x7f219333b7c8; to 'TestObject' at 0x7f21928f0ef0>

In [13]: del h

In [14]: g # dead only after both f and h have been deleted
Out[14]: <weakref at 0x7f219333b7c8; dead>

@greschd Thanks, I wasn't aware of the weakref option. That sounds like a great solution here.
I'll prepare a PR to plumpy

Err, I don't think using weak references is a good solution, which could non-deterministically dissappear when they are actually required, leading to potentially just as bad side-effects.
I think it is better to first actually clarify where Process.current() and hence these context vars are required, and see if perhaps there is a more "selective" way we can pass them around.

Really the only use for weak references is for caches, that are only kept for performance needs but won't break the code if they are garbage collected

which could non-deterministically dissappear

Right, if they need to be kept in memory by the context in general I agree it's not a useful solution. I wasn't sure if there might be a different part of the system that has "ownership" of the process; from the description above it seemed like the context shouldn't own the process.

Use cases for weakref are pretty rare, but I wouldn't go so far to say it's only for caches; it can also be useful to avoid circular referencing.

it can also be useful to avoid circular referencing.

Python already has a cycle-detecting garbage collector though, so you should not need it for that

which could non-deterministically dissappear when they are actually required, leading to potentially just as bad side-effects.

Let's put it this way: a broken reference in Process.current() will be way easier to debug than a hidden chunk of memory references dragged around by the process context.

If one can make the passing around of the context more explicit/obvious, that would be fine as well.

The only thing I would like to avoid is to have to rely on the diligence of plumpy users to always be aware of what is going on in the background - in order to avoid surprises like the one we had now in aiida.

should not need it for that

It can be needed if you want to implement a non-trivial __del__ that needs to be called in the right order.

Anyway, I agree (sort of) with @chrisjsewell, in that we don't know exactly what the consequence of making it a weakref is. We should definitely be careful with that, as it could seriously break things.
Since the automatic memory management hasn't really worked out here, we need to figure out who should own the process, and how best to fix that.

I also didn't understand 100% why the futures stay in memory in the first place, after they have been cancelled.

in that we don't know exactly what the consequence of making it a weakref is. We should definitely be careful with that, as it could seriously break things.

yep thats my key point; lets not jump out of the frying pan, and into the fire 😉

fixed for the time being in #4699
to be fixed more generally as described in #4704 (comment)