Dump report on SIGUSR2 without exiting

[craigds]

While trying to debug a long slow memory leak I found that the OOM detection wasn't quite working. So I created this script to dump a report every hour until the process is killed:

    echo "Running django-admin runserver under Fil to debug memory leaks"
    fil-profile -o "/storage/2022-04-web-fil-profile/$(hostname)" --no-browser run \
        /venv/bin/django-admin runserver 0.0.0.0:10080 \
            --noreload \
            --nostatic &
    FIL_PID=$!
    (
        while true ; do
            # once an hour
            sleep 3600
            # dump the fil profile to the output directory
            kill -s SIGUSR2 $FIL_PID
        done
    ) &
    LOOP_PID=$!
    wait $FIL_PID
    kill $LOOP_PID

Unfortunately this doesn't work as intended because Fil seems to exit the process after the first SIGUSR2, so the web server no longer handles any requests.

Is there a way to ask fil to dump a report without exiting? I can't seem to find one.

[itamarst]

This seems like a bug, SIGUSR2 should keep it running. I will try to reproduce.

[itamarst]

Separately: I'm curious about the failed OOM detection; possibly it could be improved, depending. What operating system is this? How much RAM and swap do you have? Do you use compressed RAM?

[craigds]

This was inside docker-compose container (docker desktop 4.6.1 on MacOS 12.3.1), and I added the memory limit manually using docker update --memory 3000000000 --memory-swap 3000000000 <containername>. I confirmed the memory limit had taken effect via the MEM USAGE / LIMIT column in docker stats.

I haven't yet compared behaviour with our staging k8s cluster but it looks like I'm going to have to do that. I expect the memory-limit behaviour to be quite different there

[itamarst]

Thanks! I guess I can test that directly, then, with a leaky reproducer of some sort.

[itamarst]

OK, the SIGUSR2 problem should be fixed in 2022.05.0, which should be available on PyPI in the next few minutes.