Monday, October 13, 2014

Large Dictionaries Not Released From Memory - Python + Ubuntu

I have a flask application running on Ubuntu which reads from a file and creates several large dictionaries. After the dictionaries are created and flask returns the request, the memory used for the dictionaries is not released back to the OS. This example shows the problem:

It's not really a Flask issue, but the Flask guys were super helpful when I asked why my large dictionary wasn't released from memory:

I dug even deeper and learned this is a Linux thing:
"Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."

My solution:
Turning it into a function and running it as a separate process

If you want to go even deeper down the rabbit hole, read these:


  1. That really helped me. I was stuck in similar issue from almost a week.

  2. Thanks this helped me. Using Flask 1.0.2 with Python 3.7 and still faced the issue. Solved it by using a subprocess.

  3. Thank you for your post,

    I have a very big memory leak problem on Google App Engine with python 37. Could I ask that if there are multi post request come to your python web server, still process run seperately on each post request (let say session).

    Thank you very much