Paul's Programming Notes     Archive     Feed     Github

Python Excel Library Benchmark Comparison

https://github.com/swistakm/python-excel-benchmarks

benchmark_csv                  0.026495
benchmark_excellent            1.784107
benchmark_openpyxl             2.897072
benchmark_openpyxl_rows        7.025895
benchmark_pyexcelerate         0.550225
benchmark_xlsxcessive          1.430242
benchmark_xlsxwriter           1.611668
benchmark_xlwt                 1.275144

http://xlsxwriter.readthedocs.org/en/latest/working_with_memory.html

pyexcelerate          :  10.11
xlwt                  :  15.67
xlsxwriter (optimised):  19.70
xlsxwriter            :  23.50
openpyxl   (optimised):  95.82
openpyxl              :  95.90

Large Dictionaries Not Released From Memory - Python + Ubuntu

I have a flask application running on Ubuntu which reads from a file and creates several large dictionaries. After the dictionaries are created and flask returns the request, the memory used for the dictionaries is not released back to the OS. This example shows the problem: https://gist.github.com/pawl/c3ed7663b94abec01d75

It's not really a Flask issue, but the Flask guys were super helpful when I asked why my large dictionary wasn't released from memory: https://github.com/mitsuhiko/flask/issues/1202

I dug even deeper and learned this is a Linux thing:
"Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."
http://deeplearning.net/software/theano/tutorial/python-memory-management.html

My solution:
Turning it into a function and running it as a separate processhttps://gist.github.com/pawl/95769724848269cff890

If you want to go even deeper down the rabbit hole, read these:

Virtualenvwrapper Basics

This video does a good job of describing the basics of Virtualenvwrapper: https://www.youtube.com/watch?v=thHNYVrY0lU

You'll need to have used virtualenv to appreciate this.

Basically, you create virtual environments with "mkvirtualenv" and they're stored in a central location. When you want to switch to your virtual environment, you use the "workon" command.

You can also create a postactivate script that loads environmental variables when you switch to your virtual environment.

Note: There is also a module called Pew that seems to have a simpler installation process: https://github.com/berdario/pew

Meetup API address_1_error

I was getting this error while using the POST /venues method of the meetup API.

The error seems to happen when you duplicate the address of another venue. I was able to add the venue, but when I tried to add it a second time - I would get this error.

I ended up getting the list of venues and trying to find a match first.