Python Excel Library Benchmark Comparison

15 Oct 2014

I used swistakm’s python-excel-benchmarks to compare write speed across the Python Excel libraries:

benchmark_csv                  0.026495
benchmark_excellent            1.784107
benchmark_openpyxl             2.897072
benchmark_openpyxl_rows        7.025895
benchmark_pyexcelerate         0.550225
benchmark_xlsxcessive          1.430242
benchmark_xlsxwriter           1.611668
benchmark_xlwt                 1.275144

xlsxwriter’s Working with Memory and Performance page has its own comparison:

pyexcelerate          :  10.11
xlwt                  :  15.67
xlsxwriter (optimised):  19.70
xlsxwriter            :  23.50
openpyxl   (optimised):  95.82
openpyxl              :  95.90

Large Dictionaries Not Released From Memory - Python + Ubuntu

13 Oct 2014

I have a flask application running on Ubuntu which reads from a file and creates several large dictionaries. After the dictionaries are created and flask returns the request, the memory used for the dictionaries is not released back to the OS. This example shows the problem:

import random import optparse import gc from flask import redirect, Flask from memory_profiler import memory_usage # Create flask app app = Flask(__name__) app.debug = True @app.route('/') def hello_world(): ### # The test: # Generate a 2440 mb dictionary and return a blank page. ### large_dict = {} for x in xrange(125000): large_row = {} for y in xrange(125): large_row[random.randint(1, 10000000000)] = random.randint(1, 10000000000) large_dict[random.randint(1, 10000000000)] = large_row large_dict = {} gc.collect() return '' @app.after_request def after(response): print memory_usage(-1, interval=.2, timeout=1), "after request" return response def main(): app.config['SECRET_KEY'] = "BLAH" parser = optparse.OptionParser() parser.add_option("--host", default="0.0.0.0") # can't use -h, it's already taken by help parser.add_option("-p", "--port", default="5123") options, args = parser.parse_args() if len(args) > 0: parser.error("Unexpected arguments: %s" % args) app.run(host=options.host, port=int(options.port)) if __name__ == '__main__': # Start app main()

It’s not really a Flask issue, but the Flask guys were super helpful when I asked why my large dictionary wasn’t released from memory: Flask issue #1202

I dug even deeper and learned this is a Linux thing:
“Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase.”
Theano: Python memory management

My solution:
Turning it into a function and running it as a separate process:

import random import optparse import gc from flask import redirect, Flask from memory_profiler import memory_usage from multiprocessing import Process # Create flask app app = Flask(__name__) app.debug = True @app.route('/') def hello_world(): ### # The test: # Generate a 2440 mb dictionary and return a blank page. ### def separate_process(): large_dict = {} for x in xrange(125000): large_row = {} for y in xrange(125): large_row[random.randint(1, 10000000000)] = random.randint(1, 10000000000) large_dict[random.randint(1, 10000000000)] = large_row p = Process(target=separate_process) p.start() p.join() # this blocks until the process terminates return '' @app.after_request def after(response): print memory_usage(-1, interval=.2, timeout=1), "after request" return response def main(): app.config['SECRET_KEY'] = "BLAH" parser = optparse.OptionParser() parser.add_option("--host", default="0.0.0.0") # can't use -h, it's already taken by help parser.add_option("-p", "--port", default="5123") options, args = parser.parse_args() if len(args) > 0: parser.error("Unexpected arguments: %s" % args) app.run(host=options.host, port=int(options.port)) if __name__ == '__main__': # Start app main()

If you want to go even deeper down the rabbit hole, read these:

“most malloc implementations will not release memory to the operating system, and the few that do, do not do it very easily” -http://stackoverflow.com/questions/2215259/will-malloc-implementations-return-free-ed-memory-back-to-the-system
“We compiled a version of Python with TCMalloc that only uses mmap. When testing the new Python in one of our largest projects, we found that not only did Python give back memory to the OS correctly, it also had a reduced memory usage and no apparent CPU penalty for using mmap instead of brk.” - pushingtheweb.com: Python and TCMalloc

Renaming Screen - Ubuntu

08 Oct 2014

http://superuser.com/a/521112

screen -S 8890.foo -X sessionname bar

Initializing Dictionary Elements Inside A Loop - Python

04 Oct 2014

There’s a really interesting tip in the “Initializing Dictionary Elements” section of this page: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

“it is cheaper to use a try statement” when checking to see if a key exists in a dictionary (inside of a loop).

maximum recursion depth exceeded - PyIntervalTree

02 Oct 2014

I was getting the “maximum recursion depth exceeded” when I tried to run large datasets through PyIntervalTree.

I never fixed the issue or found anyone else with the issue. I ended up switching to bx-python instead: https://www.biostars.org/p/99/#100

Group By Apply Occurring Twice On First Item - Pandas

02 Oct 2014

https://github.com/pydata/pandas/issues/7739#issuecomment-48823964

If Pandas’s Group By is applying the function in Apply twice to the first item, it’s actually expected behavior according to the warning in the documentation: http://pandas.pydata.org/pandas-docs/stable/groupby.html#flexible-apply

Virtualenvwrapper Basics

28 Sep 2014

This video does a good job of describing the basics of Virtualenvwrapper: https://www.youtube.com/watch?v=thHNYVrY0lU

You’ll need to have used virtualenv to appreciate this.

Basically, you create virtual environments with mkvirtualenv and they’re stored in a central location. When you want to switch to your virtual environment, you use the workon command.

You can also create a postactivate script that loads environmental variables when you switch to your virtual environment.

Note: There is also a module called Pew that seems to have a simpler installation process: https://github.com/berdario/pew

Meetup API address_1_error

25 Sep 2014

I was getting this error while using the POST /venues method of the meetup API.

The error seems to happen when you duplicate the address of another venue. I was able to add the venue, but when I tried to add it a second time - I would get this error.

I ended up getting the list of venues and trying to find a match first.

Profiling Python Scripts

20 Sep 2014

what-studio/profiling on GitHub

Easiest Way To Install Python Libraries In Windows (that require compiling)

18 Sep 2014

Gohlke’s prebuilt Windows Python libraries (archived)

Older Newer