Paul's Programming Notes     Archive     Feed     Github

SQLAlchemy Performance Tip

http://www.mobify.com/blog/sqlalchemy-memory-magic/

"If you can process the results of database queries iteratively (and very often you can), stream the results"

"Since the data needs to travel over the network from the database whether it's streamed or not, this doesn't add a huge overhead, but we'll see that it reduces memory requirements."

Example (using Flask-SQLAlchemy):
db.engine.execution_options(stream_results=True).execute(query)

However, that's not the full story when it comes to MySQLdb (MySQL-python). http://stackoverflow.com/a/3699677

If you're going to do this with mysql, I recommend using oursql: https://pypi.python.org/pypi/oursql