Tuesday, June 27, 2017

Backing up or dumping a memcached server

I was needing to move from an old cache server to a larger one, but I wanted to do it without flushing cache.

The first thing I came across was this "memcached-tool" which has a dump command: https://github.com/memcached/memcached/blob/master/scripts/memcached-tool

There's another article that mentions using memdump and memcat: http://www.dctrwatson.com/2010/12/how-to-dump-memcache-keyvalue-pairs-fast/

Unfortunately, those methods only dumped a few mb of data. This post explains why: https://stackoverflow.com/a/13941700

You can only dump one page per slab class (1MB of data)
So, I ended up writing a script that loops through the expected cache keys, gets the data in cache, then sets the data in the new cache server.

Monday, June 12, 2017

Gunicorn - "Resource temporarily unavailable"

Are you seeing this error in your logs while your server is under high load?:
[error] 10#0: *14843 connect() to unix:/tmp/gunicorn.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 96.44.145.186, server: , request: "GET / HTTP/1.0", upstream: "http://unix:/tmp/gunicorn.sock:/", host: "45.55.46.84"
I ended up making an example dockerfile with nginx + gunicorn + flask to reproduce this problem: https://github.com/pawl/somaxconn_test

Bumping the "net.core.somaxconn" setting ended up fixing it.

Friday, March 17, 2017

SQLAlchemy - empty lists + in_() causing crazy queries (before 1.2.0)

Before SQLAlchemy 1.2.0, if you use an empty list with in_(), it will emit some crazy SQL that will query your entire table. The best solution is probably to upgrade to SQLAlchemy 1.2.0.


Monday, March 13, 2017

The Robustness Principle

I learned about this at a talk called "Implementing Evolvable APIs" at SXSW: https://en.wikipedia.org/wiki/Robustness_principle

For example, making an API that throws errors when an unexpected parameter is provided is a bad idea. What if you need to make changes to the client to add the new parameter? You will need to make sure you deploy the code on the server side first, otherwise it will cause errors.

Tuesday, February 14, 2017

SQLAlchemy - DISTINCT, LIMIT, or OFFSET Causing Subqueries

This section at the bottom of The Zen of Eager loading is really important:
When using joined eager loading, if the query contains a modifier that impacts the rows returned externally to the joins, such as when using DISTINCT, LIMIT, OFFSET or equivalent, the completed statement is first wrapped inside a subquery, and the joins used specifically for joined eager loading are applied to the subquery. SQLAlchemy’s joined eager loading goes the extra mile, and then ten miles further, to absolutely ensure that it does not affect the end result of the query, only the way collections and related objects are loaded, no matter what the format of the query is.
I made an example to illustrate this here: https://gist.github.com/pawl/cb57e0ddbdd0b2e64b75e94116873367

On MySQL this can be responsible for some really poor query performance, because it can cause it to use temporary tables and filesort.

The best way I've found to prevent the subqueries is by first querying for the ids only, then running another query that includes all relations. For example:
ids = session.query(Product.id).limit(20)
Product.query.filter(Product.id.in_(ids))

Sunday, February 12, 2017

SQLAlchemy - "Base.query = db_session.query_property()"

I just saw this line in the SQLAlchemy flask patterns example:
Base.query = db_session.query_property()
I went searching for what it does and found this: https://eoyilmaz.blogspot.com/2014/01/i-feel-so-much-hungry-about-posting.html

Now I know how Flask-SQLAlchemy made queries work with Model.query instead of session.query(Model).