Tuesday, June 27, 2017

Backing up or dumping a memcached server

I was needing to move from an old cache server to a larger one, but I wanted to do it without flushing cache.

The first thing I came across was this "memcached-tool" which has a dump command: https://github.com/memcached/memcached/blob/master/scripts/memcached-tool

There's another article that mentions using memdump and memcat: http://www.dctrwatson.com/2010/12/how-to-dump-memcache-keyvalue-pairs-fast/

Unfortunately, those methods only dumped a few mb of data. This post explains why: https://stackoverflow.com/a/13941700

You can only dump one page per slab class (1MB of data)
So, I ended up writing a script that loops through the expected cache keys, gets the data in cache, then sets the data in the new cache server.

Monday, June 12, 2017

Gunicorn - "Resource temporarily unavailable"

Are you seeing this error in your logs while your server is under high load?:
[error] 10#0: *14843 connect() to unix:/tmp/gunicorn.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 96.44.145.186, server: , request: "GET / HTTP/1.0", upstream: "http://unix:/tmp/gunicorn.sock:/", host: "45.55.46.84"
I ended up making an example dockerfile with nginx + gunicorn + flask to reproduce this problem: https://github.com/pawl/somaxconn_test

Bumping the "net.core.somaxconn" setting ended up fixing it.