Gevent + Requests Performance With verify=True/False

11 Aug 2017

If you use gevent with requests.get on a HTTPS URL with the default verify=True enabled, you’ll see almost 2x longer execution times than with verify=False.

I made a script to test:

import urllib3 import timeit import gevent import requests from gevent import monkey monkey.patch_all() urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) def test_request(verify): resp = requests.get( 'https://www.instagram.com/instagram/?__a=1', verify=verify, ) resp.raise_for_status() start_time = timeit.default_timer() for x in range(100): test_request(verify=True) print "verify=True took:", timeit.default_timer() - start_time, "secs" start_time = timeit.default_timer() for x in range(100): test_request(verify=False) print "verify=False took:", timeit.default_timer() - start_time, "secs" start_time = timeit.default_timer() gevent.wait([gevent.spawn(test_request, True) for x in range(100)]) print "gevent verify=True took:", timeit.default_timer() - start_time, "secs" start_time = timeit.default_timer() gevent.wait([gevent.spawn(test_request, False) for x in range(100)]) print "gevent verify=False took:", timeit.default_timer() - start_time, "secs"

Here are the results:

verify=True took: 40.3454630375 secs verify=False took: 39.3803040981 secs gevent verify=True took: 2.23735189438 secs gevent verify=False took: 1.58263015747 secs

I suspect that gevent is having trouble using pyopenssl concurrently because it’s a C library.

Backing up or dumping a memcached server

28 Jun 2017

I was needing to move from an old cache server to a larger one, but I wanted to do it without flushing cache.

The first thing I came across was this “memcached-tool” which has a dump command: https://github.com/memcached/memcached/blob/master/scripts/memcached-tool

There’s another article that mentions using memdump and memcat: How to dump memcached key/value pairs fast (archived)

Unfortunately, those methods only dumped a few mb of data. This post explains why: https://stackoverflow.com/a/13941700

You can only dump one page per slab class (1MB of data)

So, I ended up writing a script that loops through the expected cache keys, gets the data in cache, then sets the data in the new cache server.

Microservice Best Practices

27 Jun 2017

Best Practices for Building a Microservice Architecture

Gunicorn - "Resource temporarily unavailable"

13 Jun 2017

Updated 2026-08-08: explained why gunicorn’s own backlog setting doesn’t fix this.

Are you seeing this error in your logs while your server is under high load?:

[error] 10#0: *14843 connect() to unix:/tmp/gunicorn.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 192.0.2.10, server: , request: "GET / HTTP/1.0", upstream: "http://unix:/tmp/gunicorn.sock:/", host: "198.51.100.20"

I ended up making an example dockerfile with nginx + gunicorn + flask to reproduce this problem: https://github.com/pawl/somaxconn_test

Bumping the net.core.somaxconn setting ended up fixing it.

Error 11 is EAGAIN, and on a connect() to a unix socket it means the listening socket’s accept queue is full. Connections sit in that queue after the kernel accepts them and before gunicorn calls accept(), so it fills up whenever requests arrive faster than the workers drain them. Once it’s full the kernel refuses new connections instead of queueing them, and nginx reports the refusal as this error.

net.core.somaxconn is the ceiling on how deep that queue is allowed to be. Linux capped it at 128 until kernel 5.4 raised the default to 4096, so on anything older this is a low bar to hit.

The part that cost me the most time is that gunicorn’s own --backlog defaults to 2048, which looks like plenty. listen(2) silently truncates whatever a process asks for down to somaxconn, so gunicorn requested 2048 and got 128, with nothing in any log to say so. Raising the sysctl is what actually changes the queue:

sudo sysctl -w net.core.somaxconn=4096

Put it in a file under /etc/sysctl.d/ to survive a reboot. In a container it’s a property of the network namespace rather than the image, so it’s docker run --sysctl net.core.somaxconn=4096, or set on the host if the container shares its network namespace.

Worth saying that a full accept queue is usually a symptom. If the workers can’t keep up, the queue depth buys headroom for a traffic spike, not for a slow application.

SQLAlchemy - in_() and notin_() with an empty list

18 Mar 2017

Updated 2026-07-16: clarified the current behavior and cited the SQLAlchemy docs.

Before SQLAlchemy 1.2.0, an empty list passed to in_() emitted SQL that queried your entire table. Since 1.2.0 this is handled: an empty in_() renders a backend-specific empty-set expression that matches no rows, and an empty notin_() one that matches all rows, so neither scans the table. If you are stuck on an older version, upgrading is the fix.

The gist below shows the old behavior and the query it produced:

# demonstrates the issue fixed in: https://bitbucket.org/zzzeek/sqlalchemy/issues/3907 from sqlalchemy import create_engine, Column, Integer from sqlalchemy.orm import scoped_session, sessionmaker from sqlalchemy.ext.declarative import declarative_base engine = create_engine('mysql://root@localhost/test?charset=utf8mb4', convert_unicode=True, echo=True) session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine)) Base = declarative_base() Base.query = session.query_property() class Post(Base): __tablename__ = 'posts' id = Column(Integer, primary_key=True) #Base.metadata.drop_all(engine) Base.metadata.create_all(engine) # create new rows if database is empty for x in range(10): session.add(Post()) session.commit() # empty lists trigger insane queries before 1.2.0 Post.query.filter(Post.id.in_([])).all() """ SELECT posts.id AS posts_id FROM posts WHERE posts.id != posts.id """

The Robustness Principle

14 Mar 2017

I learned about this at a talk called “Implementing Evolvable APIs” at SXSW: Wikipedia: Robustness principle

For example, making an API that throws errors when an unexpected parameter is provided is a bad idea. What if you need to make changes to the client to add the new parameter? You will need to make sure you deploy the code on the server side first, otherwise it will cause errors.

SQLAlchemy - DISTINCT, LIMIT, or OFFSET Causing Subqueries

15 Feb 2017

Updated 2026-07-16: fixed the docs link, confirmed it still applies in 2.0, and added the 2.0 syntax alongside the original.

This part of The Zen of Joined Eager Loading is really important, and still applies in SQLAlchemy 2.0:

When using joined eager loading, if the query contains a modifier that impacts the rows returned externally to the joins, such as when using DISTINCT, LIMIT, OFFSET or equivalent, the completed statement is first wrapped inside a subquery, and the joins used specifically for joined eager loading are applied to the subquery. SQLAlchemy’s joined eager loading goes the extra mile, and then ten miles further, to absolutely ensure that it does not affect the end result of the query, only the way collections and related objects are loaded, no matter what the format of the query is.

I made an example to illustrate this:

from sqlalchemy import create_engine, Column, ForeignKey, Integer from sqlalchemy.orm import relationship, scoped_session, sessionmaker from sqlalchemy.ext.declarative import declarative_base engine = create_engine('mysql://root@localhost/test?charset=utf8mb4', convert_unicode=True) session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine)) Base = declarative_base() Base.query = session.query_property() class Post(Base): __tablename__ = 'posts' id = Column(Integer, primary_key=True) products = relationship( 'Product', lazy='joined', ) class Product(Base): __tablename__ = 'products' id = Column(Integer, primary_key=True) post_id = Column(Integer, ForeignKey('posts.id'), index=True) #Base.metadata.drop_all(engine) Base.metadata.create_all(engine) first_result = Post.query.first() if not first_result: for x in range(50): products = [Product() for y in range(12)] session.add(Post(products=products)) session.commit() print(Post.query.limit(20))

On MySQL this can be responsible for some really poor query performance, because it can cause it to use temporary tables and filesort.

The best way I’ve found to prevent the subqueries is by first querying for the ids only, then running another query that includes all relations:

ids = session.query(Product.id).limit(20)
Product.query.filter(Product.id.in_(ids))

In SQLAlchemy 2.0, the same approach with select():

ids = select(Product.id).limit(20)
products = session.scalars(
    select(Product).where(Product.id.in_(ids))
).all()

SQLAlchemy - "Base.query = db_session.query_property()"

13 Feb 2017

I just saw this line in the SQLAlchemy flask patterns example:

Base.query = db_session.query_property()

I went searching for what it does and found this: https://eoyilmaz.blogspot.com/2014/01/i-feel-so-much-hungry-about-posting.html

Now I know how Flask-SQLAlchemy made queries work with Model.query instead of session.query(Model).

SQLAlchemy - Lost connection to MySQL server during query

11 Feb 2017

Here’s was my situation:

The database was set up behind behind an AWS ELB and HAProxy.
The idle connection timeout on the ELB was set to 60 mins.
All the relevant timeouts on HAProxy seemed to be set to 60 mins too.
The pool_recycle in SQLAlchemy was set to 30 mins.
I was still seeing the occasional “Lost connection to MySQL server during query” when small queries were running after the connection had some time to sit around.

The solution ended up being setting my pool_recycle down to 5 mins, but I’m still not sure what was causing connections to time out after 5 mins.

There are definitely other things that can cause this problem too. For example, it can happen if your data exceeds max_allowed_packet. See this page for more details: https://dev.mysql.com/doc/refman/5.7/en/error-lost-connection.html

Most of this also applies for “MySQL server has gone away”.

You should also make sure your pool_recycle is set lower than your ‘interactive_timeout’ and ‘wait_timeout’ properties in the mysql config file to the values you need.

SQLAlchemy - Unexpected Lazy Loading

27 Jan 2017

If you’re seeing unexpected lazy loading on a lazy=”joined” relationship in SQLAlchemy, it might be because you’re accessing those relationships after you’ve already run session.commit(). By default, session.commit() will expire the data on your relationships, meaning it will try to fetch it again next time you try to access those attributes.

The relevant section of the docs for session.commit():

By default, the Session also expires all database loaded state on all ORM-managed attributes after transaction commit. This so that subsequent operations load the most recent data from the database. This behavior can be disabled using the expire_on_commit=False option to sessionmaker or the Session constructor.

Older Newer