Paul's Programming Notes PostsRSSGithub

Django - RelatedManager.set not removing models

Python - Pipenv to pip-tools

I’ve been using Pipenv for the last few months and my biggest issue is that --keep-outdated has been broken in the latest release (2018.11.26) for a while. I’ve needed to install Pipenv from the master branch to make it functional. However, the last time I used --keep-outdated from the master branch, it wouldn’t automatically update the hash of the dependency being updated.

Updating specific requirements is something I need to do pretty often, and it’s not fun to explain all the Pipenv quirks to the team.

Pip-tools looks like it does everything I need and has fewer quirks, so I ended up making the switch.

Pipenv uses pip-tools under the hood, so the migration to pip-tools was very smooth. The migration process was:

  1. Copy the dev-packages and packages sections of the Pipfile to their own requirements.in files.
  2. Run pip-compile
  3. Copy over the specific versions and hashes from the Pipfile.lock to the generated requirements.txt.

I did have a small issue where updating a specific package with pip-tools removed a bunch of dependencies from the requirements.txt unexpectedly, but running pip-compile with --rebuild fixed it.

ESP32 Plant Sensor

I set up a ESP32 houseplant soil water + temperature + humidity + light sensor that sends me a daily status update message.

Here’s the code for it:

esp32 plant sensor 1

esp32 plant sensor 2

esp32 plant sensor 3

AWS - redirecting domain to url using a 302 redirect (without running a server)

I wanted to make a domain name (heckingoodboys.com) redirect to a multisubreddit for dog pictures, but I didn’t want to run a web server for it.

Here’s what I did:

  1. Purchase the domain using Route53.
  2. Create two public s3 buckets (www.heckingoodboys.com and heckingoodboys.com)
  3. Enable “Static website hosting” on www.heckingoodboys.com and redirect to heckingoodboys.com.

  4. Enable “Static website hosting” on heckingoodboys.com, select “use this bucket to host this website”, and use routing rules similar to this:

    <RoutingRules>
  <RoutingRule>
    <Redirect>
      <Protocol>https</Protocol>
      <HostName>www.reddit.com</HostName>
      <HttpRedirectCode>302</HttpRedirectCode>
      <ReplaceKeyPrefixWith>user/heckingoodboys/m/heckingoodboys/</ReplaceKeyPrefixWith>
    </Redirect>
  </RoutingRule>
</RoutingRules>
  5. Back to Route53 - Create an A record for both www.heckingoodboys.com and heckingoodboys.com using the alias to their respective buckets. (this will be the first option in autocomplete)

For more details: https://medium.com/@P_Lessing/single-page-apps-on-aws-part-1-hosting-a-website-on-s3-3c9871f126

Why not just use a CNAME from www.heckingoodboys.com to heckingoodboys.com? AWS says they don’t charge for aliases, but they do charge for CNAMEs. So, I used an alias to a bucket instead.

django-celery-email - Reducing memory usage on large email batches

I got a pull request merged into django-celery-email that cuts peak memory usage by about 74% when sending large batches of emails with attachments.

CeleryEmailBackend.send_messages used to serialize every message up front, then split the serialized list into chunks, then hand each chunk to a Celery task:

def send_messages(self, email_messages):
    result_tasks = []
    messages = [email_to_dict(msg) for msg in email_messages]
    for chunk in chunked(messages, settings.CELERY_EMAIL_CHUNK_SIZE):
        result_tasks.append(send_emails.delay(chunk, self.init_kwargs))

email_to_dict copies the whole message into a dict, attachments included. Doing that for the entire batch before chunking means every serialized email sits in memory at once. With attachments, that gets expensive fast.

The fix is to chunk the messages first, then serialize one chunk at a time:

def send_messages(self, email_messages):
    result_tasks = []
    for chunk in chunked(email_messages, settings.CELERY_EMAIL_CHUNK_SIZE):
        chunk_messages = [email_to_dict(msg) for msg in chunk]
        result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))

Same tasks get queued, but only one chunk’s worth of serialized emails exists at a time, so the earlier chunks can be garbage collected once their task is queued.

Benchmarks on 80 emails with a 5 MB attachment each:

  • Before: 1028 MB peak memory usage
  • After: 269 MB peak memory usage

Python 3 - Comparing a version string to an int

I added Twilio backward compatibility to django-sendsms so its Twilio backend would work with both the old (v5) and new (v6+) Twilio clients. The version detection looked like this:

import twilio
if twilio.__version__ > 5:
    from twilio.rest import Client as TwilioRestClient
else:
    from twilio.rest import TwilioRestClient

That works on Python 2, but it’s broken on Python 3. twilio.__version__ is a string like "6.5.0", and comparing a string to an int raises a TypeError on Python 3:

>>> "6.5.0" > 5
TypeError: '>' not supported between instances of 'str' and 'int'

Python 2 lets you order any two objects (ints always sort before strings), so the same comparison silently returns True there. The bug stayed hidden until the code ran under Python 3.

The fix pulls the major version out of __version_info__ and compares ints to ints:

if int(twilio.__version_info__[0]) > 5:

That same PR added Python 3.5 and 3.6 to the Travis build, so this kind of thing gets caught next time.