Paul's Programming Notes PostsRSSGithub

django-celery-email - Reducing memory usage on large email batches

I got a pull request merged into django-celery-email that cuts peak memory usage by about 74% when sending large batches of emails with attachments.

CeleryEmailBackend.send_messages used to serialize every message up front, then split the serialized list into chunks, then hand each chunk to a Celery task:

def send_messages(self, email_messages):
    result_tasks = []
    messages = [email_to_dict(msg) for msg in email_messages]
    for chunk in chunked(messages, settings.CELERY_EMAIL_CHUNK_SIZE):
        result_tasks.append(send_emails.delay(chunk, self.init_kwargs))

email_to_dict copies the whole message into a dict, attachments included. Doing that for the entire batch before chunking means every serialized email sits in memory at once. With attachments, that gets expensive fast.

The fix is to chunk the messages first, then serialize one chunk at a time:

def send_messages(self, email_messages):
    result_tasks = []
    for chunk in chunked(email_messages, settings.CELERY_EMAIL_CHUNK_SIZE):
        chunk_messages = [email_to_dict(msg) for msg in chunk]
        result_tasks.append(send_emails.delay(chunk_messages, self.init_kwargs))

Same tasks get queued, but only one chunk’s worth of serialized emails exists at a time, so the earlier chunks can be garbage collected once their task is queued.

Benchmarks on 80 emails with a 5 MB attachment each:

  • Before: 1028 MB peak memory usage
  • After: 269 MB peak memory usage
pythoncelerydjango