Django - RelatedManager.set not removing models

14 Feb 2020

In my opinion, one of django's biggest gotchas is using `RelatedManager.set` with models that have non-nullable ForeignKey fields. ### Example Models.py (with a one-to-many relationship and a non-nullable ForeignKey): ``` class Reporter(models.Model): name = models.CharField(max_length=255) class Article(models.Model): title = models.CharField(max_length=255) reporter = models.ForeignKey(Reporter, on_delete=models.CASCADE) ``` Guess how many articles are on the reporter after you do this?: ``` reporter = Reporter.objects.create(name="paul") # add 2 articles to the reporter article_1 = reporter.article_set.create(title="blah") reporter.article_set.create(title="blah2") # set only 1 article on the reporter reporter.article_set.set([article_1]) ``` I expected just one article to be on the reporter. Nope: ``` reporter.article_set.count() 2 ``` How do the docs say it should work? >This method accepts a clear argument to control how to perform the operation. If False (the default), the elements missing from the new set are removed using remove() and only the new ones are added. Well, crap, what's going on? It turns out you need to read the docs for `RelatedManager.remove` for a hint: >For ForeignKey objects, this method only exists if null=True. So, it turns out that `set()` is using `remove()` and `remove()` is only available on models with ForeignKeys that are `null=True`. What does `set()` do with things it needs to `remove()` when `remove()` isn't available? Nothing...

Django - ChoiceField Display Text

14 Jan 2020 I made a repo to demonstrate how to get a django ChoiceField's current selection's display text: https://github.com/pawl/django_choicefield_display_example/

I came across a few other examples, but they didn't do a good job of displaying the initial value or handling choices with integer values.

Python - Pipenv to pip-tools

20 Dec 2019 I've been using Pipenv for the last few months and my biggest issue is that "--keep-outdated" has been broken in the latest release (2018.11.26) for a while. I've needed to install Pipenv from the master branch to make it functional. However, the last time I used "--keep-outdated" from the master branch, it wouldn't automatically update the hash of the dependency being updated.

Updating specific requirements is something I need to do pretty often, and it's not fun to explain all the Pipenv quirks to the team.

Pip-tools looks like it does everything I need and has fewer quirks, so I ended up making the switch.

Pipenv uses pip-tools under the hood, so the migration to pip-tools was very smooth. The migration process was:

Copy the dev-packages and packages sections of the Pipfile to their own requirements.in files.
Run pip-compile
Copy over the specific versions and hashes from the Pipfile.lock to the generated requirements.txt.

I did have a small issue where updating a specific package with pip-tools removed a bunch of dependencies from the requirements.txt unexpectedly, but running pip-compile with "--rebuild" fixed it.

HTML5 - video playsinline

25 Jul 2019 I added "autoplay muted" to my <video> tag to make a video autoplay in a carousel. It worked on desktop Chrome or Firefox, but didn't work on iOS Chrome or Safari.

I learned about the "playsinline" fix from here: https://webkit.org/blog/6784/new-video-policies-for-ios/

Adding "playsinline" to my video tag fixed it.

ESP32 Plant Sensor

12 Jul 2019

I set up a ESP32 houseplant soil water + temperature + humidity + light sensor that sends me a daily status update message.

Here’s the code for it: https://gist.github.com/pawl/6a408afde1411c6b1f58980e3d1dff83

esp32 plant sensor 1

esp32 plant sensor 2

esp32 plant sensor 3

AWS - redirecting domain to url using a 302 redirect (without running a server)

11 Jul 2019 I wanted to make a domain name (heckingoodboys.com) redirect to a multisubreddit for dog pictures, but I didn't want to run a web server for it.

Here's what I did:

Purchase the domain using Route53.
Create two public s3 buckets (www.heckingoodboys.com and heckingoodboys.com)
Enable "Static website hosting" on www.heckingoodboys.com and redirect to heckingoodboys.com.
Enable "Static website hosting" on heckingoodboys.com, select "use this bucket to host this website", and use routing rules similar to this:
<RoutingRules>
<RoutingRule>
    <Redirect>
      <Protocol>https</Protocol>
      <HostName>www.reddit.com</HostName>
      <HttpRedirectCode>302</HttpRedirectCode>
      <ReplaceKeyPrefixWith>user/heckingoodboys/m/heckingoodboys/</ReplaceKeyPrefixWith>
    </Redirect>
</RoutingRule>
</RoutingRules>
Back to Route53 - Create an A record for both www.heckingoodboys.com and heckingoodboys.com using the alias to their respective buckets. (this will be the first option in autocomplete)

For more details: https://medium.com/@P_Lessing/single-page-apps-on-aws-part-1-hosting-a-website-on-s3-3c9871f126

Why not just use a CNAME from www.heckingoodboys.com to heckingoodboys.com? AWS says they don't charge for aliases, but they do charge for CNAMEs. So, I used an alias to a bucket instead.

Django - When To Use Signals?

08 Feb 2018 According to this blog post, almost never: https://lincolnloop.com/blog/django-anti-patterns-signals/

Django - Custom Storage Backends

08 Feb 2018 I just learned about the custom storage systems in Django.

What can you do with custom storage systems?:

"Django abstracts file storage using storage backends, from simple filesystem storage to things like S3. This can be used for processing file uploads, storing static assets, and more." -https://tartarus.org/james/diary/2013/07/18/fun-with-django-storage-backends

Sphinx Search - Lessons Learned

20 Jan 2018 Here are a few things I've learned while working on a project that uses Sphinx search:

It's important to know the difference between fields and attributes. Attributes are basically unindexed columns and you should try to avoid filtering only on these columns. Fields support full text search.
It supports its own custom binary protocol and the MySQL protocol (recently they also added a HTTP API). When you see "listen = localhost:9306:mysql41" in the config, that means it's listening for MySQL protocol traffic on port 9306.
https://github.com/a1tus/sphinxapi-py3 appears to be the best Python client for the binary api at the moment. This doesn't support INSERTing things into the index (you'll need to use the MySQL protocol for that).
The version of sphinxapi-py3 on pypi is a fork with just a few minor fixes and appears to be safe.
It does not match partial words by default. Turning on partial matching can also increase the size of your index dramatically. You can also limit the fields that support partial matching with the "infix_fields" and "prefix_fields" setting.
Stemmers aren't turned on by default. So, searching for "dog" will not match "dogs".
Most special characters ($, @, &, etc) are ignored by default. You will need to add them to charset_table if you want them to be searchable.
Ruby's thinking-sphinx looks much more battle tested than all of the Python binary api clients: https://github.com/pat/thinking-sphinx
You will need to use a real-time index if you want to INSERT/DELETE records immediately.
If you're using a real-time index, you will probably need to increase the "rt_mem_limit" from its default of 128mb. If this limit is too low, you'll see a high number of "disk chunks" when you run the "SHOW INDEX rtindex STATUS" query. More info: http://sphinxsearch.com/blog/2014/02/12/rt_performance_basics/
You have to use a special dialect if you want to use SQLAlchemy with sphinx: https://github.com/conversant/sqlalchemy-sphinx
This appears to be the best Dockerfile for sphinx: https://github.com/leodido/dockerfiles

I probably won't be using Sphinx search for any new projects. Elasticsearch seems preferable these days.

MySQL - Duplicate Errors & Trailing Whitespace

12 Dec 2017 I had a unique constraint on a VARCHAR column and I inserted two rows with the following values:

"name" (without trailing whitespace)
"name " (with trailing whitespace)

To my surprise, I got a duplicate error on that 2nd insert. It turns out that MySQL ignores that trailing whitespace when it makes comparisons.

The MySQL docs say this: "All MySQL collations are of type PAD SPACE. This means that all CHAR, VARCHAR, and TEXT values are compared without regard to any trailing spaces. “Comparison” in this context does not include the LIKE pattern-matching operator, for which trailing spaces are significant." (https://dev.mysql.com/doc/refman/5.7/en/char.html)

The solution? You should probably be trimming trailing whitespace in your API endpoints and on your front-end.

Older Newer