Ansible Vault - storing secrets in repos

04 Mar 2016 When others are deploying a project for you, it's easy for mistakes to be made when secrets must be updated in environmental variables. Ansible-vault takes a different approach and encrypts the secrets - allowing you to store the secrets in your repo.

To encrypt a file: ansible-vault encrypt secrets.py
To decrypt a file: ansible-vault decrypt secrets.py

More documentation is available here: http://docs.ansible.com/ansible/playbooks_vault.html

Python - isort, useful tool for sorting python imports

04 Mar 2016 http://timothycrosley.github.io/isort/

I saw it in the django coding style documentation: https://docs.djangoproject.com/en/1.9/internals/contributing/writing-code/coding-style/

To make an entire project's imports easier to read, all you need to do is:

pip install isort
isort -rc .

There's also a sublime plugin for it here: https://github.com/thijsdezoete/sublime-text-isort-plugin#install

git commit --amend -C HEAD

19 Feb 2016 Ever committed something and needed to make changes later? Just rebase and squash it, right? Or maybe "git reset --soft HEAD~1" and commit again?

There's an even better solution: "git commit --amend -C HEAD"

It will add the combine your commit with your last commit. If you already pushed, you will need to force push your change.

pip install -e .

19 Feb 2016 I've wasted a lot of time running "python setup.py install" before testing my changes to flask-admin. It turns out you can pip install a project as "editable", which points the install toward your local directory instead of copying where the rest of your python modules are.

To install a project as editable, navigate to the repo and run "pip install -e .".

https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs

Python - Dropping Into pdb From Nose

14 Feb 2016 This document describes how to drop into pdb when your Python nose tests fail: http://nose.readthedocs.org/en/latest/plugins/debug.html

It just requires running "nosetests --pdb".

Very helpful if you're debugging why a test failed.

Great Hadoop + Spark Tutorial

14 Feb 2016 This great tutorial takes you through installing hadoop and spark to analyze Reddit comment data: http://blog.insightdatalabs.com/spark-cluster-step-by-step/

Hadoop Namenode Not Starting

14 Feb 2016 My namenode was not starting because I had the wrong host configured in yarn-site.xml, mapred-site.xml, and core-site.xml.

When you're running start-dfs.sh on your namenode, ensure the line that says "starting namenode" shows "/usr/local/hadoop/logs/hadoop-ubuntu-namenode-<your namenode's hostname>.out" in the output. This is how you know your configuration is correct.

You can check your server's hostname on Ubuntu by running "echo $(hostname)".

Python - "The Darker Side of type"

26 Dec 2015 https://www.jeffknupp.com/blog/2013/12/28/improve-your-python-metaclasses-and-dynamic-classes-with-type/

That blog post has a great description of how the type keyword can be used for another purpose, creating new types.

Here's an example of the syntax: Foo = type(str('Foo'), (object, ), {})

This page has more details: http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html

SQLAlchemy's "scale" Argument & WTForms

23 Dec 2015

Note: Unless otherwise stated, this applies to SQLAlchemy 1.0.

Here are a few things I learned while researching the fix for this Flask-Admin Issue #1141:

With SQLAlchemy's Generic Float type, the "scale" argument is ignored. "scale" is not listed as an argument and the docs say "Additional arguments here are ignored by the default Float type.". However, the object does have a "scale", but it's always "None".
WTForms' "places" default for DecimalField's is 2.
The MySQL float does have a "scale" argument, it defaults to "None". MySQL and Postgres have default limits on the length of the precision of floats, MySQL only shows 6 digits after the decimal and Postgres has a default column length of 17.
Numeric columns do have a "scale" argument, and the default is "None" in SQLAlchemy 0.7 and 1.0.
"decimal_return_scale" was added to both Float and Numeric in SQLAlchemy 0.9.
If you raise the number of "places" in WTForms' DecimalField greater than "decimal_return_scale" in the SQLAlchemy field, the digits after the decimal place set in "decimal_return_scale" will show 0's.
In SQLAlchemy, "_default_decimal_return_scale = 10" is only used in a property/method called "_effective_decimal_return_scale". The default for "decimal_return_scale" is "None" and not 10. Since "_default_decimal_return_scale" and "_effective_decimal_return_scale" start with an underscore, so it's only intended for internal use.
MySQL's "FLOAT" in SQLAlchemy doesn't have "decimal_return_scale". It probably should? REAL and DOUBLE have this.
WTForms' FloatField does not have "places".
With SQLAlchemy 1.0, "db.DECIMAL()" will create a numeric(28, 6) in Postgres, but it creates a DECIMAL(10,0) in MySQL. This seemed pretty odd, since the default "scale" is "None". Maybe it's just using whatever the default is in each backend.

Sources:

SQLAlchemy 0.7 docs for Float type: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html?highlight=float#sqlalchemy.types.Float
Latest SQLAlchemy docs for Float type: http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.Float
Latest docs for MySQL Float type: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=float#sqlalchemy.dialects.mysql.FLOAT

Python Gotcha - Copying Nested Dicts

27 Oct 2015 Here's the best blog post I've found that talks about the need for deep or shallow copy when using nested dicts in Python:
http://www.peterbe.com/plog/must__deepcopy__

Older Newer