Python - Dropping Into pdb From Nose

Great Hadoop + Spark Tutorial

Hadoop Namenode Not Starting

Python - "The Darker Side of type"

Most people know type() as the one-argument form that tells you an object's type: type(1) returns <class 'int'>. Called with three arguments, it does something else entirely: it creates a new class at runtime.

type(name, bases, namespace) is equivalent to a class statement. These two produce the same class:

class Foo(object):
    pass

Foo = type('Foo', (object,), {})

The arguments are the class name, a tuple of base classes, and a dict of attributes and methods. A function put in that dict becomes a method on the class:

def always_false(self):
    return False

Foo = type('Foo', (object,), {'always_false': always_false})

You reach for this when you don't know the classes you'll need until runtime, so you can't write them out ahead of time. The usual example is reflecting an unknown database schema and building a model class for each table on the fly.

I first ran into this in Jeff Knupp's post "Improve Your Python: Metaclasses and Dynamic Classes With Type." His blog is no longer up (archived copy). The three-argument type() is documented in the Python docs, and there's a deeper walkthrough in Python 3 Patterns, Recipes and Idioms.

SQLAlchemy's "scale" Argument & WTForms

• With SQLAlchemy's Generic Float type, the "scale" argument is ignored. "scale" is not listed as an argument and the docs say "Additional arguments here are ignored by the default Float type.". However, the object does have a "scale", but it's always "None".
• WTForms' "places" default for DecimalField's is 2.
• The MySQL float does have a "scale" argument, it defaults to "None". MySQL and Postgres have default limits on the length of the precision of floats, MySQL only shows 6 digits after the decimal and Postgres has a default column length of 17.
• Numeric columns do have a "scale" argument, and the default is "None" in SQLAlchemy 0.7 and 1.0.
• "decimal_return_scale" was added to both Float and Numeric in SQLAlchemy 0.9.
• If you raise the number of "places" in WTForms' DecimalField greater than "decimal_return_scale" in the SQLAlchemy field, the digits after the decimal place set in "decimal_return_scale" will show 0's.
• In SQLAlchemy, "_default_decimal_return_scale = 10" is only used in a property/method called "_effective_decimal_return_scale". The default for "decimal_return_scale" is "None" and not 10. Since "_default_decimal_return_scale" and "_effective_decimal_return_scale" start with an underscore, so it's only intended for internal use.
• MySQL's "FLOAT" in SQLAlchemy doesn't have "decimal_return_scale". It probably should? REAL and DOUBLE have this.
• WTForms' FloatField does not have "places".
• With SQLAlchemy 1.0, "db.DECIMAL()" will create a numeric(28, 6) in Postgres, but it creates a DECIMAL(10,0) in MySQL. This seemed pretty odd, since the default "scale" is "None". Maybe it's just using whatever the default is in each backend.

• SQLAlchemy 0.7 docs for Float type: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html?highlight=float#sqlalchemy.types.Float
• Latest SQLAlchemy docs for Float type: http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.Float
• Latest docs for MySQL Float type: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=float#sqlalchemy.dialects.mysql.FLOAT

Python Gotcha - Copying Nested Dicts

Image Duplicate Detection In Python

GoDaddy To Amazon Route 53 - Lessons Learned

Awesome-ETL List

Python Idioms