Hadoop Namenode Not Starting
My namenode was not starting because I had the wrong host configured in yarn-site.xml, mapred-site.xml, and core-site.xml.When you're running start-dfs.sh on your namenode, ensure the line that says "starting namenode" shows "/usr/local/hadoop/logs/hadoop-ubuntu-namenode-<your namenode's hostname>.out" in the output. This is how you know your configuration is correct.
You can check your server's hostname on Ubuntu by running "echo $(hostname)".
Python - "The Darker Side of type"
https://www.jeffknupp.com/blog/2013/12/28/improve-your-python-metaclasses-and-dynamic-classes-with-type/That blog post has a great description of how the type keyword can be used for another purpose, creating new types.
Here's an example of the syntax: Foo = type(str('Foo'), (object, ), {})
This page has more details: http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html
SQLAlchemy's "scale" Argument & WTForms
Note: Unless otherwise stated, this applies to SQLAlchemy 1.0.
Here are a few things I learned while researching the fix for this Flask-Admin Issue #1141:
- With SQLAlchemy's Generic Float type, the "scale" argument is ignored. "scale" is not listed as an argument and the docs say "Additional arguments here are ignored by the default Float type.". However, the object does have a "scale", but it's always "None".
- WTForms' "places" default for DecimalField's is 2.
- The MySQL float does have a "scale" argument, it defaults to "None". MySQL and Postgres have default limits on the length of the precision of floats, MySQL only shows 6 digits after the decimal and Postgres has a default column length of 17.
- Numeric columns do have a "scale" argument, and the default is "None" in SQLAlchemy 0.7 and 1.0.
- "decimal_return_scale" was added to both Float and Numeric in SQLAlchemy 0.9.
- If you raise the number of "places" in WTForms' DecimalField greater than "decimal_return_scale" in the SQLAlchemy field, the digits after the decimal place set in "decimal_return_scale" will show 0's.
- In SQLAlchemy, "_default_decimal_return_scale = 10" is only used in a property/method called "_effective_decimal_return_scale". The default for "decimal_return_scale" is "None" and not 10. Since "_default_decimal_return_scale" and "_effective_decimal_return_scale" start with an underscore, so it's only intended for internal use.
- MySQL's "FLOAT" in SQLAlchemy doesn't have "decimal_return_scale". It probably should? REAL and DOUBLE have this.
- WTForms' FloatField does not have "places".
- With SQLAlchemy 1.0, "db.DECIMAL()" will create a numeric(28, 6) in Postgres, but it creates a DECIMAL(10,0) in MySQL. This seemed pretty odd, since the default "scale" is "None". Maybe it's just using whatever the default is in each backend.
Sources:
- SQLAlchemy 0.7 docs for Float type: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html?highlight=float#sqlalchemy.types.Float
- Latest SQLAlchemy docs for Float type: http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.Float
- Latest docs for MySQL Float type: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=float#sqlalchemy.dialects.mysql.FLOAT
Python Gotcha - Copying Nested Dicts
Here's the best blog post I've found that talks about the need for deep or shallow copy when using nested dicts in Python:http://www.peterbe.com/plog/must__deepcopy__
Image Duplicate Detection In Python
Great article on detecting duplicate images in Python: http://7webpages.com/blog/image-duplicates-detection-python/GoDaddy To Amazon Route 53 - Lessons Learned
The first thing you'll want to do is created a "Hosted Zone" with the same A and CNAME records (and any other relevant records that aren't specific to GoDaddy).When you transfer the domain and it asks you if you want to use the old name servers - answer no. I messed this one up and kept using GoDaddy's "domaincontrol.com" name servers and DNS stopped resolving eventually. If you mess this up too, you'll need to go to the "Hosted Zone" for your site and copy the records in the "NS" list to domain's name servers on the registered domains page.
Awesome-ETL List
Finally decided to put all the interesting ETL software I find into a list: https://github.com/pawl/awesome-etlPython Idioms
Here's a great page describing Python idioms: https://david.goodger.org/projects/pycon/2007/idiomatic/handout.htmlThis gist of a Raymond Hettinger talk is also great: https://gist.github.com/JeffPaine/6213790
Best "Command Line" Beginners Guide
http://www.learnenough.com/command-lineIt's a short guide, but I think it covers everything a beginner needs to know.