Saturday, December 26, 2015

Python - "The Darker Side of type"

https://www.jeffknupp.com/blog/2013/12/28/improve-your-python-metaclasses-and-dynamic-classes-with-type/

That blog post has a great description of how the type keyword can be used for another purpose, creating new types.

Here's an example of the syntax: Foo = type(str('Foo'), (object, ), {})

This page has more details: http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html

Tuesday, December 22, 2015

SQLAlchemy's "scale" Argument & WTForms


Note: Unless otherwise stated, this applies to SQLAlchemy 1.0.

Here are a few things I learned while researching the fix for this Flask-Admin Issue #1141:
  • With SQLAlchemy's Generic Float type, the "scale" argument is ignored. "scale" is not listed as an argument and the docs say "Additional arguments here are ignored by the default Float type.". However, the object does have a "scale", but it's always "None".
  • WTForms' "places" default for DecimalField's is 2.
  • The MySQL float does have a "scale" argument, it defaults to "None". MySQL and Postgres have default limits on the length of the precision of floats, MySQL only shows 6 digits after the decimal and Postgres has a default column length of 17.
  • Numeric columns do have a "scale" argument, and the default is "None" in SQLAlchemy 0.7 and 1.0.
  • "decimal_return_scale" was added to both Float and Numeric in SQLAlchemy 0.9.
  • If you raise the number of "places" in WTForms' DecimalField greater than "decimal_return_scale" in the SQLAlchemy field, the digits after the decimal place set in "decimal_return_scale" will show 0's.
  • In SQLAlchemy, "_default_decimal_return_scale = 10" is only used in a property/method called "_effective_decimal_return_scale". The default for "decimal_return_scale" is "None" and not 10. Since "_default_decimal_return_scale" and "_effective_decimal_return_scale" start with an underscore, so it's only intended for internal use.
  • MySQL's "FLOAT" in SQLAlchemy doesn't have "decimal_return_scale". It probably should? REAL and DOUBLE have this.
  • WTForms' FloatField does not have "places".
  • With SQLAlchemy 1.0, "db.DECIMAL()" will create a numeric(28, 6) in Postgres, but it creates a DECIMAL(10,0) in MySQL. This seemed pretty odd, since the default "scale" is "None". Maybe it's just using whatever the default is in each backend.


Sources:

Monday, October 26, 2015

Python Gotcha - Copying Nested Dicts

Here's the best blog post I've found that talks about the need for deep or shallow copy when using nested dicts in Python:
http://www.peterbe.com/plog/must__deepcopy__

Saturday, October 10, 2015

GoDaddy To Amazon Route 53 - Lessons Learned

The first thing you'll want to do is created a "Hosted Zone" with the same A and CNAME records (and any other relevant records that aren't specific to GoDaddy).

When you transfer the domain and it asks you if you want to use the old name servers - answer no. I messed this one up and kept using GoDaddy's "domaincontrol.com" name servers and DNS stopped resolving eventually. If you mess this up too, you'll need to go to the "Hosted Zone" for your site and copy the records in the "NS" list to domain's name servers on the registered domains page.

Wednesday, October 7, 2015

Saturday, September 19, 2015

Monday, September 14, 2015

Current State Of Workflow Engines / Frameworks

This is a great article about the current state of workflow engines (and frameworks?): http://stackstorm.com/2015/04/10/the-return-of-workflows/

Taskflow looks especially interesting. The article is unfortunately missing Airbnb's Airflow: https://github.com/airbnb/airflow

Sunday, July 26, 2015

New Macbook Pro Setup

I'm finally switching to a Macbook Pro as my personal computer after a few months of using one at work. Local development on OSX is a lot easier and bug-free than on a Windows computer.

I started a gist with what I did for setup:

Saturday, July 11, 2015

In Order to Configure TCP/IP, You Must Install and Enable a Network Adapter Card

If you get this error message, you need to disable and uninstall your network adapter. Once you've done that, you can reinstall the driver to fix it.

Cygwin - Python Compiled Without SSL Support

Download error on https://pypi.python.org/simple/pip/: unknown url type:
https -- Some packages may not be found!

ImportError: cannot import name HTTPSHandler

I was trying to install pip in cygwin when this happened. Searches say openssl-devel needed to be installed, but it already was. Apparently python was compiled without ssl support.

My solution was to install cygwin-x86 instead of the 64x one.

Friday, July 10, 2015

md5sum: standard input: no properly formatted MD5 checksum lines found

If you see this error when you're using apt-cyg, it means you need to update apt-cyg.

You need to run the following commands:

lynx -source rawgit.com/transcode-open/apt-cyg/master/apt-cyg > apt-cyg
install apt-cyg /bin

Saturday, June 27, 2015

Automatically Remove Long-Running Docker Containers

The following can be used in cron or run with & at the end of the command:


I found this didn't work: http://www.ahtik.com/blog/removing-detached-docker-containers-automatically/

Tuesday, June 2, 2015

Docker Run In Crontab

"0 5 * * 1 docker run --rm --name=mycontainer ubuntu:13.10 /opt/bin/job"
The above command is an example of how "docker run" would be used in crontab to run once every week at 5am.

  • --rm will delete the container once the job is finished running
  • --name will name the container and prevent duplicate jobs from running
  • You don't need to use "&" at the end, because crons already run in the background.

Friday, May 8, 2015

MySQL Batch Updates Not Working

Looking at "SHOW PROCESSLIST" and it looks like your queries are running individually instead of in batches like you sent?

This happens because MySQL runs each update statement individually, but you should still be able to see the batches when the queries are in the "init" state.

Use this query to see the batched queries:
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST where state="init";

Sunday, March 22, 2015

Thursday, February 12, 2015

"Error response from daemon: 404 page not found" - Docker

This required setting the following in my /etc/environment:
no_proxy=/var/run/docker.sock

Wednesday, January 28, 2015

When Not to Use Generator Expressions

http://stackoverflow.com/a/26635939

I found that post very helpful. I didn't know it's a better idea to use a list comprehension rather than a generator expression for "".join().

mod_wsgi + Flask

Here's my mod_wsgi configuration for a flask app that uses Flask-Admin.

Trailing slashes matter! If you set a trailing slash, it will only set the alias for that directory and not all of the directories below it.

Monday, January 26, 2015

Request.Args Empty During Tests - Flask

https://github.com/mitsuhiko/flask/issues/801#issuecomment-21513278

If you're using app.test_client().get, it's very important to use a relative url or your parameters will be removed.

# will not pass request.args
rv = self.client.get('http://0.0.0.0:5000/view/?foo=bar')

# works just fine
rv = self.client.get('/view/?foo=bar')


Thursday, January 22, 2015

Self-Hosting Fonts

https://github.com/CSSLint/csslint/wiki/Bulletproof-font-face

That's the most helpful guide I found for self-hosting your own fonts and preventing compatibility issues.

Wednesday, January 21, 2015

Dokku Change Nginx Timeout

None of the plug-ins for doing custom nginx configurations were working for me.

So, I changed the following two files:
/var/lib/dokku/plugins/nginx-vhosts/templates/nginx.ssl.conf
/var/lib/dokku/plugins/nginx-vhosts/templates/nginx.conf

I added the following under "location    / {":
proxy_connect_timeout 300s;
proxy_read_timeout 300s;

Tuesday, January 20, 2015

Supervisord Tips and Gotchas

If you make configuration changes, the changes will not take effect until you:
  • supervisorctl reread
  • supervisorctl update

"redirect_stderr=true" will cause both stderr and stdout to appear in the "stdout" log.

You get to select which user runs the command with the "user" parameter.
For example: user=root
Supervisord keeps its own set of environmental variables with the "environment" parameter. It won't pick up env variables from /etc/environment.
For example: environment=A="1",B="2"
For Gunicorn, use a configuration file rather than putting all your arguments into the "command" parameter.
For example, -t will be ignored here:
command=/path/to/gunicorn main:application -t 300
Instead use:
command=/path/to/gunicorn main:application -c /path/to/gunicorn.conf.py

Monday, January 19, 2015

Installing python-ldap on Dokku

Install this plugin: https://github.com/F4-Group/dokku-apt#installation

Create an apt-packages file in the root directory of your repository with the following lines:
python-dev
libldap2-dev
libsasl2-dev
libssl-dev

Now, commit your new apt-packages file and python-ldap should install successfully.

I also had trouble installing pymssql, and got an error saying "_mssql.c:314:22: fatal error: sqlfront.h: No such file or directory". Similar fix, just add "freetds-dev" to your apt-packages file.

Enabling SSL On Dokku

  1. It's important that you have the newest version of Dokku: http://progrium.viewdocs.io/dokku/upgrading
  2. Zip your certificate and key: sudo tar cvf archive_name.tar.gz server.crt server.key
  3. Import using dokku nginx:import-ssl appname < ~/archive_name.tar.gzhttps://github.com/progrium/dokku/blob/master/docs/nginx.md#importing-ssl-certificates

Expecting: TRUSTED CERTIFICATE

I was trying to add my certificate to Dokku using "dokku nginx:import-ssl" when I got the error message "Expecting: TRUSTED CERTIFICATE".

This ended up being an issue with line endings and the dos2unix utility fixed it: http://serverfault.com/a/317038

Sunday, January 18, 2015

Thursday, January 15, 2015

Nmap Not Finding Hostnames

http://serverfault.com/a/153779

The secret ended up being running the command as sudo. It's odd how "socket.gethostbyaddr()" worked just fine without sudo, but was slower than nmap.

Wednesday, January 7, 2015

Slow Disk Performance On Dell r620

Update:

I installed openmanage with the instructions on this page and used it to check/update bios settings: http://linux.dell.com/repo/community/deb/OMSA_7.1/

The fix ended up being BIOS power profile settings. It was set to Performance Per Watt (DAPC) and needed to be set to "Performance".
It's very interesting that the BIOS power profile settings didn't seem to affect Ubuntu 14.10's disk read speed.

Troubleshooting:

I have three of the same server. All the servers are Dell r620's with a PERC H710 Mini RAID controller (21.2.0-0007_A04 firmware) and have the exact same hard drives (Seagate ST300MM0006) in RAID 1 configurations. I'm seeing about 50% worse disk read performance on some versions of Ubuntu.

They only differences I can find in the hardware:
  • The BIOS version on the poorly performing machines is 1.4.8, the machine that's performing well is on 1.6.0. The BIOS change log doesn't appear to have any changes between the two versions that would affect anything.
  • The chipset version on the RAID controller of the poorly performing machines is rev 01 (ChipRevision: B0). The machine that's performing well has the rev 05 chipset (ChipRevision: D1). The motherboard chipset revision is also different.
Here are the benchmark results for the different Ubuntu versions and RAID controller chipsets:

(mysql read speed test - using the same version of mysql across all 3 machines)
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run:
(13.04 - rev 05): read/write requests:                 4996362 (83271.11 per sec.)
(14.04 - rev 01): read/write requests:                 1906520 (31773.58 per sec.)
(14.10 - rev 01): read/write requests:                 5166798 (86111.50 per sec.)

dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output:
(13.04 - rev 05): 83886080 bytes (84 MB) copied, 0.0437543 s, 1.9 GB/s
(13.04 - rev 01) 83886080 bytes (84 MB) copied, 0.116811 s, 718 MB/s
(13.10 - rev 01): 83886080 bytes (84 MB) copied, 0.129371 s, 648 MB/s
(14.04 - rev 01): 83886080 bytes (84 MB) copied, 0.157808 s, 532 MB/s
(14.10 - rev 01): 83886080 bytes (84 MB) copied, 0.102355 s, 820 MB/s

hdparm -tT /dev/sda1:
(13.04 - rev 05): Timing cached reads:23154 MB in  2.00 seconds = 11589.16 MB/sec
(13.04 - rev 01): Timing cached reads:17934 MB in  2.00 seconds = 8973.68 MB/sec
(13.10 - rev 01): Timing cached reads:18102 MB in  2.00 seconds = 9058.08 MB/sec
(14.04 - rev 01): Timing cached reads:17846 MB in  1.99 seconds = 8956.28 MB/sec
(14.10 - rev 01): Timing cached reads:21538 MB in  2.00 seconds = 10777.93 MB/sec

At one point, 2 of the rev 01 servers were using Ubuntu 14.04 and were getting similar bad benchmark results. After the upgrading them to 14.10, I immediately saw the disk read speed increase.

The performance on rev 01 machine with 13.04 was bad, but somehow the rev 05 machine with the same OS version was performing well. This told me it's not a problem with the megaraid_sas driver, because the rev 01 and rev 05 machines with 13.04 had the same version.

Here's the closest I can find to a similar issue (it's different hardware though): http://en.community.dell.com/support-forums/servers/f/906/t/19596533

Update 3/1/2015:
Saw an interesting checklist on Hacker News today that also mentioned disabling BIOS power saving settings: http://odetodata.com/2015/02/installation-and-configuration-checklist-for-microsoft-sql-server/

Tuesday, January 6, 2015

MegaCli64 / MegaCli Only showing "Exit Code: 0x00"

Problem: You see MegaCli64 or MegaCli only showing "Exit Code: 0x00" when you try to run commands.

Solution: Try running as sudo.

Thursday, January 1, 2015

Changing Putty Default Settings

If you need to change Putty's defaults, press "Load" on the saved session for "Default Settings", make your changes, and "Save".