Paul's Programming Notes     Archive     Feed     Github

New Machine Setup For Flask Development On Ubuntu

Here are the commands I use when I bring up a new Ubuntu server for flask development:

sudo apt-get update && time sudo apt-get dist-upgrade
sudo timedatectl set-timezone America/Chicago
apt-get install git fail2ban htop nano
mkdir -p .ssh
nano .ssh/authorized_keys
eval `ssh-agent -s`
ssh-add ~/.ssh/id_rsa
apt-get install python-setuptools
easy_install pip
pip install virtualenvwrapper
sudo mkdir ~/virtualenvs
sudo sh -c "echo 'WORKON_HOME=~/virtualenvs' >> /etc/environment"
for line in $( cat /etc/environment ) ; do export $line ; done
mkdir -p $WORKON_HOME
sudo sh -c "echo 'source /usr/local/bin/virtualenvwrapper.sh' >> ~/.bashrc"
bash
mkvirtualenv default
apt-get install python-dev
view raw gistfile1.txt hosted with ❤ by GitHub

MySQL Performance LXC Container vs Host

I have a linux virtual container (LXC) and the server hosting the container running the same version of MySQL with the same database.

I tried running some select queries on both of the both the virtual container and host for benchmark purposes. My conclusion? MySQL read performance inside and outside the LXC container is the exactly the same.

WHMCS Cronjob Error

If you get the following error while you try to run the WHMCS cronjob from the command line:
Site error: the file <b>/var/www/accounts/admin/cron.php</b> requires the ionCube PHP Loader ioncube_loader_lin_5.3.so to be installed by the website operator. If you are the website operator please use the <a href="http://www.ioncube.com/lw/">ionCube Loader Wizard</a> to assist with installation.

You need to add your the ioncube zend extension to the CLI php.ini too: /etc/php5/cli/php.ini


Sendmail/STARTTLS verify=FAIL



Oct 19 14:04:34 Billing sm-mta[17583]: STARTTLS=client, relay=aspmx.l.google.com., version=TLSv1/SSLv3, verify=FAIL, cipher=ECDHE-RSA-RC4-SHA, bits=128/128

If you see those errors in your mail.log and your emails are failing to send, you need to add your ssl cert from apache to the sendmail config in /etc/mail/sendmail.cf:
# CA directory
O CACertPath=/etc/apache2/ssl
# CA file
O CACertFile=/etc/apache2/ssl/example.ca-bundle
# Server Cert
O ServerCertFile=/etc/apache2/ssl/example.cert
# Server private key
O ServerKeyFile=/etc/apache2/ssl/example.key

Adding Virtualhost In Apache For Port

Say we wanted to route the <your-domain>.com:8072/misc to /var/www/misc...

Add the following to /etc/apache2/ports.conf:
NameVirtualHost *:8072
Listen 8072

Add a file that describes your site to /etc/apache2/sites-available with this:
<VirtualHost *:8072>
   ServerName <your-domain>.com
   DocumentRoot /var/www/misc/
</VirtualHost>

Run this command: sudo a2ensite misc

And restart apache: sudo service apache2 restart

Deploying Nginx + Gunicorn + Flask

Why use both Nginx and Gunicorn?: http://serverfault.com/a/220047

Simple explanation: http://prakhar.me/articles/flask-on-nginx-and-gunicorn/

Nginx SSL configuration: https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-with-ssl-as-a-reverse-proxy-for-jenkins

The thing I spent the longest on was getting supervisor to work. Gunicorn kept giving me "gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>", because I was missing the "directory" parameter. I needed to tell supervisor which directory to start from.

[program:my_app]
command=/<path>/venv/prod/bin/gunicorn -w 8 main:app -b 127.0.0.1:8080
user=me
autostart=true
directory=/home/me/my_dir/
stdout_logfile=/home/me/my_dir/gunicorn_supervisor.log
stdout_logfile_maxbytes=20MB
redirect_stderr=true

You will need to create an upstart script to get supervisor to run when the system starts: http://serverfault.com/a/96500

Python Excel Library Benchmark Comparison

https://github.com/swistakm/python-excel-benchmarks

benchmark_csv                  0.026495
benchmark_excellent            1.784107
benchmark_openpyxl             2.897072
benchmark_openpyxl_rows        7.025895
benchmark_pyexcelerate         0.550225
benchmark_xlsxcessive          1.430242
benchmark_xlsxwriter           1.611668
benchmark_xlwt                 1.275144

http://xlsxwriter.readthedocs.org/en/latest/working_with_memory.html

pyexcelerate          :  10.11
xlwt                  :  15.67
xlsxwriter (optimised):  19.70
xlsxwriter            :  23.50
openpyxl   (optimised):  95.82
openpyxl              :  95.90

Large Dictionaries Not Released From Memory - Python + Ubuntu

I have a flask application running on Ubuntu which reads from a file and creates several large dictionaries. After the dictionaries are created and flask returns the request, the memory used for the dictionaries is not released back to the OS. This example shows the problem: https://gist.github.com/pawl/c3ed7663b94abec01d75

It's not really a Flask issue, but the Flask guys were super helpful when I asked why my large dictionary wasn't released from memory: https://github.com/mitsuhiko/flask/issues/1202

I dug even deeper and learned this is a Linux thing:
"Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."
http://deeplearning.net/software/theano/tutorial/python-memory-management.html

My solution:
Turning it into a function and running it as a separate process: https://gist.github.com/pawl/95769724848269cff890

If you want to go even deeper down the rabbit hole, read these: