Friday, December 26, 2014

Best Django Boilerplate/Skeleton

This is my favorite boilerplate for new projects so far: https://github.com/kirpit/django-sample-app

Because:

If you're not a fan of bootstrap, django-html5-boilerplate also looks good.

Thursday, December 11, 2014

Setting A WTForm Default Date To Today/Now

http://stackoverflow.com/a/27424739/1364191

It's definitely a common mistake for people using WTForms. When you create a form and use something like datetime.date.today() as a default value, that value will always stay the same. The fix is to use "datetime.date.today" instead of "datetime.date.today()".

Friday, December 5, 2014

Dokku Flask Port and Host

In your flask application, you need to:

  • Set default port=5000 (this seems to be Dokku's default too)
  • Set default host=0.0.0.0 (this allows your host server to connect to the container)

In your Procfile:
web: python <your-app>.py

If you're using gunicorn, it shouldn't matter because it's not using app.run().

If you still can't connect to your application after it deploys, see this: http://www.paulsprogrammingnotes.com/2014/12/troubleshooting-dokku-not-accessible.html

Troubleshooting Dokku Not Accessible

First, make sure your application isn't giving any errors when it attempts to run:
dokku run <your-app> "<whatever is in your proc file>"

Get the :port number from the line with "upstream" in the following file: /home/dokku/<your-app>/nginx.conf

Try to "wget 127.0.0.1:<your-port>" from the host. Will it connect?

Thursday, December 4, 2014

"Requested runtime not available for this stack" - Dokku

While using Dokku, I was getting the following error message: Requested runtime (python-2.7.8) is not available for this stack (cedar-14).

This was due to a connection error between my docker container and the internet. To determine this, I ran "docker images" and got the image id of "programium/buildstep", then got into a container with the following command: sudo docker run -t -i <your image id> /bin/bash

Once in the container, I ran:
curl http://lang-python.s3.amazonaws.com/cedar/runtimes/python-2.7.8.tar.gz -s

If that is unsuccessful, that should confirm it's a connection issue.

This was the solution to the problem: http://www.paulsprogrammingnotes.com/2014/12/rebuilding-buildstep-image-for-dokku.html

Tuesday, December 2, 2014

Rebuilding Buildstep Image For Dokku From Behind Proxy

I had to change the buildstep image's docker file to add my proxy. This is because the buildstep image required a proxy before the build to download the python buildpack.

I was rebuilding the buildstep image for dokku with these instructions: https://github.com/progrium/dokku/commit/9ebf453b72cab3a16ea261284236bb7c20ca3a1a#diff-04c6e90faac2675aa89e2176d2eec7d8R101

I'm behind a proxy, so I was getting this error when I ran "sudo make build":
Cloning into 'heroku-buildpack-multi'...
fatal: unable to access 'https://github.com/ddollar/heroku-buildpack-multi.git/': Failed to connect to github.com port 443: Connection timed out

The solution? Replace the Dockerfile in the root of the folder you git cloned with this:

Monday, December 1, 2014

Troubleshooting Pip Timeout - Python

Timeout: (<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x27330d0>, 'Connection to pypi.python.org timed out. (connect timeout=15)')

If you're installing several requirements from a requirements.txt file, it helps to use "-v" after your pip install command to figure out which module has the timeout. Most likely, it's trying to download a module from a URL that doesn't exist.

Friday, November 28, 2014

Tuesday, November 25, 2014

NULL values excluded from NOT IN - SQLAlchemy+SQLite

Solution: AND column IS NOT NULL

Example: query.filter(or_(~self.column.in_(value), self.column == None))

This thread explains "SQL Server in treating NULL as a value": http://stackoverflow.com/questions/11491831/null-values-are-excluded-why

Github Asking For Password After Adding SSH Key

Run: git remote -v

If the output has a URL that starts with https://github.com and not git@github.com, you need to change that by running a command similar to this: git remote set-url origin git@github.com:pawl/flask-admin.git

Friday, November 21, 2014

Guide To Syncing Fork With Original Project

https://help.github.com/articles/syncing-a-fork/

From my fork, I ran these commands:

  1. git remote add upstream https://github.com/mrjoes/flask-admin.git
  2. git fetch upstream
  3. git merge upstream/master
  4. git push
Create new branch and sync with upstream (change the bolded text):
  1. git remote add upstream https://github.com/mrjoes/flask-admin.git
  2. git fetch upstream
  3. git checkout -b issue_xxxx upstream/master

Sunday, November 16, 2014

Configuring Notepad++ For Python

On the Settings->"Preferences", "Tab Settings" tab, I set "[Default]" at Tab size: 8, uncheck Replace by space; and set "Python" to uncheck Use default value, Tab size: 4, check Replace by space. This causes inserts into a python source to use 4 spaces for indents, and indent with spaces instead of tabs. 
If I end up with any tabs in the source, I use Edit->Blank Operations->TAB to Space to convert them. I also clean up trailing blanks with Edit->Blank Operations->Trim Trailing Space. 
I install and use the pep8 package to verify standard formatting.
http://www.reddit.com/r/Python/comments/2mfrn6/python_in_notepad/cm3tdj5

Monday, November 10, 2014

Using Bootstrap-DateRangePicker As A Timepicker

Here's the jsfiddle: http://jsfiddle.net/d8uomdjv/

It seems like the author is opposed to adding this sort of functionality: https://github.com/dangrossman/bootstrap-daterangepicker/issues/295

re.match vs re.search - Python

http://stackoverflow.com/a/180993/1364191

  • re.match - If zero or more characters at the beginning of string match the regular expression pattern.
  • re.search - Scan through string looking for a location where the regular expression pattern produces a match


Saturday, November 8, 2014

New Machine Setup For Flask Development On Ubuntu

Here are the commands I use when I bring up a new Ubuntu server for flask development:

Wednesday, October 22, 2014

MySQL Performance LXC Container vs Host

I have a linux virtual container (LXC) and the server hosting the container running the same version of MySQL with the same database.

I tried running some select queries on both of the both the virtual container and host for benchmark purposes. My conclusion? MySQL read performance inside and outside the LXC container is the exactly the same.

Tuesday, October 21, 2014

WHMCS Cronjob Error

If you get the following error while you try to run the WHMCS cronjob from the command line:
Site error: the file <b>/var/www/accounts/admin/cron.php</b> requires the ionCube PHP Loader ioncube_loader_lin_5.3.so to be installed by the website operator. If you are the website operator please use the <a href="http://www.ioncube.com/lw/">ionCube Loader Wizard</a> to assist with installation.

You need to add your the ioncube zend extension to the CLI php.ini too: /etc/php5/cli/php.ini


Sunday, October 19, 2014

Sendmail/STARTTLS verify=FAIL



Oct 19 14:04:34 Billing sm-mta[17583]: STARTTLS=client, relay=aspmx.l.google.com., version=TLSv1/SSLv3, verify=FAIL, cipher=ECDHE-RSA-RC4-SHA, bits=128/128

If you see those errors in your mail.log and your emails are failing to send, you need to add your ssl cert from apache to the sendmail config in /etc/mail/sendmail.cf:
# CA directory
O CACertPath=/etc/apache2/ssl
# CA file
O CACertFile=/etc/apache2/ssl/example.ca-bundle
# Server Cert
O ServerCertFile=/etc/apache2/ssl/example.cert
# Server private key
O ServerKeyFile=/etc/apache2/ssl/example.key

Saturday, October 18, 2014

Securing External LDAP Connections

https://help.ubuntu.com/community/SecuringOpenLDAPConnections

I was getting the following error when I tried testing with openssl:
error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177

The solution? It turns out that /etc/default/slapd needs to be configured to use /etc/ldap/ldap.conf using the following:

SLAPD_OPTIONS="-f /etc/ldap/slapd.conf"

Friday, October 17, 2014

Adding Virtualhost In Apache For Port

Say we wanted to route the <your-domain>.com:8072/misc to /var/www/misc...

Add the following to /etc/apache2/ports.conf:
NameVirtualHost *:8072
Listen 8072

Add a file that describes your site to /etc/apache2/sites-available with this:
<VirtualHost *:8072>
   ServerName <your-domain>.com
   DocumentRoot /var/www/misc/
</VirtualHost>

Run this command: sudo a2ensite misc

And restart apache: sudo service apache2 restart

Thursday, October 16, 2014

Deploying Nginx + Gunicorn + Flask

Why use both Nginx and Gunicorn?: http://serverfault.com/a/220047

Simple explanation: http://prakhar.me/articles/flask-on-nginx-and-gunicorn/

Nginx SSL configuration: https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-with-ssl-as-a-reverse-proxy-for-jenkins

The thing I spent the longest on was getting supervisor to work. Gunicorn kept giving me "gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>", because I was missing the "directory" parameter. I needed to tell supervisor which directory to start from.

[program:my_app]
command=/<path>/venv/prod/bin/gunicorn -w 8 main:app -b 127.0.0.1:8080
user=me
autostart=true
directory=/home/me/my_dir/
stdout_logfile=/home/me/my_dir/gunicorn_supervisor.log
stdout_logfile_maxbytes=20MB
redirect_stderr=true

You will need to create an upstart script to get supervisor to run when the system starts: http://serverfault.com/a/96500

Tuesday, October 14, 2014

Python Excel Library Benchmark Comparison

https://github.com/swistakm/python-excel-benchmarks

benchmark_csv                  0.026495
benchmark_excellent            1.784107
benchmark_openpyxl             2.897072
benchmark_openpyxl_rows        7.025895
benchmark_pyexcelerate         0.550225
benchmark_xlsxcessive          1.430242
benchmark_xlsxwriter           1.611668
benchmark_xlwt                 1.275144

http://xlsxwriter.readthedocs.org/en/latest/working_with_memory.html

pyexcelerate          :  10.11
xlwt                  :  15.67
xlsxwriter (optimised):  19.70
xlsxwriter            :  23.50
openpyxl   (optimised):  95.82
openpyxl              :  95.90

Monday, October 13, 2014

Large Dictionaries Not Released From Memory - Python + Ubuntu

I have a flask application running on Ubuntu which reads from a file and creates several large dictionaries. After the dictionaries are created and flask returns the request, the memory used for the dictionaries is not released back to the OS. This example shows the problem: https://gist.github.com/pawl/c3ed7663b94abec01d75

It's not really a Flask issue, but the Flask guys were super helpful when I asked why my large dictionary wasn't released from memory: https://github.com/mitsuhiko/flask/issues/1202

I dug even deeper and learned this is a Linux thing:
"Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."
http://deeplearning.net/software/theano/tutorial/python-memory-management.html

My solution:
Turning it into a function and running it as a separate processhttps://gist.github.com/pawl/95769724848269cff890

If you want to go even deeper down the rabbit hole, read these:

Saturday, October 4, 2014

Initializing Dictionary Elements Inside A Loop - Python

There's a really interesting tip in the "Initializing Dictionary Elements" section of this page: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

"it is cheaper to use a try statement" when checking to see if a key exists in a dictionary (inside of a loop).

Thursday, October 2, 2014

maximum recursion depth exceeded - PyIntervalTree

I was getting the "maximum recursion depth exceeded" when I tried to run large datasets through PyIntervalTree.

I never fixed the issue or found anyone else with the issue. I ended up switching to bx-python instead: https://www.biostars.org/p/99/#100

Group By Apply Occurring Twice On First Item - Pandas

https://github.com/pydata/pandas/issues/7739#issuecomment-48823964

If Pandas's Group By is applying the function in Apply twice to the first item, it's actually expected behavior according to the warning in the documentation: http://pandas.pydata.org/pandas-docs/stable/groupby.html#flexible-apply

Sunday, September 28, 2014

Virtualenvwrapper Basics

This video does a good job of describing the basics of Virtualenvwrapper: https://www.youtube.com/watch?v=thHNYVrY0lU

You'll need to have used virtualenv to appreciate this.

Basically, you create virtual environments with "mkvirtualenv" and they're stored in a central location. When you want to switch to your virtual environment, you use the "workon" command.

You can also create a postactivate script that loads environmental variables when you switch to your virtual environment.

Note: There is also a module called Pew that seems to have a simpler installation process: https://github.com/berdario/pew

Wednesday, September 24, 2014

Meetup API address_1_error

I was getting this error while using the POST /venues method of the meetup API.

The error seems to happen when you duplicate the address of another venue. I was able to add the venue, but when I tried to add it a second time - I would get this error.

I ended up getting the list of venues and trying to find a match first.

Tuesday, September 16, 2014

Eventbrite "Application Key Error"

The error message looks like this:
{u'status': {u'http_code': 200}, u'contents': {u'error': {u'error_type': u'Application Key Error', u'error_message': u'Please provide your Application Key in the URL as "?app_key=<APP_KEY>".'}}}

After a few hours of trying to get past the Eventbrite API's "Application Key Error". I finally figured out that I needed to switch my request URL from:
"https://developer.eventbrite.com/json/event_new"
to:
"https://www.eventbrite.com/json/event_new"

Wednesday, September 10, 2014

Wednesday, September 3, 2014

SQLAlchemy Performance Tip

http://www.mobify.com/blog/sqlalchemy-memory-magic/

"If you can process the results of database queries iteratively (and very often you can), stream the results"

"Since the data needs to travel over the network from the database whether it's streamed or not, this doesn't add a huge overhead, but we'll see that it reduces memory requirements."

Example (using Flask-SQLAlchemy):
db.engine.execution_options(stream_results=True).execute(query)

However, that's not the full story when it comes to MySQLdb (MySQL-python). http://stackoverflow.com/a/3699677

If you're going to do this with mysql, I recommend using oursql: https://pypi.python.org/pypi/oursql

Enable Threading - Flask

http://stackoverflow.com/a/14823968/1364191

If you're going to be expecting more than one user on your flask server, you need to enable threading like this: app.run(host='0.0.0.0', port=5000, threaded=True)

Unfortunately, the threading makes Ctrl+C not work sometimes and I'll have to kill the thread manually.

For better solutions for production, check out these WSGI solutions: http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/

Monday, September 1, 2014

Mosh - SSH Without Dropped Connections

https://mosh.mit.edu/

Warning: As of 10/17/2014, it's still in development and they haven't implemented scrollback yet. However, you can use screen and view the scrollback with Ctrl+A then Esc.

Sunday, August 31, 2014

Things I Learned From DEFCON

Pentoo - Linux distro for pen testing.
VMware fusion - Most of the presenters were using a Macbook with VMware fusion managing their VMs.
Chinavasion - One of the presentations mentioned this in the context of dealextreme, I'm assuming this site is similar.
Kali linux - Another distro for pen testing.
Open bts
Sqlmap (SQLi injection) - Python tool used to tell if php pages were vulnerable to SQLi injection.
C99 shell - PHP interface for shell level system functions.
b374k shell - Another shell which allows an user to run file system, database, and shell commands from a web interface.
Accunetix (xss)
Business logic flaws in mobile operators
Doskey /history - Will show you all the previous commands typed into command prompt.
Maria DB - When oracle took over MySQL, it forked and maria db is still run by the previous developers.
Firefox imacro - A presenter said he uses this for all of his bots.
Burp repeater - Looks similar to fiddler, allows repeating http packets. Maybe only for Mac.
Fritzing
Groupie and Geocouch (couch dB) - This was used in combination with the unity engine to display clusters of map points in a video game.
Steganography - The word for hiding data inside of other files. A presenter showed how histograms could be used.
Femtocells - A personal cell tower. One of the presentations talked about how to use a femtocell to intercept text messages to a Verizon (CDMA) phone.
Shark - Has something to do with a hadoop, good for storing large amounts of data.
SuperTimeline

Wednesday, August 27, 2014

Alembic Cheat Sheet - Python/SQLAlchemy

alembic revision --autogenerate -m "<your message>"
alembic upgrade head

Problem: FAILED: Target database is not up to date.
Solution: alembic upgrade head

Problem: No such revision '5000106def16'
Solution: 
  1. sqlite3 db.sqlite3
  2. drop table alembic_version; (and exit, ctrl+d)
  3. alembic upgrade head

Failed to fetch http://security.ubuntu.com

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/quantal-security/main/binary-amd64/Packages  404  Not Found [IP: 91.189.91.14 80]

This is happening because you're on an unsupported/old version of Ubuntu.

One way to fix it:
  1. nano /etc/apt/sources.list
  2. Change all entries referring to http://archive.ubuntu.com/ubuntu to http://old-releases.ubuntu.com/ubuntu

Thursday, August 21, 2014

Best Python Tricks

https://gist.github.com/JeffPaine/6213790

Most of the examples involve making your code much faster by using iterators instead of creating copies of the data. Example: xrange

Here's the video: https://www.youtube.com/watch?v=OSGv2VnC0go

CSS Selectors - ~ + >

http://css-tricks.com/child-and-sibling-selectors/

That's an excellent article describing what angle brackets, plus signs, and tildes do in CSS selectors. The illustrations make it especially clear.

Wednesday, August 6, 2014

Wednesday, July 30, 2014

Overwriting module with same version in Heroku - Python

I needed to install a newer version of a module from a github commit, but it had the same version number. Heroku just used the cached version of the module and didn't overwrite it with the newer version.

The easiest fix for this:

  1. Create a file in your application's root directory called "runtime.txt" with only "python-3.4.1" written inside.
  2. Add, commit, and push the new runtime.txt
  3. Change runtime.txt back to "python-2.7.6"
  4. Add, commit, and push to clear your virtualenv and reinstall all modules

Friday, June 27, 2014

The server quit without updating PID file (/var/run/mysqld/mysqld.pid). - MySQL

. * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

If you're getting this error message, try using sudo to start mysql.

Thursday, June 19, 2014

Submit Form When User Clicks Option (autosubmit) - Typeahead.js

http://stackoverflow.com/a/18483816/1364191

The bolded section below will automatically submit your form when you select an option from the Typeahead.js dropdown:
$('#remote .typeahead').typeahead(null, {
 displayKey: 'value',
 source: site_search.ttAdapter()
}).on('typeahead:selected', function(e){
      e.target.form.submit();
    });

Using Enter Key With Typeahead.js - Javascript

https://github.com/twitter/typeahead.js/issues/255

If your text input with typeahead.js is not letting you use the enter key, you need to add a submit button (if you don't already have one).

You can hide it if you want:
<input type="submit" value="Submit" style="display: none">

Thursday, June 12, 2014

WinSCP vs Samba For Windows <---> Ubuntu Development

I've always used WinSCP to transfer files between my Windows development environment and Ubuntu. It worked pretty well. All you had to do was open the file with your editor by clicking the file in WinSCP, and when you saved the file it would save to the server in the background.

The main negative was the lack of feedback when a file fails to save to the server. WinSCP could be logged out and asking for a password in the background, but you were expecting that save to fix the bug you were working on. It ended up being another thing to debug.

A coworker suggested using Samba instead. I installed it with this simple guide: How to Create a Network Share Via Samba Via CLI Then I mapped the network drive in Windows by going to the "Map Network Drive" option in the tools menu of the file explorer:

The only negative I've found with Samba so far is that it makes Notepad++ a little less responsive. But it helps a small amount if you turn off "File Status Auto-Detection":

Tuesday, June 10, 2014

Relative Paths In Scripts Run By Crontab - Python

I learned that a script run by crontab can't have a relative path.

This fix was here: http://stackoverflow.com/a/10422444

The following example opens my_file.txt in a path relative to your script:
import os
dir_path = os.path.dirname(os.path.abspath(__file__))
with open (os.path.join(dir_path, 'my_file.txt'), "r") as myfile:
    mytext = myfile.read()

Monday, June 9, 2014

ELSE in FOR loops - Python

http://stackoverflow.com/a/9980160

Turns out that you can use ELSE after a FOR loop to make it give an output if it didn't have anything to iterate.

Undoing git rm -r --cached

I followed these instructions to gitignore an already committed file: http://stackoverflow.com/questions/6535362/gitignore-after-commit

When I wanted to start tracking it again, I needed to following these instructions: https://gist.github.com/laszlomiklosik/3833968

Make sure you save a backup of your current version before you check out the file again.

Thursday, June 5, 2014

Multiple DSNs All Connecting To The Same IP - Sybase ODBC

The problem: You have multiple DSNs defined in your unixODBC's odbc.ini file, and it always connects to the same IP when you try testing with iSQL.

The solution:
DO NOT use the following line in your odbc.ini file to define your DSN's port:
CommLinks       = tcpip(host=localhost;port=9502)

DO use the following format:
CommLinks       = tcpip(HOST=localhost:9502)

Compress - Sybase ODBC

Sybase has a odbc parameter called "Compress" that is disabled by default. It's very useful if you're running queries on something with slow network transfer speeds and large datasets.

Example:
Compress = yes

This improve the performance of my query by about 40%.

Monday, June 2, 2014

[IM004] [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed (0) (SQLDriverConnect) - Ubuntu

The link to download SQLanywhere is here: http://www.sybase.com/detail?id=1087327

To install it, you need to unzip the file and run "./setup" in the top level directory in the unzipped folder.

I also installed:
https://code.google.com/p/pyodbc/ (use the .zip file on the left)
http://onefinepub.com/2014/03/05/installing-unixodbc-2-3-2-higher-ubuntu-12-04-lts/

The first error message I received was:
sqlanydb.InterfaceError: Could not load dbcapi.  Tried: dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib
Exception AttributeError: "'Root' object has no attribute 'api'" in <bound method Root.__del__ of <sqlanydb.Root object at 0x3061150>> ignored

This error is resolved by creating a file called "/etc/ld.so.conf.d/sqlanywhere.conf" and adding a line with "/opt/sqlanywhere12/lib64". Then you will need to run ldconfig -v.

The error message in the title occurred on Ubuntu because unixODBC was unable to find all of the libraries it needed to run.

This helped to find the issue: http://blogs.msdn.com/b/dataaccesstechnologies/archive/2014/01/22/sqlallochandle-on-sql-handle-henv-from-linux-sqlncli-driver.aspx

strace -t -f -o trace_out.txt isql -v <DSN>< USERNAME> <PASSWORD>

It will show there are a bunch of missing files. The issue was resolved when isql was run from the /opt/sqlanywhere12/bin64 directory. This indicated that it's not looking in the right place for the library files. Run the following: export LD_LIBRARY_PATH='/opt/sqlanywhere12/lib64'

Not having the environmental variable above will also cause "sqlanydb.InterfaceError: dbcapi version 1 required." if you're using python's sqlanydb library.

It will give a "file not found" error if you try using bin32/lib32 files on a 64 bit system. Make sure you use the bin64/lib64 files in the "Driver=" fild in your /etc/odbc.ini file (if you're using a 64 bit system)!

Now you will receive an error saying "Parse error: DSN does not exist". You need to add an ODBC_INI environmental variable to where your DSNs are defined. I ran the following: export ODBC_INI='/etc/odbc.ini'

To make those variables we exported work system-wide and continue to work even after we log out:

  1. sudo nano /etc/environment
  2. Add:
    ODBC_INI='/etc/odbc.ini'
    LD_LIBRARY_PATH='/opt/sqlanywhere12/lib64'

These articles were helpful:
https://hynek.me/articles/twisted-sybase/
https://hynek.me/articles/a-short-summary-on-sybase-sql-anywhere-python/

Monday, May 26, 2014

Heroku's Advantages Over Google App Engine

Pros:

  • Postgres (you can even use pgAdmin to view your database)
  • Uses Git to push your application into production
  • More monitoring features and 3rd party apps
  • As long as your project uses one dyno, it runs for free (similar to google, but with a less confusing billing structure)
Cons:
  • 10k row limit for your database before they start charging you $9 per month 

Saturday, May 24, 2014

AttributeError: 'QuerySelectField' object has no attribute '_sa_instance_state'

I came across this error when I was trying to make a custom validator in Flask-Admin with wtforms form objects.

This code didn't work:
def location_must_not_conflict(form, field):
if Event.query.filter(Event.location == form.location).first():
raise wtforms.validators.ValidationError('Location conflicts with another request for the same venue.')

Because I was using form.location and not form.location.data.

This will work:
def location_must_not_conflict(form, field):
if Event.query.filter(Event.location == form.location.data).first():
raise wtforms.validators.ValidationError('Location conflicts with another request for the same venue.')

Tuesday, May 20, 2014

Individual Form Fields In Jinja2 - Flask-Admin

On your edit and create templates in Flask-Admin, you can print individual form fields like this (the field's name is location):
{{ form.location }}

Sunday, May 18, 2014

Google Calendar Api bad request 400 - Python Authomatic Library

I was finally able to get the request to go through by adding a header saying it's json format. Here's an example:

requestbody = """
{
      "end": {
      "dateTime": "2014-05-18T01:35:00Z"
      },
      "start": {
      "dateTime": "2014-05-18T01:35:00Z"
      },
      "description": "test2",
      "location": "test venue",
      "summary": "test2"
      }
"""
response = authomatic.access(credentials, url, method='POST', headers={'Content-Type': 'application/json'}, body=requestbody)


Saturday, May 17, 2014

Add user_id for modified_by field before save - Flask-Admin


In Flask-Admin, you can override the "on_model_change" function to to automatically populate a "modified_by" field in your model.

class LocationView(ModelView):
def on_model_change(self, form, model):
model.user_id = g.user.id

admin = Admin(app)
admin.add_view(LocationView(Location, db.session, name="Locations"))

Favorite Flask Boilerplate/Skeleton/Templates




Those do a lot of the work of starting a new Flask project for you.

With cookiecutter flask, all you need to do to build a new project is: cookiecutter https://github.com/sloria/cookiecutter-flask.git

As a warning, cookiecutter flask and overholt include a lot of stuff. I ended up not using it because it was too much work to change everything to work with oAuth.

Saturday, May 10, 2014

Reverse PDF Page Order

I created an app that allows you to upload a pdf and it returns a copy of your PDF with the page order reversed. It's available here: http://reversepdf.appspot.com/

Initially when I googled how to do this in Adobe Acrobat, the first few results had fairly complicated solutions. After I made the app, I ended up finding an easier way to do it in Adobe (in the 6th from the top Google result on "reverse pdf order acrobat"...):

Oh well, it was a fun project.

Monday, April 28, 2014

Discourse As Potential Mailing List Replacement

While tested out Discourse, I only found a few negative things:
  1. To make it act like a mailing list, it currently requires each user to change settings on their account: https://github.com/discourse/discourse/blob/master/docs/MAILING-LIST-SETUP.md#suggested-user-preferences
    I commented on a thread about a new feature that allows setting defaults for this: https://meta.discourse.org/t/default-email-settings-for-a-new-user/14980
  2. It doesn't allow setting defaults for which categories the user has "Watched". For example, I would want to default the "General" list to "Watched", but more obscure lists would only be watched if the user chooses to.
  3. It uses a different e-mail address for each e-mail it sends you (this is to keep track of which thread it belongs to), here's an example of a "From" address: testdiscourse+43854630fb3a3d9492505fa5a23db196@gmail.com
    Google Groups always has the same to/from e-mail address and magically figures this out. This might be weird in some peoples' email clients.
As a forum, this beats the crap out of PhpBB, but as a mailing list it's not a clear winner over Google Groups yet. It's open source, has a better web interface, and the "Categories" are a cool feature for keeping all the lists in the same place. But, the "email in" functionality was too recently implemented, and it might take a few months before it's as polished as Google Groups.

It's going to be a very tempting replacement if they add default settings for user preferences.

Screenshots:






Data source name not found and no default driver specified - SQLalchemy/Sybase

I was getting this error when trying to connect to a sybase database using Microsoft's odbc driver by using the connection string shown in the SqlAlchemy examples:
DBAPIError: (Error) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)') None None

The following code worked to connect:
import pyodbc, sqlalchemy

def connect():
    return pyodbc.connect('DSN=<dsn name>;UID=<username>;PWD=<password>')

srcEngine = sqlalchemy.create_engine('sybase+pyodbc://', creator=connect, echo=True)

Saturday, April 26, 2014

Docker Not Starting - Ubuntu

I was getting this error message because docker was refusing to start: Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

This issue was occurring because my kernel was refusing to upgrade, because I was on a VPS. (you need to get your VPS provider to upgrade it for you)

Run uname -r, if it returns something like this: "2.6.32-042stab084.20" That's probably the reason why docker isn't starting. Your kernel isn't compatible.

If you're not on a VPS and can upgrade your kernel, try instructions on this page: http://docs.docker.io.s3-website-us-west-2.amazonaws.com/installation/ubuntulinux/

E: Sub-process /usr/bin/dpkg returned an error code (1)

I was getting this error message while I was trying to upgrade the kernel on my VPS:
/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/linux-image-3.8.0-34-generic.postrm line 328.
dpkg: error processing linux-image-3.8.0-34-generic (--remove):
 subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
 linux-image-3.8.0-34-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)

You might not be able to upgrade the kernel on your VPS. Contact your VPS provider and get them to upgrade the kernel for you.

WHMCS Cronjob Not Running On Ubuntu

My issue was the user with the cronjob did not have access to run the cronjob php file. So, I needed to make the new cronjob for the correct user.

Solution:

  1. Find out the owner of your apache process (usually www-data).
  2. Look at the top of the "Automation Settings" page in your WHMCS admin area. It should say something like: php -q /var/www/whmcs/admin/cron.php (keep this handy for step 4)
  3. In terminal run: crontab -e -u www-data
  4. Ensure you have a line that says:
    0   6   *   *   *    php -q /var/www/whmcs/admin/cron.php (note: this should be the same file path you got for your own install in step 2)



Http Port 443 Working But Https Port 443 Not Working In CherryPy - Python

I was able to visit my site using HTTP and port 443, but nothing would even show in the log when I tried to visit with HTTPS and port 443.

This seems to only be an issue with CherryPy 3.3.0, I switched back to CherryPy 3.2.3 and it worked again.

I noted it on their bitbucket issue here: https://bitbucket.org/cherrypy/cherrypy/issue/1298/ssl-not-working

OpenSSL DLL load failed On Windows - Python

The DLL load failed error happened when cherrypy tried the following import: "from OpenSSL import SSL"

ChannelFailures: VerificationError("importing 'C:\\\\Python27\\\\lib\\\\site-packages\\\\cryptography\\\\_Cryptography_c
ffi_48bbf0ebx93c91939.pyd': DLL load failed: %1 is not a valid Win32 application.",)

These error messages are solved with this install: http://slproweb.com/download/Win32OpenSSL-1_0_1g.exe

If you have a 32-bit version of python, that install will move the correct DLLs to your system32 folder.

Thursday, April 24, 2014

My First Python Library

https://github.com/pawl/Chinese-RFID-Access-Control-Library

I purchased a chinese access controller and reverse engineered some functionality of the windows software that came with it. I made a python library that does some of the same functions as the desktop software: https://github.com/pawl/Chinese-RFID-Access-Control-Library/

I use this library to tie our access controller into with our billing system (called WHMCS) and another web application (which is meant to be a self-documenting way to activate/deactivate RFID badges): https://github.com/pawl/MakerManager

Now, whenever someone stops paying for 5+ days, their RFID badge is deactivated because a "hook" in WHMCS sends a request to a webservice that uses the access control library.

Sunday, April 20, 2014

Android Bitcoin OutOfMemoryError Or Crashing On Startup

This page explains the problem: https://code.google.com/p/bitcoin-wallet/wiki/OutOfMemory

At one point, it got so bad that the application would not even start anymore.

Solution:

  1. Get Root Explorer
  2. Navigate to your system folder and open "build.prop" in the text editor
  3. Edit "dalvik.vm.heapsize" and "dalvik.vm.heapgrowthlimit" to the following:
    dalvik.vm.heapsize=256m
    dalvik.vm.heapgrowthlimit=128m
  4. Restart your phone, open the bitcoin application, and backup your keys before it crashes again.

Monday, April 14, 2014

Lock wait timeout exceeded - MySQL

OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')

After trying a bunch of different solutions in the code, I ended up having to restart MySQL to get this error message to go away.

Solution To Apache Triggering OOM-Killer

Update: We switched to Prefork apache and have been experiencing significantly fewer problems.

In this instance, we were getting hit with web crawlers according to the access log and the syslog was saying this: /usr/sbin/apach invoked oom-killer

This issue occurs when apache opens up too many child processes, uses up too much memory, then OOM killer starts shutting down random processes like mysql.

The problem ended up being that our apache2.conf did not have any configuration limiting the number of processes that Apache could open. It might have had a default, but the default was too high. We were using Apache ITK MPM and the apache2.conf only mentioned prefork, worker, and event MPMs. You can check which MPM you are running with "apache2 -V".

Adding this to the configuration solved the problem:
<IfModule mpm_itk_module>
    StartServers 5
    MinSpareServers 5
    MaxSpareServers 10
    ServerLimit 75
    MaxClients 75
    MaxRequestsPerChild 5000
</IfModule>

At first I had the ServerLimit and MaxClients set to 150 and it seemed like apache was ignoring ServerLimit and Maxclients. However, ITK MPM creates an extra fork for each request (according to their site), that's why there was 200+ processes running for my limit of 150. So, the process count is going to be 2x the limit you set.

Thursday, April 3, 2014

Edit A Sharepoint List Page

Add the following to the end of the List page's URL:  ?ToolPaneView=2

Tuesday, April 1, 2014

Override ModelView Class - Flask-Admin

This has an awesome example of how to override your base ModelView: http://code.vokor.org/vokorbb/src/04de9dcb666d2fb35513ca29c97654b6eef80648/vokorbb/admin.py?at=tip

You would want to do this if you wanted to add something to all of your admin classes. Use cases:

  • If you wanted to require login by overriding is_accessible for all of your ModelViews.
  • If you wanted to add an export method to all of your ModelViews.

Flask-Admin Export CSV

Edited: This was recently added to Flask-Admin as a feature.

You can add "can_export = True" to your ModelView to enable it.

More info here: http://flask-admin.readthedocs.org/en/latest/api/mod_model/#flask_admin.model.BaseModelView.can_export

Print URL With Request Parameters/Arguments - Python Flask / Jinja2

You need to add the following to your jinja2 template:
{{ request.path ~ '?' ~ request.query_string }}

You can also print only the Get Variables / URL Parameters / Argument String with the following:
print request.query_string

The example is available here: http://stackoverflow.com/a/11781872/1364191

Thursday, March 27, 2014

Use A Temporary File (StringIO) for Pandas's to_csv - Python Flask

This link helped me use a temporary file for an excel export in flask (using pandas): https://coderwall.com/p/nwbuhq

Get Column and Row Labels In selectHandler - Google Charts / Visualization API

For a more complete example of using selectHandler, see this page: https://developers.google.com/chart/interactive/docs/basic_interactivity

My example below get the labels from the row and the columns. I used this to send the user to another link and drill-down.

// this is an example of a bar chart's selectHandler function
function selectHandler() {
var selection = chart.getSelection();
var item = selection[0];

if (item.row != null && item.column != null) {
var rowLabel = parseInt(data.getValue(item.row, 0));
var columnLabel = data.getColumnLabel(item.column);
        }
}
google.visualization.events.addListener(chart, 'select', selectHandler);

Saturday, March 22, 2014

Ordered Post Data - Python Requests

Here's an example of how to send ordered data with python's Requests library, for when the order of your post request data matters:

import requests
from collections import OrderedDict

payload = OrderedDict([("username", "admin"), ("pwd", "password")]) # the order matters!
r = requests.post('http://1.1.1.1/example', data=payload)
print r.text

Monday, March 17, 2014

Multiple Views Based On Same Model - Flask-Admin

You need to manually change the endpoint like MrJoes says in issue 480: https://github.com/mrjoes/flask-admin/issues/480

Here's an example:
admin.add_view(MyModelView(model.CsvTable, db.session, endpoint="duplicate_view"))

You can now visit your view by going to 0.0.0.0/admin/duplicate_view

Tuesday, March 4, 2014

Checkmark "column_formatters" - Flask-Admin

In your custom ModelView, you need to add the following lines:
from flask.ext.admin.model.template import macro
column_formatters = dict(your_column=macro('render_your_column'))
list_template = 'list.html'

In your custom list.html template, you need to add the following jinja2 code:
{% macro render_your_column(model, column) %}
{% if model.your_column%}<i class="icon-ok"></i>{% endif %}
{% endmacro %}

If your row has a value in "your_column", then it will show as a checkmark. You need to make a new macro for each column.


Flask-Migrate

https://github.com/miguelgrinberg/Flask-Migrate

Flask-Migrate is exactly what I've been looking for when it comes to database migrations. When I create PHP applications with MVC frameworks, the changes in the database need to be done manually when the model is changed. Flask-Migrate handles changes to the database (like adding/removing columns) so your models reflect your database.

This is a big advantage Flask has over PHP frameworks.

Tuesday, February 25, 2014

Pip Not Working From Powershell

If you have strawberry perl installed, you may get one of these errors when you try to use pip from powershell:

  • Did not provide a command
  • Unknown or unsupported command 'install'
This is the result of a conflict between strawberry perl's pip and python's pip. You need to remove anything related to strawberry perl from your path environmental variable and ensure you have C:\Python27\Scripts and C:\Python27\Scripts in your environmental variables.

Tableau's Frustrating Licensing Program

Tableau's licensing program is crazy.

I was planning to use "Trusted Authentication" to share an embedded chart with lots of users (not all of whom have licenses). Some of those users will only look at the chart maybe once a year, and we can't justify buying them an entire license to look at a chart once.

This page says "everyone who visits the page must be a licensed user on Tableau Server": http://onlinehelp.tableausoftware.com/v7.0/server/en-us/trusted_auth.htm

If you want more details on how to implement Trusted Authentication using PHP, first visit the following folder on your server: C:\Program Files\Tableau\Tableau Server\8.1\extras\embedding\php

It has some sample code, but you need to install version 1.x of pecl_http to make it work. The guides for that are:

Possible Solution To Slow SQLalchemy + Flask Application

This stackoverflow thread talks about why you want to use scoped_session:
http://stackoverflow.com/questions/6519546/scoped-sessionsession-maker-or-plain-session-maker-in-sqlalchemy

Basically, your web application isn't thread-safe if you're not using scoped_session. It's much better than having you application use a new thread for every instance.

An example of scoped_session in use is available here: https://github.com/mjhea0/flask-boilerplate/blob/master/models.py

Tuesday, February 18, 2014

Hide Zero Value Points On Google Chart

I have a line chart which needed a series of points. I don't want points to show if the value is zero.

Here's my solution:

  • Set the series's lineWidth to zero and make the point larger than 0:
    series: {0: {lineWidth: 0, pointSize: 5}}
  • Make the viewWindow setting's min higher than 0 (to hide the 0 points):
    vAxis: { viewWindowMode:'explicit', viewWindow:{ min: 0.5 } },

Monday, February 17, 2014

Change Home Flask-Admin

This was the only thing I was able to get working: http://www.marteinn.se/blog/?p=637

Just putting index.html into an admin/ directory wouldn't work.

There's also an even shorter way to do it, for example:
admin = Admin(app, "G-Cal Manager", index_view=AdminIndexView(name='Home', template='admin/home.html', url='/'))

The example above will change the root url to / instead of /admin and it will use home.html from your templates/admin folder.

To fix the 404 errors on your stylesheets after the change, you will also need to change the static_folder in your flask object:
app = Flask(__name__, static_folder='admin')


Thursday, January 30, 2014

invalid byte sequence for encoding "UTF8" - sqlalchemy

Try putting ?charset=utf8 at the end of your connection string, like:
'mysql+mysqldb://user:pass@111.111.111.111/rf?charset=utf8'

Tuesday, January 28, 2014

SQLalchemy - Print Tables In Database (Show Tables)

from sqlalchemy import create_engine, Table, MetaData

engine = create_engine('mysql+mysqldb://username:password@111.111.111.111/databaseName')
metadata = MetaData(bind=engine)
metadata.reflect(engine)
print metadata.tables.keys()

You could also use that for looping through a list of the tables in the database.

Print Flask-Restless Routes

# show api routes
for route in app.url_map.iter_rules():
print route.rule

Monday, January 27, 2014

pip install MySQLdb

In case you haven't already noticed, the command in the title won't work.

You need to use: pip install MySQL-python

This article explains the correct approach: http://mysql-python.blogspot.com/2012/11/is-mysqldb-hard-to-install.html

Tuesday, January 21, 2014

Friday, January 17, 2014

Best VNC Installation Guide

https://wiki.amahi.org/index.php/Install_VNC_server_on_Ubuntu_Server_12.04

That guide gave me the least amount of hassle when I installed VNC on a server. I started by trying to get it working with 13.10, but had issues with a grey screen when tightvnc connected.

Wednesday, January 15, 2014

"Could not connect to database: Database connection "Mysql" is missing, or could not be created." - CakePHP

I got rid of this error just by creating the database manually. Shouldn't Cake do this?

"Not Found The requested URL /app/install was not found on this server." - CakePHP

Rather than dealing with my .htaccess problems to fix the error, I thought this would be a better approach: http://wwdj.wijndaele.com/getting-started-with-cakephp-without-mod_rewrite/

Unfortunately, it messed up quite a few links on my site.

I ended up solving the problem by changing "AllowOverride" None to "AllowOverride All" in my /etc/apache2/sites-available/default file.

Why the heck is CakePHP using .htaccess files by default anyway? Yii has this more streamlined.

CakePHP Boilerplate

While I was searching for a some good boilerplate to start my first CakePHP project with, I found this: https://github.com/hugodias/cakeStrap

There's also this one, but it has much fewer commits: https://github.com/zynesis/cakephp-boilerplate

Tuesday, January 7, 2014

Exception in thread "main" java.lang.NoClassDefFoundError - Java

Got this error when I was missing a library that came with the build.

If you're getting this error, you may need to move the lib folder from your build directory into the folder of your JAR file.

Friday, January 3, 2014

How To Use Endpoint Parameter - Flask-Admin

You can access the endpoint parameter by using self.endpoint.

class displayResults(BaseView):
@expose('/',methods=('GET','POST'))
def index(self):
print self.endpoint

admin.add_view(displayResults(name='test', endpoint='test2'))

That will print "test2".

Thursday, January 2, 2014

UndefinedError: 'admin_base_template' is undefined - Flask Admin

This happens when you try to render a template like this:
return render_template('admin/upload.html')

You need to use self.render instead:
return self.render('admin/upload.html')