Paul's Programming Notes

Macvlan private mode for container L2 isolation

2026-04-08T23:00:00+00:00

Macvlan is a Docker networking mode that gives each container its own MAC and IP on the network the host port is plugged into. The container shows up on the subnet as its own device, gets a DHCP lease from the router, and talks to the LAN directly instead of going through the Docker bridge.

I have a box with multiple Ethernet ports, each port on a different VLAN. Some containers sit on the port assigned to a guest-style VLAN that can only reach the internet, not other hosts on the LAN. That worked fine in the default bridge macvlan mode while each port only had one service on it. Once I added more services to the same port, I hit a problem I hadn’t thought about: containers sharing a macvlan parent in bridge mode can freely ARP and talk to each other inside the kernel, which defeats the VLAN-level isolation.

Macvlan has a private mode for exactly this. The kernel drops frames going between siblings on the same parent. Outbound traffic via the gateway still works. No switch cooperation, no extra VLANs.

docker network create -d macvlan \
  --subnet=192.0.2.0/24 \
  --gateway=192.0.2.1 \
  -o parent=eth0 \
  -o macvlan_mode=private \
  guest_private

Attach a container to it with a static IP:

services:
  app:
    networks:
      guest_private:
        ipv4_address: 192.0.2.16
networks:
  guest_private:
    external: true

One caveat: the host itself can’t reach its own containers over that interface. That’s true for any macvlan setup, not specific to private mode. For my case it’s fine since the containers just need internet access and to be reachable from other subnets through the router.

More on the kernel feature underneath: Linux kernel macvlan docs. Docker’s guide: Docker macvlan driver.

Python - Inline Script Dependencies With PEP 723

2026-03-28T18:00:00+00:00

Distributing one-off Python scripts has gotten a lot better recently. PEP 723 added a standard way to declare dependencies and a minimum Python version inside a single .py file, and uv makes running them painless. I used this to share a script with the team and it just worked across everyone’s machines.

Before this, you’d end up writing some wrapper shell script that creates a virtualenv, makes sure the right Python version is being used, installs the dependencies, and tries to keep everything in sync. Brew-installed Python was especially bad for this because brew upgrade can silently swap your Python version and break every virtualenv pointing at it. Even version managers like asdf or pyenv don’t really help because the recipient still needs the right version installed through their manager. A lot of ceremony for a single file.

PEP 723 lets you declare all of that inline:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "requests",
# ]
# ///

import requests

resp = requests.get("https://example.com/api/health", timeout=10)
print(resp.json())

The # /// script block is TOML. You don’t have to write it by hand either, uv add --script example.py 'requests' will add it for you.

The recipient just needs uv installed:

uv run check_status.py

uv reads the inline metadata and installs the dependencies in an isolated environment. If the required Python version isn’t on their machine, uv downloads it automatically. No virtualenv, no version manager, no wrapper script.

You can also lock dependencies with uv lock --script example.py for reproducible runs. For bigger tools that span multiple files, uv tool install is the next step up and can install from a private PyPI index.

Switching Open WebUI from Ollama to llama.cpp for Qwen 3.5

2026-03-22T18:00:00+00:00

I finally got to a local LLM setup that feels pretty usable within a 16 GB VRAM constraint (around 40 tok/s).

As far as I can tell, Open WebUI is still the best open source chat interface for local models. I am running it on a Ryzen 5 5600 box with an RTX 5060 Ti 16 GB card, with Qwen3.5-9B GGUF at Q8_0 as the main local model. I started with Ollama because it is the popular default. For Qwen 3.5 though, I hit a few open issues at the time that made llama.cpp easier for my setup: much slower inference than llama.cpp with the same model, long stalls on later turns in a conversation, and broken /no_think handling for Qwen. It is a very new model, and I am sure the Ollama project will get those fixed.

llama-server exposes an OpenAI-compatible API, so Open WebUI mostly treated it like a config swap.

The more interesting part is that Qwen 3.5’s newer hybrid architecture seems to help a lot here. The 9B model feels much better than I would have expected for the size, and this is the first local setup I have had on this machine that felt worth keeping around.

From what I had been reading, thinking mode also did not seem worth the wait for a setup like this, and that matched what I was seeing. Qwen 3.5 defaults to thinking, but for my use it was usually slower, often added 30-60 seconds, and sometimes got stuck in long thinking loops without noticeably improving the answer. What finally worked reliably here was a newer llama.cpp build with LLAMA_ARG_JINJA=1 and LLAMA_ARG_THINK_BUDGET=0.

Most of the other tuning was about using the remaining VRAM well once the 9B model itself had already taken roughly 9 GB. LLAMA_ARG_N_PARALLEL=1, LLAMA_ARG_FLASH_ATTN=1, q8_0 KV cache, LLAMA_ARG_BATCH_SIZE=4096, and LLAMA_ARG_UBATCH_SIZE=1024 were what got the whole setup to around 14 GB and let me spend the rest on context instead of spilling into system memory. The next size up looked much more likely to spill into RAM. I wanted to see what I could get done with this Nvidia card first, since it still seemed like the safest compatibility bet.

For serious work I would still use frontier hosted models. The local setup is useful enough to keep around for narrower cases, especially redacting or cleaning text before sending it to a cloud model.

Full example gist with the commented minimal two-container compose file:

services: llama-server: image: ghcr.io/ggml-org/llama.cpp:server-cuda-b8369 container_name: llama-server restart: unless-stopped environment: # Model - auto-downloads from Hugging Face on first run. - LLAMA_ARG_HF_REPO=unsloth/Qwen3.5-9B-GGUF:Q8_0 - LLAMA_ARG_N_GPU_LAYERS=99 # VRAM tuning for a 16 GB card. - LLAMA_ARG_CTX_SIZE=131072 - LLAMA_ARG_N_PARALLEL=1 - LLAMA_ARG_FLASH_ATTN=1 - LLAMA_ARG_CACHE_TYPE_K=q8_0 - LLAMA_ARG_CACHE_TYPE_V=q8_0 - LLAMA_ARG_BATCH_SIZE=4096 - LLAMA_ARG_UBATCH_SIZE=1024 # Qwen 3.5 non-thinking mode. - LLAMA_ARG_JINJA=1 - LLAMA_ARG_THINK_BUDGET=0 # Qwen-recommended sampling defaults for non-thinking mode. - LLAMA_ARG_TEMP=0.6 - LLAMA_ARG_TOP_K=20 - LLAMA_ARG_TOP_P=0.95 - LLAMA_ARG_MIN_P=0 # Shared gaming PC - unload after 5 minutes idle. - LLAMA_ARG_SLEEP_IDLE_SECONDS=300 - LLAMA_ARG_PORT=11434 - LLAMA_ARG_HOST=0.0.0.0 volumes: - ./models:/root/.cache/llama.cpp ports: - "11434:11434" deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] open-webui: image: ghcr.io/open-webui/open-webui:v0.8.9 container_name: open-webui restart: unless-stopped ports: - "3000:8080" volumes: - ./open-webui:/app/backend/data environment: - ENABLE_OLLAMA_API=false # llama-server exposes an OpenAI-compatible API. - OPENAI_API_BASE_URLS=http://llama-server:11434/v1 - OPENAI_API_KEYS=no-key - WEBUI_AUTH=true depends_on: - llama-server

Kubernetes - Fixing GitOps Drift in Metrics Server

2025-10-15T18:00:00+00:00

I recently got a small fix merged into metrics-server, which powers kubectl top and is used for horizontal pod autoscaling in Kubernetes. It’s not exactly a core component, but most production clusters have it installed.

I’ve been using ArgoCD at work lately for deploying Helm charts through a GitOps flow, and I noticed that the metrics-server APIService resource kept showing as “OutOfSync” even though nothing had actually changed.

The issue was that the Helm template always rendered the insecureSkipTLSVerify field explicitly, but Kubernetes omits it from live resources when it’s false (the API default). This caused ArgoCD to see a constant diff.

The fix was to conditionally render the field using {{- with .Values.apiService.insecureSkipTLSVerify }} so it only appears when set to true. Same approach other projects like KEDA have used.

It’s a tiny fix, but it’s satisfying to have a change merged into something as widely deployed as metrics-server.

PR: kubernetes-sigs/metrics-server#1727

Building a Lightweight Linux Torrent Seeder for Low-End VPS Hosting

2025-10-15T18:00:00+00:00

Earlier this year, I wanted to help seed Linux distribution torrents using cheap VPS servers that offer terabytes of monthly bandwidth for less than $20/year. To maximize cost efficiency, I needed something as memory-efficient as possible.

After trying qBittorrent, rTorrent, and Transmission (via Docker) on a 1GB RAM VPS, I kept running into OOM issues and configuration headaches. Those clients are great, but running them in 1GB RAM doesn’t seem to be a design goal. It’s probably still possible with enough tweaking.

I ended up building distro-seed, a lightweight Go-based BitTorrent seeder using anacrolix/torrent. It’s a Go library that handles all the BitTorrent protocol details while letting you build exactly what you need without the overhead of a full client.

So far, it’s seeded 1.25 TB of Linux distributions.

The project includes an Ansible playbook to set up a fresh Ubuntu VPS for automatic seeding. The whole setup is simple: configure your torrent sources in a YAML file, run the playbook, and let it seed.

Repo: pawl/distro-seed

Linux - Forgetting Exec

2022-08-25T18:11:00+00:00

Today, I came across a bug where a Celery worker wasn’t gracefully shutting down, and it was causing some odd “Connection Refused” errors from requests within the task being run by the worker. It was also shutting down before it could send errors to Rollbar/Sentry for the team to know they need to address it.

This was happening because the entrypoint had a script that effectively did this:

echo "starting celery worker"
celery -A tasks worker

The problem with that entrypoint: it’s not using exec to run the child process. Without exec, it will not forward signals like SIGTERM to the process the entrypoint is waiting on. The Celery worker child process won’t know it needs to shut down gracefully if the SIGTERM is not forwarded to it.

The fixed example would look like this:

echo "starting celery worker"
exec celery -A tasks worker

Linux - Dig's +short Option

2022-07-19T18:11:00+00:00

I learned Linux’s dig has a “+short” option that returns only the IP address for a hostname:

$ dig +short google.com
142.251.32.174

Linux - Ping Not Showing Lost Packets

2022-06-02T18:11:00+00:00

I was recently troubleshooting some packet loss with ping on linux, and I noticed by default ping won’t explicitly show lost packets:

$ ping x.x.x.x
64 bytes from x.x.x.x: icmp_seq=8 ttl=52 time=18.1 ms
64 bytes from x.x.x.x: icmp_seq=11 ttl=52 time=21.2 ms

(notice the skipped 8-10, those were lost packets)

To fix this, you can add run ping with -O:

$ ping -O x.x.x.x
64 bytes from x.x.x.x: icmp_seq=11 ttl=52 time=19.0 ms
no answer yet for icmp_seq=12
no answer yet for icmp_seq=13
64 bytes from x.x.x.x: icmp_seq=14 ttl=52 time=18.9 ms

Related askubuntu.com Thread

Linux - Improving Traceroute Output

2022-06-02T18:11:00+00:00

Usually when I use traceroute without any options, it gets stuck showing output like this:

 *  *  * 
 *  *  * 
 *  *  * 

For more info about why this is happening: Link

To improve this output, try adding -I (traceroute -I) to make it use ICMP ECHO instead of UDP for probes.

If you want better statistics about packet loss, you can also use a tool called mtr which combines traceroute and ping. If you’re on Ubuntu, you can install this with sudo apt-get install mtr.

Docker - WORKDIR Creates Directories

2022-04-01T18:11:00+00:00

Which user do you think owns the src/ directory in this Dockerfile example?:

FROM alpine:3.13.2
RUN adduser -D wendy
USER wendy
WORKDIR src/
COPY --chown=wendy . src/

The answer?: root

This happens because WORKDIR will create directories that don’t exist with the root user.

What if you wanted wendy to be the owner of that src/ directory?

You would need to COPY the src/ directory into place before setting it as the WORKDIR. For example:

FROM alpine:3.13.2
RUN adduser -D wendy
USER wendy
COPY --chown=wendy . src/
WORKDIR src/

More info: Closed GitHub Issue & Docker Code

DIY NAS

2022-03-13T18:11:00+00:00

If you have an old computer with spare hard drives, it might be useful to use it to share files with computers on your network by turning it into a NAS (network attached storage). There are several popular DIY NAS options:

Unraid - Paid, Slackware linux based
FreeNAS (now TrueNAS Core) - FreeBSD based
OpenMediaVault - Debian linux based

I ended up going with OpenMediaVault due to it being Debian based and super popular. To set it up, I followed this incredibly thorough guide: link

I had a few issues that caused it to stop working after restarting.

The first issue was related to the disk being encrypted. OpenMediaVault started in emergency mode due to being unable to access the encrypted disk. Emergency mode unfortunately doesn’t allow SSH access, which means you need to plug a monitor and keyboard into it. You have to make sure you have nofail in the options section of the lines of /etc/fstab starting with /dev/disk. You also need to add nonempty and remove x-systemd.requires=<disk> from the options section of the lines starting with /srv/dev-disk-by-uuid (for mergerfs). I’ve had to redo the removing x-systemd.requires part every time I apply settings from the web UI. More information about this: 1 2 3 4 5 6 7

The second issue involved losing network adapter configuration after restarting. I had to run omv-firstaid and configure the network adapter with a static IP to resolve this. This may have something to do with an empty configuration from the web UI ovewriting the existing working configuration.

Docker - ufw rules ignored

2022-01-01T18:11:00+00:00

I recently learned that docker will ignore ufw (uncomplicated firewall) rules by default. This means that it will still expose ports that are blocked by ufw.

The fix involved adding this to /etc/docker/daemon.json:

{
    "iptables": false,
    "ip6tables": false
}

Then I restarted the docker daemon with sudo systemctl restart docker.

More details

Raspberry Pi Zero Not Connecting to Unifi AP

2021-12-27T18:11:00+00:00

I recently had a lot of trouble getting my Raspberry Pi Zero W 2 to connect to WiFi on my Ubiquiti Unifi AP AC Lite after a firmware upgrade to 5.43.46. It’s a 2.4Ghz-only device, and actually all of my 2.4Ghz-only devices wouldn’t connect.

Apparently something I did set a setting called PMF (Protected Management Frames) to “Required”, and the Pi Zero W 2 doesn’t support that. In theory, this setting should help prevent clients from getting disconnected through de-auth packets from an attacker. However, not all devices support this.

To fix the issue, you need to find the PMF setting under Settings -> Wifi -> <your network> -> Advanced -> Security and change it to “Optional”.

Fixing Memory Leaks In Popular Python Libraries

2021-12-20T02:11:00+00:00

I was recently able to make a minimal example that reproduced a Celery memory leak. The memory leak would happen on the main Celery worker process that’s forked to make child processes, which makes the leak especially bad. Issue #4843 has been around for 3+ years and has 140+ comments, so this one has been causing a lot of problems for Celery users for a while.

The memory leak has been causing a lot of issues at my work too, and I was able to get some help resolving the issue during a work hackathon. My coworker Michael Lazar was able to find the root cause of the issue and make a pull request to fix it in py-amqp (a celery dependency when using RabbitMQ as a broker). The code with the issue was 10 years old!

Here’s what the bug looks like:

try:
    sock.shutdown(socket.SHUT_RDWR)
    sock.close()
except OSError:
    pass

The problem occurs when socket.shutdown fails on an OSError and doesn’t proceed to socket.close to clean up the socket and allow garbage collection to release the memory used for it. The OSError on shutdown can occur when the remote side of the connection closes the connection first.

The fixed example (with separate try/except blocks):

try:
    sock.shutdown(socket.SHUT_RDWR)
except OSError:
    pass

try:
    sock.close()
except OSError:
    pass

I was able to make the same fix to a few other popular Python libraries too:

I also found another way to reduce memory usage of Connections in py-amqp and librabbitmq by changing how active channel IDs are stored.

Update 12/20: Hacker News user js2 pointed out that Python will automatically close the socket when all the references to the socket are gone.

Update 12/23: I got a pull request merged into Kombu with the same memory usage reduction fix I made to py-amqp and librabbitmq. I also opened another pull request to Kombu that should fix a memory leak issue when using Celery with Redis.

Update 12/25: I wrote a section for the Celery docs about optimizing memory usage. I also fixed another leak in Celery that happens when connection errors occur on a prefork worker.

Fn Key At The Software Level

2021-12-04T02:11:00+00:00

I was trying to remap the “Fn” key on my keyboard and learned something interesting: The “Fn” key is not a real key at the software level.

However, you can remap combinations of “Fn” + other keys.

More details: https://askubuntu.com/a/827953

Firefox - arkenfox user.js

2021-12-04T02:11:00+00:00

I recently learned about the arkenfox user.js project, which is an easy way to update Firefox’s settings to improve security and privacy.

It suggests you add your own overrides to the bottom of the file. I added these:

/* re-enable location bar using search ***/
user_pref("keyword.enabled", true); 

/* override recipe: enable session restore ***/
user_pref("browser.startup.page", 3); // 0102
user_pref("browser.privatebrowsing.autostart", false); // 0110 required if you had it set as true
user_pref("places.history.enabled", true); // 0862 required if you had it set as false
user_pref("browser.sessionstore.privacy_level", 0); // 1003 optional [to restore cookies/formdata]
user_pref("network.cookie.lifetimePolicy", 0); // 2801  optional [so cookies persist]
user_pref("privacy.clearOnShutdown.history", false); // 2811
user_pref("privacy.cpd.history", false); // 2812 to match when you use Ctrl-Shift-Del

Comparing Tags On Different Branches With GitHub

2021-11-23T02:12:00+00:00

I recently compared two tags on github using the “compare” dropdown, and it displayed unexpectedly few changes between the tags. However, one of the tags was from a different branch and by default GitHub doesn’t show the other differences between the branches. To see other differences besides the tagged commit compared to the main/master branch, you need to change the “…” in the compare url to “..”.

Now, I’m thinking it’s probably a good idea to avoid making tags from non-master/main branches.

More info: GitHub Docs

Air Quality Monitor

2021-11-23T02:11:00+00:00

I build the air quality monitor from Jeff Geerling’s post:

Python - Singleton

2021-09-21T06:00:00+00:00

I saw some interesting Python singleton code in the wild today. It was used for a Redis connection pool.

It looked like this.

Starting To Learn About GraphQL

2021-08-19T06:00:00+00:00

I’ve started learning about GraphQL.

The best summary I’ve seen so far is the graphene-python “Getting Started” guide.

This post also talks about the dataloader pattern to helps solve n+1 query issues.

Apollo Codegen also looks like a good way to share GraphQL types with the front-end.

Celery - Tuning worker_prefetch_multiplier

2021-08-19T06:00:00+00:00

I came across an interesting article that talks about speeding up the processing of fast running celery tasks by tuning worker_prefetch_multiplier: https://www.lorenzogil.com/blog/2020/03/01/celery-tasks/

The default scheduling strategy in Celery is now “fair”, so part of the article doesn’t apply anymore.

Raspberry Pi Imager - Secret Menu

2021-07-23T06:00:00+00:00

The v1.6 update for the Raspberry Pi Imager added a new shortcut (CTRL + Shift + X) that allows enabling SSH, configuring WiFi, setting hostname, and more.

More Info

I'm on Hackaday!

2021-07-23T06:00:00+00:00

Hackaday posted an article about my Raspberry Pi 1U Server project.

ssh-keygen -R hostname

2021-07-12T00:11:00+00:00

I’ve been manually removing the line from .known_hosts when I saw this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:<key>.
Please contact your system administrator.
Add correct host key in /Users/<user>/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/<user>/.ssh/known_hosts:85
ECDSA host key for <ip> has changed and you have requested strict checking.
Host key verification failed.

Today I found out about this command to remove entries from known_hosts: ssh-keygen -R hostname

Raspberry Pi 1U Server

2021-06-24T02:11:00+00:00

I’ve been working on a power efficient 1U server made with Raspberry Pi’s to take advantage of cheap server colocation options:

More details: Github

Python - Automatic Type Hinting With Monkeytype

2021-05-12T02:11:00+00:00

I’ve been poking around with adding type hints to Python code automatically. Here’s my process so far:

1. `pip install MonkeyType` 1. run tests with `monkeytype run` instead of `python` 1. pytest: `monkeytype run -m pytest` 1. `mkdir stubs` 1. `monkeytype list-modules | xargs -n1 -I{} sh -c 'monkeytype stub {} > stubs/{}.pyi'` 1. `pip install pytype` 1. `python move_files.py` (file in this gist) 1. `mypy <code folder name>` 1. remove duplicate imports added by merge-pyi, fix import sorting issues 1. clean up super long type hints by inlining with params

Python - setup.py project_urls

2021-05-08T02:11:00+00:00

I saw the attrs project has a link to their changelog on pypi:

It turns out that’s controlled by the project_urls section of the setup.py.

More details: Python Docs

Python - virtualenv & Dockerfiles

2021-04-23T23:43:00+00:00

I recently saw a Dockerfile based on the official docker Debian image that was installing dependencies into a virtualenv.

I’m pretty sure using a virtualenv in an official Debian-based Dockerfile is unnecessary, because there’s no system Python to isolate from:

$ docker run -it debian /bin/bash
root@21ca17310079:/# python
bash: python: command not found

git pickaxe

2021-04-23T23:43:00+00:00

I learned about the “git pickaxe” this week, and I used it to find the first commit with a line of code that had been moved between a few different files.

More details: https://remireuvekamp.nl/blog/the-git-pickaxe.html

apt-get install --no-install-recommends

2021-04-19T23:43:00+00:00

I learned about apt-get install’s “–no-install-recommends” flag, and I used it to prevent unnecessary “recommended” packages from getting installed. This helped reduce some unnecesary bloat in a Docker image.

More details: https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends

WLED

2021-03-14T01:43:00+00:00

Today I learned about the amazing WLED project: https://github.com/Aircoookie/WLED

All you need to do is flash their binary to an ESP32 and it will bring up a WIFI access point that allows you to control the LEDs through an app or web page.

Event Driven Architecture Talk

2021-01-22T05:21:00+00:00

I really liked this talk by Martin Fowler about Event Driven Architecture: https://youtu.be/STKCRSUsyP0

It’s interesting to think about Git as an example of event sourcing.

Microservice Communication

2021-01-20T18:21:00+00:00

I’ve been learning more about async communication between microservices and came across this article that talks about some of the trade-offs between sync and async communication: https://dzone.com/articles/patterns-for-microservices-sync-vs-async

Summary

Sync

Pros

“synchronous calls are simpler to grasp, debug and implement”

Cons

“A temporary burst at one component can flood other services with requests.”
Risk of Cascading Failures (domino effect of failures if one service experiences an otuage)
tighter coupling (requires endpoint versioning in the owning service)

Async

Pros

“removes the need to wait for a response thereby decoupling the execution of two or more services”
“deals better with sporadic bursts of traffic”

Cons

“Asynchronous systems tend to be significantly more complex than synchronous ones.”
“consumers need to adapt to work with an asynchronous system”
“the message bus the Achilles heel of the system as it remains a central point of failure”
Eventual Consistency (potentially out of date data)

Python - Mypy

2021-01-08T18:21:00+00:00

My new job has me working on a larger codebase than previous jobs, and it’s also my first time using mypy.

I’m starting to understand why Guido’s work on mypy had a lot to do with Dropbox’s Python 3 migration. Dropbox wrote an article with details on how they used mypy on their “4 million lines of Python”. With that much code, I can understand why they needed to treat code like “cattle, not pets”.

Python 3 disallows some comparisons with None types, makes huge changes to strings/bytes, and has many other changes involving types. When your codebase is massive, you have to reach for automated tooling to consistently find and prevent those bugs. With the same tooling you can also prevent entire categories of other bugs from reaching production. The type hints can also be helpful documentation. For those reasons and more, I think implementing type hints on a large codebase like Dropbox’s will definitely be worth it in the long run.

It has me wondering if I should have been taking the extra time to use type hints and checking (or a statically typed language) this whole time. Was going without type hints one less distraction? Or will be the price be paid in maintenance difficulties and bug fixes later?

At this point, not much of the Python ecosystem has type hints (not even Python 2 compatible comment-style type hints). I’m starting to think it would be a good use of time to work on changing that.

Ruby - Jekyll

2020-12-25T08:21:00+00:00

I migrated this blog from blogger/blogspot to a static site generated with Jekyll and hosted with Netlify.

This blog post describes the process: http://joshualande.com/jekyll-github-pages-poole

Initially, I started using Python’s Pelican, but pelican-import (pelican’s tool for migrating from blogger) doesn’t work as well as jekyll-import. It turned comments into posts and threw exceptions while processing draft posts without content. Also, the first docs that show up on google for Pelican aren’t the latest docs. This causing issues when I followed the old docs that said to use Python 2.7, but Pelican only supports Python 3.6+ now.

"A (multi-) monorepo setup with Git Submodules"

2020-12-25T00:21:00+00:00

https://www.jannikbuschke.de/blog/git-submodules/

Arduino - Oven Thermometer

2020-12-20T20:38:00+00:00

Parts:

HiLetgo DC 3-5V MAX6675 Module + K Type Thermocouple Temperature Sensor
UCTRONICS 0.96 Inch OLED Module 12864 128x64 Yellow Blue SSD1306 Driver I2C
ESP32

Code: https://gist.github.com/pawl/73a1ccfbf2b2be6f934e651b39c51082

Linux - Sparse Files

2020-12-12T23:54:00+00:00

Found a massive faillog/lastlog file recently and thought it was responsible for using up all my disk space. Turns out it's actually a sparse file and doesn't take up that much physical space: https://unix.stackexchange.com/a/530157

Arduino - Coffee Roasting Monitor

2020-12-12T23:42:00+00:00

Parts:

Arduino Uno
2x MAX6675 w/ thermocouple

Code: https://gist.github.com/pawl/5ed1f61e0b0b27aa839fc2b4499a92c2

Django - Duplicate Unformatted Logs

2020-09-04T20:27:00+00:00

Recently I worked on a bug that was causing duplicate unformatted log messages to appear in a Django app's logs. I made a repository that demonstrates the issue: https://github.com/pawl/django_duplicate_unformatted_logs_example

The problem was caused by an accidental call to logging.info (without using logging.getLogger to get a specific logger) while the root logger isn't already configured.

The solution ended up being to get rid of the accidental calls to logging.info and configuring the root logger to prevent it from accidentally happening again. I go into more details in that repo.

Vue - vue-custom-element

2020-05-12T01:46:00+00:00

Vue-custom-element seems like the best way to integrate vue into an existing application that uses server side templates.

This approach allows you to use vue components (built with the build pipeline) in your existing html without making your entire application a SPA.

Initially I started following this guide: https://robinverton.de/blog/2018/06/22/django-vue.js-integration-as-a-widget/

Vue-custom-element turned that code into 3 lines and allowed using components like normal (without the data attributes).

This approach seems much better than: https://vsupalov.com/vue-js-in-django-template/ Which doesn't allow for using a build pipeline for babel, linting, testing, single page components, live reload, modules with import/require, etc.

Targeting your builds for web components might do the same thing: https://cli.vuejs.org/guide/build-targets.html#web-component It says it's not compatible with IE11, but that might not be a big deal in most situations.

Django - Requiring Prefetching

2020-05-06T03:21:00+00:00

https://github.com/pawl/django_require_prefetch_select_related_example

I made a repository to try to figure out the best way to require prefech_related/select_related when fetching a model's related objects. I also want to figure out a good pattern for avoiding duplicated prefetch_related/select_related for multiple places in a codebase that need the related objects eager loaded in the same way.

Nginx - WebDAV

2020-04-29T23:44:00+00:00

Just learned Nginx has WebDAV support: https://nginx.org/en/docs/http/ngx_http_dav_module.html

It's a lot faster than the Python package I was using: https://github.com/mar10/wsgidav

Wsgidav is great though. It's super customizable and the documentation is great too.

Django - ModelChoiceField queryset caching

2020-04-23T03:58:00+00:00

It would be nice to know a better way to cache the ModelChoiceField’s queryset when it’s used in a form that runs “is_valid()” in a loop (like Formsets do). The best way I know how at the moment is by not using a ModelChoiceField at all. The solution requires using a ChoiceField, running the query for choices outside of the loop, then passing the choices into the form to override the ChoiceField choices.

Here’s an example:

# See full example: https://github.com/pawl/django_modelchoicefield_caching/blob/master/myapp/views.py#L31-L51 for song in playlist: form_data = {'title': song["title"], 'artist': song["artist"]} song_form = forms.SongFormWithModelChoiceField(data=form_data) song_form.is_valid() # runs a query to get the ModelChoiceField queryset each time print('ModelChoiceField - query count AFTER validating all songs:', len(connection.queries)) # 5 queries # query for choices outside of the loop to prevent unnecessary queries artist_choices = [(artist.pk, artist.name) for artist in models.Artist.objects.all()] for song in playlist: form_data = {'title': song["title"], 'artist': song["artist"]} # pass choices into the Form song_form = forms.SongFormWithChoiceField( artist_choices=artist_choices, data=form_data) song_form.is_valid() print('ChoiceField w/ choices passed in - query count AFTER validating all songs:', len(connection.queries)) # 6 queries (only 1 more query!)

Django API Forms

2020-04-23T03:06:00+00:00

Just found a really cool library for using Django's built-in forms to validate JSON: https://github.com/Sibyx/django_api_forms

I made a small example with it to show how it works with ModelChoiceFields: https://github.com/pawl/django_api_forms_modelchoicefield_example

Seems like a more lightweight version of DRF's serializers (but only for deserializing and validating) and with a similar API to Django's built-in forms.

Learning Vue Through Terrible Pull Requests

2020-04-18T18:58:00+00:00

I made a pull request asking to "remove the unused public/index.html" file from django-vue-template: https://github.com/gtalarico/django-vue-template/pull/53

Well, the author responded and it turns out it's definitely used (by vue magic): https://cli.vuejs.org/guide/html-and-static-assets.html

Oops, definitely should have googled "vue public/index.html" before making that pull request. I made another pull request to add a link about it to the django-vue-template docs: https://github.com/gtalarico/django-vue-template/pull/54

Server Side Templates Vs Front End Framework

2020-04-12T04:57:00+00:00

These Reddit comments have some of the best descriptions I've seen for the type of scenario where you'd use a front end framework over server side templates:

"Server side templating can work really well when the website is static. The problem is that as you add more and more dynamic elements, you either have to go to the server and re-render everything with every change (which can be slow), or you can make the changes on the client side with something like jQuery. But the latter option presents a challenge because you'll eventually find yourself duplicating functionality.

For example, imagine a to-do list website that you render on the server with a templating language. It fetches the user's to-do items from the database and then renders the list. Now let's say you want to let a user add a new item without re-rendering the entire page. You could use AJAX to tell the server to add the new item, but now you need to update your UI list accordingly. You could use jQuery to construct a new DOM element and append it, but now that list element exists in two different places: in your jQuery code and in your template on the server. If your website becomes very dynamic, this situation can get really unwieldy from a programming perspective.

Both Angular (from what I've read) and React (from personal experience) are highly suitable for creating single page apps, with the goal of creating a dynamic, fast, and maintainable website. There are other benefits, as well as downsides, to using Angular and React, but I hope this clears it up a little bit. Both of them make server side templating unnecessary by moving UI logic/rendering from your server to the client's browser. So a user who visits your React powered to-list website would get the React code as a JS bundle, it would make an AJAX call to your server to get just the data, and then it would render the UI based on the data." -/u/VertiGuo (https://www.reddit.com/r/node/comments/6t22cr/back_end_templating_engine_or_front_end_framework/dlhukkf/)

"for example putting together a form, and handle the response is easy, but if you have to manage a list of items where the user can add/remove items, or change the schema of items based on a selector (e.g. phone type or address) you have to split the logic, and put some on client side scripts, which can become very messy. not to mention live updates like chat, comments etc.

creating a separate backend and frontend solves this problem, as the backend's responsibility will be only manipulating the database, retrieve data and run jobs providing an api and the frontend's will be the only the representation of these data, providing the user interface. this separation lets developers use different technologies and languages on the two side, as well as specializing in these. for the beginning this approach is a bit hard to learn, but it will worth it on the long run." -/u/bdvx (https://www.reddit.com/r/webdev/comments/cbrdte/html_templating_vs_spa/etifiar/)

Integrating Django and Vue.js

2020-04-08T01:03:00+00:00

This is a great guide for including vue bundles in server side templates: https://medium.com/@rodrigosmaniotto/integrating-django-and-vuejs-with-vue-cli-3-and-webpack-loader-145c3b98501a

Otherwise, if you're starting a fresh project with a separate front-end, this is a good template: https://github.com/gtalarico/django-vue-template/

The deepest reason why modern JavaScript frameworks exist

2020-04-04T02:16:00+00:00

Another really good article about why using a front-end framework is a good idea:
https://medium.com/dailyjs/the-deepest-reason-why-modern-javascript-frameworks-exist-933b86ebc445

Make your life easier with Vue.js

2020-04-02T16:37:00+00:00

This is a really great article that shows how vue can make dynamic forms so much better: https://dev.to/tsanak/make-your-life-easier-with-vuejs-4mj5

Django - RelatedManager.set not removing models

2020-02-14T03:36:00+00:00

In my opinion, one of django's biggest gotchas is using `RelatedManager.set` with models that have non-nullable ForeignKey fields. ### Example Models.py (with a one-to-many relationship and a non-nullable ForeignKey): ``` class Reporter(models.Model): name = models.CharField(max_length=255) class Article(models.Model): title = models.CharField(max_length=255) reporter = models.ForeignKey(Reporter, on_delete=models.CASCADE) ``` Guess how many articles are on the reporter after you do this?: ``` reporter = Reporter.objects.create(name="paul") # add 2 articles to the reporter article_1 = reporter.article_set.create(title="blah") reporter.article_set.create(title="blah2") # set only 1 article on the reporter reporter.article_set.set([article_1]) ``` I expected just one article to be on the reporter. Nope: ``` reporter.article_set.count() 2 ``` How do the docs say it should work? >This method accepts a clear argument to control how to perform the operation. If False (the default), the elements missing from the new set are removed using remove() and only the new ones are added. Well, crap, what's going on? It turns out you need to read the docs for `RelatedManager.remove` for a hint: >For ForeignKey objects, this method only exists if null=True. So, it turns out that `set()` is using `remove()` and `remove()` is only available on models with ForeignKeys that are `null=True`. What does `set()` do with things it needs to `remove()` when `remove()` isn't available? Nothing...

Django - ChoiceField Display Text

2020-01-14T01:57:00+00:00

I made a repo to demonstrate how to get a django ChoiceField's current selection's display text: https://github.com/pawl/django_choicefield_display_example/

I came across a few other examples, but they didn't do a good job of displaying the initial value or handling choices with integer values.

Python - Pipenv to pip-tools

2019-12-20T17:31:00+00:00

I've been using Pipenv for the last few months and my biggest issue is that "--keep-outdated" has been broken in the latest release (2018.11.26) for a while. I've needed to install Pipenv from the master branch to make it functional. However, the last time I used "--keep-outdated" from the master branch, it wouldn't automatically update the hash of the dependency being updated.

Updating specific requirements is something I need to do pretty often, and it's not fun to explain all the Pipenv quirks to the team.

Pip-tools looks like it does everything I need and has fewer quirks, so I ended up making the switch.

Pipenv uses pip-tools under the hood, so the migration to pip-tools was very smooth. The migration process was:

Copy the dev-packages and packages sections of the Pipfile to their own requirements.in files.
Run pip-compile
Copy over the specific versions and hashes from the Pipfile.lock to the generated requirements.txt.

I did have a small issue where updating a specific package with pip-tools removed a bunch of dependencies from the requirements.txt unexpectedly, but running pip-compile with "--rebuild" fixed it.

HTML5 - video playsinline

2019-07-25T04:05:00+00:00

I added "autoplay muted" to my <video> tag to make a video autoplay in a carousel. It worked on desktop Chrome or Firefox, but didn't work on iOS Chrome or Safari.

I learned about the "playsinline" fix from here: https://webkit.org/blog/6784/new-video-policies-for-ios/

Adding "playsinline" to my video tag fixed it.

ESP32 Plant Sensor

2019-07-12T01:43:00+00:00

I set up a ESP32 houseplant soil water + temperature + humidity + light sensor that sends me a daily status update message.

Here’s the code for it: https://gist.github.com/pawl/6a408afde1411c6b1f58980e3d1dff83

AWS - redirecting domain to url using a 302 redirect (without running a server)

2019-07-11T00:26:00+00:00

I wanted to make a domain name (heckingoodboys.com) redirect to a multisubreddit for dog pictures, but I didn't want to run a web server for it.

Here's what I did:

Purchase the domain using Route53.
Create two public s3 buckets (www.heckingoodboys.com and heckingoodboys.com)
Enable "Static website hosting" on www.heckingoodboys.com and redirect to heckingoodboys.com.
Enable "Static website hosting" on heckingoodboys.com, select "use this bucket to host this website", and use routing rules similar to this:
<RoutingRules>
<RoutingRule>
    <Redirect>
      <Protocol>https</Protocol>
      <HostName>www.reddit.com</HostName>
      <HttpRedirectCode>302</HttpRedirectCode>
      <ReplaceKeyPrefixWith>user/heckingoodboys/m/heckingoodboys/</ReplaceKeyPrefixWith>
    </Redirect>
</RoutingRule>
</RoutingRules>
Back to Route53 - Create an A record for both www.heckingoodboys.com and heckingoodboys.com using the alias to their respective buckets. (this will be the first option in autocomplete)

For more details: https://medium.com/@P_Lessing/single-page-apps-on-aws-part-1-hosting-a-website-on-s3-3c9871f126

Why not just use a CNAME from www.heckingoodboys.com to heckingoodboys.com? AWS says they don't charge for aliases, but they do charge for CNAMEs. So, I used an alias to a bucket instead.

Django - When To Use Signals?

2018-02-08T04:18:00+00:00

According to this blog post, almost never: https://lincolnloop.com/blog/django-anti-patterns-signals/

Django - Custom Storage Backends

2018-02-08T04:17:00+00:00

I just learned about the custom storage systems in Django.

What can you do with custom storage systems?:

"Django abstracts file storage using storage backends, from simple filesystem storage to things like S3. This can be used for processing file uploads, storing static assets, and more." -https://tartarus.org/james/diary/2013/07/18/fun-with-django-storage-backends

Sphinx Search - Lessons Learned

2018-01-20T02:26:00+00:00

Here are a few things I've learned while working on a project that uses Sphinx search:

It's important to know the difference between fields and attributes. Attributes are basically unindexed columns and you should try to avoid filtering only on these columns. Fields support full text search.
It supports its own custom binary protocol and the MySQL protocol (recently they also added a HTTP API). When you see "listen = localhost:9306:mysql41" in the config, that means it's listening for MySQL protocol traffic on port 9306.
https://github.com/a1tus/sphinxapi-py3 appears to be the best Python client for the binary api at the moment. This doesn't support INSERTing things into the index (you'll need to use the MySQL protocol for that).
The version of sphinxapi-py3 on pypi is a fork with just a few minor fixes and appears to be safe.
It does not match partial words by default. Turning on partial matching can also increase the size of your index dramatically. You can also limit the fields that support partial matching with the "infix_fields" and "prefix_fields" setting.
Stemmers aren't turned on by default. So, searching for "dog" will not match "dogs".
Most special characters ($, @, &, etc) are ignored by default. You will need to add them to charset_table if you want them to be searchable.
Ruby's thinking-sphinx looks much more battle tested than all of the Python binary api clients: https://github.com/pat/thinking-sphinx
You will need to use a real-time index if you want to INSERT/DELETE records immediately.
If you're using a real-time index, you will probably need to increase the "rt_mem_limit" from its default of 128mb. If this limit is too low, you'll see a high number of "disk chunks" when you run the "SHOW INDEX rtindex STATUS" query. More info: http://sphinxsearch.com/blog/2014/02/12/rt_performance_basics/
You have to use a special dialect if you want to use SQLAlchemy with sphinx: https://github.com/conversant/sqlalchemy-sphinx
This appears to be the best Dockerfile for sphinx: https://github.com/leodido/dockerfiles

I probably won't be using Sphinx search for any new projects. Elasticsearch seems preferable these days.

MySQL - Duplicate Errors & Trailing Whitespace

2017-12-12T01:36:00+00:00

I had a unique constraint on a VARCHAR column and I inserted two rows with the following values:

"name" (without trailing whitespace)
"name " (with trailing whitespace)

To my surprise, I got a duplicate error on that 2nd insert. It turns out that MySQL ignores that trailing whitespace when it makes comparisons.

The MySQL docs say this: "All MySQL collations are of type PAD SPACE. This means that all CHAR, VARCHAR, and TEXT values are compared without regard to any trailing spaces. “Comparison” in this context does not include the LIKE pattern-matching operator, for which trailing spaces are significant." (https://dev.mysql.com/doc/refman/5.7/en/char.html)

The solution? You should probably be trimming trailing whitespace in your API endpoints and on your front-end.

Gevent + Requests Performance With verify=True/False

2017-08-11T00:48:00+00:00

If you use gevent with requests.get on a HTTPS URL with the default verify=True enabled, you'll see almost 2x longer execution times than with verify=False.

I made a script to test: https://gist.github.com/pawl/56100a4ef958374a433840be8037b11b

Here are the results:

verify=True took: 40.3454630375 secs
verify=False took: 39.3803040981 secs
gevent verify=True took: 2.23735189438 secs
gevent verify=False took: 1.58263015747 secs

I suspect that gevent is having trouble using pyopenssl concurrently because it's a C library.

Backing up or dumping a memcached server

2017-06-28T03:35:00+00:00

I was needing to move from an old cache server to a larger one, but I wanted to do it without flushing cache.

The first thing I came across was this "memcached-tool" which has a dump command: https://github.com/memcached/memcached/blob/master/scripts/memcached-tool

There's another article that mentions using memdump and memcat: http://www.dctrwatson.com/2010/12/how-to-dump-memcache-keyvalue-pairs-fast/

Unfortunately, those methods only dumped a few mb of data. This post explains why: https://stackoverflow.com/a/13941700

You can only dump one page per slab class (1MB of data)

So, I ended up writing a script that loops through the expected cache keys, gets the data in cache, then sets the data in the new cache server.

Microservice Best Practices

2017-06-27T05:59:00+00:00

http://www.vinaysahni.com/best-practices-for-building-a-microservice-architecture

Gunicorn - "Resource temporarily unavailable"

2017-06-13T05:14:00+00:00

Are you seeing this error in your logs while your server is under high load?:

[error] 10#0: *14843 connect() to unix:/tmp/gunicorn.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 96.44.145.186, server: , request: "GET / HTTP/1.0", upstream: "http://unix:/tmp/gunicorn.sock:/", host: "45.55.46.84"

I ended up making an example dockerfile with nginx + gunicorn + flask to reproduce this problem: https://github.com/pawl/somaxconn_test

Bumping the "net.core.somaxconn" setting ended up fixing it.

SQLAlchemy - empty lists + in_() causing crazy queries (before 1.2.0)

2017-03-18T02:31:00+00:00

Before SQLAlchemy 1.2.0, if you use an empty list with in_(), it will emit some crazy SQL that will query your entire table. The best solution is probably to upgrade to SQLAlchemy 1.2.0.

# demonstrates the issue fixed in: https://bitbucket.org/zzzeek/sqlalchemy/issues/3907 from sqlalchemy import create_engine, Column, Integer from sqlalchemy.orm import scoped_session, sessionmaker from sqlalchemy.ext.declarative import declarative_base engine = create_engine('mysql://root@localhost/test?charset=utf8mb4', convert_unicode=True, echo=True) session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine)) Base = declarative_base() Base.query = session.query_property() class Post(Base): __tablename__ = 'posts' id = Column(Integer, primary_key=True) #Base.metadata.drop_all(engine) Base.metadata.create_all(engine) # create new rows if database is empty for x in range(10): session.add(Post()) session.commit() # empty lists trigger insane queries before 1.2.0 Post.query.filter(Post.id.in_([])).all() """ SELECT posts.id AS posts_id FROM posts WHERE posts.id != posts.id """

The Robustness Principle

2017-03-14T06:05:00+00:00

I learned about this at a talk called "Implementing Evolvable APIs" at SXSW: https://en.wikipedia.org/wiki/Robustness_principle

For example, making an API that throws errors when an unexpected parameter is provided is a bad idea. What if you need to make changes to the client to add the new parameter? You will need to make sure you deploy the code on the server side first, otherwise it will cause errors.

SQLAlchemy - DISTINCT, LIMIT, or OFFSET Causing Subqueries

2017-02-15T02:26:00+00:00

This section at the bottom of The Zen of Eager loading is really important:

When using joined eager loading, if the query contains a modifier that impacts the rows returned externally to the joins, such as when using DISTINCT, LIMIT, OFFSET or equivalent, the completed statement is first wrapped inside a subquery, and the joins used specifically for joined eager loading are applied to the subquery. SQLAlchemy’s joined eager loading goes the extra mile, and then ten miles further, to absolutely ensure that it does not affect the end result of the query, only the way collections and related objects are loaded, no matter what the format of the query is.

I made an example to illustrate this here: https://gist.github.com/pawl/cb57e0ddbdd0b2e64b75e94116873367

On MySQL this can be responsible for some really poor query performance, because it can cause it to use temporary tables and filesort.

The best way I've found to prevent the subqueries is by first querying for the ids only, then running another query that includes all relations. For example:

ids = session.query(Product.id).limit(20)
Product.query.filter(Product.id.in_(ids))

SQLAlchemy - "Base.query = db_session.query_property()"

2017-02-13T00:23:00+00:00

I just saw this line in the SQLAlchemy flask patterns example:

Base.query = db_session.query_property()

I went searching for what it does and found this: https://eoyilmaz.blogspot.com/2014/01/i-feel-so-much-hungry-about-posting.html

Now I know how Flask-SQLAlchemy made queries work with Model.query instead of session.query(Model).

SQLAlchemy - Lost connection to MySQL server during query

2017-02-11T18:55:00+00:00

Here's was my situation:

The database was set up behind behind an AWS ELB and HAProxy.
The idle connection timeout on the ELB was set to 60 mins.
All the relevant timeouts on HAProxy seemed to be set to 60 mins too.
The pool_recycle in SQLAlchemy was set to 30 mins.
I was still seeing the occasional "Lost connection to MySQL server during query" when small queries were running after the connection had some time to sit around.

The solution ended up being setting my pool_recycle down to 5 mins, but I'm still not sure what was causing connections to time out after 5 mins.

There are definitely other things that can cause this problem too. For example, it can happen if your data exceeds max_allowed_packet. See this page for more details: https://dev.mysql.com/doc/refman/5.7/en/error-lost-connection.html

Most of this also applies for "MySQL server has gone away".

You should also make sure your pool_recycle is set lower than your 'interactive_timeout' and 'wait_timeout' properties in the mysql config file to the values you need.

SQLAlchemy - Unexpected Lazy Loading

2017-01-27T01:32:00+00:00

If you're seeing unexpected lazy loading on a lazy="joined" relationship in SQLAlchemy, it might be because you're accessing those relationships after you've already run session.commit(). By default, session.commit() will expire the data on your relationships, meaning it will try to fetch it again next time you try to access those attributes.

The relevant section of the docs for session.commit():

By default, the Session also expires all database loaded state on all ORM-managed attributes after transaction commit. This so that subsequent operations load the most recent data from the database. This behavior can be disabled using the expire_on_commit=False option to sessionmaker or the Session constructor.

Python - David Beazley's "Understanding The Python GIL"

2017-01-20T01:05:00+00:00

This talk explains a lot about Python's multithreading: http://www.dabeaz.com/python/UnderstandingGIL.pdf

In Python 2.7, it knows to switch to different thread whenever it starts waiting on IO. And, it will check if it needs to switch threads every 100 bytecode instructions.

SQLAlchemy - Is is necessary to commit after session.execute?

2017-01-17T04:50:00+00:00

The docs for session.execute say:

Execute a SQL expression construct or string statement within the current transaction.

So, the answer is yes, you need to issue a commit after running session.execute().

Here’s a code example along with the output from “echo=True” showing it begins the session but doesn’t end it unless you run session.commit():

from sqlalchemy import create_engine from sqlalchemy.orm import scoped_session, sessionmaker engine = create_engine('mysql://root@localhost/test') db_session = scoped_session(sessionmaker(bind=engine)) db_session.execute(''' INSERT INTO user (first_name, last_name, username, email) VALUES ("blah", "blah", "blah", "blah@blah.com"); ''')

Global .gitignore

2016-11-27T02:37:00+00:00

https://help.github.com/articles/ignoring-files/#create-a-global-gitignore

SQLAlchemy 1.1.4's New "server_side_cursors" Option

2016-11-18T05:43:00+00:00

Before SQLAlchemy 1.1.4, if you wanted to stream your MySQL query using server side cursors, you would need to run your query using "execution_options(stream_results=True)" and pass SSCursor into create_engine's connect_args. Now, all you need to do is pass "server_side_cursors=True" into create_engine and it will automatically stream the results for your select queries.

http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#server-side-cursors

Is it necessary to run SQLAlchemy's session.remove()?

2016-10-23T22:21:00+00:00

If you're using flask-sqlalchemy, then the answer is no - it's already doing it for you when the request finishes: https://github.com/mitsuhiko/flask-sqlalchemy/blob/2.1/flask_sqlalchemy/__init__.py#L823

If you're using scoped_session and not explicitly running session.remove(), then your connections will only be returned to the pool after your thread finishes and garbage collection occurs. It's not a good idea to leave this to garbage collection, because you can't guarantee your connections will be returned to the pool when your application is busy. This will often lead to errors like this: "TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30"

The solution is to make sure session.remove() runs when the work finishes (like flask-sqlalchemy does), as described here: http://docs.sqlalchemy.org/en/latest/orm/contextual.html#using-thread-local-scope-with-web-applications

Another alternative is not making the session global and using a context manager, as described at the bottom of this section: http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it However, this can make things difficult when you need to access the query results outside of the context manager. If you access the query results outside of the context manager without running session.expunge_all(), you'll see all kinds of errors like this: "DetachedInstanceError: Instance <ReportingJob at 0xa41cd8c> is not bound to a Session; attribute refresh operation cannot proceed"

Another option is setting autocommit to True, but that has quite a few gotchas. Turning on autocommit will acquire connections from the engine on an as-needed basis and return them immediately after their use. You will have to explicitly start transactions with session.begin() if you enable it. It seems like this setting should be called "autotransaction" instead. It's also not very efficient, because it needs to get and return a connection each time you run a query.

PHP - max_execution_time doesn't work for socket operations

2016-09-02T23:07:00+00:00

In new relic transaction logs, I was seeing some PHP requests last as long as an hour. It turns out that time spent waiting on sockets doesn't apply toward max_execution_time, and loops that involved waiting on sockets could end up taking a really long time without timing out.

PHP docs have this to say about max_execution_time:
"The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real."

The solution ended up being setting request_terminate_timeout in php-fpm.

Python - Cachetools LRUCache KeyError

2016-08-27T22:34:00+00:00

If your application is threaded and you're getting a "KeyError" while using the non-decorator version of cachetool's LRUCache, then you need to put whatever is manipulating the cache object inside of a lock. Also, since LRUCache is modified when values are gotten from it, you will also need to make sure you're locking when you get values from cache too. If you can use the decorator version of LRUCache, that's preferred since it has built-in locking.

Here's an example of the error:

And an example of the fix: https://bitbucket.org/zzzeek/dogpile.cache/pull-requests/32/add-a-cachetools-lru-lfu-in-memory-backend/diff#comment-22242704

Javascript's Round vs PHP's Round

2016-06-22T06:33:00+00:00

Today I learned that by default PHP rounds differently than javascript.

PHP (using the default PHP_ROUND_HALF_UP)

php > echo round(-1.5);
-2

"Round val up to precision decimal places away from zero, when it is half way there. Making 1.5 into 2 and -1.5 into -2."

https://secure.php.net/manual/en/function.round.php

Javascript

Math.round(-1.5);
-1

"For negative numbers, if the decimal portion is exactly -0.5, the return value is the smallest integer that is greater than the number."
https://msdn.microsoft.com/en-us/library/5cza0web(v=vs.94).aspx

Django-Moderation

2016-05-21T22:41:00+00:00

It would be nice if there were a sqlalchemy/flask equivalent to this: https://github.com/dominno/django-moderation

The equivalent might use sqlalchemy_utils for its generic relationships: https://sqlalchemy-utils.readthedocs.io/en/latest/generic_relationship.html

Modified Preorder Tree Traversal

2016-05-20T03:25:00+00:00

This was a great explanation of Modified Preorder Tree Traversal (or MPTT): https://www.sitepoint.com/hierarchical-data-database-2/

A common use-case for MPTT is categorization data, like with the way django-oscar uses django-mptt: https://github.com/django-mptt/django-mptt

Datatables - Uncaught TypeError: Cannot read property 'style' of undefined

2016-05-09T00:52:00+00:00

If you see this error in datatables: "Uncaught TypeError: Cannot read property 'style' of undefined"

It's likely that you're missing a header column in your table. Make sure you have the same number of <th> elements as items in the "columns" section of your datatables initialization.

Select2 Replacement

2016-05-08T23:52:00+00:00

This selectize.js library looks like a good replacement for Select2: https://selectize.github.io/selectize.js/

Javascript Data Tables

2016-05-05T04:04:00+00:00

https://fooplugins.github.io/FooTable/index.html - good, no inline editing though
https://datatables.net/ - editable plugin requires purchase
https://github.com/mleibman/SlickGrid - unmaintained
https://handsontable.com/ - responsiveness doesn't work very well
https://github.com/mindmup/editable-table - only supports editing
https://github.com/David-Mulder/paper-datatable - pretty, but doesn't seem very battle-tested
https://www.ag-grid.com/ - no tests
https://github.com/angular-ui/ui-grid - requires angular
http://ng-table.com/ - requires angular
http://backgridjs.com/ - requires backbone
https://github.com/daniel-nagy/md-data-table - pretty, but requires angular
https://www.dynatable.com/ - not editable
http://www.jtable.org/ - last updated in 2014, styling looks dated
https://datazenit.com/static/sensei-grid/examples/ - simple, editable

Falsehoods programmers believe about time and time zones

2016-04-19T05:31:00+00:00

This article has a lot of great lessons learned about timezones: http://www.creativedeletion.com/2015/01/28/falsehoods-programmers-date-time-zones.html

SQLAlchemy - JSON not changing on commit

2016-04-10T23:36:00+00:00

If your changes to a JSON, JSONB, or HSTORE field aren't saving after you commit, then you need to wrap your field in the MutableDict shown here: http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#sqlalchemy.dialects.postgresql.HSTORE

SQLAlchemy - AttributeError: type object 'JSONB' has no attribute 'lower'

2016-04-10T05:59:00+00:00

This error happened because I mistyped "db.column" instead of "db.Column" when I was creating my database model using SQLAlchemy.

This might be your problem if db.create_all() isn't creating all the columns you expected.

PHP - Problems With Long Running Processes

2016-03-19T07:18:00+00:00

Great article about the problems with using PHP for long running processes: https://software-gunslinger.tumblr.com/post/48215406921/php-is-meant-to-die-continued

Describes the main reason why I started using Python for long-running jobs.

Ansible Vault - storing secrets in repos

2016-03-04T03:52:00+00:00

When others are deploying a project for you, it's easy for mistakes to be made when secrets must be updated in environmental variables. Ansible-vault takes a different approach and encrypts the secrets - allowing you to store the secrets in your repo.

To encrypt a file: ansible-vault encrypt secrets.py
To decrypt a file: ansible-vault decrypt secrets.py

More documentation is available here: http://docs.ansible.com/ansible/playbooks_vault.html

Python - isort, useful tool for sorting python imports

2016-03-04T03:44:00+00:00

http://timothycrosley.github.io/isort/

I saw it in the django coding style documentation: https://docs.djangoproject.com/en/1.9/internals/contributing/writing-code/coding-style/

To make an entire project's imports easier to read, all you need to do is:

pip install isort
isort -rc .

There's also a sublime plugin for it here: https://github.com/thijsdezoete/sublime-text-isort-plugin#install

git commit --amend -C HEAD

2016-02-19T04:29:00+00:00

Ever committed something and needed to make changes later? Just rebase and squash it, right? Or maybe "git reset --soft HEAD~1" and commit again?

There's an even better solution: "git commit --amend -C HEAD"

It will add the combine your commit with your last commit. If you already pushed, you will need to force push your change.

pip install -e .

2016-02-19T04:19:00+00:00

I've wasted a lot of time running "python setup.py install" before testing my changes to flask-admin. It turns out you can pip install a project as "editable", which points the install toward your local directory instead of copying where the rest of your python modules are.

To install a project as editable, navigate to the repo and run "pip install -e .".

https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs

Python - Dropping Into pdb From Nose

2016-02-14T01:09:00+00:00

This document describes how to drop into pdb when your Python nose tests fail: http://nose.readthedocs.org/en/latest/plugins/debug.html

It just requires running "nosetests --pdb".

Very helpful if you're debugging why a test failed.

Great Hadoop + Spark Tutorial

2016-02-14T01:07:00+00:00

This great tutorial takes you through installing hadoop and spark to analyze Reddit comment data: http://blog.insightdatalabs.com/spark-cluster-step-by-step/

Hadoop Namenode Not Starting

2016-02-14T01:05:00+00:00

My namenode was not starting because I had the wrong host configured in yarn-site.xml, mapred-site.xml, and core-site.xml.

When you're running start-dfs.sh on your namenode, ensure the line that says "starting namenode" shows "/usr/local/hadoop/logs/hadoop-ubuntu-namenode-<your namenode's hostname>.out" in the output. This is how you know your configuration is correct.

You can check your server's hostname on Ubuntu by running "echo $(hostname)".

Python - "The Darker Side of type"

2015-12-26T08:31:00+00:00

https://www.jeffknupp.com/blog/2013/12/28/improve-your-python-metaclasses-and-dynamic-classes-with-type/

That blog post has a great description of how the type keyword can be used for another purpose, creating new types.

Here's an example of the syntax: Foo = type(str('Foo'), (object, ), {})

This page has more details: http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html

SQLAlchemy's "scale" Argument & WTForms

2015-12-23T01:24:00+00:00

Note: Unless otherwise stated, this applies to SQLAlchemy 1.0.

Here are a few things I learned while researching the fix for this Flask-Admin Issue #1141:

With SQLAlchemy's Generic Float type, the "scale" argument is ignored. "scale" is not listed as an argument and the docs say "Additional arguments here are ignored by the default Float type.". However, the object does have a "scale", but it's always "None".
WTForms' "places" default for DecimalField's is 2.
The MySQL float does have a "scale" argument, it defaults to "None". MySQL and Postgres have default limits on the length of the precision of floats, MySQL only shows 6 digits after the decimal and Postgres has a default column length of 17.
Numeric columns do have a "scale" argument, and the default is "None" in SQLAlchemy 0.7 and 1.0.
"decimal_return_scale" was added to both Float and Numeric in SQLAlchemy 0.9.
If you raise the number of "places" in WTForms' DecimalField greater than "decimal_return_scale" in the SQLAlchemy field, the digits after the decimal place set in "decimal_return_scale" will show 0's.
In SQLAlchemy, "_default_decimal_return_scale = 10" is only used in a property/method called "_effective_decimal_return_scale". The default for "decimal_return_scale" is "None" and not 10. Since "_default_decimal_return_scale" and "_effective_decimal_return_scale" start with an underscore, so it's only intended for internal use.
MySQL's "FLOAT" in SQLAlchemy doesn't have "decimal_return_scale". It probably should? REAL and DOUBLE have this.
WTForms' FloatField does not have "places".
With SQLAlchemy 1.0, "db.DECIMAL()" will create a numeric(28, 6) in Postgres, but it creates a DECIMAL(10,0) in MySQL. This seemed pretty odd, since the default "scale" is "None". Maybe it's just using whatever the default is in each backend.

Sources:

SQLAlchemy 0.7 docs for Float type: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html?highlight=float#sqlalchemy.types.Float
Latest SQLAlchemy docs for Float type: http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.Float
Latest docs for MySQL Float type: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html?highlight=float#sqlalchemy.dialects.mysql.FLOAT

Python Gotcha - Copying Nested Dicts

2015-10-27T03:18:00+00:00

Here's the best blog post I've found that talks about the need for deep or shallow copy when using nested dicts in Python:
http://www.peterbe.com/plog/must__deepcopy__

Image Duplicate Detection In Python

2015-10-25T06:02:00+00:00

Great article on detecting duplicate images in Python: http://7webpages.com/blog/image-duplicates-detection-python/

GoDaddy To Amazon Route 53 - Lessons Learned

2015-10-11T05:52:00+00:00

The first thing you'll want to do is created a "Hosted Zone" with the same A and CNAME records (and any other relevant records that aren't specific to GoDaddy).

When you transfer the domain and it asks you if you want to use the old name servers - answer no. I messed this one up and kept using GoDaddy's "domaincontrol.com" name servers and DNS stopped resolving eventually. If you mess this up too, you'll need to go to the "Hosted Zone" for your site and copy the records in the "NS" list to domain's name servers on the registered domains page.

Awesome-ETL List

2015-10-07T08:00:00+00:00

Finally decided to put all the interesting ETL software I find into a list: https://github.com/pawl/awesome-etl

Python Idioms

2015-09-24T09:13:00+00:00

Here's a great page describing Python idioms: https://david.goodger.org/projects/pycon/2007/idiomatic/handout.html

This gist of a Raymond Hettinger talk is also great: https://gist.github.com/JeffPaine/6213790

Best "Command Line" Beginners Guide

2015-09-19T21:06:00+00:00

http://www.learnenough.com/command-line

It's a short guide, but I think it covers everything a beginner needs to know.

Current State Of Workflow Engines / Frameworks

2015-09-15T06:49:00+00:00

This is a great article about the current state of workflow engines (and frameworks?): http://stackstorm.com/2015/04/10/the-return-of-workflows/

Taskflow looks especially interesting. The article is unfortunately missing Airbnb's Airflow: https://github.com/airbnb/airflow

Installing Postgres 9.5 On OSX

2015-08-02T04:51:00+00:00

Edit 12/25/2020: I’d recommend using: https://postgresapp.com/

Ended up being surprisingly easy:

brew uninstall postgresql brew install postgresql-9.5 brew link -f postgresql-9.5 mv /usr/local/var/postgres /usr/local/var/postgres.old initdb -D /usr/local/var/postgres # restore old database? # pg_upgrade -b /usr/local/Cellar/postgresql/9.0.4/bin -B /usr/local/Cellar/postgresql/9.1.2/bin -d /usr/local/var/postgres.old -D /usr/local/var/postgres # start postgres pg_ctl -D /usr/local/var/postgres -l logfile start

New Macbook Pro Setup

2015-07-26T22:21:00+00:00

I’m finally switching to a Macbook Pro as my personal computer after a few months of using one at work. Local development on OSX is a lot easier and bug-free than on a Windows computer.

I started a gist with what I did for setup:

Important tips: * to open applications quickly, command + space and type the application name * mac doesn't have an address bar in finder, you need to use command + shift + g to type a directory path * mac doesn't have "snap window to screen" like windows does, you need to install moom for this * activity monitor = task manager * brew install = apt get install (you'll need to install brew) * use command + tilde to switch between windows for the same application Setup: * Upgrade MacOS to newest through the app store if you haven't already * Install chrome (https://www.google.com/chrome/browser/desktop/) * Install chrome plugins (adblock, etc) * Install brew * /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" * Install iterm2 (https://www.iterm2.com/) * Fix alt + left/right arrow in iterm: http://apple.stackexchange.com/a/136931 * `brew install git` * [github ssh setup](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent) * Install xcode and agree to terms of service (it's required by some things) * Install sublime text * Install sublime package manager and plugins * command + shift + p then "install plugin manager" * install plugin sidebarenhancements * restore sublime text settings from other computer * Security hardening (especially turn off bluetooth): http://docs.hardentheworld.org/OS/OSX_10.11_El_Capitan/index.html * Open dock settings and check "Automatically hide and show dock" * Open trackpad settings and enable "Tap to Click" * Open keyboard settings and set "key repeat" to "Fast" and "delay until repeat" to "Short" * install docker * install vlc

In Order to Configure TCP/IP, You Must Install and Enable a Network Adapter Card

2015-07-12T04:44:00+00:00

If you get this error message, you need to disable and uninstall your network adapter. Once you've done that, you can reinstall the driver to fix it.

Cygwin - Python Compiled Without SSL Support

2015-07-11T23:11:00+00:00

Download error on https://pypi.python.org/simple/pip/: unknown url type:
https -- Some packages may not be found!

ImportError: cannot import name HTTPSHandler

I was trying to install pip in cygwin when this happened. Searches say openssl-devel needed to be installed, but it already was. Apparently python was compiled without ssl support.

My solution was to install cygwin-x86 instead of the 64x one.

md5sum: standard input: no properly formatted MD5 checksum lines found

2015-07-11T06:22:00+00:00

If you see this error when you're using apt-cyg, it means you need to update apt-cyg.

You need to run the following commands:

lynx -source rawgit.com/transcode-open/apt-cyg/master/apt-cyg > apt-cyg
install apt-cyg /bin

Automatically Remove Long-Running Docker Containers

2015-06-28T00:54:00+00:00

The following can be used in cron or run with & at the end of the command:

I found this didn’t work: http://www.ahtik.com/blog/removing-detached-docker-containers-automatically/

Docker Run In Crontab

2015-06-03T04:17:00+00:00

"0 5 * * 1 docker run --rm --name=mycontainer ubuntu:13.10 /opt/bin/job"
The above command is an example of how "docker run" would be used in crontab to run once every week at 5am.

--rm will delete the container once the job is finished running
--name will name the container and prevent duplicate jobs from running
You don't need to use "&" at the end, because crons already run in the background.

MySQL Batch Updates Not Working

2015-05-08T07:51:00+00:00

Looking at "SHOW PROCESSLIST" and it looks like your queries are running individually instead of in batches like you sent?

This happens because MySQL runs each update statement individually, but you should still be able to see the batches when the queries are in the "init" state.

Use this query to see the batched queries:
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST where state="init";

New Flask Template

2015-03-23T00:29:00+00:00

https://github.com/xen/flask-project-template/

It's also an interesting example of using docker for a flask project.

"Error response from daemon: 404 page not found" - Docker

2015-02-13T05:16:00+00:00

This required setting the following in my /etc/environment:

no_proxy=/var/run/docker.sock

When Not to Use Generator Expressions

2015-01-28T21:04:00+00:00

http://stackoverflow.com/a/26635939

I found that post very helpful. I didn't know it's a better idea to use a list comprehension rather than a generator expression for "".join().

mod_wsgi + Flask

2015-01-28T08:52:00+00:00

Here’s my mod_wsgi configuration for a flask app that uses Flask-Admin.

WSGIDaemonProcess makermanager2 user=www-data group=www-data threads=5 WSGIScriptAlias /makermanager2 /var/www/makermanager2/runserver.wsgi <Directory /var/www/makermanager2> WSGIProcessGroup makermanager2 WSGIApplicationGroup %{GLOBAL} Order deny,allow Allow from all </Directory> Alias /makermanager2/assets /var/www/makermanager2/application/static <Directory /var/www/makermanager2/application/static> Order allow,deny Allow from all </Directory> Alias /makermanager2/static/admin /opt/Envs/prod/src/flask-admin-master/flask_admin/static/ <Directory /opt/Envs/prod/src/flask-admin-master/flask_admin/static/> Order allow,deny Allow from all </Directory>

Trailing slashes matter! If you set a trailing slash, it will only set the alias for that directory and not all of the directories below it.

Request.Args Empty During Tests - Flask

2015-01-27T04:01:00+00:00

https://github.com/mitsuhiko/flask/issues/801#issuecomment-21513278

If you're using app.test_client().get, it's very important to use a relative url or your parameters will be removed.

# will not pass request.args
rv = self.client.get('http://0.0.0.0:5000/view/?foo=bar')

# works just fine
rv = self.client.get('/view/?foo=bar')

Self-Hosting Fonts

2015-01-23T05:51:00+00:00

https://github.com/CSSLint/csslint/wiki/Bulletproof-font-face

That's the most helpful guide I found for self-hosting your own fonts and preventing compatibility issues.

Dokku Change Nginx Timeout

2015-01-21T20:36:00+00:00

None of the plug-ins for doing custom nginx configurations were working for me.

So, I changed the following two files:
/var/lib/dokku/plugins/nginx-vhosts/templates/nginx.ssl.conf
/var/lib/dokku/plugins/nginx-vhosts/templates/nginx.conf

I added the following under "location / {":
proxy_connect_timeout 300s;
proxy_read_timeout 300s;

Supervisord Tips and Gotchas

2015-01-21T05:22:00+00:00

If you make configuration changes, the changes will not take effect until you:

supervisorctl reread
supervisorctl update

"redirect_stderr=true" will cause both stderr and stdout to appear in the "stdout" log.

You get to select which user runs the command with the "user" parameter.

For example: user=root

Supervisord keeps its own set of environmental variables with the "environment" parameter. It won't pick up env variables from /etc/environment.

For example: environment=A="1",B="2"

For Gunicorn, use a configuration file rather than putting all your arguments into the "command" parameter.

For example, -t will be ignored here:
command=/path/to/gunicorn main:application -t 300

Instead use:
command=/path/to/gunicorn main:application -c /path/to/gunicorn.conf.py

Installing python-ldap on Dokku

2015-01-20T06:20:00+00:00

Install this plugin: https://github.com/F4-Group/dokku-apt#installation

Create an apt-packages file in the root directory of your repository with the following lines:
python-dev
libldap2-dev
libsasl2-dev
libssl-dev

Now, commit your new apt-packages file and python-ldap should install successfully.

I also had trouble installing pymssql, and got an error saying "_mssql.c:314:22: fatal error: sqlfront.h: No such file or directory". Similar fix, just add "freetds-dev" to your apt-packages file.

Enabling SSL On Dokku

2015-01-20T05:13:00+00:00

It's important that you have the newest version of Dokku: http://progrium.viewdocs.io/dokku/upgrading
Zip your certificate and key: sudo tar cvf archive_name.tar.gz server.crt server.key
Import using dokku nginx:import-ssl appname < ~/archive_name.tar.gzhttps://github.com/progrium/dokku/blob/master/docs/nginx.md#importing-ssl-certificates

Expecting: TRUSTED CERTIFICATE

2015-01-20T04:59:00+00:00

I was trying to add my certificate to Dokku using "dokku nginx:import-ssl" when I got the error message "Expecting: TRUSTED CERTIFICATE".

This ended up being an issue with line endings and the dos2unix utility fixed it: http://serverfault.com/a/317038

Downloading Files With Python's Requests Module

2015-01-19T00:28:00+00:00

http://docs.python-requests.org/en/latest/user/quickstart/#binary-response-content

From the example:

>>> from PIL import Image
>>> from StringIO import StringIO
>>> i = Image.open(StringIO(r.content))

Nmap Not Finding Hostnames

2015-01-16T06:22:00+00:00

http://serverfault.com/a/153779

The secret ended up being running the command as sudo. It's odd how "socket.gethostbyaddr()" worked just fine without sudo, but was slower than nmap.

Slow Disk Performance On Dell r620

2015-01-07T16:47:00+00:00

Update:

I installed openmanage with the instructions on this page and used it to check/update bios settings: http://linux.dell.com/repo/community/deb/OMSA_7.1/

The fix ended up being BIOS power profile settings. It was set to Performance Per Watt (DAPC) and needed to be set to "Performance".

It's very interesting that the BIOS power profile settings didn't seem to affect Ubuntu 14.10's disk read speed.

Troubleshooting:

I have three of the same server. All the servers are Dell r620's with a PERC H710 Mini RAID controller (21.2.0-0007_A04 firmware) and have the exact same hard drives (Seagate ST300MM0006) in RAID 1 configurations. I'm seeing about 50% worse disk read performance on some versions of Ubuntu.

They only differences I can find in the hardware:

The BIOS version on the poorly performing machines is 1.4.8, the machine that's performing well is on 1.6.0. The BIOS change log doesn't appear to have any changes between the two versions that would affect anything.
The chipset version on the RAID controller of the poorly performing machines is rev 01 (ChipRevision: B0). The machine that's performing well has the rev 05 chipset (ChipRevision: D1). The motherboard chipset revision is also different.

Here are the benchmark results for the different Ubuntu versions and RAID controller chipsets:

(mysql read speed test - using the same version of mysql across all 3 machines)
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run:

(13.04 - rev 05): read/write requests: 4996362 (83271.11 per sec.)
(14.04 - rev 01): read/write requests: 1906520 (31773.58 per sec.)
(14.10 - rev 01): read/write requests: 5166798 (86111.50 per sec.)

dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output:

(13.04 - rev 05): 83886080 bytes (84 MB) copied, 0.0437543 s, 1.9 GB/s
(13.04 - rev 01) 83886080 bytes (84 MB) copied, 0.116811 s, 718 MB/s
(13.10 - rev 01): 83886080 bytes (84 MB) copied, 0.129371 s, 648 MB/s

(14.04 - rev 01): 83886080 bytes (84 MB) copied, 0.157808 s, 532 MB/s

(14.10 - rev 01): 83886080 bytes (84 MB) copied, 0.102355 s, 820 MB/s

hdparm -tT /dev/sda1:
(13.04 - rev 05): Timing cached reads:23154 MB in 2.00 seconds = 11589.16 MB/sec
(13.04 - rev 01): Timing cached reads:17934 MB in 2.00 seconds = 8973.68 MB/sec
(13.10 - rev 01): Timing cached reads:18102 MB in 2.00 seconds = 9058.08 MB/sec
(14.04 - rev 01): Timing cached reads:17846 MB in 1.99 seconds = 8956.28 MB/sec
(14.10 - rev 01): Timing cached reads:21538 MB in 2.00 seconds = 10777.93 MB/sec

At one point, 2 of the rev 01 servers were using Ubuntu 14.04 and were getting similar bad benchmark results. After the upgrading them to 14.10, I immediately saw the disk read speed increase.

The performance on rev 01 machine with 13.04 was bad, but somehow the rev 05 machine with the same OS version was performing well. This told me it's not a problem with the megaraid_sas driver, because the rev 01 and rev 05 machines with 13.04 had the same version.

Here's the closest I can find to a similar issue (it's different hardware though): http://en.community.dell.com/support-forums/servers/f/906/t/19596533

Update 3/1/2015:
Saw an interesting checklist on Hacker News today that also mentioned disabling BIOS power saving settings: http://odetodata.com/2015/02/installation-and-configuration-checklist-for-microsoft-sql-server/

MegaCli64 / MegaCli Only showing "Exit Code: 0x00"

2015-01-06T18:23:00+00:00

Problem: You see MegaCli64 or MegaCli only showing "Exit Code: 0x00" when you try to run commands.

Solution: Try running as sudo.

Changing Putty Default Settings

2015-01-02T06:10:00+00:00

If you need to change Putty’s defaults, press “Load” on the saved session for “Default Settings”, make your changes, and “Save”.

Best Django Boilerplate/Skeleton

2014-12-27T07:29:00+00:00

This is my favorite boilerplate for new projects so far: https://github.com/kirpit/django-sample-app

Because:

It's up-to-date (as of 12/27/2014). Django-skel is currently at only up to 1.5.9.
It's minimal. It doesn't make many design decisions for you or include a bunch of junk. Django-base-template isn't as minimal.
Great structure, it follows this fairly closely: http://stackoverflow.com/a/23469321/1364191

If you're not a fan of bootstrap, django-html5-boilerplate also looks good.

Setting A WTForm Default Date To Today/Now

2014-12-11T17:14:00+00:00

http://stackoverflow.com/a/27424739/1364191

It's definitely a common mistake for people using WTForms. When you create a form and use something like datetime.date.today() as a default value, that value will always stay the same. The fix is to use "datetime.date.today" instead of "datetime.date.today()".

Dokku Flask Port and Host

2014-12-05T20:07:00+00:00

In your flask application, you need to:

Set default port=5000 (this seems to be Dokku's default too)
Set default host=0.0.0.0 (this allows your host server to connect to the container)

In your Procfile:

web: python <your-app>.py

If you're using gunicorn, it shouldn't matter because it's not using app.run().

If you still can't connect to your application after it deploys, see this: http://www.paulsprogrammingnotes.com/2014/12/troubleshooting-dokku-not-accessible.html

Troubleshooting Dokku Not Accessible

2014-12-05T20:02:00+00:00

First, make sure your application isn't giving any errors when it attempts to run:
dokku run <your-app> "<whatever is in your proc file>"

Get the :port number from the line with "upstream" in the following file: /home/dokku/<your-app>/nginx.conf

Try to "wget 127.0.0.1:<your-port>" from the host. Will it connect?

"Requested runtime not available for this stack" - Dokku

2014-12-04T15:32:00+00:00

While using Dokku, I was getting the following error message: Requested runtime (python-2.7.8) is not available for this stack (cedar-14).

This was due to a connection error between my docker container and the internet. To determine this, I ran "docker images" and got the image id of "programium/buildstep", then got into a container with the following command: sudo docker run -t -i <your image id> /bin/bash

Once in the container, I ran:
curl http://lang-python.s3.amazonaws.com/cedar/runtimes/python-2.7.8.tar.gz -s

If that is unsuccessful, that should confirm it's a connection issue.

This was the solution to the problem: http://www.paulsprogrammingnotes.com/2014/12/rebuilding-buildstep-image-for-dokku.html

Rebuilding Buildstep Image For Dokku From Behind Proxy

2014-12-02T20:52:00+00:00

I had to change the buildstep image’s docker file to add my proxy. This is because the buildstep image required a proxy before the build to download the python buildpack.

I was rebuilding the buildstep image for dokku with these instructions: https://github.com/progrium/dokku/commit/9ebf453b72cab3a16ea261284236bb7c20ca3a1a#diff-04c6e90faac2675aa89e2176d2eec7d8R101

I’m behind a proxy, so I was getting this error when I ran “sudo make build”:

Cloning into ‘heroku-buildpack-multi’… fatal: unable to access ‘https://github.com/ddollar/heroku-buildpack-multi.git/’: Failed to connect to github.com port 443: Connection timed out

The solution? Replace the Dockerfile in the root of the folder you git cloned with this:

FROM progrium/cedarish:cedar14 MAINTAINER Jeff Lindsay <progrium@gmail.com> ENV http_proxy http://proxy.myproxy.com:8080/ ENV https_proxy http://proxy.myproxy.com:8080/ ADD ./stack/configs/etc-profile /etc/profile ADD ./builder/ /build RUN xargs -L 1 /build/install-buildpack /tmp/buildpacks < /build/config/buildpacks.txt

Troubleshooting Pip Timeout - Python

2014-12-01T16:39:00+00:00

Timeout: (<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x27330d0>, 'Connection to pypi.python.org timed out. (connect timeout=15)')

If you're installing several requirements from a requirements.txt file, it helps to use "-v" after your pip install command to figure out which module has the timeout. Most likely, it's trying to download a module from a URL that doesn't exist.

django-admin2

2014-11-29T03:09:00+00:00

This project looks similar more similar to Flask-Admin than regular Django Admin: https://github.com/pydanny/django-admin2

There's an example admin app here: https://github.com/pydanny/django-admin2/tree/develop/example

NULL values excluded from NOT IN - SQLAlchemy+SQLite

2014-11-26T01:47:00+00:00

Solution: AND column IS NOT NULL

Example: query.filter(or_(~self.column.in_(value), self.column == None))

This thread explains "SQL Server in treating NULL as a value": http://stackoverflow.com/questions/11491831/null-values-are-excluded-why

Github Asking For Password After Adding SSH Key

2014-11-25T23:21:00+00:00

Run: git remote -v

If the output has a URL that starts with https://github.com and not git@github.com, you need to change that by running a command similar to this: git remote set-url origin git@github.com:pawl/flask-admin.git

https://help.github.com/articles/changing-a-remote-s-url/#switching-remote-urls-from-ssh-to-https

Guide To Syncing Fork With Original Project

2014-11-21T08:14:00+00:00

https://help.github.com/articles/syncing-a-fork/

From my fork, I ran these commands:

git remote add upstream https://github.com/mrjoes/flask-admin.git
git fetch upstream
git merge upstream/master
git push

Create new branch and sync with upstream (change the bolded text):

git remote add upstream https://github.com/mrjoes/flask-admin.git
git fetch upstream
git checkout -b issue_xxxx upstream/master

https://docs.djangoproject.com/en/1.6/internals/contributing/writing-code/working-with-git/#working-on-a-ticket

Auto-Fixing Syntax Guideline Problems With Autopep8 - Python

2014-11-18T19:50:00+00:00

find . -name '*.py' -exec autopep8 --in-place '{}' \;
https://github.com/hhatto/autopep8/issues/30#issuecomment-22458121

Configuring Notepad++ For Python

2014-11-16T08:53:00+00:00

On the Settings->"Preferences", "Tab Settings" tab, I set "[Default]" at Tab size: 8, uncheck Replace by space; and set "Python" to uncheck Use default value, Tab size: 4, check Replace by space. This causes inserts into a python source to use 4 spaces for indents, and indent with spaces instead of tabs.

If I end up with any tabs in the source, I use Edit->Blank Operations->TAB to Space to convert them. I also clean up trailing blanks with Edit->Blank Operations->Trim Trailing Space.

I install and use the pep8 package to verify standard formatting.

http://www.reddit.com/r/Python/comments/2mfrn6/python_in_notepad/cm3tdj5

Redirecting From Within Helper Functions - Flask

2014-11-15T23:33:00+00:00

from flask import Flask, redirect from werkzeug.routing import RoutingException, HTTPException app = Flask(__name__) @app.route('/') def hello_world(): def helper_function(): # Attempt #1 # this obviously won't work: return redirect('www.google.com') # the program would just return 'Hello World!' # Attempt #2 # THIS IS PERMANENT!: raise RequestRedirect('example2') # your browser will cache the status 301 and ALWAYS redirect # explanation: http://stackoverflow.com/a/16371787/1364191 # Attempt #3 class RequestRedirect(HTTPException, RoutingException): """Raise if the map requests a redirect. This is for example the case if `strict_slashes` are activated and an url that requires a trailing slash. The attribute `new_url` contains the absolute destination url. The attribute `code` is returned status code. """ def __init__(self, new_url, code=301): RoutingException.__init__(self, new_url) self.new_url = new_url self.code = code def get_response(self, environ): return redirect(self.new_url, self.code) raise RequestRedirect('example3', code=302) helper_function() return 'Hello World!' @app.route('/example2') def example2(): return 'Example 2' @app.route('/example3') def example3(): return 'Example 3' if __name__ == '__main__': app.run(host="0.0.0.0", port=9999, debug=True)

Reload Enviromental Variables Without Logging Out - Ubuntu

2014-11-15T21:54:00+00:00

This doesn't actually work: "source /etc/environment"
This did the trick though:

for line in $( cat /etc/environment ) ; do export $line ; done

http://superuser.com/questions/339617/how-to-reload-etc-environment-without-rebooting#comment371112_339617

Open Source Charting Library With Vertical Markers/Annotations

2014-11-15T18:42:00+00:00

Saw this charting library from the Mozilla Metrics Team on hacker news recently: http://metricsgraphicsjs.org/interactive-demo.htm

Their interactive example is here: http://metricsgraphicsjs.org/interactive-demo.htm

I'm not aware of any other charting libraries besides highcharts that do vertical marker lines this easily.

Python datetime.strftime Cheat Sheet

2014-11-14T17:52:00+00:00

http://strftime.org/

Using Bootstrap-DateRangePicker As A Timepicker

2014-11-11T07:32:00+00:00

Here's the jsfiddle: http://jsfiddle.net/d8uomdjv/

It seems like the author is opposed to adding this sort of functionality: https://github.com/dangrossman/bootstrap-daterangepicker/issues/295

re.match vs re.search - Python

2014-11-11T01:13:00+00:00

http://stackoverflow.com/a/180993/1364191

re.match - If zero or more characters at the beginning of string match the regular expression pattern.
re.search - Scan through string looking for a location where the regular expression pattern produces a match

New Machine Setup For Flask Development On Ubuntu

2014-11-09T01:34:00+00:00

Here are the commands I use when I bring up a new Ubuntu server for flask development:

sudo apt-get update && time sudo apt-get dist-upgrade sudo timedatectl set-timezone America/Chicago apt-get install git fail2ban htop nano mkdir -p .ssh nano .ssh/authorized_keys eval `ssh-agent -s` ssh-add ~/.ssh/id_rsa apt-get install python-setuptools easy_install pip pip install virtualenvwrapper sudo mkdir ~/virtualenvs sudo sh -c "echo 'WORKON_HOME=~/virtualenvs' >> /etc/environment" for line in $( cat /etc/environment ) ; do export $line ; done mkdir -p $WORKON_HOME sudo sh -c "echo 'source /usr/local/bin/virtualenvwrapper.sh' >> ~/.bashrc" bash mkvirtualenv default apt-get install python-dev

MySQL Performance LXC Container vs Host

2014-10-23T05:25:00+00:00

I have a linux virtual container (LXC) and the server hosting the container running the same version of MySQL with the same database.

I tried running some select queries on both of the both the virtual container and host for benchmark purposes. My conclusion? MySQL read performance inside and outside the LXC container is the exactly the same.

WHMCS Cronjob Error

2014-10-22T00:54:00+00:00

If you get the following error while you try to run the WHMCS cronjob from the command line:
Site error: the file <b>/var/www/accounts/admin/cron.php</b> requires the ionCube PHP Loader ioncube_loader_lin_5.3.so to be installed by the website operator. If you are the website operator please use the <a href="http://www.ioncube.com/lw/">ionCube Loader Wizard</a> to assist with installation.

You need to add your the ioncube zend extension to the CLI php.ini too: /etc/php5/cli/php.ini

Sendmail/STARTTLS verify=FAIL

2014-10-19T19:33:00+00:00

Oct 19 14:04:34 Billing sm-mta[17583]: STARTTLS=client, relay=aspmx.l.google.com., version=TLSv1/SSLv3, verify=FAIL, cipher=ECDHE-RSA-RC4-SHA, bits=128/128

If you see those errors in your mail.log and your emails are failing to send, you need to add your ssl cert from apache to the sendmail config in /etc/mail/sendmail.cf:

# CA directory

O CACertPath=/etc/apache2/ssl

# CA file

O CACertFile=/etc/apache2/ssl/example.ca-bundle

# Server Cert

O ServerCertFile=/etc/apache2/ssl/example.cert

# Server private key

O ServerKeyFile=/etc/apache2/ssl/example.key

Securing External LDAP Connections

2014-10-19T03:58:00+00:00

https://help.ubuntu.com/community/SecuringOpenLDAPConnections

I was getting the following error when I tried testing with openssl:
error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177

The solution? It turns out that /etc/default/slapd needs to be configured to use /etc/ldap/ldap.conf using the following:

SLAPD_OPTIONS="-f /etc/ldap/slapd.conf"

Adding Virtualhost In Apache For Port

2014-10-17T16:26:00+00:00

Say we wanted to route the <your-domain>.com:8072/misc to /var/www/misc...

Add the following to /etc/apache2/ports.conf:
NameVirtualHost *:8072
Listen 8072

Add a file that describes your site to /etc/apache2/sites-available with this:

<VirtualHost *:8072>

ServerName <your-domain>.com

DocumentRoot /var/www/misc/

</VirtualHost>

Run this command: sudo a2ensite misc

And restart apache: sudo service apache2 restart

Deploying Nginx + Gunicorn + Flask

2014-10-16T19:33:00+00:00

Why use both Nginx and Gunicorn?: http://serverfault.com/a/220047

Simple explanation: http://prakhar.me/articles/flask-on-nginx-and-gunicorn/

Nginx SSL configuration: https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-with-ssl-as-a-reverse-proxy-for-jenkins

The thing I spent the longest on was getting supervisor to work. Gunicorn kept giving me "gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>", because I was missing the "directory" parameter. I needed to tell supervisor which directory to start from.

[program:my_app]
command=/<path>/venv/prod/bin/gunicorn -w 8 main:app -b 127.0.0.1:8080
user=me
autostart=true
directory=/home/me/my_dir/
stdout_logfile=/home/me/my_dir/gunicorn_supervisor.log
stdout_logfile_maxbytes=20MB
redirect_stderr=true

You will need to create an upstart script to get supervisor to run when the system starts: http://serverfault.com/a/96500

Python Excel Library Benchmark Comparison

2014-10-15T05:01:00+00:00

https://github.com/swistakm/python-excel-benchmarks

benchmark_csv 0.026495
benchmark_excellent 1.784107
benchmark_openpyxl 2.897072
benchmark_openpyxl_rows 7.025895
benchmark_pyexcelerate 0.550225
benchmark_xlsxcessive 1.430242
benchmark_xlsxwriter 1.611668
benchmark_xlwt 1.275144

http://xlsxwriter.readthedocs.org/en/latest/working_with_memory.html

pyexcelerate : 10.11
xlwt : 15.67
xlsxwriter (optimised): 19.70
xlsxwriter : 23.50
openpyxl (optimised): 95.82
openpyxl : 95.90

Large Dictionaries Not Released From Memory - Python + Ubuntu

2014-10-13T18:27:00+00:00

I have a flask application running on Ubuntu which reads from a file and creates several large dictionaries. After the dictionaries are created and flask returns the request, the memory used for the dictionaries is not released back to the OS. This example shows the problem: https://gist.github.com/pawl/c3ed7663b94abec01d75

It's not really a Flask issue, but the Flask guys were super helpful when I asked why my large dictionary wasn't released from memory: https://github.com/mitsuhiko/flask/issues/1202

I dug even deeper and learned this is a Linux thing:
"Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."
http://deeplearning.net/software/theano/tutorial/python-memory-management.html

My solution:
Turning it into a function and running it as a separate process: https://gist.github.com/pawl/95769724848269cff890

If you want to go even deeper down the rabbit hole, read these:

"most malloc implementations will not release memory to the operating system, and the few that do, do not do it very easily" -http://stackoverflow.com/questions/2215259/will-malloc-implementations-return-free-ed-memory-back-to-the-system
"We compiled a version of Python with TCMalloc that only uses mmap. When testing the new Python in one of our largest projects, we found that not only did Python give back memory to the OS correctly, it also had a reduced memory usage and no apparent CPU penalty for using mmap instead of brk." - http://pushingtheweb.com/2010/06/python-and-tcmalloc/

Renaming Screen - Ubuntu

2014-10-08T18:23:00+00:00

http://superuser.com/a/521112

screen -S 8890.foo -X sessionname bar

Initializing Dictionary Elements Inside A Loop - Python

2014-10-04T08:29:00+00:00

There's a really interesting tip in the "Initializing Dictionary Elements" section of this page: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

"it is cheaper to use a try statement" when checking to see if a key exists in a dictionary (inside of a loop).

maximum recursion depth exceeded - PyIntervalTree

2014-10-02T17:59:00+00:00

I was getting the "maximum recursion depth exceeded" when I tried to run large datasets through PyIntervalTree.

I never fixed the issue or found anyone else with the issue. I ended up switching to bx-python instead: https://www.biostars.org/p/99/#100

Group By Apply Occurring Twice On First Item - Pandas

2014-10-02T17:10:00+00:00

https://github.com/pydata/pandas/issues/7739#issuecomment-48823964

If Pandas's Group By is applying the function in Apply twice to the first item, it's actually expected behavior according to the warning in the documentation: http://pandas.pydata.org/pandas-docs/stable/groupby.html#flexible-apply

Virtualenvwrapper Basics

2014-09-28T08:25:00+00:00

This video does a good job of describing the basics of Virtualenvwrapper: https://www.youtube.com/watch?v=thHNYVrY0lU

You'll need to have used virtualenv to appreciate this.

Basically, you create virtual environments with "mkvirtualenv" and they're stored in a central location. When you want to switch to your virtual environment, you use the "workon" command.

You can also create a postactivate script that loads environmental variables when you switch to your virtual environment.

Note: There is also a module called Pew that seems to have a simpler installation process: https://github.com/berdario/pew

Meetup API address_1_error

2014-09-25T03:10:00+00:00

I was getting this error while using the POST /venues method of the meetup API.

The error seems to happen when you duplicate the address of another venue. I was able to add the venue, but when I tried to add it a second time - I would get this error.

I ended up getting the list of venues and trying to find a match first.

Profiling Python Scripts

2014-09-20T20:24:00+00:00

https://github.com/what-studio/profiling

Easiest Way To Install Python Libraries In Windows (that require compiling)

2014-09-18T19:03:00+00:00

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Eventbrite "Application Key Error"

2014-09-16T07:29:00+00:00

The error message looks like this:
{u'status': {u'http_code': 200}, u'contents': {u'error': {u'error_type': u'Application Key Error', u'error_message': u'Please provide your Application Key in the URL as "?app_key=<APP_KEY>".'}}}

After a few hours of trying to get past the Eventbrite API's "Application Key Error". I finally figured out that I needed to switch my request URL from:
"https://developer.eventbrite.com/json/event_new"
to:
"https://www.eventbrite.com/json/event_new"

dc.js + Crosstab: Chart With Linked Table

2014-09-14T23:59:00+00:00

http://dc-js.github.io/dc.js/

Flask OAuthLib Examples

2014-09-13T06:40:00+00:00

https://github.com/lepture/flask-oauthlib/blob/master/example/github.py
https://github.com/bjourne/vvm/blob/master/app/users/views.py

Override get_query based on GET parameter - Flask-Admin

2014-09-10T18:59:00+00:00

This will change the results in a flask-admin index view based on GET parameters. In most cases, using filters to do this is probably a better idea.

from application import db from application.views.modelview import ModelView from application.models import Things # has an attribute called thing_type class MyModelView(ModelView): def get_query(self): thing_type = request.args.get('type', None) # pretending we have a GET parameter called "type" if thing_type == "type1": return super(ModelView, self).get_query().filter(Things.thing_type.like('type1 %')) elif thing_type == "type2": return super(ModelView, self).get_query().filter(Things.thing_type.like('type2 %')) else: return super(ModelView, self).get_query() # note, you can't define get_query inside of def index_view(self)

Changing A Single GET Parameter In Request - Flask

2014-09-10T16:42:00+00:00

from application import app from flask import request, url_for, redirect @app.route('/') def index(): modified_args = dict(request.args) modified_args['key'] = 'modified_value' return redirect(url_for('my_view', **modified_args))

SQLAlchemy Performance Tip

2014-09-03T18:51:00+00:00

http://www.mobify.com/blog/sqlalchemy-memory-magic/

"If you can process the results of database queries iteratively (and very often you can), stream the results"

"Since the data needs to travel over the network from the database whether it's streamed or not, this doesn't add a huge overhead, but we'll see that it reduces memory requirements."

Example (using Flask-SQLAlchemy):
db.engine.execution_options(stream_results=True).execute(query)

However, that's not the full story when it comes to MySQLdb (MySQL-python). http://stackoverflow.com/a/3699677

If you're going to do this with mysql, I recommend using oursql: https://pypi.python.org/pypi/oursql

Enable Threading - Flask

2014-09-03T18:47:00+00:00

http://stackoverflow.com/a/14823968/1364191

If you're going to be expecting more than one user on your flask server, you need to enable threading like this: app.run(host='0.0.0.0', port=5000, threaded=True)

Unfortunately, the threading makes Ctrl+C not work sometimes and I'll have to kill the thread manually.

For better solutions for production, check out these WSGI solutions: http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/

Mosh - SSH Without Dropped Connections

2014-09-01T21:59:00+00:00

https://mosh.mit.edu/

Warning: As of 10/17/2014, it's still in development and they haven't implemented scrollback yet. However, you can use screen and view the scrollback with Ctrl+A then Esc.

Things I Learned From DEFCON

2014-08-31T20:52:00+00:00

Pentoo - Linux distro for pen testing.
VMware fusion - Most of the presenters were using a Macbook with VMware fusion managing their VMs.
Chinavasion - One of the presentations mentioned this in the context of dealextreme, I'm assuming this site is similar.
Kali linux - Another distro for pen testing.
Open bts
Sqlmap (SQLi injection) - Python tool used to tell if php pages were vulnerable to SQLi injection.
C99 shell - PHP interface for shell level system functions.
b374k shell - Another shell which allows an user to run file system, database, and shell commands from a web interface.
Accunetix (xss)
Business logic flaws in mobile operators
Doskey /history - Will show you all the previous commands typed into command prompt.
Maria DB - When oracle took over MySQL, it forked and maria db is still run by the previous developers.
Firefox imacro - A presenter said he uses this for all of his bots.
Burp repeater - Looks similar to fiddler, allows repeating http packets. Maybe only for Mac.
Fritzing
Groupie and Geocouch (couch dB) - This was used in combination with the unity engine to display clusters of map points in a video game.
Steganography - The word for hiding data inside of other files. A presenter showed how histograms could be used.
Femtocells - A personal cell tower. One of the presentations talked about how to use a femtocell to intercept text messages to a Verizon (CDMA) phone.
Shark - Has something to do with a hadoop, good for storing large amounts of data.
SuperTimeline

Alembic Cheat Sheet - Python/SQLAlchemy

2014-08-27T07:34:00+00:00

alembic revision --autogenerate -m "<your message>"
alembic upgrade head

Problem: FAILED: Target database is not up to date.
Solution: alembic upgrade head

Problem: No such revision '5000106def16'

Solution:

sqlite3 db.sqlite3
drop table alembic_version; (and exit, ctrl+d)
alembic upgrade head

Failed to fetch http://security.ubuntu.com

2014-08-27T07:23:00+00:00

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/quantal-security/main/binary-amd64/Packages 404 Not Found [IP: 91.189.91.14 80]

This is happening because you're on an unsupported/old version of Ubuntu.

One way to fix it:

nano /etc/apt/sources.list
Change all entries referring to http://archive.ubuntu.com/ubuntu to http://old-releases.ubuntu.com/ubuntu

Best Python Tricks

2014-08-22T04:17:00+00:00

https://gist.github.com/JeffPaine/6213790

Most of the examples involve making your code much faster by using iterators instead of creating copies of the data. Example: xrange

Here's the video: https://www.youtube.com/watch?v=OSGv2VnC0go

CSS Selectors - ~ + >

2014-08-21T21:38:00+00:00

http://css-tricks.com/child-and-sibling-selectors/

That's an excellent article describing what angle brackets, plus signs, and tildes do in CSS selectors. The illustrations make it especially clear.

Change Percent To Numbers and Keep Color - Flot Pie Chart

2014-08-06T18:51:00+00:00

You can access the color of the section of the pie chart by using “series.color”:

options = { series: { pie: { innerRadius: 0.5, show: true, label: { show: true, formatter: function (label, series) { console.log(series); return '<div style="font-size:8pt;text-align:center;padding:2px; color: ' + series.color +';">' + label + '<br/>' + series.data[0][1] + '</div>'; }, } } }, };

Use List In Bind Parameters - SQLAlchemy

2014-08-01T00:34:00+00:00

my_list = ['peach', 'grape', 'apple'] query_parameters = {} counter = 1 for list_item in my_list: query_parameters["list_item" + str(counter)] = list_item counter += 1 where_clause = 'fruits IN(:' + ",:".join(query_parameters.keys()) + ')' # create clause to be inserted into query query_text = db.text(""" SELECT fruits FROM table WHERE """ + where_clause + """ """) result = db.engine.execute(query_text, **query_parameters)

Overwriting module with same version in Heroku - Python

2014-07-30T09:01:00+00:00

I needed to install a newer version of a module from a github commit, but it had the same version number. Heroku just used the cached version of the module and didn't overwrite it with the newer version.

The easiest fix for this:

Create a file in your application's root directory called "runtime.txt" with only "python-3.4.1" written inside.
Add, commit, and push the new runtime.txt
Change runtime.txt back to "python-2.7.6"
Add, commit, and push to clear your virtualenv and reinstall all modules

The server quit without updating PID file (/var/run/mysqld/mysqld.pid). - MySQL

2014-06-27T23:25:00+00:00

. * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

If you're getting this error message, try using sudo to start mysql.

Submit Form When User Clicks Option (autosubmit) - Typeahead.js

2014-06-19T20:56:00+00:00

http://stackoverflow.com/a/18483816/1364191

The bolded section below will automatically submit your form when you select an option from the Typeahead.js dropdown:
$('#remote .typeahead').typeahead(null, {
displayKey: 'value',
source: site_search.ttAdapter()
}).on('typeahead:selected', function(e){
e.target.form.submit();
});

Using Enter Key With Typeahead.js - Javascript

2014-06-19T20:29:00+00:00

https://github.com/twitter/typeahead.js/issues/255

If your text input with typeahead.js is not letting you use the enter key, you need to add a submit button (if you don't already have one).

You can hide it if you want:
<input type="submit" value="Submit" style="display: none">

WinSCP vs Samba For Windows <---> Ubuntu Development

2014-06-12T15:36:00+00:00

I’ve always used WinSCP to transfer files between my Windows development environment and Ubuntu. It worked pretty well. All you had to do was open the file with your editor by clicking the file in WinSCP, and when you saved the file it would save to the server in the background.

The main negative was the lack of feedback when a file fails to save to the server. WinSCP could be logged out and asking for a password in the background, but you were expecting that save to fix the bug you were working on. It ended up being another thing to debug.

A coworker suggested using Samba instead. I installed it with this simple guide: How to Create a Network Share Via Samba Via CLI Then I mapped the network drive in Windows by going to the “Map Network Drive” option in the tools menu of the file explorer:

The only negative I’ve found with Samba so far is that it makes Notepad++ a little less responsive. But it helps a small amount if you turn off “File Status Auto-Detection”:

Relative Paths In Scripts Run By Crontab - Python

2014-06-10T15:52:00+00:00

I learned that a script run by crontab can't have a relative path.

This fix was here: http://stackoverflow.com/a/10422444

The following example opens my_file.txt in a path relative to your script:
import os
dir_path = os.path.dirname(os.path.abspath(__file__))
with open (os.path.join(dir_path, 'my_file.txt'), "r") as myfile:
mytext = myfile.read()

ELSE in FOR loops - Python

2014-06-09T16:19:00+00:00

http://stackoverflow.com/a/9980160

Turns out that you can use ELSE after a FOR loop to make it give an output if it didn't have anything to iterate.

Undoing git rm -r --cached

2014-06-09T13:37:00+00:00

I followed these instructions to gitignore an already committed file: http://stackoverflow.com/questions/6535362/gitignore-after-commit

When I wanted to start tracking it again, I needed to following these instructions: https://gist.github.com/laszlomiklosik/3833968

Make sure you save a backup of your current version before you check out the file again.

Multiple DSNs All Connecting To The Same IP - Sybase ODBC

2014-06-05T19:10:00+00:00

The problem: You have multiple DSNs defined in your unixODBC's odbc.ini file, and it always connects to the same IP when you try testing with iSQL.

The solution:
DO NOT use the following line in your odbc.ini file to define your DSN's port:
CommLinks = tcpip(host=localhost;port=9502)

DO use the following format:

CommLinks = tcpip(HOST=localhost:9502)

Compress - Sybase ODBC

2014-06-05T18:20:00+00:00

Sybase has a odbc parameter called "Compress" that is disabled by default. It's very useful if you're running queries on something with slow network transfer speeds and large datasets.

Example:
Compress = yes

This improve the performance of my query by about 40%.

[IM004] [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed (0) (SQLDriverConnect) - Ubuntu

2014-06-02T22:45:00+00:00

The link to download SQLanywhere is here: http://www.sybase.com/detail?id=1087327

To install it, you need to unzip the file and run "./setup" in the top level directory in the unzipped folder.

I also installed:
https://code.google.com/p/pyodbc/ (use the .zip file on the left)
http://onefinepub.com/2014/03/05/installing-unixodbc-2-3-2-higher-ubuntu-12-04-lts/

The first error message I received was:
sqlanydb.InterfaceError: Could not load dbcapi. Tried: dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib
Exception AttributeError: "'Root' object has no attribute 'api'" in <bound method Root.__del__ of <sqlanydb.Root object at 0x3061150>> ignored

This error is resolved by creating a file called "/etc/ld.so.conf.d/sqlanywhere.conf" and adding a line with "/opt/sqlanywhere12/lib64". Then you will need to run ldconfig -v.

The error message in the title occurred on Ubuntu because unixODBC was unable to find all of the libraries it needed to run.

This helped to find the issue: http://blogs.msdn.com/b/dataaccesstechnologies/archive/2014/01/22/sqlallochandle-on-sql-handle-henv-from-linux-sqlncli-driver.aspx

strace -t -f -o trace_out.txt isql -v <DSN>< USERNAME> <PASSWORD>

It will show there are a bunch of missing files. The issue was resolved when isql was run from the /opt/sqlanywhere12/bin64 directory. This indicated that it's not looking in the right place for the library files. Run the following: export LD_LIBRARY_PATH='/opt/sqlanywhere12/lib64'

Not having the environmental variable above will also cause "sqlanydb.InterfaceError: dbcapi version 1 required." if you're using python's sqlanydb library.

It will give a "file not found" error if you try using bin32/lib32 files on a 64 bit system. Make sure you use the bin64/lib64 files in the "Driver=" fild in your /etc/odbc.ini file (if you're using a 64 bit system)!

Now you will receive an error saying "Parse error: DSN does not exist". You need to add an ODBC_INI environmental variable to where your DSNs are defined. I ran the following: export ODBC_INI='/etc/odbc.ini'

To make those variables we exported work system-wide and continue to work even after we log out:

sudo nano /etc/environment
Add:
ODBC_INI='/etc/odbc.ini'
LD_LIBRARY_PATH='/opt/sqlanywhere12/lib64'

These articles were helpful:
https://hynek.me/articles/twisted-sybase/
https://hynek.me/articles/a-short-summary-on-sybase-sql-anywhere-python/

Escape Newlines In Description For Google Calendar Api - Python

2014-06-02T06:41:00+00:00

# how you need to format the body of the request requestbody = """{ "description": %s, "summary": %s } """ % (json.dumps(description), json.dumps(summary)) # use json.dumps to escape string for the request body

Get ID of edited item within WTforms validator - Flask-Admin

2014-06-02T03:04:00+00:00

def start_must_not_conflict(form, field): print request.args.get('id') # get ID if Event.query.filter(db.and_(Event.location == form.location.data, Event.start.between(form.start.data, form.end.data), Event.id != request.args.get('id'))).first(): # exclude ID from query raise wtforms.validators.ValidationError('Start time conflicts with another request for the same time.') form_args = dict( start=dict(validators=[start_must_not_conflict]) )

Heroku's Advantages Over Google App Engine

2014-05-26T20:43:00+00:00

Pros:

Postgres (you can even use pgAdmin to view your database)
Uses Git to push your application into production
More monitoring features and 3rd party apps
As long as your project uses one dyno, it runs for free (similar to google, but with a less confusing billing structure)

Cons:

10k row limit for your database before they start charging you $9 per month

AttributeError: 'QuerySelectField' object has no attribute '_sa_instance_state'

2014-05-25T02:56:00+00:00

I came across this error when I was trying to make a custom validator in Flask-Admin with wtforms form objects.

This code didn't work:
def location_must_not_conflict(form, field):
if Event.query.filter(Event.location == form.location).first():
raise wtforms.validators.ValidationError('Location conflicts with another request for the same venue.')

Because I was using form.location and not form.location.data.

This will work:
def location_must_not_conflict(form, field):
if Event.query.filter(Event.location == form.location.data).first():
raise wtforms.validators.ValidationError('Location conflicts with another request for the same venue.')

Individual Form Fields In Jinja2 - Flask-Admin

2014-05-21T04:20:00+00:00

On your edit and create templates in Flask-Admin, you can print individual form fields like this (the field's name is location):

{{ form.location }}

Google Calendar Api bad request 400 - Python Authomatic Library

2014-05-18T19:34:00+00:00

I was finally able to get the request to go through by adding a header saying it's json format. Here's an example:

requestbody = """
{
"end": {
"dateTime": "2014-05-18T01:35:00Z"
},
"start": {
"dateTime": "2014-05-18T01:35:00Z"
},
"description": "test2",
"location": "test venue",
"summary": "test2"
}

"""

response = authomatic.access(credentials, url, method='POST', headers={'Content-Type': 'application/json'}, body=requestbody)

Add user_id for modified_by field before save - Flask-Admin

2014-05-18T01:58:00+00:00

In Flask-Admin, you can override the "on_model_change" function to to automatically populate a "modified_by" field in your model.

class LocationView(ModelView):
def on_model_change(self, form, model):
model.user_id = g.user.id

admin = Admin(app)
admin.add_view(LocationView(Location, db.session, name="Locations"))

Favorite Flask Boilerplate/Skeleton/Templates

2014-05-17T07:31:00+00:00

Those do a lot of the work of starting a new Flask project for you.

With cookiecutter flask, all you need to do to build a new project is: cookiecutter https://github.com/sloria/cookiecutter-flask.git

As a warning, cookiecutter flask and overholt include a lot of stuff. I ended up not using it because it was too much work to change everything to work with oAuth.

Remove Index Column From Pandas CSV Export

2014-05-12T15:27:00+00:00

http://stackoverflow.com/a/19782137/1364191

Reverse PDF Page Order

2014-05-11T02:13:00+00:00

I created an app that allows you to upload a pdf and it returns a copy of your PDF with the page order reversed. The code is here: https://github.com/pawl/pdfreverse

Initially when I googled how to do this in Adobe Acrobat, the first few results had fairly complicated solutions. After I made the app, I ended up finding an easier way to do it in Adobe (in the 6th from the top Google result on “reverse pdf order acrobat”…):

Oh well, it was a fun project.

Discourse As Potential Mailing List Replacement

2014-04-29T05:09:00+00:00

While tested out Discourse, I only found a few negative things:

To make it act like a mailing list, it currently requires each user to change settings on their account: https://github.com/discourse/discourse/blob/master/docs/MAILING-LIST-SETUP.md#suggested-user-preferences
I commented on a thread about a new feature that allows setting defaults for this: https://meta.discourse.org/t/default-email-settings-for-a-new-user/14980
It doesn’t allow setting defaults for which categories the user has “Watched”. For example, I would want to default the “General” list to “Watched”, but more obscure lists would only be watched if the user chooses to.
It uses a different e-mail address for each e-mail it sends you (this is to keep track of which thread it belongs to), here’s an example of a “From” address: testdiscourse+43854630fb3a3d9492505fa5a23db196@gmail.com
Google Groups always has the same to/from e-mail address and magically figures this out. This might be weird in some peoples’ email clients.

As a forum, this beats the crap out of PhpBB, but as a mailing list it’s not a clear winner over Google Groups yet. It’s open source, has a better web interface, and the “Categories” are a cool feature for keeping all the lists in the same place. But, the “email in” functionality was too recently implemented, and it might take a few months before it’s as polished as Google Groups.

It’s going to be a very tempting replacement if they add default settings for user preferences.

Screenshots:

Data source name not found and no default driver specified - SQLalchemy/Sybase

2014-04-28T19:05:00+00:00

I was getting this error when trying to connect to a sybase database using Microsoft's odbc driver by using the connection string shown in the SqlAlchemy examples:
DBAPIError: (Error) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)') None None

The following code worked to connect:
import pyodbc, sqlalchemy

def connect():
return pyodbc.connect('DSN=<dsn name>;UID=<username>;PWD=<password>')

srcEngine = sqlalchemy.create_engine('sybase+pyodbc://', creator=connect, echo=True)

Docker Not Starting - Ubuntu

2014-04-27T00:12:00+00:00

I was getting this error message because docker was refusing to start: Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

This issue was occurring because my kernel was refusing to upgrade, because I was on a VPS. (you need to get your VPS provider to upgrade it for you)

Run uname -r, if it returns something like this: "2.6.32-042stab084.20" That's probably the reason why docker isn't starting. Your kernel isn't compatible.

If you're not on a VPS and can upgrade your kernel, try instructions on this page: http://docs.docker.io.s3-website-us-west-2.amazonaws.com/installation/ubuntulinux/

E: Sub-process /usr/bin/dpkg returned an error code (1)

2014-04-27T00:02:00+00:00

I was getting this error message while I was trying to upgrade the kernel on my VPS:
/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 1
Failed to process /etc/kernel/postrm.d at /var/lib/dpkg/info/linux-image-3.8.0-34-generic.postrm line 328.
dpkg: error processing linux-image-3.8.0-34-generic (--remove):
subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
linux-image-3.8.0-34-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)

You might not be able to upgrade the kernel on your VPS. Contact your VPS provider and get them to upgrade the kernel for you.

WHMCS Cronjob Not Running On Ubuntu

2014-04-26T18:48:00+00:00

My issue was the user with the cronjob did not have access to run the cronjob php file. So, I needed to make the new cronjob for the correct user.

Solution:

Find out the owner of your apache process (usually www-data).
Look at the top of the "Automation Settings" page in your WHMCS admin area. It should say something like: php -q /var/www/whmcs/admin/cron.php (keep this handy for step 4)
In terminal run: crontab -e -u www-data
Ensure you have a line that says:
0 6 * * * php -q /var/www/whmcs/admin/cron.php (note: this should be the same file path you got for your own install in step 2)

Http Port 443 Working But Https Port 443 Not Working In CherryPy - Python

2014-04-26T15:35:00+00:00

I was able to visit my site using HTTP and port 443, but nothing would even show in the log when I tried to visit with HTTPS and port 443.

This seems to only be an issue with CherryPy 3.3.0, I switched back to CherryPy 3.2.3 and it worked again.

I noted it on their bitbucket issue here: https://bitbucket.org/cherrypy/cherrypy/issue/1298/ssl-not-working

OpenSSL DLL load failed On Windows - Python

2014-04-26T14:07:00+00:00

The DLL load failed error happened when cherrypy tried the following import: "from OpenSSL import SSL"

ChannelFailures: VerificationError("importing 'C:\\\\Python27\\\\lib\\\\site-packages\\\\cryptography\\\\_Cryptography_c
ffi_48bbf0ebx93c91939.pyd': DLL load failed: %1 is not a valid Win32 application.",)

These error messages are solved with this install: http://slproweb.com/download/Win32OpenSSL-1_0_1g.exe

If you have a 32-bit version of python, that install will move the correct DLLs to your system32 folder.

My First Python Library

2014-04-24T16:27:00+00:00

https://github.com/pawl/Chinese-RFID-Access-Control-Library

I purchased a chinese access controller and reverse engineered some functionality of the windows software that came with it. I made a python library that does some of the same functions as the desktop software: https://github.com/pawl/Chinese-RFID-Access-Control-Library/

I use this library to tie our access controller into with our billing system (called WHMCS) and another web application (which is meant to be a self-documenting way to activate/deactivate RFID badges): https://github.com/pawl/MakerManager

Now, whenever someone stops paying for 5+ days, their RFID badge is deactivated because a "hook" in WHMCS sends a request to a webservice that uses the access control library.

Android Bitcoin OutOfMemoryError Or Crashing On Startup

2014-04-21T05:47:00+00:00

This page explains the problem: https://code.google.com/p/bitcoin-wallet/wiki/OutOfMemory

At one point, it got so bad that the application would not even start anymore.

Solution:

Get Root Explorer
Navigate to your system folder and open "build.prop" in the text editor
Edit "dalvik.vm.heapsize" and "dalvik.vm.heapgrowthlimit" to the following:
dalvik.vm.heapsize=256m
dalvik.vm.heapgrowthlimit=128m
Restart your phone, open the bitcoin application, and backup your keys before it crashes again.

Network Visualization Library

2014-04-16T02:11:00+00:00

http://visjs.org/

This example looks particularly awesome: http://visjs.org/examples/graph/17_network_info.html

Lock wait timeout exceeded - MySQL

2014-04-14T15:51:00+00:00

OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')

After trying a bunch of different solutions in the code, I ended up having to restart MySQL to get this error message to go away.

Solution To Apache Triggering OOM-Killer

2014-04-14T15:49:00+00:00

Update: We switched to Prefork apache and have been experiencing significantly fewer problems.

In this instance, we were getting hit with web crawlers according to the access log and the syslog was saying this: /usr/sbin/apach invoked oom-killer

This issue occurs when apache opens up too many child processes, uses up too much memory, then OOM killer starts shutting down random processes like mysql.

The problem ended up being that our apache2.conf did not have any configuration limiting the number of processes that Apache could open. It might have had a default, but the default was too high. We were using Apache ITK MPM and the apache2.conf only mentioned prefork, worker, and event MPMs. You can check which MPM you are running with "apache2 -V".

Adding this to the configuration solved the problem:
<IfModule mpm_itk_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 75
MaxClients 75
MaxRequestsPerChild 5000
</IfModule>

At first I had the ServerLimit and MaxClients set to 150 and it seemed like apache was ignoring ServerLimit and Maxclients. However, ITK MPM creates an extra fork for each request (according to their site), that's why there was 200+ processes running for my limit of 150. So, the process count is going to be 2x the limit you set.

My Favorite Guide For Installing Python On Windows

2014-04-05T01:47:00+00:00

http://docs.python-guide.org/en/latest/starting/install/win/

If you're behind a proxy, start with this guide: https://github.com/chocolatey/chocolatey/wiki/Proxy-Settings-for-Chocolatey#using-chocolatey-behind-a-proxy-server (only the part starting with "Using Chocolatey behind a proxy server")

Edit A Sharepoint List Page

2014-04-03T18:55:00+00:00

Add the following to the end of the List page's URL: ?ToolPaneView=2

Bootstrap Wizard

2014-04-02T18:37:00+00:00

https://github.com/amoffat/bootstrap-application-wizard

That looks like it will be useful someday.

Override ModelView Class - Flask-Admin

2014-04-01T22:59:00+00:00

This has an awesome example of how to override your base ModelView: http://code.vokor.org/vokorbb/src/04de9dcb666d2fb35513ca29c97654b6eef80648/vokorbb/admin.py?at=tip

You would want to do this if you wanted to add something to all of your admin classes. Use cases:

If you wanted to require login by overriding is_accessible for all of your ModelViews.
If you wanted to add an export method to all of your ModelViews.

Flask-Admin Export CSV

2014-04-01T19:16:00+00:00

Edited: This was recently added to Flask-Admin as a feature.

You can add “can_export = True” to your ModelView to enable it.

More info here: http://flask-admin.readthedocs.org/en/latest/api/mod_model/#flask_admin.model.BaseModelView.can_export

Print URL With Request Parameters/Arguments - Python Flask / Jinja2

2014-04-01T19:00:00+00:00

You need to add the following to your jinja2 template:
{{ request.path ~ '?' ~ request.query_string }}

You can also print only the Get Variables / URL Parameters / Argument String with the following:
print request.query_string

The example is available here: http://stackoverflow.com/a/11781872/1364191

SQLalchemy Cheat Sheet

2014-03-28T16:33:00+00:00

Common Filter Operators: http://manuelmax.tumblr.com/post/289720781/common-filter-operators-sqlalchemy

Common relationships: http://docs.sqlalchemy.org/en/latest/orm/relationships.html#basic-relational-patterns

If you're using Flask-SQLalchemy, their documentation serves as an awesome quick summary of how to use SQLalchemy: http://pythonhosted.org/Flask-SQLAlchemy/

Use A Temporary File (StringIO) for Pandas's to_csv - Python Flask

2014-03-27T22:06:00+00:00

This link helped me use a temporary file for an excel export in flask (using pandas): https://coderwall.com/p/nwbuhq

Get Column and Row Labels In selectHandler - Google Charts / Visualization API

2014-03-27T15:54:00+00:00

For a more complete example of using selectHandler, see this page: https://developers.google.com/chart/interactive/docs/basic_interactivity

My example below get the labels from the row and the columns. I used this to send the user to another link and drill-down.

// this is an example of a bar chart's selectHandler function
function selectHandler() {
var selection = chart.getSelection();
var item = selection[0];

if (item.row != null && item.column != null) {
var rowLabel = parseInt(data.getValue(item.row, 0));
var columnLabel = data.getColumnLabel(item.column);
}
}
google.visualization.events.addListener(chart, 'select', selectHandler);

Ordered Post Data - Python Requests

2014-03-23T06:42:00+00:00

Here's an example of how to send ordered data with python's Requests library, for when the order of your post request data matters:

import requests
from collections import OrderedDict

payload = OrderedDict([("username", "admin"), ("pwd", "password")]) # the order matters!
r = requests.post('http://1.1.1.1/example', data=payload)
print r.text

Example of custom filters - Flask-Admin

2014-03-18T18:14:00+00:00

Update: I got this example added to the project: https://github.com/mrjoes/flask-admin/tree/master/examples/custom-sqla-filter

This Github issue has an example of how to use custom filters in Flask-Admin: https://github.com/mrjoes/flask-admin/issues/400

Multiple Views Based On Same Model - Flask-Admin

2014-03-17T21:24:00+00:00

You need to manually change the endpoint like MrJoes says in issue 480: https://github.com/mrjoes/flask-admin/issues/480

Here's an example:
admin.add_view(MyModelView(model.CsvTable, db.session, endpoint="duplicate_view"))

You can now visit your view by going to 0.0.0.0/admin/duplicate_view

Checkmark "column_formatters" - Flask-Admin

2014-03-04T21:54:00+00:00

In your custom ModelView, you need to add the following lines:
from flask.ext.admin.model.template import macro
column_formatters = dict(your_column=macro('render_your_column'))
list_template = 'list.html'

In your custom list.html template, you need to add the following jinja2 code:
{% macro render_your_column(model, column) %}
{% if model.your_column%}<i class="icon-ok"></i>{% endif %}
{% endmacro %}

If your row has a value in "your_column", then it will show as a checkmark. You need to make a new macro for each column.

Flask-Migrate

2014-03-04T19:44:00+00:00

https://github.com/miguelgrinberg/Flask-Migrate

Flask-Migrate is exactly what I've been looking for when it comes to database migrations. When I create PHP applications with MVC frameworks, the changes in the database need to be done manually when the model is changed. Flask-Migrate handles changes to the database (like adding/removing columns) so your models reflect your database.

This is a big advantage Flask has over PHP frameworks.

Pip Not Working From Powershell

2014-02-26T01:13:00+00:00

If you have strawberry perl installed, you may get one of these errors when you try to use pip from powershell:

Did not provide a command
Unknown or unsupported command 'install'

This is the result of a conflict between strawberry perl's pip and python's pip. You need to remove anything related to strawberry perl from your path environmental variable and ensure you have C:\Python27\Scripts and C:\Python27\Scripts in your environmental variables.

Tableau's Frustrating Licensing Program

2014-02-25T22:58:00+00:00

Tableau's licensing program is crazy.

I was planning to use "Trusted Authentication" to share an embedded chart with lots of users (not all of whom have licenses). Some of those users will only look at the chart maybe once a year, and we can't justify buying them an entire license to look at a chart once.

This page says "everyone who visits the page must be a licensed user on Tableau Server": http://onlinehelp.tableausoftware.com/v7.0/server/en-us/trusted_auth.htm

If you want more details on how to implement Trusted Authentication using PHP, first visit the following folder on your server: C:\Program Files\Tableau\Tableau Server\8.1\extras\embedding\php

It has some sample code, but you need to install version 1.x of pecl_http to make it work. The guides for that are:

http://www.php.net/manual/en/http.install.php#113853
http://www.mkfoster.com/2009/01/04/how-to-use-the-pecl-http-pecl_http-extension-to-make-http-requests-from-php/ (careful, this guide will install pecl_http 2.x which is too different to work)

Possible Solution To Slow SQLalchemy + Flask Application

2014-02-25T20:30:00+00:00

This stackoverflow thread talks about why you want to use scoped_session:
http://stackoverflow.com/questions/6519546/scoped-sessionsession-maker-or-plain-session-maker-in-sqlalchemy

Basically, your web application isn't thread-safe if you're not using scoped_session. It's much better than having you application use a new thread for every instance.

An example of scoped_session in use is available here: https://github.com/mjhea0/flask-boilerplate/blob/master/models.py

Hide Zero Value Points On Google Chart

2014-02-18T19:28:00+00:00

I have a line chart which needed a series of points. I don't want points to show if the value is zero.

Here's my solution:

Set the series's lineWidth to zero and make the point larger than 0:
series: {0: {lineWidth: 0, pointSize: 5}}
Make the viewWindow setting's min higher than 0 (to hide the 0 points):
vAxis: { viewWindowMode:'explicit', viewWindow:{ min: 0.5 } },

Change Home Flask-Admin

2014-02-17T18:52:00+00:00

This was the only thing I was able to get working: http://www.marteinn.se/blog/?p=637

Just putting index.html into an admin/ directory wouldn't work.

There's also an even shorter way to do it, for example:
admin = Admin(app, "G-Cal Manager", index_view=AdminIndexView(name='Home', template='admin/home.html', url='/'))

The example above will change the root url to / instead of /admin and it will use home.html from your templates/admin folder.

To fix the 404 errors on your stylesheets after the change, you will also need to change the static_folder in your flask object:
app = Flask(__name__, static_folder='admin')

invalid byte sequence for encoding "UTF8" - sqlalchemy

2014-01-30T09:38:00+00:00

Try putting ?charset=utf8 at the end of your connection string, like:
'mysql+mysqldb://user:pass@111.111.111.111/rf?charset=utf8'

Adminer - Unable to connect to PostgreSQL server: FATAL: Peer authentication failed for user

2014-01-30T07:01:00+00:00

You need to make sure you have the following line in your pg_hba.conf file:

local all postgres trust

Clone/Copy Table Schema From One Database To Another - SQLalchemy

2014-01-30T00:22:00+00:00

I got the following code from slide 36 of this slideshow: http://www.slideshare.net/Stiivi/python-business-intelligence-pydata-2012-talk

from sqlalchemy import create_engine, Table, Column, Integer, Unicode, MetaData, String, Text, update, and_, select, func, types # create engine, reflect existing columns, and create table object for oldTable srcEngine = create_engine('mysql+mysqldb://username:password@111.111.111.111/database') # change this for your source database srcEngine._metadata = MetaData(bind=srcEngine) srcEngine._metadata.reflect(srcEngine) # get columns from existing table srcTable = Table('oldTable', srcEngine._metadata) # create engine and table object for newTable destEngine = create_engine('mysql+mysqldb://username:password@localhost/database') # change this for your destination database destEngine._metadata = MetaData(bind=destEngine) destTable = Table('newTable', destEngine._metadata) # copy schema and create newTable from oldTable for column in srcTable.columns: destTable.append_column(column.copy()) destTable.create()

SQLalchemy - Print Tables In Database (Show Tables)

2014-01-28T22:57:00+00:00

from sqlalchemy import create_engine, Table, MetaData

engine = create_engine('mysql+mysqldb://username:password@111.111.111.111/databaseName')
metadata = MetaData(bind=engine)
metadata.reflect(engine)
print metadata.tables.keys()

You could also use that for looping through a list of the tables in the database.

Print Flask-Restless Routes

2014-01-28T16:43:00+00:00

# show api routes
for route in app.url_map.iter_rules():
print route.rule

pip install MySQLdb

2014-01-27T18:04:00+00:00

In case you haven't already noticed, the command in the title won't work.

You need to use: pip install MySQL-python

This article explains the correct approach: http://mysql-python.blogspot.com/2012/11/is-mysqldb-hard-to-install.html

Generate Class From Existing Table Sql-Alchemy - Python

2014-01-21T15:55:00+00:00

http://stackoverflow.com/questions/6290162/how-to-automatically-reflect-database-to-sqlalchemy-declarative

That is the key to the kingdom when it comes to using extensions like Flask-Admin and Flask-Restless on existing tables which do not have a class already.

Best VNC Installation Guide

2014-01-18T06:49:00+00:00

https://wiki.amahi.org/index.php/Install_VNC_server_on_Ubuntu_Server_12.04

That guide gave me the least amount of hassle when I installed VNC on a server. I started by trying to get it working with 13.10, but had issues with a grey screen when tightvnc connected.

"Could not connect to database: Database connection "Mysql" is missing, or could not be created." - CakePHP

2014-01-15T20:03:00+00:00

I got rid of this error just by creating the database manually. Shouldn't Cake do this?

"Not Found The requested URL /app/install was not found on this server." - CakePHP

2014-01-15T19:44:00+00:00

Rather than dealing with my .htaccess problems to fix the error, I thought this would be a better approach: http://wwdj.wijndaele.com/getting-started-with-cakephp-without-mod_rewrite/

Unfortunately, it messed up quite a few links on my site.

I ended up solving the problem by changing "AllowOverride" None to "AllowOverride All" in my /etc/apache2/sites-available/default file.

Why the heck is CakePHP using .htaccess files by default anyway? Yii has this more streamlined.

CakePHP Boilerplate

2014-01-15T17:27:00+00:00

While I was searching for a some good boilerplate to start my first CakePHP project with, I found this: https://github.com/hugodias/cakeStrap

There's also this one, but it has much fewer commits: https://github.com/zynesis/cakephp-boilerplate

Exception in thread "main" java.lang.NoClassDefFoundError - Java

2014-01-07T18:05:00+00:00

Got this error when I was missing a library that came with the build.

If you're getting this error, you may need to move the lib folder from your build directory into the folder of your JAR file.

How To Use Endpoint Parameter - Flask-Admin

2014-01-03T19:01:00+00:00

You can access the endpoint parameter by using self.endpoint.

class displayResults(BaseView):
@expose('/',methods=('GET','POST'))
def index(self):
print self.endpoint

admin.add_view(displayResults(name='test', endpoint='test2'))

That will print "test2".

UndefinedError: 'admin_base_template' is undefined - Flask Admin

2014-01-02T17:47:00+00:00

This happens when you try to render a template like this:
return render_template('admin/upload.html')

You need to use self.render instead:

return self.render('admin/upload.html')

Enable UART2 On BeagleBone Black for RS232 Cape

2013-12-30T12:21:00+00:00

Use this script to ensure UART2 is enabled in the device tree: https://github.com/victorporof/BeagleBone-SPI-UART
Enable ttyO1 it in the firmware using this guide (makes the device appear): http://www.element14.com/community/blogs/mirandasoft/2013/07/30/beaglebone-black-rapid-activation-of-ttyo1-ttyo2-ttyo4-and-ttyo5
Put a paperclip into the RS232 port like in the picture on this page: http://sensorsblog.com/serial-communications/rs232loopbacktest/
echo hello > /dev/ttyO1
cat /dev/ttyO1

Clustering With Leaflet

2013-12-30T03:59:00+00:00

http://jmduke.com/blog/2013/12/28/seattle-911

The Z Layout

2013-12-29T23:56:00+00:00

I'm definitely going to use this technique on my next business site: http://design.tutsplus.com/articles/understanding-the-z-layout-in-web-design--webdesign-28

Two Column Match - Flask + HandsOnTable

2013-12-29T23:37:00+00:00

I'm getting more comfortable with Flask, so I decided to experiment with Google App Engine + Flask + HandsOnTable.

Here's the application I made: http://twocolumnmatch.appspot.com/match

It takes the contents of the 2nd column and matches it to the rows of the first column. Just press the "Match" button to see what it does.

I've found myself doing what this application does a few times, but I manually create two python lists and compare them in IDLE. This application streamlines that process quite a bit.

Here's the code: https://github.com/pawl/2ColumnMatch

SQLalchemy - Simple Select Query

2013-12-26T19:10:00+00:00

from sqlalchemy import create_engine, MetaData, Table

engine = create_engine('sqlite:///cars.db', echo=False)
metadata = MetaData(bind=engine)

table = Table('cars', metadata)
stmt = table.select()
for row in stmt.execute():
print row

Entity-Attribute-Value - Dynamic Database Columns

2013-12-26T16:35:00+00:00

This page has a great example of when to use the Entity-Attribute-Value data model: http://stackoverflow.com/questions/7933596/django-dynamic-model-fields

I'm planning to use it for an application I'm making which has user defined fields.

SHOW COLUMNS or Get Column Names From SQLalchemy SELECT Query

2013-12-18T23:19:00+00:00

http://stackoverflow.com/questions/8947616/python-sqlalchemy-get-column-names-dynamically

Rob Wouter's answer really helped me out:

You can either find the columns by calling result.keys() or you can access them through calling v.keys() inside the for loop.
Here's an example using items():
for v in result:
for column, value in v.items():
print('{0}: {1}'.format(column, value))

AttributeError: 'dict' object has no attribute '_set_parent_with_dispatch' - SQLalchemy

2013-12-11T17:56:00+00:00

The documentation says: "Keyword arguments can be specified by specifying the last argument as a dictionary"

So you will need to change this:

__table_args__ = (

{'sqlite_autoincrement': True},

UniqueConstraint('filename', 'path')

)

to this:

__table_args__ = (
UniqueConstraint('filename', 'path'),
{'sqlite_autoincrement': True}

)

Flask-Admin Foreign Key Columns Allowing Editing

2013-12-11T17:36:00+00:00

Make sure you have a row id number as your primary key for your child table. Having a composite primary key didn't seem to save when I was editing the form.

PC Building Site

2013-12-06T06:57:00+00:00

Awesome site for finding the best prices on PC parts: http://pcpartpicker.com

jPanelMenu Causing Document.Ready() To Run Twice

2013-11-27T22:57:00+00:00

This line in jquery.jpanelmenu.js is the cause:
$('body > *').not(jP.menu + ', ' + jP.options.excludedPanelContent).wrapAll('<div class="' + 'jPanelMenu-panel' + '"/>');

You will need to comment it out and wrap everything in your body tag manually with:

</div>

The end result will look like:

<body>

all your code....

</div>

</body>

This page helped me figure it out: http://doctype.com/trying-jquery-wrapall-seems-wrap-content-twice

Best cArrayDataProvider Example - Yii

2013-11-18T16:40:00+00:00

http://queirozf.com/reminders/using-carraydataprovider-to-populate-a-cgridview-in-yii

Spreadsheet Parsing With Javascript

2013-11-13T07:04:00+00:00

http://oss.sheetjs.com/

Responsive Email Template

2013-11-08T09:18:00+00:00

Zurb Ink - http://zurb.com/ink/templates.php

Catch ldap_bind(): Unable to bind to server: Invalid credentials

2013-11-06T04:33:00+00:00

I used the tip from this page: http://bytes.com/topic/php/answers/600903-ldap_bind-warning-instead-false

"You can get around this if you supress warning and error messages by
putting a @ in front of the function name:

$ldapbind = @ldap_bind($ldapconn, $ldaprdn, $ldappass);"

Yii Google Chart Extension

2013-11-05T23:08:00+00:00

Here's YiiWheel's google chart wrapper extension: https://github.com/2amigos/yiiwheels/blob/master/widgets/google/WhVisualizationChart.php

I downloaded just the WhVisualizationChart file and put this in my view:

$this->widget('ext.WhVisualizationChart', array(
'visualization' => 'PieChart',
'data' => $chartDataProvider,
'options' => array(
'pieHole'=> '0.5',
'backgroundColor'=>array('fill'=>'transparent'),
),
'htmlOptions'=>array('style'=>'width:100%; height:375px'),
));

and this in my controller:

public function actionIndex()
{
$chartDataProvider = array(
array('Task', 'Hours per Day'),
array('Work', 11),
array('Eat', 2),
array('Commute', 2),
array('Watch TV', 2),
array('Sleep', 7)
);
$this->render('index', array('chartDataProvider'=>$chartDataProvider));
}

Examples on how to use it are at the bottom of this page: http://yiiwheels.2amigos.us/site/charts#visualizationchart

Render View From Another Controller In Yii

2013-10-31T02:39:00+00:00

http://developwithguru.com/how-to-renderpartial-a-file-in-another-controllers-view-folder-for-yii/

<?php $this->renderPartial('//library/_book', array('books'=>$books))?>

Use double slashes before the controller.

Easier Monitoring Than Nagios

2013-10-28T16:57:00+00:00

http://mmonit.com/monit/

It was not easy at all to get Nagios to monitor mysql. This only took a minute.

CSqlDataProvider Using Special Data Attribute - Yii

2013-10-23T18:55:00+00:00

Just posted this on the Yii forum after I found out that using the special data attribute in a column requires a syntax like 'value'=>'$data["issueCount"]' when you're using CSqlDataProvider:
http://www.yiiframework.com/forum/index.php/topic/48219-special-data-attribute-syntax-in-cactivedataprovider-vs-csqldataprovider/

Trick For Running Python Scripts On Reboot - Ubuntu

2013-10-18T08:15:00+00:00

http://www.raspberrypi-spy.co.uk/2013/07/running-a-python-script-at-boot-using-cron/

Here's what it looks like in crontab:
@reboot python /home/pi/MyScript.py &

Enabling SSL On Apache

2013-10-17T17:40:00+00:00

This is a quick overview of the process of adding an SSL certificate to apache (for next time...):

Generate your private key and CSR with:
openssl req -new -newkey rsa:2048 -nodes -keyout yourdomain.key -out yourdomain.csr
Request your certificate
Save requested certificate onto server in a .crt file
Download the intermediary/root cert from http://www.symantec.com/page.jsp?id=roots
Make your site file in /etc/apache2/sites-enabled/ look like this:

SSLEngine On

SSLCertificateFile /home/youruser/yourdomain.crt

SSLCertificateKeyFile /home/youruser/yourdomain.key

SSLCACertificateFile /var/www/verisign.crt

ServerName www.yourdomain.com

DocumentRoot /var/www/yourdomain

An explanation of each of those certificates:

SSLCertificateFile - This is the certificate you received after your request.

SSLCertificateKeyFile - This is the private key you generated in step 1.

SSLCACertificateFile - This is the intermediary cert you downloaded from verisign.

TbEditableColumn Inside of A TbRelationalColumn's GridView - Yii's YiiBooster Extension

2013-10-16T17:23:00+00:00

https://github.com/clevertech/YiiBooster/issues/703

Just posted an issue on github with instructions for how to get TbEditableColumn working inside of one of YiiBooster's subgrids.

Yii quickdlgs Without UrlManager Enabled

2013-10-11T19:51:00+00:00

EQuickDlgs::iframeButton(
array(
'controllerRoute' => 'controller/create',
'dialogTitle' => 'Create New Record',
'dialogWidth' => 800,
'dialogHeight' => 600,
'openButtonText' => 'Add Record',
'closeButtonText' => 'Close',
//'openButtonHtmlOptions' => array(...),
)
);

With the code above, I was getting the following error:
The system is unable to find the requested action "create?qdsClass=EFrameJuiDlg".

I had to change line 79 in EFrameJuiDlg.php inside the quickdlgs extension from:
$url .= '?' . http_build_query($this->urlParams);

to:

$url .= '&' . http_build_query($this->urlParams);

htmlspecialchars() expects parameter 1 - Yii (echmultiselect)

2013-10-09T05:38:00+00:00

htmlspecialchars() expects parameter 1 to be string, array given C:\xampp\htdocs\yii\framework\web\helpers\CHtml.php(98)

The error only occurred when trying to apply the echmultiselect filter to the filter above the gridview.

Here's what I did to fix:
1. Went to the Yii framework folder and found the following file: yii\framework\web\helpers\CHtml.php

2. Modifed line 98 from "return htmlspecialchars($text,ENT_QUOTES,Yii::app()->charset);" to:
"return htmlspecialchars(print_r($text, true),ENT_QUOTES,Yii::app()->charset);"

Then you need to implement the echmultiselect extension in the view (my example shows adding it for site_status):

3. Added this php code to the view: $data= CHtml::listData(Nvdaily::model()->findAll(), 'blahblahblah', 'blahblahblah');
(but it should probably be in the model)

4. Made my 'columns' array look like this:
'columns'=>array(
'id',
'blah',
'blahblah',
array (
'name'=>'blahblahblah',
'filter'=> $this->widget('ext.EchMultiselect.EchMultiselect',
array(
'model' => $model,
'dropDownAttribute' => 'blahblahblah',
'data' => $data,
'options' => array('buttonWidth' => 80, 'ajaxRefresh' => true),
),
true // capture output; needed so the widget displays inside the grid
),
),
array(
'class'=>'bootstrap.widgets.TbButtonColumn',
),
),

SQL Formatter

2013-10-07T06:26:00+00:00

I had no idea this existed until now: http://www.dpriver.com/pp/sqlformat.htm

It makes your SQL queries more readable.

I learned about it from this thread: http://www.reddit.com/r/PHP/comments/1n8kgz/how_do_you_write_readable_sql_queries/

Error Handling In mysqldb - Python

2013-10-06T22:56:00+00:00

I've had issues with queries failing and leaving connections open (enough to stall a server...). I know python's oursql library supports using the WITH keyword, and I think it will close the connection when there is an unexpected error. However, I'm not sure if I'm ready to move to a different library for MySQL (it's working well).

Here's what I'm currently doing to close the cursor and connection, then re-raise the error:

import MySQLdb

conn = MySQLdb.connect(user="username", passwd="secret", db="database", charset='utf8')
cur = conn.cursor()

try:
cur.execute("INSERT INTO testTable (userid) VALUES(%s);" % id)
conn.commit()
except:
cur.close()
conn.close()
raise
finally:
print "Insert Successful"

haufe.sharepoint Python Library - URLs With %20

2013-10-04T19:06:00+00:00

I changed line 25 of the haufe.sharepoint library to fix an issue I was having with URLs which had %20 (the url encoded space character):

New Code:
location = urllib.quote(self.location(), safe=":/")

Old Code:
location = self.location()

Just try using a URL with %20 with haufe.sharepoint to see what I'm talking about.

urllib2.URLError Python SUDS library

2013-10-03T21:51:00+00:00

To figure this out, you're going to need logging. Turn on logging with the following:

http://stackoverflow.com/questions/2388046/can-you-help-me-solve-this-suds-soap-issue

import logging logging.basicConfig(level=logging.INFO) logging.getLogger('suds.client').setLevel(logging.DEBUG) logging.getLogger('suds.transport').setLevel(logging.DEBUG) logging.getLogger('suds.xsd.schema').setLevel(logging.DEBUG) logging.getLogger('suds.wsdl').setLevel(logging.DEBUG)

pip 1.3 Not Working Behind Proxy

2013-10-03T19:00:00+00:00

http://pythontesting.net/python/pip/

Like this article says, it's fixed in the dev version of pip.

Bootstrap Starter Templates

2013-10-03T07:51:00+00:00

http://startbootstrap.com/all-templates

Installing Nagios

2013-10-02T21:16:00+00:00

I had a heck of a time finding the config file for nagios. It's under: /etc/nagios3/conf.d/localhost_nagios2.cfg

These guides were helpful:
http://ostatic.com/blog/monitoring-web-services-with-nagios
https://library.linode.com/server-monitoring/nagios/ubuntu-12.04-precise-pangolin

MySQL Not Starting On Reboot

2013-10-01T21:07:00+00:00

http://stackoverflow.com/questions/9859381/how-make-mysql-start-automatically-linux-cli-only

I was able to to the following, based on the post above:

sudo apt-get install sysv-rc-conf
sudo sysv-rc-conf
mark the 6 column for mysqld or mysql.server (6 = reboot)

Note: I tried sudo update-rc.d mysql.server defaults and it didn't work. It gave me the following error: System start/stop links for /etc/init.d/mysql.server already exist.

Fira Font

2013-09-25T06:28:00+00:00

https://github.com/mozilla/Fira

Mozilla's new Fira font looks great. Looking forward to use it on some future projects.

MySQL IF Statement Affect On Speed

2013-09-19T17:28:00+00:00

if(`table`.`start`='0000-00-00','',`table`.`start`)

That's the part of my query that was increasing the execution time by 10x.

I changed it to just `table`.`start` (without the IF statement) and the query was 10x faster.

Phonegap Start (for Phonegap Build)

2013-09-18T04:24:00+00:00

"A Hello World application to get started with PhoneGap Build."

https://github.com/phonegap/phonegap-start

Twilio PHP Library Using A Proxy

2013-09-17T20:48:00+00:00

https://github.com/twilio/twilio-php/issues/144

I found a spot where I can add a proxy to the twilio PHP library.

Go to TinyHttp.php and add the highlighted part (using your own URL of course):

$opts[CURLOPT_PROXY] = 'http://xxx.xxx.xxx.xxx:8080/';
try {
      if ($curl = curl_init()) {

Failed to connect to socket /var/run/dbus/system_bus_socket

2013-09-17T15:36:00+00:00

Error: Can't find out if NetworkManager is running: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory.
The following fixed the problem:
apt-get install dbus
(if you already have dbus installed: sudo service dbus start)
/etc/init.d/network-manager start

What caused the error?:
/etc/init.d/network restart

Could not initialize NMClient

2013-09-16T23:52:00+00:00

** (process:17986): WARNING **: error: could not connect to NetworkManager

Fix: Try running with sudo.

For example: sudo nm-tool | grep DNS

Check If Script Is already Running - Python

2013-09-08T18:36:00+00:00

http://pythonhosted.org/tendo/

import tendo

# If is there another instance already running:
# display the message “Another instance is already running, quitting.”, and return -1 error code.
me = SingleInstance()

Fighting WinSCP Time-Outs

2013-09-03T17:14:00+00:00

http://www.lampdocs.com/blog/2008/10/winscp-how-to-prevent-timeouts/

That's a guide to making WinSCP send a dummy command every 30 seconds to keep your connection alive. It's way better than constantly having to type in your password.

Crontab Mistake

2013-09-03T16:20:00+00:00

I wanted to run a script every 20 minutes, so I put this into my crontab:
20 * * * * sh script.sh

That's wrong. That only runs it once every hour 20 minutes into the hour. For example, "20,40 * * * * sh bash.sh" would run it twice - 20 and 40 minutes into the hour.

To run the script every 20 minutes like I was expecting:
*/20 * * * * sh script.sh

Include yiiGridView Javascript

2013-09-02T20:04:00+00:00

http://chevronscode.com/yii-include-preload-yiigridview-javascript/

I ended up with the following code at the top of the page (since I'm not using a layout):

Yii::app()->clientScript->registerCoreScript('jquery');
$assetsScriptUrl = Yii::app()->getAssetManager()->publish(Yii::getPathOfAlias('zii.widgets.assets'));
Yii::app()->getClientScript()->registerScriptFile($assetsScriptUrl .'/gridview/jquery.yiigridview.js');

PHP Best Practices

2013-08-20T06:52:00+00:00

https://phpbestpractices.org/

Things I learned:

It's easy to install PHP-APC for quick opcode caching, and it apparently serves the same purpose as memcached for a single server.
Memcache and Memcached are two different libraries.
There's a different (a small one) in speed between single and double quotes.
Use phpass to store passwords.

Set Up Wordpress For FTP

2013-08-16T22:19:00+00:00

If it's asking you for FTP login information when you try to update wordpress or a plugin, you don't need to do that! Chances are you don't have permissions correctly configured on that directory. Make sure apache owns the directory. On ubuntu, this will probably work (depending on which user owns apache):

sudo chown -R www-data:www-data /var/www

Add Multiple Unique Indexes - MySQL

2013-08-15T01:35:00+00:00

alter table users add unique index(id, name, phone);

Bootstrap Span Is Overlapping

2013-08-15T01:21:00+00:00

Try setting a static width on the span. This ensures responsive knows when to put it on another line. It solved my issues with the spans overlapping.

example: add style="width: 520px" to your div with class="span6"

mysql-client 5.6 or phpmyadmin for mysql 5.6

2013-08-15T00:02:00+00:00

Mysql-client 5.6 doesn't seem to exist, and you don't need it. Phpmyadmin installs mysql-client 5.5 automatically, and that works with mysql server 5.6. If you're having issues, you probably need to run "sudo dpkg-reconfigure phpmyadmin".

Turn Off MySQL Strict Mode

2013-08-14T23:59:00+00:00

http://nickbartlett.com/wordpress/how-to-turn-off-mysql-strict-mode/

SET @@global.sql_mode= ”; will set it temporarily. (to see if it will fix your problem)

To turn it off permanently, add this setting to your my.cnf (probably in /etc/mysql/my.cnf if you're using ubuntu):

[mysqld]

sql-mode =

If you're having issues getting this setting to work in your my.cnf, look for another my.cnf in /opt/mysql/server-5.6/my.cnf or elsewhere on your system using the "find / -name 'my.cnf'" command.

Using Dictionaries In Python's cursor.execute

2013-08-09T17:05:00+00:00

http://stackoverflow.com/questions/10983616/parameterized-queries-in-oursql

pymssql SELECT Always Returns None - Python

2013-08-07T18:54:00+00:00

apt-get remove python-pymssql

The version from apt-get has this error. Just use easy_install to install pymssql and select statements will work.

error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

2013-08-07T18:49:00+00:00

sudo apt-get install python-dev

Running that command on ubuntu will fix the issue.

If you're installing lxml:

Also try this:

sudo apt-get install libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev

Defcon Ace Badge

2013-08-02T17:10:00+00:00

WHMCS Hook - Trigger E-mail On New User Sign-Up

2013-07-28T19:48:00+00:00

This script needs to go into your WHMCS hooks folder (root/includes/hooks/):

<?php

function send_invitiation_email($vars) {

$command = "sendemail";
$adminuser = "admin"; // Your admin username
$values["messagename"] = "Google Groups Invitation"; // exact name of e-mail template
$values["id"] = $vars['userid'];

$results = localAPI($command,$values,$adminuser);

}

add_hook("ClientAdd",1,"send_invitiation_email");

?>

Bootstrap Form Builder

2013-07-24T21:53:00+00:00

http://minikomi.github.io/Bootstrap-Form-Builder/

This looks great for non-technical users who want a form built. It's a great response to "Hey, make me a checklist!"...

"You make the checklist the way you want it using this site, then I'll put it on the web."

PHPmyAdmin Loading Slowly (or not at all)

2013-07-23T19:04:00+00:00

http://future500.nl/phpmyadmin-slow-on-startup/

UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 12: ordinal not in range(128)

2013-07-19T19:49:00+00:00

The error in the title will occur on pre-0.12.0 versions of python’s Requests library. It seems to be fixed on later versions and maybe even earlier.

Here’s an example of how you save a zip file from behind a corporate proxy with python’s Requests library:

sess = requests.session() # you may have other get/post requests before you can get to the download page for the file url = 'https://example.com/report.zip' # this may be different for you, i just copied this from an existing request (using fiddler) file_headers = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.8', 'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)', 'Host': 'example.com', 'Accept-Encoding': 'gzip,deflate,sdch', 'Connection': 'Keep-Alive', 'Content-Type': 'application/octet-stream;charset=text/html;charset=UTF-8', } with open('report.zip', mode='wb') as handle: report_file = sess.get(url, headers=file_headers, prefetch=False, verify=False) handle.write(report_file.content)

That’s it! Just use the write function with the content of the returned request.

requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

2013-07-19T18:55:00+00:00

This happens with the Requests library in python. All you need to do is set verify=false.

For example:
cookieRequest = s.post(url, data=json.dumps(payload).strip('"'), allow_redirects=False, headers=headers, verify=False)

src/pycurl.c:42:20: fatal error: Python.h: No such file or directory

2013-07-19T17:14:00+00:00

The following helped to resolve this issue: sudo apt-get install python-dev

Writing To File - UnicodeEncodeError: 'ascii' codec can't encode characters

2013-07-17T18:10:00+00:00

I was getting a lot of errors when I was trying to download a file with the python requests library. The errors looked like the following:

UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 2-6: ordinal not in range(128)

This code ended up fixing it:

import codecs import requests import time # this could be different, but it works for my application, but the content-type definitely matters fileHeaders = { 'Content-Type': 'application/octet-stream;charset=text/html;charset=UTF-8' } url = 'www.yourfile.com/your.zip' with codecs.open('your.zip', mode='wb', encoding='utf-8') as handle: zip_file = requests.get(url, headers=fileHeaders, prefetch=False) for block in zip_file.iter_content(1024): if block: handle.write(block)

Prevent URL Encoding - Python Requests Library

2013-07-15T23:48:00+00:00

Use json.dumps in the data parameter.

for example:
r = s.post(url, data=json.dumps(payload), allow_redirects=False, cookies=cookieRequest.cookies)

Google Maps Event Firing Order

2013-07-11T19:49:00+00:00

http://gmaps-samples-v3.googlecode.com/svn/trunk/map_events/map_events.html

Tunnel Traffic Through Webserver Using Your Browser

2013-07-10T05:56:00+00:00

Use the following command in Cygwin (using your IP and Username, of course):

ssh yourUsername@198.168.xxx.xxx -D 2280 -i your/certificate

Set the following settings in Firefox (connect settings, under the advanced tab):

Dream Factory

2013-07-10T05:51:00+00:00

https://www.dreamfactory.com/

It's suspiciously free. It's a ready to go back-end that can be accessed with APIs.

Secure MySQL Command

2013-07-03T16:02:00+00:00

/usr/bin/mysql_secure_installation

Easyphp Change Directory To Github

2013-07-02T21:01:00+00:00

1. Right click on the easyphp icon --> configuration --> apache (this should open the httpd.conf in your easyphp apache directory)

2. Find the section labelled DocumentRootDirectory and overwrite with this:

# DocumentRootDirectory
<Directory "C:/Users/your_username/Documents/GitHub/your_project/">
#
# Possible values for the Options directive are "None", "All",
# or any combination of:
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important. Please see
# http://httpd.apache.org/docs/2.4/mod/core.html#options
# for more information.
#
Options Indexes FollowSymLinks Includes ExecCGI

#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
# Options FileInfo AuthConfig Limit
#
AllowOverride All

#
# Controls who can get stuff from this server.
#
Require all granted

</Directory>

Setting Up Github Windows Client Behind a Proxy

2013-07-02T20:59:00+00:00

1. Navigate to your git.exe directory (C:\Users\your_username\AppData\Local\GitHub\PortableGit~\bin)

2. git config --global http.proxy http://username:password@host:port/

3. git config --system http.sslcainfo /bin/curl-ca-bundle.crt

4. Open the github client (it should work now)

move_uploaded_file To Current Path - PHP

2013-06-29T23:23:00+00:00

$uploadfile = './' . basename($_FILES['file']['name']);

if (move_uploaded_file($_FILES['file']['tmp_name'], $uploadfile)) {
echo "File was successfully uploaded.\n";
} else {
echo "There was an issue uploading the file.\n";
}

View In Use and Will Not DROP - MySQL

2013-06-29T03:16:00+00:00

I had several views in my MySQL database that were showing "In Use" and phpmyadmin would not let me drop them through the interface.

The solution:
Open the MySQL in terminal and "DROP VIEW yourView;"

Include Array From File - PHP

2013-06-29T03:04:00+00:00

file.php
<?php
return array('item1','item2');
?>

Your php code:

<?php
$thisVariable = include('file.php');
print_r($thisVariable); //prints the array
?>

Fixing "Copying To Temp Table" - MySQL

2013-06-17T07:54:00+00:00

http://themanbehindthecode.com/2011/08/12/avoid-mysql-copying-to-tmp-table/

Don't Use Nested IFNULL, Use COALESCE - MySQL

2013-06-16T22:53:00+00:00

If you're looking for how to do a nested IFNULL, you probably need to be looking at how to use COALESCE instead. It returns the first non-null item in a list.

Example:

SELECT COALESCE(NULL,1);

Returns 1

IFNULL - MySQL

2013-06-16T22:41:00+00:00

IFNULL(`field1`, `field2`)

That returns field1 if it's not null. However, it will return field2 if it is null.

Notes On JOIN - MySQL

2013-06-16T21:43:00+00:00

If your INNER join is returning too few values and duplicates, you're probably using the wrong kind of join. Try a LEFT join.
INNER JOIN is the same as JOIN
http://i.stack.imgur.com/GbJ7N.png (diagram explaining joins)
LEFT JOIN is used - this will return ALL rows from Table1, regardless of whether or not there is a matching row in Table2.

Indexes Help... A Lot

2013-06-16T15:54:00+00:00

I just brought 4 queries from 15 seconds to less than 1 second by adding indexes to the columns in the ON() portion on the join.

Indexes help... A lot.

Yii-Booster Incorrect Width

2013-06-15T20:38:00+00:00

Set public $responsiveCss = false; to public $responsiveCss = true;

group_concat - MySQL

2013-06-13T19:38:00+00:00

http://mahmudahsan.wordpress.com/2008/08/27/mysql-the-group_concat-function/

This seems like it will be super useful.

If you're using "group by" and you still want to get data out of the grouped rows, then you can use this to combine multiple rows into a single row.

Looping Through Associative Array - PHP

2013-06-13T01:32:00+00:00

http://stackoverflow.com/questions/1951690/php-how-to-loop-through-a-associative-array-and-get-the-key-name

This was totally not very intuitive. I was expecting something like this:

foreach($array as $sub) {
echo(key($sub));
}

Instead, it's this:

foreach ($decodedOutput as $key => $value) {
echo $key;
}

Search Directory For String - Linux

2013-06-03T19:36:00+00:00

grep -rl "search_for_this_string" /var/www

PHPExcel Convert Excel Serial Date To MM/DD/YY

2013-05-31T08:31:00+00:00

PHPExcel_Shared_Date::ExcelToPHPObject($value)->format("m/d/y")

Easiest Way To Consume .Net Web Services - Python

2013-05-19T21:48:00+00:00

http://www.jansipke.nl/python-soap-client-with-suds/

That article was very helpful for learning to use Python's SUDS library for consuming soap web services.

Here's an example of my simple client:

from suds.client import Client
client = Client("http://xxx.xxx.xxx.xxx/API/Info.asmx?WSDL")
client.service.getInfo("05/12/2013","05/12/2013")

"05/12/2013" and "05/12/2013" are both parameters

New Vocabulary Word

2013-05-11T09:34:00+00:00

CRUD
-Create
-Read
-Update
-Delete

An example of a CRUD application would be something that tracks orders in a database. All 4 actions are going to be necessary to track those orders. CRUD: The basic database operations.

Dynamically Import Data From Other Workbooks - Google Spreadsheet

2013-05-08T05:26:00+00:00

I made a google spreadsheet which pulls data from two other workbooks using "importRange" and QUERY function. You can use either one if you just want to pull a few columns from another spreadsheet. QUERY provides more flexibility by allowing you to use a SQL statement. Here is an example of a query which pulls 2 columns from another workbook and filters out the blank rows in column 41:

=QUERY(importRange("yourSpreadsheetKeyHere", "Donations!A3:BB200000"), "select Col41, Col2 WHERE Col41!='' ", 1)

One of the biggest pains during the entire process was google spreadsheet's vlookup. I learned the vlookup function only searches for a value in the first column in the Array. It's also helpful if you format the results of your vlookup as "Plain Text" under the numbers format drop-down. Of course, something formatted as a number won't match something formatted as plain text.

You could use this technique if you have a list of customer id's with names, and a separate spreadsheet with more customer information. With this, you could combine that information.

Javascript Excel Parser

2013-05-07T08:20:00+00:00

http://niggler.github.io/js-xlsx/

That's an impressive project that allows parsing excel files in the browser.

Green Mountain Energy's B2B Site

2013-04-28T03:29:00+00:00

Green Mountain Energy has the worst B2B portal I’ve seen in my life. The worst parts:

It only works in IE (barely, and switching between browser versions in the F12 menu doesn’t help)
The page only takes up ~30% of the browser window (see the picture)
Nothing is functional, I can’t change my automated billing agreement because there’s no submit button.
If you have more than one address on your account, just give up.

Picture: http://i.imgur.com/cP4CLTl.png

Update: They sent a letter saying they made a new business portal site. www.businessportal.greenmountain.com My first visit gives me an “Our site is currently down as we upgrade our systems.” error message.

Don't Use == null - Python

2013-04-26T18:04:00+00:00

http://stackoverflow.com/questions/3289601/null-object-in-python

It says not to use == to check for "none-ness". It's better to use:

if variable is none:
<your code>

Why? "And the reason for choosing egg is None over egg == None: The latter can be overloaded, and is likely to break when comparing valid object with None (depends on how it's implemented, but you don't expect everyone to take comparisions with None into account, do you?), while is always works the same."

King Of Button Generators

2013-04-26T05:07:00+00:00

http://buttonoptimizer.com/

Something I Need To Drill Into My Brain - Python

2013-04-24T03:36:00+00:00

The operation of the slicing operator:

>>> word[:2]    # The first two characters
'He'
>>> word[2:]    # Everything except the first two characters
'lpA'

Prezi Video With No Audio

2013-04-22T22:06:00+00:00

I converted a wmv video to a few different formats (mp4 and flv) and still couldn’t get the audio to play in Prezi. However, videos with the following codec settings will work:

Solution: Open your video with Handbrake and convert it to H264 with AAC as the selected audio codec and it will play just fine.

Start Ser2Sock.c On Boot

2013-04-16T21:54:00+00:00

Make a shell script saying the following:
./ser2sock -b 115200 -c -w 5000 -p 10000 -s /dev/ttyUSB0 -d -g 3

Follow these instructions: http://jonathonhill.net/2009-04-23/auto-start-a-shell-script-on-ubuntu-server/

Easiest To Set Up FTP Server

2013-04-09T07:23:00+00:00

http://askubuntu.com/questions/1722/basic-ubuntu-ftp-server

"I'm going to recommend PureFTPD because it's been the simplest and easiest to use in my opinion. You'll need to install it first: sudo apt-get install pure-ftpd once it's installed it'll start itself up."

Useful Bootstrap Links

2013-04-08T22:00:00+00:00

http://bootstraphero.com/the-big-badass-list-of-twitter-bootstrap-resources

Also, an user named Billy Cravens on Meetup.com left a ton of good bootstrap links (bootsnip looks especially awesome) in this post: http://www.meetup.com/HoustonCFUG/events/79862792/

Twitter Bootstrap:
http://twitter.github.com/bootstrap/

Bootswatch: (free Bootstrap themes)
http://www.bootswatch.com

Wrap Bootstrap (premium Bootstrap themes)
http://www.wrapbootstrap.com

Common Bootstrap Snippets
http://www.bootsnipp.com

Some nice date plugins for Bootstrap
http://www.eyecon.ro/bootstrap-datepicker/ https://github.com/dangrossman/bootstrap-daterangepicker

Bootstrap WYSIWYG editor plugin
http://jhollingworth.github.com/bootstrap-wysihtml5/

Integrating Bootstrap with jQueryValidate
http://alittlecode.com/jquery-form-validation-with-styles-from-twitter-bootstrap/

Nice data grids:
http://www.datatables.net

Integrate Bootstrap with Data Tables:
http://datatables.net/blog/Twitter_Bootstrap_2

HTML5 In < IE9

2013-04-08T16:43:00+00:00

https://code.google.com/p/html5shiv/

I saw this javascript library in the code for the Mozilla Support page (really well done). It looks like it adds HTML5 functionality to older versions of IE. Looks like it should be really useful in the future.

PHP - Verify File Is Uploaded And Zip

2013-04-02T16:33:00+00:00

if($_FILES["zip_file"]["name"]) {
$filename = $_FILES["zip_file"]["name"];

$name = explode(".", $filename);

$continue = strtolower(end($name)) == 'zip' ? true : false;
if(!$continue) {
$message = "The file you are trying to upload is not a .zip file. Please try again.";
} else {
$message = "File Is .zip";
}
}

PHP - Loop Through Excel Document and Print Cells

2013-03-29T15:21:00+00:00

require_once 'excel_reader2.php';

foreach($reader->sheets as $k=>$data)
{
echo "\n\n ".$reader->boundsheets[$k]."\n\n";

foreach($data['cells'] as $row)
{
foreach($row as $cell)
{
echo "$cell\t";
}
echo "\n";
}
}

Head Tracking In The Browser

2013-03-17T01:55:00+00:00

http://learningthreejs.com/blog/2013/03/12/move-a-cube-with-your-head/

Had no idea this was even possible, but that's an example of head tracking in the browser done with WEBRTC.

Demo: http://jeromeetienne.github.com/tquery/plugins/headtrackr/examples/demo.html

Check If Time Is Between - Python

2013-03-16T23:28:00+00:00

http://stackoverflow.com/questions/1907196/how-do-i-check-if-its-monday-to-friday-and-the-time-is-between-10-am-to-3-pm

From an user named miku:

>>> import datetime
>>> d = datetime.datetime.now() 
# => datetime.datetime(2009, 12, 15, 13, 50, 35, 833175)

# check if weekday is 1..5
>>> d.isoweekday in range(1, 6)
True

# check if hour is 10..15
>>> d.hour in range(10, 15)
True

# check if minute is 30
>>> d.minute==30
False

Prevent Character Escaping - Python

2013-03-15T21:57:00+00:00

http://stackoverflow.com/questions/2042342/network-path-and-variables-in-python

If you want to use network paths without adding extra slashes, you can use python's raw string function.

Here's an example of something you want to make a raw string:

source_path = r"\\mynetworkshare"

Canvas JS Charts - Google Charts Killer?

2013-03-07T06:13:00+00:00

http://canvasjs.com/docs/charts/intro/high-performance-javascript-charts/

The most impressive part is how fast it renders all that data.

Check If Workbook Is Open By Name - VBA

2013-03-05T23:13:00+00:00

This code from StackOverflow helped me find whether a workbook was open with its name. Here's the link to the thread: Link

Function BookOpen(strBookName As String) As Boolean

Dim oBk As Workbook

On Error Resume Next

Set oBk = Workbooks(strBookName)

On Error GoTo 0

If oBk Is Nothing Then

BookOpen = False

Else

BookOpen = True

End If

End Function

Sub testbook()

Dim strBookName As String

strBookName = "myWork.xls"

If BookOpen(strBookName) Then

MsgBox strBookName & " is open", vbOKOnly + vbInformation

Else

MsgBox strBookName & " is NOT open", vbOKOnly + vbExclamation

End If

End Sub

"The link you have used to enter the PayPal system is invalid. Please review the link and try again."

2013-03-04T03:08:00+00:00

Using an incorrect e-mail address in the hidden "business" field in your HTML form can cause this issue.

Example of the hidden field with the problem:
<input type="hidden" name="business" value="dave@davesgarage.com">

Overdue Member Calculator

2013-03-01T12:07:00+00:00

https://github.com/pawl/OverdueMemberCalculator

This should help ease the pain of dealing with suspended paypal recurring payment agreements.

The page accepts variables from the URL, then generates a paypal payment button based on the person's last payment date.

HTML Skeleton / Boilerplate

2013-03-01T09:09:00+00:00

Here's a good HTML skeleton/boilerplate tool: http://www.htmlshell.com/

WebForms.js or WebUIValidation.js CDN

2013-03-01T07:23:00+00:00

In the console, it was giving me two errors saying it could not GET these two files from the CDN:

http://ajax.microsoft.com/ajax/4.0/2/WebForms.js

http://ajax.microsoft.com/ajax/4.0/2/WebUIValidation.js

To fix it, you need to delete the <script></script> references for both of the files above and make a single reference to:

Raspberry Pi Text To Speech IRC Service

2013-02-27T20:17:00+00:00

These are my notes for installing Eastein's announce. It's a IRC bot that will receive a message and do text to speech on the message: https://github.com/eastein/announce

Installation:

Download announce and extract it to it's own folder.
apt-get install python-irclib
apt-get install python-pip
sudo apt-get install python-dev
apt-get install festival festlex-cmu festlex-poslex festlex-oald libestools1.2 unzip
apt-get install speech-tools
pip install pyzmq --install-option="--zmq=bundled"
or (if you have easy_install installed):
easy_install pyzmq
download mediorc and move mediorc into directory
easy_install dnspython
apt-get install mplayer
apt-get install sox
Fix permissions:
chmod 777 saypitchprase
chmod 777 pitchphrase2wav

Running The Program:

Run process in background by starting a screen (first navigate to the Announce folder):

(start a new screen)
python announced "tcp://*:4900"
(detach from screen)
(start a new screen)
python announcebot "chat.freenode.net" "voicebot" "#yourircchannel" "tcp://0:4901"
(detack from screen)

Starting the program using mp3 files in the JSON file (as always, start in the folder which has Announce):

wget http://fake.com/fake/eye_tiger.mp3 (use a real URL with an mp3 file)
nano default.json
enter into the json file: [ ["eye of the tiger", "eye_tiger.mp3"] ]
ctrl+x and save
(start a new screen)
python announced "tcp://0:4900" default.json
(detach from screen)
(start a new screen)
python announcebot "chat.freenode.net" "voicebot" "#yourircchannel" "tcp://0:4900"
(detach from screen)
now "!say eye of the tiger" in the IRC channel will trigger the mp3

Troubleshooting:

Ensure your Raspberry Pi audio is working: http://jeffskinnerbox.wordpress.com/2012/11/15/getting-audio-out-working-on-the-raspberry-pi/

Testing Festival (create a text file named mytext.txt with some words first):

echo "This is a test." | festival --tts

echo "this is a test" > mytext.txt

text2wave -o myaudio.wav mytext.txt

aplay myaudio.wav

better voices:
http://ubuntuforums.org/showthread.php?t=677277

http://ubuntuforums.org/showthread.php?t=751169

adjust the volume:
http://blog.scphillips.com/2013/01/sound-configuration-on-raspberry-pi-with-alsa/
or use amixer?

Find An Image Font

2013-02-27T20:01:00+00:00

This page will tell you what font an image is using: http://www.myfonts.com/WhatTheFont/

I wish this feature was included in GIMP or Photoshop.

Regex Debugger

2013-02-23T00:38:00+00:00

http://www.debuggex.com/?re=&str=

Don't want to lose this link, it's going to make regex related programming a lot easier.

Tableizer

2013-02-19T22:48:00+00:00

Freaking brilliant tool that converts spreadsheet format things to HTML: http://tableizer.journalistopia.com/tableizer.php

PhoneGap Build - cordova-2.4.0.js

2013-02-19T20:39:00+00:00

If you're using Phonegap Build, you don't need to include cordova-2.4.0.js or any other specific version of cordova/phonegap. Just use phonegap.js, but you don't actually need to include phonegap.js in the file you upload to Phonegap Build.

Quote from the Phonegap Build site:
"Once you've included the necessary assets, remove the phonegap.js (cordova.js) as Build will automatically inject it during compile time."

Sharepoint Lists Crashing

2013-02-05T23:03:00+00:00

If your sharepoint lists crash IE as soon as they open, try disabling the following IE addons:

Sharepoint spreadsheet Launcher

Sharepoint export database~

Open File With Excel Macro

2013-02-05T15:31:00+00:00

This macro will ask the user to input an excel file, then open that excel file:

Sub RunMacro()

Dim vaFiles As Variant
Dim i As Long

vaFiles = Application.GetOpenFilename _
(FileFilter:="Excel Filer (*.xls),*.xls", _
Title:="Open File(s)", MultiSelect:=False)

If Not IsArray(vaFiles) Then Exit Sub

With Application
.ScreenUpdating = False

For i = 1 To UBound(vaFiles)
Workbooks.Open vaFiles(i)
Next i

.ScreenUpdating = True
End With

End Sub

Java - Swing's Nimbus Look & Feel

2013-02-01T16:32:00+00:00

Since Java SE 6 Update 10, Java has had an updated interface called Nimbus. Comparison pictures are below.

All I needed to do to use it was add this import to the top of the .java file with my main() method: import javax.swing.UIManager.*;

And this code to the top of the main() method (http://stackoverflow.com/questions/4617615/how-to-set-nimbus-look-and-feel-in-main):

public static void main(String[] args) {
try {
    for (LookAndFeelInfo info : UIManager.getInstalledLookAndFeels()) {
        if ("Nimbus".equals(info.getName())) {
            UIManager.setLookAndFeel(info.getClassName());
            break;
        }
    }
} catch (Exception e) {
    // If Nimbus is not available, fall back to cross-platform
    try {
        UIManager.setLookAndFeel(UIManager.getCrossPlatformLookAndFeelClassName());
    } catch (Exception ex) {
        // not worth my time
    }
}
new Controller();
}

Default:

Nimbus:

Note: This blog post says it is slightly slower than the default: http://www.pushing-pixels.org/2008/06/17/lightbeam-measuring-performance-of-swing-look-and-feels.html

"Protected mode is currently turned off for the Internet zone"

2013-01-23T17:58:00+00:00

Want to disable the IE7 pop-up saying “Protected mode is currently turned off”?

Solution: Just right-click the message and click “Don’t show this again”.

Ljava.lang.String; Error

2013-01-22T18:49:00+00:00

This is a simple one. You're asking java to return an Array as a string.

Solution: Put Arrays.toString( ) around your Array.

Java Exceptions - Catching All Errors

2013-01-22T18:42:00+00:00

"Exception is the base class for all exceptions"

"Thus any exception that may get thrown is an Exception (Uppercase 'E')."

Meaning, catch(Exception e) will catch any type of exception that is thrown, as opposed to specific Exceptions.

Example:
try {
// code
}catch(Exception e)
{
//catch all exceptions
}

http://stackoverflow.com/questions/1075895/how-can-i-catch-all-the-exceptions-that-will-be-thrown-through-reading-and-writi

Meaning Of "l" In LDAP

2013-01-16T20:53:00+00:00

"The lower case 'l' refers to the locality (city)."

Jquery Slideup Jerking Effect In IE9

2013-01-15T20:21:00+00:00

This website has code to help get rid of the jerking effect on Jquery SlideUp in IE9: https://siderite.dev/blog/jquery-slideup-flickers-in-internet.html

You just add this code after you include your Jquery library:

(function(){
// Define overriding method.
jQuery.fx.prototype.hide = function(){

// Remember where we started, so that we can go back to it later
this.options.orig[this.prop] = jQuery.style( this.elem, this.prop );
this.options.hide = true;

// Begin the animation
this.custom(this.cur(), 1);
}
})();

Raspberry Pi - No Ethernet Caused By Charger

2013-01-15T04:34:00+00:00

I was using the following charger, but I couldn't get my Raspberry Pi to connect over Ethernet (ethernet lights showed no link): http://www.amazon.com/gp/product/B0038HYPZS

The charger was definitely the problem. I swapped it with a Motorola OEM charger and the Ethernet started working again. I did some quick googling, and haven't seen many reports of this issue yet. I have the Model B, Rev 2.0 board with 512MB RAM.

WHMCS LDAP User Creation Action Hook

2013-01-14T07:44:00+00:00

I created an Action Hook for WHMCS which will create an user when they are added as a client. With this script, we no longer need to add our users to LDAP manually. I'm thinking about turning it into a module, and making it easier to other users to use.

https://github.com/pawl/WHMCS_LDAP

LDAP_ADD PHP - Object Class Violation

2013-01-14T07:11:00+00:00

If you are trying to use php's ldap_add to add an user and are getting an object class violation, try to copy all the fields of a working user exactly. Most likely you are missing some of the required fields of the objectclass you are trying to add.

Reading RFID With Raspberry Pi (or any linux device) + Reader

2013-01-14T06:59:00+00:00

I'm using the following Parallax USB RFID reader: http://www.parallax.com/Portals/0/Downloads/docs/prod/audiovis/28140-28340-RFIDreader-v2.2.pdf

First you will need to ensure you have PySerial installed by trying to "import PySerial" while running Python's IDLE. If you receive an error:

Install PySerial:

Install python's setuptools: apt-get install python-setuptools
Install PySerial: pip install pyserial

Once you have PySerial installed, open IDLE again and use the following code to see output from the reader:

Why didn't ser.read(12) return the same unique identifier as what is written on the fob?:

If the key on the card says 30788590 and the reader returns 3501D5CBEE. The "1D5CBEE" part is the card number. Try going from the card number in DEC to HEX it in the windows calculator while it's in scientific mode if you don't believe me.

"margin-left: auto;" and "margin-right: auto;" Not Working In IE8

2013-01-07T22:51:00+00:00

To get something centered within a div in IE8, it requires doing something unusual. I wasn't able to get the "display: block;" trick working. But this worked:

#super-wrap { position: absolute; width: 100%; text-align: center; }

#page-wrap { position: relative; width: 970px; margin: auto auto; }

I had to do this to get the InfoGrid (http://css-tricks.com/examples/InfoGrid/) to center in IE8. The div with the ID of #page-wrap already exists, but the super-wrap div needs to be created outside of the page-wrap div.

Using "nth-of-type" with IE8

2013-01-07T16:44:00+00:00

IE8 apparently doesn't support the CSS selector "nth-of-type".

You can fix it by adding this script to your page: https://github.com/keithclark/JQuery-Extended-Selectors/blob/master/jquery-extra-selectors.js

You will also need to add the following script to the bottom of your page (of course you will need to swap the numbers with your own):

$("div:nth-of-type(1)").css("background", "#b44835");

I used this to fix an IE8 compatibility issue with something called InfoGrid: http://css-tricks.com/examples/InfoGrid/

PHP - Generate SSHA (sha1) Password For LDAP User

2013-01-07T02:48:00+00:00

$info['userpassword'][0] = "{SHA}" . base64_encode(sha1("pass", TRUE));

Function To Find Largest UID Number In LDAP

2013-01-07T02:43:00+00:00

I got most of this function from http://bakery.cakephp.org/articles/UncleBill/2006/10/15/using-ldap-as-a-database. It will find all the uidNumbers, sorting them largest to smallest, then return the largest number.

$ds = ldap_connect("localhost"); // assuming the LDAP server is on this host

function findLargestUidNumber($ds)
{
$s = ldap_search($ds, "ou=people,dc=yourdomain,dc=com", 'uidnumber=*');
if ($s)
{
// there must be a better way to get the largest uidnumber, but I can't find a way to reverse sort.
ldap_sort($ds, $s, "uidnumber");

$result = ldap_get_entries($ds, $s);
$count = $result['count'];
$biguid = $result[$count-1]['uidnumber'][0];
return $biguid;
}
return null;
}

$largestUID = findLargestUidNumber($ds);

LDAP shadowlastchange - Weird Date Format

2013-01-07T02:09:00+00:00

I'm working on a PHP script that adds an user to a directory using LDAP.

Here's the formula required to get the date format that shadowlastchange uses:
$unixTimeDays = floor(time()/86400);

It's the days since the last epoch.

First Android App

2012-12-31T13:04:00+00:00

https://play.google.com/store/apps/details?id=com.DallasTollCalculator

Py2exe Issues

2012-12-28T17:21:00+00:00

When getting the directory of the file you're executing in python use the correct example:

CORRECT: os.path.abspath(os.path.dirname(sys.argv[0]))
INCORRECT: os.path.dirname(inspect.getsourcefile( lambda:None ))

If you're using py2exe, the incorrect example will just give error messages.

Outlook 2010 - Run a Script (Blank)

2012-12-27T15:47:00+00:00

Apparently a script does not equal a macro in outlook. When you're setting up a rule, you have the option to run a script. However, your macro will not show up in the script menu unless it has (MyMail As MailItem) as an argument. Example:

Sub Save(MyMail As MailItem)

SaveEmailAttachmentsToFolder "AgentReports", "html", "C:\Users\EBRNPAL\Documents\Agent Reports\Saved"

End Sub

Save Outlook Attachments Automatically With A Macro

2012-12-21T22:26:00+00:00

Source: http://www.rondebruin.nl/mail/folder2/saveatt.htm

Sub Test()
'Arg 1 = Folder name in your Inbox
'Arg 2 = File extension, "" is every file
'Arg 3 = Save folder, "C:\Users\Ron\test" or ""
'If you use "" it will create a date/time stamped
'folder for you in the "My Documents" folder.
'Note: If you use this "C:\Users\Ron\test" the folder must exist

SaveEmailAttachmentsToFolder "AgentReports", "html", "C:\Users\Paul\Documents\Agent Reports\Saved"

End Sub

Sub SaveEmailAttachmentsToFolder(OutlookFolderInInbox As String, _
ExtString As String, DestFolder As String)
Dim ns As NameSpace
Dim Inbox As MAPIFolder
Dim SubFolder As MAPIFolder
Dim Item As Object
Dim Atmt As Attachment
Dim FileName As String
Dim MyDocPath As String
Dim I As Integer
Dim wsh As Object
Dim fs As Object

On Error GoTo ThisMacro_err

Set ns = GetNamespace("MAPI")
Set Inbox = ns.GetDefaultFolder(olFolderInbox)
Set SubFolder = Inbox.Folders(OutlookFolderInInbox)

I = 0
' Check subfolder for messages and exit of none found
If SubFolder.Items.Count = 0 Then
MsgBox "There are no messages in this folder : " & OutlookFolderInInbox, _
vbInformation, "Nothing Found"
Set SubFolder = Nothing
Set Inbox = Nothing
Set ns = Nothing
Exit Sub
End If

'Create DestFolder if DestFolder = ""
If DestFolder = "" Then
Set wsh = CreateObject("WScript.Shell")
Set fs = CreateObject("Scripting.FileSystemObject")
MyDocPath = wsh.SpecialFolders.Item("mydocuments")
DestFolder = MyDocPath & "\" & Format(Now, "dd-mmm-yyyy hh-mm-ss")
If Not fs.FolderExists(DestFolder) Then
fs.CreateFolder DestFolder
End If
End If

If Right(DestFolder, 1) <> "\" Then
DestFolder = DestFolder & "\"
End If

' Check each message for attachments and extensions
For Each Item In SubFolder.Items
For Each Atmt In Item.Attachments
If LCase(Right(Atmt.FileName, Len(ExtString))) = LCase(ExtString) Then
FileName = DestFolder & Item.SenderName & " " & Atmt.FileName
Atmt.SaveAsFile FileName
I = I + 1
End If
Next Atmt
Next Item

' Show this message when Finished
If I > 0 Then
MsgBox "You can find the files here : " _
& DestFolder, vbInformation, "Finished!"
Else
MsgBox "No attached files in your mail.", vbInformation, "Finished!"
End If

' Clear memory
ThisMacro_exit:
Set SubFolder = Nothing
Set Inbox = Nothing
Set ns = Nothing
Set fs = Nothing
Set wsh = Nothing
Exit Sub

' Error information
ThisMacro_err:
MsgBox "An unexpected error has occurred." _
& vbCrLf & "Please note and report the following information." _
& vbCrLf & "Macro Name: SaveEmailAttachmentsToFolder" _
& vbCrLf & "Error Number: " & Err.Number _
& vbCrLf & "Error Description: " & Err.Description _
, vbCritical, "Error!"
Resume ThisMacro_exit

End Sub

Declaring 2 Variables At The Same Time In Python

2012-12-21T16:57:00+00:00

I thought this was interesting:

import os

fileName, fileExtension = os.path.splitext('/path/to/somefile.ext')

Since os.path.splitext returns a list of 2 items, you can assign those 2 list items to variables at the same time with the code above.

Path To Sharepoint

2012-12-18T23:23:00+00:00

http://www.pathtosharepoint.com

A lot of that stuff looks useful for adding functionality to Sharepoint. Also, this is one of the first public sites I've seen which use sharepoint (and it's not too bad). I saw it mentioned on this site: http://sharepointjavascript.wordpress.com/

UTDesign - Xbee Claims vs Reality

2012-12-15T08:55:00+00:00

The picture above shows some research done by some UTD students for their UT Design program.

Their results showed the specs claimed by Xbee modules were far superior to reality. They said they experienced an overwhelming amount of noise at about half of the rated distance and both the peak and idle current ratings were inaccurate.

Unheap - Uncluttering Plugins

2012-12-13T16:48:00+00:00

http://www.unheap.com/

That website is an excellent list of a lot of good jquery plugins out there.

Raspberry Pi - Mining Bitcoins

2012-12-13T07:58:00+00:00

Rather than letting my Raspberry Pi lay dormant in its cabinet at a data center, I've decided to use it for mining bitcoins.

I followed these instructions: http://www.youtube.com/watch?v=rrdzam7voOg

I'm using Deepbit as the pool, but I'm not 100% sure what a pool is or if there are better pools out there.

Update: In about 5 days, I have only mined 0.00003898 BTC.

Stop IE Alert Messages

2012-12-04T23:30:00+00:00

The following snippet of Jquery will prevent alerts from showing in internet explorer:

if ($.browser.msie) { window.alert = function() { }; };

It detects if the browser is IE, then disables the alert function with a blank if it is IE. The alert happens only in IE, that's why it checks to see if the browser is IE.

I did this because there is some code I can't change on a sharepoint page which is causing an alert (which doesn't actually cause any real issues). Adding this code to the part of the page I can change will stop the error.

About the alert:
It says "Hit error fn!" and it happens in the following span after an ajax call fails: <span id="ctl00_LogUserActivity1">

It's definitely a hack, but it works so far.

Raspberry Pi Webserver w/ Wordpress Slow

2012-11-28T08:22:00+00:00

I installed the LAMP stack on a Raspberry Pi, then moved a small Wordpress site to it. It then got plugged into a fast connection at a data center.

Before the move (shared hosting webserver at Siteground):

1 second load times on the main page

After the move (Raspberry Pi):

8-9 second load times on the main page
5 second load times on single pages

The Pi does a good job at displaying static pages, but it does not run more intensive things like a CMS quickly enough.

LAMP Stack Ubuntu

2012-11-28T03:54:00+00:00

It's crazy easy to install the LAMP stack on ubuntu with this command:

sudo apt-get install lamp-server^

SPJS Charts - Dynamically Select Filter Item

2012-11-26T20:53:00+00:00

I've been using SPJS charts to make google charts with data from sharepoint lists. If you use a custom drop-down filter, you may want to dynamically select items in the drop-down like I did.

An example: You're using a drop-filter which filters the results by week, but you always want it to select last week.

Snippet:

function manualLoad(){
loadManually = false;
spjs_GenerateChart();

Date.prototype.getWeek = function() {
var onejan = new Date(this.getFullYear(),0,1);
var today = new Date(this.getFullYear(),this.getMonth(),this.getDate());
var dayOfYear = ((today - onejan + 1)/86400000);
return Math.ceil(dayOfYear/7)
};

$("#MyChart1_CustomFilterSelect_Week").val(new Date().getWeek()-1).change();
}

2nd Place - Ericsson & North Texas Food Bank Hackathon

2012-11-13T22:54:00+00:00

The “Hackathon” was a programming competition sponsored by Ericsson to solve some of the North Texas Food Bank’s problems.

We made a scheduling system that allows people to see which days need more volunteers, then sign up to volunteer. The coolest part is that it would call people with a prerecorded message reminding them of their appointment.

Calculate Weeks In Sharepoint

2012-11-13T22:51:00+00:00

=INT((Created-DATE(YEAR(Created),1,6)+(TEXT(WEEKDAY(DATE(YEAR(Created),0,1)),"d")))/7)+1

This is a modified version of this formula which didn't give me the correct week number (maybe due to regional settings?): http://lamahashim.blogspot.com/2009/10/sharepoint-calculated-field-week-number.html

It matches the number of weeks calculated in excel and all the calculators I could find on the internet.

Update: I also needed to add a 0 before the "Week" if the number is less than 10, because it will cause an error in sorting otherwise.

Here's the formula to add a 0 if the number is less than 10:
=IF((INT((Date-DATE(YEAR(Date),1,6)+(TEXT(WEEKDAY(DATE(YEAR(Date),0,1)),"d")))/7)+1)<10,"0"&(INT((Date-DATE(YEAR(Date),1,6)+(TEXT(WEEKDAY(DATE(YEAR(Date),0,1)),"d")))/7)+1),(INT((Date-DATE(YEAR(Date),1,6)+(TEXT(WEEKDAY(DATE(YEAR(Date),0,1)),"d")))/7)+1))&" "

(Note: the &" " part forces it to be a string rather than a floating point value)

Impressive Javascript

2012-11-01T19:36:00+00:00

https://github.com/mbostock/d3/wiki/Gallery

"D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation."

Google Charts - GeoMap vs GeoChart

2012-10-25T18:33:00+00:00

GeoMap - The map is rendered in the browser using an embedded Flash player.

GeoChart - Rendered using SVG

I noticed the GeoChart loads faster (at least in Chrome).

"Object date(2012,10,4) has no method 'getTimezoneOffset'" Google Charts

2012-10-24T17:41:00+00:00

This error was caused by not lowercase date in date(2012,10,4). It should be Date(2012,10,4).

Sizeof() vs Count() in PHP

2012-10-22T18:27:00+00:00

http://php.net/manual/en/function.sizeof.php

They're the same. According to the link above: the sizeof function is an alias of: count().

PHP Date Variables

2012-10-19T19:48:00+00:00

http://php.about.com/od/learnphp/ss/php_functions_3.htm

That's a helpful post for controlling the output of PHP's date().

Difference Between Echo and Print In PHP

2012-10-19T15:58:00+00:00

http://stackoverflow.com/questions/3615679/what-is-basic-difference-echo-vs-print

"echo can print more than 1 argument, print can only print 1 argument."

Triple Equals Sign In Php

2012-10-19T15:50:00+00:00

http://stackoverflow.com/questions/80646/how-do-the-equality-double-equals-and-identity-triple-equals-comparis

Since PHP uses $ to declare variables (and not int, etc), one can run into issues with an if statement being unintentionally true.

For example, 0 == null.

The triple equals sign (===) allows you to prevent these cases by only returning true if the two inputs are not the same type.

So, 0 === null is not true. Also, 0 === "0" is not true either.

SQLSRV30.EXE is not a valid Win32 application

2012-10-15T15:41:00+00:00

If you want to install the PDO drivers for PHP 5.4, you will need to extract SQLSRV30.EXE with Winrar.

Since I was using EasyPHP, I needed to extract the files into the following directory (there's also another php directory in EasyPHP, not sure why, but it's worth adding the files under "ext" there too): C:\Program Files\EasyPHP-12.1\php\php546x120827090829\ext

You will also need to add the following lines to your PHP.ini file under PHPExt:

extension=php_pdo_sqlsrv_54_ts.dll
extension=php_sqlsrv_54_ts.dll

Managing SQL Server 2008 with Management Studio 2005

2012-10-12T16:38:00+00:00

When you google "SQL Management Studio", one of the first things that comes up is this (the 2005 version): http://www.microsoft.com/en-us/download/details.aspx?id=8961

I recommend not installing 2005 if you have SQL Server 2008 Express installed! You will not be able to manage your 2008 server, and you will need to uninstall management studio to install the 2008 version here: http://www.microsoft.com/en-us/download/details.aspx?id=22985

Find Values With VBA

2012-10-04T15:07:00+00:00

http://www.rondebruin.nl/find.htm

The above website has some great snippets of code which can:

Get Folder Path From User Input

2012-09-28T15:26:00+00:00

This is some very useful code from Stack Overflow which allows the user to select a folder in VBA and reads the folder path into a variable. I'm currently making a macro which opens several XML files in a folder, and this will help users to input the folder where those XML files are located:

Sub SelectFolder()
Dim diaFolder As FileDialog

' Open the file dialog
Set diaFolder = Application.FileDialog(msoFileDialogFolderPicker)
diaFolder.AllowMultiSelect = False
diaFolder.Show

MsgBox diaFolder.SelectedItems(1)

Set diaFolder = Nothing
End Sub

VBA - Delete Worksheet With A Specific Name If It Exists

2012-09-17T18:39:00+00:00

Application.DisplayAlerts = False

On Error Resume Next

ThisWorkbook.Sheets("Sheet1").Delete

On Error Goto 0

Application.DisplayAlerts = True

The code between "On Error Resume Next" and "On Error Goto 0" will be skipped if an error is generated.
It also won't display the error to the user because of the code on line 1 (Application.DisplayAlerts = False).
"ThisWorkbook" will need to be declared.

Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object

2012-09-14T14:51:00+00:00

If you get this error when you just type “java” into the command prompt, then it can be solved by typing “PATH=C:\Program Files\Java\jdk1.7.0_06\bin” (the folder of your java.exe).

Modifying PhoneGap Childbrowser View

2012-08-14T16:21:00+00:00

In IOS, removing the back/forward, refresh, and history buttons from ChildBrowser requires opening its .xib file.

I removed everything except the done button:

Childbrowser Plugin

2012-08-14T14:11:00+00:00

http://blog.digitalbackcountry.com/2012/03/installing-the-childbrowser-plugin-for-ios-with-phonegapcordova-1-5/

I thought that was a really good guide for getting started on Phonegap's Childbrowser plugin.

I'm using it to open PDFs from phonegap.

My First Stack Overflow Answer

2012-08-07T18:59:00+00:00

http://stackoverflow.com/questions/11783402/iscroll-and-similar-will-not-scroll-horizontally-phonegap-jqtouch-iphone-app/11852309#11852309

I use Stack Overflow a ton, and I'm going to try to make more of an effort to contribute.

I was having trouble scrolling horizontally with the iscroll plugin, and I solved it by enabling the hScroll option (which I thought should have been enabled by default). I use iscroll's zoom function to enable zooming on a single div, while the rest of the page does not zoom.

SigPad API

2012-08-07T13:59:00+00:00

I had difficulties getting Thomas J Bradley's HTML5 Signature Pad to let me to use its API without resetting the field. This Github issue explains a workaround:

https://github.com/thomasjbradley/signature-pad/issues/13

He is saving the signature, declaring the api variable, then regenerating the signature.

I needed to use the .getSignatureImage() api function which allows me to turn the signature into a base64 string so I could store the signature in a database.

PHP Inline HTML

2012-07-24T18:20:00+00:00

http://www.4webhelp.net/tutorials/php/echo.php

The article in the link above describes the process of adding inline HTML to PHP code instead of "echo '<html>';". It seems like it's much faster to not process each echo statement with PHP.

This is definitely a big "oh!" moment for someone still learning PHP.

Jquery Mobile - Only Load Once

2012-07-24T14:28:00+00:00

If you only want something to load once, you can use the "pageinit" event.

Here's an example of some code with "pageinit":

$(document).delegate("#yourPage", "pageinit", function(event) { alert( "ALERT!"); });

The code above would not run if you visited the page again.

Make Phonegap Responsive

2012-07-23T16:34:00+00:00

The following was absolutely mandatory for me to get my phonegap app to be responsive: https://github.com/cargomedia/jquery.touchToClick

Without modification, Phonegap will wait for a "double-click" after the user taps. This results in a noticeable delay. With the plugin above included, it removes the delay.

Other plugins I tested wanted me to use selectors for each element I didn't want delayed.

Prevent Scrolling

2012-07-16T13:03:00+00:00

http://hankchizljaw.co.uk/tutorials/prevent-window-bounce-phonegap-1-7-ios-tip/13/05/2012/

The above link is the most simple solution I've found to prevent phone gap from scrolling outside of the bounds.

Magnifying Glass For Images Plugin

2012-07-13T19:07:00+00:00

Here's a link to a cool magnifying glass plugin which uses Jquery:
http://www.gcmingati.net/wordpress/wp-content/lab/jquery/minizoompan/

Using Quotes In mysql_fetch_array()

2012-07-05T13:28:00+00:00

According to this blog post: http://www.securityandcaffeine.com/2008/04/03/php-mysql-and-mysql_fetch_array/

mysql_fetch_array() can be sped up dramatically by including quotes, like this:

$result['2'] rather than this $result[2]

I just changed my code to use quotes, and I haven't noticed a big speed increase (that's probably not where the bottleneck is).

Edit: This is probably misleading according to the good people at stackoverflow. http://stackoverflow.com/questions/11345894/mysql-fetch-array-quotes-increase-speed

Sisyphus.js

2012-07-03T16:56:00+00:00

I'm really impressed with sisyphus.js and its usage of HTML5 localStorage. It makes it so users won't need to re-enter data into a form when they close the window.

It's super easy to implement, you only need to include the following in your header:

<script type="text/javascript" src="sisyphus.min.js"></script>

And, you will need to activate it on the form you want to save with something like this:

$('form').sisyphus();

Source: http://simsalabim.github.com/sisyphus/

Write To Closest Div (with certain class)

2012-06-26T15:56:00+00:00

The following code makes it so I don't have to mention specifically what page needs to load before a script runs. This is good for having the same function on several different pages in Jquery Mobile.

var pageId = $(this).closest('div.diagramPage').attr('id');

$(document).delegate(pageId, 'pageshow', function() { }

I included the class because I don't want to end up moving code around and having it write to a different div.

Fixing: "The requested URL /jquery-1.6.3.min.js was not found on this server."

2012-06-19T05:58:00+00:00

In Wordpress, the following is the error message that occurs at the top of the page:

Not Found

The requested URL /jquery-1.6.3.min.js was not found on this server.

www.4llw4d.freefilesblog.com

I fixed this error by editing the all-in-one SEO plugin's code (all_in_one_seo_pack.php). I'm pretty sure the following part of the code is the culprit:

$url = "http://www.4llw4d.freefilesblog.com/jquery-1.6.3.min.js";

$ch = curl_init();

$timeout = 5;

curl_setopt($ch,CURLOPT_URL,$url);

curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);

curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);

$data = curl_exec($ch);

curl_close($ch);

echo "$data";

}

I tried changing it to something else, like: http://code.jquery.com/jquery-1.6.3.min.js However, it just prints the entire contents onto the page.

The solution: Switch from the Pro 1.72 version of All-In-One SEO to the newest unpaid version.

Jquery Mobile - Display Loading Message

2012-06-18T15:14:00+00:00

I've been displaying the ajax loading message when loading a php file (especially when there's a chance it won't run instantly).

I've been doing the following:

$.mobile.showPageLoadingMsg ();

$('#phpDiv').load('file.php',function(){

$('#phpDiv').trigger('create');

});

$.mobile.hidePageLoadingMsg ();

"showPageLoadingMsg" will start to display the ajax loading message and "hidePage~" will close it. The stuff in the middle loads what the php file returns into a div, then creates DOM contents for it with the trigger('create') function.

Best Jquery Mobile Datebox Plugin

2012-06-14T15:13:00+00:00

The currently reigning champion of datebox plugins for Jquery Mobile: http://dev.jtsage.com/jQM-DateBox/

Jquery Mobile - Close Jqm-Datebox When Dialog Closes

2012-06-14T15:12:00+00:00

// this will make sure the datebox closes when the dialog closes
$(document).delegate('#closeButton', 'click', function() {
$(".ui-datebox-container").hide();
$(".ui-datebox-screen").hide();
});

Make sure your #closeButton is the same as the button which closes the dialog.

TodoMVC with MySQL Database

2012-06-13T14:37:00+00:00

https://github.com/bartcc/app.todos

This project is a modification of the TodoMVC (used to sample how a language is structured by making a To-Do list). It adds database support and ends up being a very cool To-Do list.

Good Jquery Mobile Blog Post

2012-06-13T13:39:00+00:00

Just got finished reading the following blog post: http://roughlybrilliant.com/jquery_mobile_best_practices#0b

Most of it's obvious, but it's definitely on my list of good blog posts for Jquery Mobile beginners.

Reminds me how Jquery Mobile needs an official "Best Practices" document.

Loading Scripts Dynamically In Jquery Mobile

2012-06-11T19:05:00+00:00

At first, I loaded all of my scripts in the header (even when the scripts weren't needed when the page was being loaded).

I found this website that partially explained how to load scripts when the page loads. However, Jquery Mobile needs to use the delegate function to load the script when a page is showing.

I copied the function from the website above:

function loadjscssfile(filename, filetype){
 if (filetype=="js"){ //if filename is a external JavaScript file
  var fileref=document.createElement('script')
  fileref.setAttribute("type","text/javascript")
  fileref.setAttribute("src", filename)
 }
 else if (filetype=="css"){ //if filename is an external CSS file
  var fileref=document.createElement("link")
  fileref.setAttribute("rel", "stylesheet")
  fileref.setAttribute("type", "text/css")
  fileref.setAttribute("href", filename)
 }
 if (typeof fileref!="undefined")
  document.getElementsByTagName("head")[0].appendChild(fileref)
}

Now, you need to add code to the top of the data-role="page" where you want the script to load:

$(document).delegate('#page', 'pageshow', function() {
 loadjscssfile("your.css", "css")
 loadjscssfile("your.js", "js")

});

Best Jquery Calendar Plugin

2012-06-05T18:47:00+00:00

FullCalendar can be found here: http://arshaw.com/fullcalendar/

I thought it looked the best and had the most API functions out of the HTML5 calendars I saw.

Jquery Add/Remove Class

2012-06-05T18:25:00+00:00

Problem: In Jquery Mobile, I used the mobile.changePage (method) to switch pages. Now the button for the previous page is still blue.

Solution: Add Id's to the list items you'll be dealing with. Then use Jquery to remove the class "ui-btn-active" from the old button, and add the "ui-btn-active" class to the new button.

Here's the code:
$('#listItem1').removeClass('ui-btn-active')
$('#listItem2').addClass('ui-btn-active')

Global Variables In Javascript

2012-06-05T15:14:00+00:00

Problem: I had two different <script> (javascript) sections, and I was unable to pass a variable from one set of <script> tags to the other.

Turns out it doesn't have anything to do with the script tags, and I just needed to learn a bit more about scope in Javascript. Apparently global variables are a common thing (maybe still not a good practice?) and as long as a variable is used outside of a function.. You'll be able to access the variable anywhere on the page.

Solution: Take the variable outside of the function.

Jquery Serialize

2012-06-05T15:05:00+00:00

I spent a while looking for a good way to pass variables from HTML to Javascript (ajax) to PHP. Turns out Jquery’s serialize function is exactly what I needed.

Here’s an example of the code:

//this will send all the inputs in the form to php with ajax
$('#form').submit( function() {
     var data = form.$('input').serialize();
     $.ajax({
          type: "POST",
          url: "process.php",
          data: data,
          success: function(msg){
          alert( "Your quote request has been submitted!" );
          }
     });
     return false;
} );

http://api.jquery.com/serialize/

Also, you can use serializeArray() to get the data in a better format. Here’s a picture of the console.log output: