Blog

SSH session slow to start? It's the DNS stupid!

Liraz Siri — Mon, 08 Mar 2010 11:10:42 +0000

Ever tried logging into a machine with ssh and found you have to wait much longer than reasonable for the session to start? This happened to me a few times and was especially annoying with machines on my local network (or a VM attached to a virtual network) that should be letting me in immediately.

I eventually got mad enough to strace the SSH daemon and debug what was going on and it turns out it's a DNS thing. Basically the session is slow to start because the SSH server is trying to lookup the hostname of the SSH client and for whatever reason it's timing out (e.g., it can't reach a nameserver, because you happen to be offline)

There are a couple of very simple ways to fix that:

add "UseDNS no" to /etc/ssh/sshd_config
add the client's net address to the server's /etc/hosts

Was that helpful? Has this ever happened to you? Post a comment!

Tweeting for news-a-holics

Alon Swartz — Wed, 03 Mar 2010 10:30:17 +0000

Hi, my name is Alon and I'm a news-a-holic.

Lets face it, some of us have addictions, be it alcoholism, nicotine, gambling, over-eating, television, or even just mowing the lawn. Mine is being up to date with the latest news.

Some addictions are worse than others in being self-destructive, and as you already know, identifying and acknowledging you have a problem is the first step. I am also a work-a-holic, so when I realized that having to know the latest news was impacting my workflow, I needed to find a solution.

Luckily, with a little self discipline I was able make a rule and stick with it: "Access to RSS reader is off limits from work laptop". The problem was how to still consume news, with the added bonus of easily sharing interesting articles on Twitter that I come across (I am reading the news anyway, so why not share it).

So, this is what I came up with.

I have been a long fan of RSS, and have a long list of feeds set up in Google Reader, which I now only access from my mobile. When I come across something that is really interesting, I hit "share".

Everything I share in Google Reader appears on my shared items page, which has an RSS feed of its own.

Updating Twitter with an RSS feed is simple thanks to services like twitterfeed. I have also configured twitterfeed to use my bit.ly account (URL shortener) so I can get click-through stats.

Initially I planned on tweaking news items I share according to the stats, but in practice I don't. I don't even checkup on the stats, so I guess its just nice-to-have.

So there you have it:

Sites that interest me are fed into Google Reader.
Articles that I share appear in my RSS feed.
Twitterfeed picks up the RSS feed, shortens the links with bit.ly, and sends them to Twitter.

Do you have an addiction? How do you tweet? Leave a comment!

Smack yourself if you don't use generic shell script hooks

Liraz Siri — Mon, 01 Mar 2010 07:01:10 +0000

Smack yourself in the forehead if you don't use the following snippet (or an equivalent) in all custom shell scripts that could benefit from a hooking mechanism:

run_scripts()
{
    for script in $1/*; do

        # skip non-executable snippets
        [ -x "$script" ] || continue

        # execute $script in the context of the current shell
        . $script
    done
}

run_scripts path/to/hooks.d

I've found this pattern useful in nearly every sufficiently complex shell script I maintain, and I even use it to help manage my bashrc and xsession configurations:

$ ls .bashrc.d/
paths git editor pager autologout qemu tmpdirs scratch

$ ls .xsession.d/
10-lang 10-tmpdir 90-bell 90-kbdrate 91-xscreensaver 99-fvwm-conf

Features:

Modular: add or remove code without having to edit a big monolithic file.
Order of execution is determined by the filename. Changing the order is as simple as changing a number prefix.
Disable execution by removing execution bit:
```
chmod -x .bashrc.d/git
```

These days breaking down configurations into separate modular files like this is common in the Linux world, so by now I expect many experienced users are wondering why I'm channeling Captain Obvious. Just keep in mind that many Linux newbies haven't yet learned all our best practices and some lessons are worth reteaching.

Why not share your own tricks? Post a comment!

Background task processing and deferred execution in Django

Alon Swartz — Wed, 24 Feb 2010 06:15:09 +0000

Or, Celery + RabbitMQ = Django awesomeness!

As you know, Django is synchronous, or blocking. This means each request will not be returned until all processing (e.g., of a view) is complete. It's the expected behavior and usually required in web applications, but there are times when you need tasks to run in the background (immediately, deferred, or periodically) without blocking.

Some common use cases:

Give the impression of a really snappy web application by finishing a request as soon as possible, even though a task is running in the background, then update the page incrementally using AJAX.
Executing tasks asynchronously and using retries to make sure they are completed successfully.
Scheduling periodic tasks.
Parallel execution (to some degree).

There have been multiple requests to add asynchronous support to Django, namely via the python threading module, and even the multiprocessing module released in Python2.6, but I doubt it will happen any time soon, actually I doubt it will ever happen.

This is a common problem for many, and after scouring over many forum posts the following proposed solution keeps popping up, which reminds of me of the saying "when all you have is a hammer, everything looks like a nail".

Create a table in the database to store tasks.
Setup a cron job to trigger processing of said tasks.
Bonus: Create an API for task management and monitoring.

Well, you can do it like that, but it usually leads to ugly, coupled code, which can become very complex over time, not very flexible, doesn't scale well, and generally a bad idea.

In my opinion, it ultimately comes down to seperation of concerns. I recently fell in love with the message queuing world (AMQP), in particular RabbitMQ, which can be used as an integral part of a really elegant solution for this issue, especially when coupled with Celery.

Define a task.
Send it to a processing queue.
Let other code handle the processing.

What is Celery

Celery is a task queue system based on distributed message passing. Originally developed for Django, it can now be used in any Python project.

It's focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single (or multiple) worker server. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Celery provides a powerful and flexible interface to defining, executing, managing and monitoring tasks. If you have a use-case, chances are you can do it with Celery.

Installation and configuration

Install Celery

One of Celery's dependencies is the multiprocessing module released in Python2.6. If you have an earlier version, such as Python2.5, you're in luck as the module has been backported.

When installing the backported module, it will need to be compiled, so lets install the required support.

apt-get install gcc python-dev

Now we are ready to install celery, lets install a few more dependencies and let easy_install take care of the rest.

apt-get install python-setuptools python-simplejson
easy_install celery

Install RabbitMQ

Celery's recommended message broker is RabbitMQ.

RabbitMQ is a complete and highly reliable enterprise messaging system based on the emerging AMQP standard. It is based on a proven platform, offers exceptionally high reliability, availability and scalability.

In the below example, I will download and install the latest release (at time of writing), but you should check their download page for newer versions and/or support for your platform.

Note: Installation will fail if there are missing dependencies. Because of this, we use the --fix-broken workaround.

wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.7.2/rabbitmq-server_...
dpkg -i rabbit-server_1.7.2-1_all.deb
apt-get --fix-broken install

The default installation includes a guest user with the password of guest. Don't be fooled by the wording of the account, guest has full permissions on the default virtual host called /.

We will use the default configuration below, but you are encouraged to tweak your setup.

Configure Django project to use Celery/RabbitMQ

Add the following to settings.py

BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672
BROKER_VHOST = "/"
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"

INSTALLED_APPS = (
    ...
    'celery',
)

Synchronize the database

python manage.py syncdb

Sample code

Now that everything is installed and configured, here is some sample code to get you started. But, I recommend taking a look at the Celery documentation to get acquainted with its power and flexibility.

fooapp/tasks.py

from celery.task import Task
from celery.registry import tasks

class MyTask(Task):
    def run(self, some_arg, **kwargs):
        logger = self.get_logger(**kwargs)
        ...
        logger.info("Did something: %s" % some_arg)

tasks.register(MyTask)

fooapp/views.py

from fooapp.tasks import MyTask

def foo(request):
    MyTask.delay(some_arg="foo")
    ...

Now start the daemon and test your code.

python manage.py celeryd -l INFO

For convenience, there is a shortcut decorator @task which makes simple tasks that much cleaner.

A note on state: Since Celery is a distributed system, you can't know in which process, or even on what machine the task will run. So you shouldn't pass Django model objects as arguments to tasks, its almost always better to re-fetch the object from the database instead, as there are possible race conditions involved.

Have you ever needed to use background/deferred execution in Django? Post a comment!

Good automation vs bad automation

Liraz Siri — Mon, 22 Feb 2010 12:30:52 +0000

I recently eliminated a bit of code that was supposed to handle upgrading our build infrastructure from using one distribution (e.g., Ubuntu 8.04 LTS) to another (e.g., Ubuntu 10.04 LTS). That got me thinking about how to decide (and then explain) when it's a good idea to automate and when it isn't.

Since writing and maintaining any piece of software comes with a cost, you always have to weigh the costs against the benefits.

I boiled it down to the following:

Good automation: the best kind of automation can reliably handle tasks that it is slow, error prone and labor intensive to perform manually. That kind of automation is usually a good idea because once it works well enough you can basically forget about it and enjoy the benefits from an accelerated development cycle (AKA tightened developer feedback loop), reduced amount of friction from debugging human mistakes. It basically frees up your mind and precious labor for other tasks.
Bad automation: the worst kind of automation handles in an unreliable, error-prone way tasks that are infrequently performed and simple to handle manually. Not only do you have to pay for the cost of developing and maintaining this kind of bad automation, you also get an overall decrease in productivity for your efforts because the automation mechanism will require more maintenance and attention then the tasks it supposedly handles.

A litmus test for good vs bad automation is the ability to test. For example, if you try to automate something that happens infrequently there is a good chance that the assumptions embedded in your automation will not hold over time.

So case in point, since there is no way to test that a bit of automation will work for a future release transition, it's safe to assume it probably won't.

Django Signals: Be lazy, let stuff happen magically

Alon Swartz — Wed, 17 Feb 2010 10:10:46 +0000

When I first learned about Django signals, it gave me the same warm fuzzy feeling I got when I started using RSS. Define what I'm interested in, sit back, relax, and let the information come to me.

You can do the same in Django, but instead of getting news type notifications, you define what stuff should happen when other stuff happens, and best of all, the so-called stuff is global throughout your project, not just within applications (supporting decoupled apps).

For example:

Before a blog comment is published, check it for spam.
After a user logs in successfully, update his twitter status.

What you can do with signals are plentiful, and up to your imagination, so lets get into it.

What are Django signals?

In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place. They're especially useful when many pieces of code may be interested in the same events.

This might remind you of Event driven programming, and rightfully so, the core concepts are very similar.

Django provides a set of built-in signals that let user defined code get notified by Django itself of certain actions, for example:

# Sent before or after a model's save() method is called.
django.db.models.signals.pre_save | post_save

# Sent before or after a model's delete() method is called.
django.db.models.signals.pre_delete | post_delete

# Sent when a ManyToManyField on a model is changed.
django.db.models.signals.m2m_changed

# Sent when Django starts or finishes an HTTP request.
django.core.signals.request_started | request_finished

Built-in signals are really useful, but what I really like is the ability to define custom signals, and due to the way the "signal dispatcher" works, it allows decoupled applications to be notified when actions occur elsewhere in the framework.

In other words, one of your apps can send a signal when something happens, and a different app can listen for the signal and do something when it's received.

Using Django signals

Defining and sending a signal

All signals are django.dispatch.Signal instances. The providing_args is a list of the names of arguments the signal will provide to listeners.

Application: foo

signals.py

from django.dispatch import Signal
user_login = Signal(providing_args=["request", "user"])

views.py

from foo import signals

def login(request):
    ...
    if request.user.is_authenticated():
        signals.user_login.send(sender=None, request=request, user=request.user)
    ...

In the above example, a signal will be sent once the user logs in successfully.

Note: The above is just for exemplary purposes, it should be very rare to create your own authentication system as opposed to leveraging django.contrib.auth or one of the great apps out there.

Just as a side note, sender is usually defined as self when sending the signal from a class, such as a model class. This gives the intercepting handler instant access to the related class instance.

Listening to signals

To receive a signal, you need to register a receiver function that gets called when the signal is sent.

Application: bar

tasks.py

from foo.signals import user_login

def user_login_handler(sender, **kwargs):
    """signal intercept for user_login"""
    user = kwargs['user']
    ...

user_login.connect(user_login_handler)

Now, when a user logs in successfully and the signal is sent, it will be intercepted and the handler can then do what ever is required. Note that multiple receivers can be registered for a single signal.

Where should the code live?

You can put signal handling and registration code anywhere you like.

However, you'll need to make sure that the module it's in gets imported early on so that the signal handling gets registered before any signals need to be sent. This makes your apps models.py a good place to put registration of signal handlers.

Integrating with django.contrib.auth

Seeing as I opened the door to this, let me digress a little.

The login and logout methods of django.contrib.auth do not send signals, and is discussed in this ticket. The main reason is that signals are synchronous (discussed in the next section) and would slow down the logout/logout process.

If you do need authentication related signals, there are a few ways of accomplishing it, each with their own pros and cons, such as:

Patch django.contrib.auth
Create a custom auth backend
Create a wrapping view

Creating a wrapping view provides the most flexibility, while writing less code and leveraging great code that already exists.

Application: foo

urls.py

from django.conf.urls.defaults import *
from foo.views import login

urlpatterns = patterns('',
   url(r'^login/$', login,
       {'template_name': 'fooapp/login.html'}, name='auth_login'),
   )

views.py

from django.contrib.auth.views import login as auth_login
from foo import signals

def login(request, **kwargs):
    """workaround wrapper for auth.login that sends user_login signal"""
    response = auth_login(request, **kwargs)
    if request.user.is_authenticated():
        signals.user_login.send(sender=None, request=request, user=request.user)

    return response

The above will wrap the auth.login method, and send a signal when a user successfully logs in. We are also passing a custom login template, but of course you don't need to.

Signals are synchronous

As I mentioned above, signals are synchronous, or blocking (just like everything else in Django). This means that the request will not be returned until all signal handlers are done.

There have been multiple requests to add asynchronous support to Django, namely via the Python threading module, but I doubt it will happen anytime soon, if at all. In my opinion it comes down to separation of concerns, and there is a whole other world called message queuing, namely AMQP which is designed for these sort of things.

In an upcoming post I will discuss implementing AMQP (Advanced Message Queuing Protocol) with RabbitMQ and Celery.

Ever needed to use signaling in your Django webapp? Post a comment!

We don't need no stinking SSL

Liraz Siri — Mon, 15 Feb 2010 06:34:57 +0000

Why we disabled SSL and use an SSH tunnel for web site administration

Content managements systems like the one we're using for the web site (Drupal) need to provide a privileged administration interface which you usually want to access securely. Due to the insecure nature of the Internet, it's reasonable to assume your traffic may be intercepted at some point. So how do you prevent that?

Up until recently, we used SSL. You could access the web site from both:

Unfortunately, as the site grew in complexity this created a range of subtle but annoying paper-cut type problems.

For example, I configured the site to use a content filter which translates local urls (e.g., /lamp) into absolute urls (e.g., http://www.turnkeylinux.org/lamp ). This is necessary for links and images in blog posts to work correctly in RSS feeds and email updates (e.g., such as those provided by FeedBurner). We didn't want to do that by hand because we were using FCKeditor and IMCE which create local links by default and it wasn't any fun to have to translate every single link into an absolute URL by hand.

The problem is that Drupal cache saves pages AFTER the content filter, so guess what happens to links in pages you access via SSL? The situation could be a mix-up of http and https links which would be potentially confusing to both human visitors and search engines.

More importantly while in theory SSL should have protected our administration credentials in practice it didn't.

Drupal, along with many other web applications doesn't support secure SSL cookies. That means even if you login via SSL your cookie is still transmitted over the clear when you access the site, say accidentally, via HTTP. In a perfect world that might never happen but in practice it's quite a frequent mistake.

An attacker that can intercept your traffic can then intercept the cookie containing your session key and use that to access the site with your privileges.

So using SSL in this way doesn't really add that much security.

OTOH, session keys are temporary where passwords have much longer lifetimes so you still don't want to transmit your password in the clear.

Alternative: access the site through an SSH tunnel

So perhaps somewhat unintuitively, after considering our options we decided to turn off SSL and access the web site through an SSH tunnel which serves as a poor man's VPN.

That sounds a bit complicated but it really isn't.

From my ~/.ssh/config:

host tkl
DynamicForward 1081

Then when I ssh to tkl, which is the VPS which hosts the web site, this creates a virtual socks proxy bound to 127.0.0.1:1081. Constructing the tunnel in this way can be a bit of a hassle but you can set that up to happen automatically (future post).

We then configure SwitchProxy FireFox extension to make it easy to switch to browsing within the tunnel.

Configuration tips:

disable the SwitchProxy toolbar (Views->Toolbars)
added my staging server test.turnkeylinux.org to "No proxy for..."

And then to switch into the tunnel: Tools->SwitchProxy->tkl

A nice bonus is that in our case this setup seems to be noticeably snappier than using SSL.

Converting a virtual disk image: VDI or VMDK to an ISO you can distribute

Alon Swartz — Thu, 11 Feb 2010 07:21:03 +0000

Why would anyone in their right mind want to convert a VM into an ISO?

Good question, the answer for Conor Fox (who was the inspiration for this post - thanks Conor!) was to distribute his customized TurnKey PostgreSQL image so others could use it.

Distributing an ISO as opposed to a VM image allows it to be installed on any virtualization platform, as well as on bare metal, with the added bonus of running live.

I suppose that's a good enough reason, so lets get to it.

Convert VM disk to raw image and mount it

First we need to get qemu-img, a tool bundled with qemu (KVM's virtualization backend) to convert the VM disk to a raw image, and TKLPatch, the TurnKey customization mechanism to package the ISO.

If you are not using a TurnKey installation, see the TKLPatch installation notes.

apt-get install qemu
apt-get install tklpatch

I'll show how to convert a VMWare VMDK image into raw disk format. If you are using a different virtualization platform such as Virtualbox, see this post on converting a VDI to a raw image.

qemu-img convert -f vmdk turnkey-core.vmdk -O raw turnkey-core.raw

Next, mount the raw disk as a loopback device.

mkdir turnkey-core.mount
mount -o loop turnkey-core.raw turnkey-core.mount

GOTCHA 1: If your VM has partitions, it's a little tricker. You'll need to setup the loop device, partition mappings and finally mount the rootfs partition. You will need kpartx to setup the mappings.

loopdev=$(losetup -s -f turnkey-core.raw)

apt-get install kpartx
kpartx -a $loopdev

# p1 refers to the first partition (rootfs)
mkdir turnkey-core.mount
mount /dev/mapper/$(basename $loopdev)p1 turnkey-core.mount

Extract root filesystem and tweak for ISO configuration

Now, make a copy of the root filesystem and unmount the loopback.

mkdir turnkey-core.rootfs
rsync -a -t -r -S -I turnkey-core.mount/ turnkey-core.rootfs

umount -d turnkey-core.mount

# If your VM had partitions (GOTCHA 1):
kpartx -d $loopdev
losetup -d $loopdev

Because the VM is an installed system as opposed to the ISO, the file system table needs to be updated.

cat>turnkey-core.rootfs/etc/fstab<<EOF
aufs / aufs rw 0 0
tmpfs /tmp tmpfs nosuid,nodev 0 0
EOF

GOTCHA 2: If your VM uses a kernel optimized for virtualization (like the one included in the TurnKey VM builds), you need to replace it with a generic kernel, and also remove vmware-tools if installed.

tklpatch-chroot turnkey-core.rootfs

# inside the chroot
apt-get update
apt-get install linux-image-generic
dpkg --purge $(dpkg-query --showformat='${Package}\n' -W 'vmware-tools*')
dpkg --purge $(dpkg-query --showformat='${Package}\n' -W '*-virtual')

exit

Generate the ISO

Finally, prepare the cdroot and generate the ISO.

tklpatch-prepare-cdroot turnkey-core.rootfs/
tklpatch-geniso turnkey-core.cdroot/

Thats it!

Bonus: By default the ISO will boot automatically. If you want to include the TurnKey bootsplash and bootmenu, extract the cdroot from a TurnKey ISO and tell tklpatch-prepare-cdroot to use it as a template.

tklpatch-extractiso turnkey-core.iso
tklpatch-prepare-cdroot turnkey-core.rootfs/ turnkey-core.cdroot/
tklpatch-geniso turnkey-core.cdroot/

Ever needed to package a VM as a distributable ISO? Post a comment!

Converting a virtual disk image: VDI to VMDK to a raw loopback file you can mount

Liraz Siri — Tue, 09 Feb 2010 18:00:00 +0000

By default, VirtualBox creates virtual disk images in a special format called VDI, which is unique to VirtualBox. Disk images are stored in $HOME/.VirtualBox/HardDisks.

You'll need to convert VDI into another format if you want to run a VirtualBox VM on another virtualization platform, such as VMWare or KVM.

The VMDK virtual disk format is a good choice because even though it originated with VMWare it is supported by other virtualization platforms including VirtualBox and KVM.

VirtualBox enables the conversion using the low-level "VBoxManage clonehd" command:

VBoxManage list hdds # prints a list of disk image UUIDs
VBoxManage clonehd <UUID> -o converted.vmdk format VMDK
cd ~/.VirtualBox/HardDisks/
ls -la converted.vmdk

Once you have converted to VMDK you can use qemu-img, a tool bundled with qemu (KVM's virtualization backend) to further convert VMDK to other formats.

A particularly useful format to convert to is 'raw' which you can then mount as a loopback device:

apt-get install qemu
qemu-img convert -f vmdk converted.vmdk -O raw converted.raw
mount -o loop converted.raw /mnt

Ever had to convert between disk image formats? Post a comment!

Friends don't let friends program in shell script

Liraz Siri — Tue, 02 Feb 2010 05:38:54 +0000

Lately I've been going over a hellish patch-work of old shell scripts we wrote to automate some internal processes and I realized something: friends shouldn't let friends program in shell script.

Why?

Shell script is ugly.
Shell script is unreliable.
Shell script doesn't have exceptions.
Shell script is hard to debug.
Shell script is hard to test.
Shell script isn't object oriented.
Shell script is full of idiosyncratic cruft.
Shell script doesn't have any decent data structures.
Shell script doesn't scale.
Shell script encourages code duplication.

The problem with shell script is that it's so deceptively easy to get started. Once you've started adding each bit of extra complexity seems easier than trashing the whole thing and starting from scratch in a decent language (e.g., Python) with a decent design.

Speaking from personal experience, nearly every time I have used shell script in recent times for something that isn't completely trivial, I have come to regret it. Ugh.

Sometimes you don't have a choice. For example, one time I had to use shell script to create a transparent command line hijacking function for a utility I was working on, and just look at the abomination I am ashamed to have been forced to write:

hijack_command() {
    CALLBACK=$1
    COMMAND=$2
    NEW_COMMAND=$3

    ORIG_COMMAND=$(which $COMMAND 2>/dev/null)

    eval "

        function $COMMAND() {
            args=()
            for arg; do
                args[\${#args[*]}]=\"'\$arg'\"
            done
            if $CALLBACK; then
                eval $NEW_COMMAND \${args[*]}
            else
                if [ \"$ORIG_COMMAND\" ]; then
                    eval $ORIG_COMMAND \${args[*]}
                fi
            fi
        }"
}

When a language forces you to do nested metaprogramming something as trivial as this, run and don't look back!

Have you ever regretted using shell script? Share your thoughts!