Blog

Announcing TurnKey OpenStack optimized builds

Alon Swartz — Wed, 01 Feb 2012 11:51:51 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for us. Unfortunately there are too many players in the cloud software space for us to support every single one. It's much easier to put effort into making TurnKey work well with the winning horses.

TurnKey has been supported on the leading public cloud platform Amazon EC2 from early on, not to mention simplifying management and deployment via the Hub.

OpenStack is particularly interesting, because as it is most likely the future of open source clouds.

I originally got intrigued when I heard about NASA planning to open source Nebula in 2009, which has become the basis for Nova, the compute component in OpenStack. Since then, I've been following OpenStack development from a far and have been itching to develop support for TurnKey on the platform.

The time has finally arrived, and I'm pleased to announce TurnKey optimized builds are hot out of our build farm, and available for immediate download and deployment.

You can get them from the "Download -> More Builds" link on the appliance pages.

TurnKey OpenStack optimized builds

EBS auto-mounting support: we've updated our custom EBSmount mechanism for OpenStack, which automatically mounts EBS devices when attached.
Support for automating instance setup: via the user-data scripts mechanism.
Automatic APT configuration on boot: saves bandwidth costs by using the closest package archive for maximum performance.
SSH key support: instances that are launched with a key-pair will be configured accordingly.
SSH host key fingerprints displayed in system log: verification of server to prevent man-in-the-middle (mitm) attacks.
Randomly generated root password: is set on first boot, and displayed in the system log **.
Randomly generated mysql/postgres passwords: the MySQL root and/or PostgreSQL postgres passwords are set to to the same random password as root **.
Instance metadata python library and CLI: used internally, but useful for advanced users. (learn more).

** Because OpenStack builds are used in headless deployments (without a console), they include an inithook which preseeds default values, and random passwords:

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=$(mcookie | cut --bytes 1-8)

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=turnkey
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE

Depending on your use case, you can utilize user-data (note the security implications) to preseed during boot, or once the system has booted by executing turnkey-init.

Exemplary import of TurnKey Core on OpenStack

There are several ways of uploading an image into an OpenStack deployment, below is one way to get you started.


# cd /tmp
# tar -zxf turnkey-core-11.3-lucid-x86-openstack.tar.gz
# ls turnkey-core-11.3-lucid-x86
    turnkey-core-11.3-lucid-x86-initrd
    turnkey-core-11.3-lucid-x86-kernel
    turnkey-core-11.3-lucid-x86.img

# IMG=turnkey-core-11.3-lucid-x86

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=aki \
    disk_format=aki \
    name="$IMG-kernel" \
    < /tmp/$IMG/$IMG-kernel

Added new image with ID: 5

# KERNEL_ID=5

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=ami \
    disk_format=ami \
    kernel_id=$KERNEL_ID \
    name="$IMG" \
    < /tmp/$IMG/$IMG.img

Added new image with ID: 6

# glance -A $GLANCE_TOKEN index

ID  Name                                Disk Format  Container Format  Size
--  ----------------------------------  -----------  ----------------  ---------
6   turnkey-core-11.3-lucid-x86         ami          ami               688498688
5   turnkey-core-11.3-lucid-x86-kernel  aki          aki               4179712

# euca-describe-images

IMAGE   ami-00000006    turnkey-core-11.3-lucid-x86         available  public                  machine aki-00000005
IMAGE   aki-00000005    turnkey-core-11.3-lucid-x86-kernel  available  public                  kernel

Announcing TurnKey OpenVZ optimized builds (+ Proxmox VE channel)

Alon Swartz — Mon, 16 Jan 2012 14:30:41 +0000

OpenVZ and Proxmox VE has been a recurring topic of discussion on the forums, for which we have Jeremy to ~~blame~~ thank. He's done tons of research, testing, preaching, and then some.

What I love about Open Source is that if you have an itch, and the drive to scratch it yourself, you can.

That's exactly what Jeremy and Adrian did. They wanted OpenVZ optimized builds for their Proxmox VE deployments, so they developed a TKLPatch that would convert an ISO into an OpenVZ container. And if that wasn't enough, took the time to upload some of the builds to sourceforge so it would be easier for others to leverage their work.

Hats off to you guys, you rock!

TurnKey OpenVZ optimized builds

Based on Adrian's and Jeremy's work, we were able to add OpenVZ support to our build infrastructure in no time, and after some initial testing, triggered the whole appliance library to be built as optimized OpenVZ containers.

You can get them from the "Download -> More Builds" link on the appliance pages.

Pre-seeding / default passwords

Because OpenVZ builds are used in headless deployments (without a console), they include an inithook which preseeds default values and passwords (excluding the root password which is handled by the VZ CLI tools).

/usr/lib/inithooks/firstboot.d/29preseed

DB_PASS=turnkey
APP_PASS=turnkey
APP_EMAIL=admin@example.com
APP_DOMAIN=DEFAULT
HUB_APIKEY=SKIP
SEC_UPDATES=FORCE

Depending on your use case, you can preseed the values before the system is booted for the first time, or once the system has booted by executing turnkey-init.

It would be great if someone would add preseeding support to PVE...

TurnKey Proxmox VE channel

A while back the Proxmox folks came up with the idea of adding a TurnKey channel to PVE, to allow users to download TKL appliances in the same way their custom built appliances are downloaded.

It was a great idea, but unfortunately it never got off the ground.

As I mentioned above, the great thing about Open Source is that you can scratch your own itch, and I was curious how the channel mechanism worked - so I dived in. When I came up for air I had added minimal third party channel support and a TurnKey Linux channel (github).

What this basically means is you can now download and deploy any TurnKey appliance on your PVE server in a couple of clicks without leaving your browser.

I hope to see this integrated in the upcoming PVE 2.0 release. If you're running PVE 1.9 then you can add the TurnKey channel as follows:

cd /usr/share/perl5/PVE
mv APLInfo.pm APLInfo.pm.bak
wget https://raw.github.com/turnkeylinux/pve-patches/master/PVE/APLInfo.pm

# update appliance list
pveam update

The DDoS spam bot from hell (a suburb of China)

Liraz Siri — Fri, 30 Dec 2011 14:50:32 +0000

Happy new year everyone,

I'm back online to put out a fire. My inbox was full of alerts that the CPU on the server that runs the site was maxing out.

Well boys and girls, it turns out www.turnkeylinux.org has been under an escalating distributed denial of service attack that started about two weeks ago. To the best of my knowledge the site continued operating normally. We use a ton of caching. Did any of you notice a slowdown?

Lucky for us the "attack" was braindead simple so it was easy to figure out what was happening and block the offending IPs. 32 nodes from 4 Chinese /16 network blocks which I sincerely hope aren't home to any TurnKey fans:

60.169.73.186
222.186.24.101
60.169.78.19
60.169.75.168
61.160.232.38
222.186.26.164
60.169.78.57
60.169.78.174
61.160.232.22
60.169.78.193
60.169.78.177
222.186.25.134
60.169.78.15
60.169.78.52
60.169.75.50
60.169.78.54
61.160.232.39
60.169.78.7
61.160.232.58
61.160.232.4
61.160.232.10
60.169.75.161
60.169.78.42

All using the same User Agent:

Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

Supposedly identifies as Firefox but from the logs it's transparent it isn't behaving like a real browser. For example, a real browser gets CSS and image files. This just crawls all over the site and POSTs a zillion times the kind of predictable crap our spam filter blocks half-asleep.

What does that sound like? Ah yes, a poorly programmed, incredibly persistent spam bot network from hell. None of the spam attempts went through our countermeasures but it still took up a ton of CPU time.

Being naturally inquisitive I investigated the offending IPs and it turns out most of them are running a remotely exploitable version of SSH (SSH-2.0-OpenSSH_4.3). I'm half tempted to run metasploit to get into these systems and clean away the spambot software as a public service but that's illegal and I'm a bit busy besides.

Wouldn't it be neat though if we had a net equivalent of the Justice League to deal with the kind of lowlife scum who commandeer hapless machines to run very low quality spam software?

Note that I tried doing the right thing and looked up the abuse contact for the network that was attacking us (and presumably thousands/millions of other sites) on WHOIS:

person:         Jinneng Wang
address:        17/F, Postal Building No.120 Changjiang
address:        Middle Road, Hefei, Anhui, China
country:        CN
phone:          +86-551-2659073
fax-no:         +86-551-2659287
e-mail:         wang@mail.hf.ah.cninfo.net
nic-hdl:        JW89-AP
mnt-by:         MAINT-NEW
changed:        wang@mail.hf.ah.cninfo.net 19990818
source:         APNIC

Then instead of sending off an angry e-mail into the void I actually picked up the phone, dialed the number, and listened to some funky Chinese elevator music until some guy (Mr. Jinneng Wang I presume?) who didn't speak English picked up and eventually hung up on me after an akward mutually incomprehensible exchange. Of course. How could it be any different?

I don't get it, what's the point of putting up an abuse contact in the WHOIS records if the person listed doesn't speak English? Just list the abuse contact in Mandarin and get it over with.

Sometimes I feel like a character in a Neal Stephenson novel.

Mapping AWS data centers for fastest connection

Alon Swartz — Thu, 29 Dec 2011 12:25:00 +0000

Yes, that's 'fastest', not closest.

Background

A while back I published a blog post entitled Finding the closest data center using GeoIP and indexing, which described how we automatically determine the AWS regional data center to be used for storing encrypted server backups.

We used the same solution to determine the preferred region when launching cloud servers in the Hub, as well as selecting the closest APT package archive for all TurnKey deployments.

New and improved

Since the original publication, Amazon built new regional data centers in Oregon, Sao Paulo and Tokyo, so the indexes needed to be updated.

While adding support for the new regions I decided to take it a step further and add some improvements.

Improvement #1: Automatic association (distance)

The method originally used to perform automatic association of countries/states to data centers was lacking some what and needed to be improved.

We are now using the Haversine formula, which is used to determine great-circle distances between two points on a sphere from their longitudes and latitudes.

Improvement #2: Incorporated world wide underwater cables (latency)

Originally we relied on user feedback of connection latency to tweak the indexes. This didn't scale very well, so we needed a way to make it easier.

Based on Gregs Cable map, we could mashup the automatic associations and tweak the index overrides based on expected latency.

It turns out that this was a crucial part of the equasion, as a user might be physically closer to data center X, but in reality the connection to data center Y is faster. For example, previously Australia was allocated to Singapore but has been moved to California as the pipe is much fatter (see the visual map below).

Improvement #3: Open source

We originally published the indexes, but have now open sourced the whole project on github in hope that others might find it useful, and make collaboration easier.

Putting it all together

The below screenshot plots countries/states to their associated AWS regional data centers, and overlays the world wide underwater cables for reference:

Want to zoom in? Toggle active and future cables? Check out the live mashup.

TurnKey 11.3 maintenance release - next stop Ubuntu 12.04!

Liraz Siri — Tue, 06 Dec 2011 07:34:59 +0000

Ho ho ho, happy holidays everyone! I know most of you are already shifting into holiday mode, so I'll keep it short and sweet.

We've just pushed out TurnKey 11.3 - the final maintenance release based on Ubuntu 10.04. The next release will be based on Ubuntu 12.04. We're already shifting into high gear for that. There will be surprises. Hopefully good ones!

Anyhow the new images we just pushed out from our CloudTask automation swarm include fixes for various bruises and scrapes, as well as the very latest security updates.

If you've already installed a previous version of TurnKey 11, you don't need to download anything because by default TurnKey is configured to automatically install all of the security updates over the network.

The maintenance release will mainly be of interest to new users and existing users doing new deployments. Especially those of you who are super impatient like mua and don't care to wait long minutes after deployment for the system to pull over a ton of security updates. This cuts down the time it takes to fully deploy Core in the cloud from 5 minutes to just 30 seconds.

New Hub feature: Cloud server monitoring

Alon Swartz — Fri, 02 Dec 2011 02:15:11 +0000

Ladies and gentle geeks, I'm proud to announce we've just pushed out 100% free basic server monitoring to all TurnKey Hub accounts. This should make it easier to keep tabs on the health and performance of your cloud servers. Existing Hub users don't need to do anything to enjoy this new feature. It just works.

A better server dashboard

As you can see in the screenshot below, the server dashboard now includes thumbnail graphs of CPU utilization, disk IO and network traffic for the last hour:

Give me more!

Alright, so instead of the last hour, you want data on how your server was doing last night? Or last week? No problem. CloudWatch samples performance at 5 minute intervals, and stores up to two weeks worth of data.

Clicking on the thumbnail graph pops up a larger interactive graph that lets you zoom in and out, sample performance metrics ondifferent timescales (e.g., hourly, daily, weekly, etc.) and move back and forward in time:

No installation, monitoring agents required

You don't need to install or configure any monitoring agents, because the Hub pulls statistics directly from Amazon's CloudWatch API. CloudWatch in turn gets its data directly from the virtualized hardware layer running underneath your server's operating system.

Why Rackspace open sourced OpenStack

Liraz Siri — Wed, 30 Nov 2011 13:19:18 +0000

Making TurnKey easy to deploy on as many public and private clouds is an important goal for us. We're going to soon be expanding the number of image formats TurnKey officially support to include more major contenders in this space. We'd also like to establish a mirror network that supports rsync so that service providers will find it easy to get up-to-date images of all TurnKey appliances. By the way, if an rsync mirror is something you would find valuable, drop me a line.

Unfortunately there are too many players in the cloud software space for us to support every single one. It's much easier to put effort into making TurnKey work well with the winning horses. OpenStack is particularly interesting, because as I've said before it is most likely the future of open source clouds.

It was a very interesting move on RackSpace's part to put their weight behind the commoditization of cloud technology. You'd think after all of the effort and money they invested in developing their own cloud technology they'd want to keep it for themselves. Instead they form an alliance with their competitors and decide to give it away.

My take on this is that this is in fact a very smart move, but not necessarily for the official reasons RackSpace made public.

You see, well before OpenStack there was a fury of open source activity in this space and a few years down the road one of these projects (e.g., before OpenStack, Eucalyptus was my favorite candidate) would have inevitably become the "Linux" of cloud. This would have left RackSpace stuck with the costs of developing their own proprietary cloud operating system and robbed them of the ability to ride the wave of free innovation that almost magically happens once an open source project gathers enough momentum.

I think RackSpace realized, hey if it's going to happen anyway, we might as well lead the camp. I suspect this is a strong indicator that RackSpace really do believe their competitive advantage is based on branding, corporate culture (e.g., "fanatical support") and economies of scale rather than software development, which in this case is just a cost they would love to share with the community. They're probably right!

Also, having internal clouds run on the same platform as RackSpace's cloud will make interoperability much easier, unlocking a huge source of revenue as corporations finally begin trusting cloud providers, but not just any cloud provider, to offload (or outsource) their own internal IT infrastructure. Interesting how open source collaboration as a tool for reducing costs and commoditizing key technology components can make sense even to a large profit-oriented company. It doesn't have to be about the ideology, and these days I think it usually isn't - it just makes business cents.

How TKLBAM hooks work

Liraz Siri — Wed, 23 Nov 2011 10:47:11 +0000

Most TKLBAM users probably don't realize this, but TKLBAM has a nifty, general purpose hooks mechanism you can use to trigger useful actions on backup and restore.

Examples of hooks:

Cleaning up temporary files
Stopping/starting services to increase data consistency
Encoding/decoding data from non-supported databases
Using LVM to create/restore a snapshot of a fast changing volume

Originally I developed the hooks mechanism so we could fix a few issues indirectly related to the usability of TKLBAM. In particular, our very first beta users reported that sometimes tklbam-restore would fail to find any backup volumes. When we investigated this turned out to be a clock discrepancy. The obvious solution was to sync the clock before starting the restore, but the more I thought about it the more the idea of hardwiring that ntpdate stuff rubbed me the wrong way. For a few reasons:

It's an auxiliary problem, not a core issue with TKLBAM's logic
I'm offline much of the time during development so I needed some way to turn this off, but I don't want to add more testing-specific code unless it's absolutely necessary.
It's OK if a specific server (e.g., pool.ntp.org) is the default, but there should be some way to configure it if a user, for example, wants to use an internal NTP server.

I tried to think of a clean way to achieve these simple goals in a clean way (e.g., cli options, environment variables, configuration files), but everything I came up with was just so darn ugly.

Then I realized that a hooks mechanism would solve this problem in a simple, generic way.

Implementation

/etc/tklbam/hooks.d may contains executables (e.g., scripts) that will be run by tklbam before and after two operations (currently):

backup
restore

Two arguments are passed to the hooks:

operation: restore/backup
state: pre/post

Non zero exitcodes raise a HookError is raised.

Advantages

In one stroke, solve the clock problem and also lets advanced users define their own hooks to take care of things TKLBAM doesn't (e.g., stopping IO intensive processes before backup, encoding/decoding unsupported databases, etc.)

Example fixclock hook

	#!/usr/bin/python
# hook that runs ntpdate before duplicity to sync clock to UTC

import os
import sys
import executil
from string import Template

NTPSERVER = os.environ.get("NTPSERVER", "pool.ntp.org")

ERROR_TPL = """\
##########################
## FIXCLOCK HOOK FAILED ##
##########################

Amazon S3 and Duplicity need a UTC synchronized clock so we invoked the
following command::

    $COMMAND

Unfortunately, something went wrong...

$ERROR
"""

def fixclock():
    command = "ntpdate -u " + NTPSERVER

    try:
        executil.getoutput(command)
    except executil.ExecError, e:
        msg = Template(ERROR_TPL).substitute(COMMAND=command,
                                             ERROR=e.output)

        print >> sys.stderr, msg,
        sys.exit(1)

def main():
    op, state = sys.argv[1:]

    if op in ('restore', 'backup') and state == 'pre':
        fixclock()

if __name__ == "__main__":
    main()

On my Kindle I am root

Liraz Siri — Thu, 17 Nov 2011 04:56:18 +0000

Starting from the end

That's my Kindle in the screenshot running a full screen terminal. I'm about to run nmap (a network mapping program) inside a chrooted Debian ARM installation I put on the device. Having Debian on the device isn't really necessary for hacking the Kindle but it does make it easier to install ARM binaries of just about any of the 25,000 packages in Debian. Yep, apt-get works on my Kindle!

More practically I can now SSH into the device over the WIFI, use SFTP to transfer over new books without having to mess around with a USB cable, etc.

The device can still gets books from Amazon, but I've disabled its ability to auto-update firmware. Now that I control my device I'd like to keep it that way, even if there's no immediate practical benefit.

Besides, it's one thing to know on a theoretical level that the device runs Linux, and being able to see for yourself which processes are running:

Rewinding back to the beginning

Besides my workstation, my Kindle is the device I use the most. By far.

So much that it's almost wearable computing by now. When I take a break I stick it in my pocket and have Tom Glynn's synthesized voice quickly humming whatever I'm reading to me while my hands are free to eat my meals, take care of boring errands, etc.

It's maybe the only mobile device I feel has unambiguously improved my quality of life in a net positive way (I'll leave my gripes with smart phones for another time).

My only major concern with the Kindle is that I'm not supposed to have full control over it:

If it's connected to a network, Amazon can update my firmware remotely at any time without asking me first, possibly changing the device's behavior in undesirable ways. They can spy on my reading (how would I know?), delete my books, etc.
I can't customize its behavior. I keep having these ideas on little features that would make the device even more useful to me but probably wouldn't make sense for the average user. I don't expect Amazon (or any other consumer company for that matter) to design a product that fits perfectly with my needs out of the box.
I know there's Linux under the hood and I want root on it. On principle dammit!

OK, maybe not just on principle. The Kindle is a very low cost, super lightweight, ARM Linux machine with an eInk display that can be easily read in bright sunlight, a great text-to-speech system, amazing battery life, WIFI / 3G access, a nice bit of storage, sound output and even a hidden microphone. There are endless creative off-label things you could do with it.

Considering all the features packed into the Kindle the price is jaw dropping. Amazon probably isn't making a profit on the hardware. Heck the "special offers" Kindle now costs just $79. That's $20 less the $99 ARM SheevaPlug which doesn't have nearly as many features.

So over the weekend I took a look and it turns out that since I last checked a nice Kindle hacking community has sprung up, discovered that the Kindle doesn't have any real security, and made available all the tools you need to take full control over your device.

Kindle hacking is at its infancy but there's already a pretty sweet list of homebrew hacks that let you for example, replace the dead people in your screensavers, change/add new fonts, etc.

I found everything online. Mostly on the excellent mobileread forums but it took time to make sense of it all. The documentation is often a somewhat confusing and dodgy patchwork so I took notes, tested what worked on my Kindle and figured it would be useful to summarize my "crystallized" understanding for the benefit of others who might want to go down the same road.

Rooting your Kindle

Under the hood Amazon's firmware updates are just glorified shell scripts in a proprietary package format that contains an embedded Amazon signature.

The first thing we need to do to get control of the device is "jailbreak" it, which really just adds a "hacked" key to the keyring used to verify the package signature.

Install the Jailbreak

http://wiki.mobileread.com/wiki/Kindle_Screen_Saver_Hack_for_all_2.x_and_3.x_Kindles

See the "How to install Jailbreak Hack" section.

Currently the latest version of the JailBreak is 0.7. To install it you just transfer over the bin that's right for your version of the Kindle (I.e., update_jailbreak_0.7.N_k3w_install.bin = Kindle 3 Wifi) into the device root and then update the device:

Home > Settings > Menu > Update Kindle

Now you can install packages signed by a non-secret hacked key. The Jailbreak contains a whitelist of md5sums of known good hacks.

Install usbnet hack

I downloaded the usbnet hack from an attachment on this forum thread:

http://www.mobileread.com/forums/showthread.php?t=88004

What's usbnet?

The Kindle 2 has a hidden USB network mode, probably left over from development. When activated, the Kindle would behave as a USB network device rather than a USB mass storage device. This allowed you to do neat things such as tethering the device to your laptop.

Kindle 3 seems to have removed this feature, but the usbnet hack reactivates it and installs busybox (a micro shell environment), dropbear (a micro SSH server) and a few other utilities to allow you to SSH into your device and explore its insides.

After installation, usbnet creates a usbnet directory in your kindle root which contains its configuration files:

$ cd /mnt/kindle/usbnet
$ find

./info.txt
./DISABLED_auto
./run
./run/telnetd.pid
./run/sshd.pid
./etc
./etc/dropbear_rsa_host_key
./etc/htoprc
./etc/dropbear_dss_host_key
./etc/config
./etc/terminfo
./etc/terminfo/x
./etc/terminfo/x/xterm
./etc/authorized_keys
./etc/bak
./etc/bak/dropbear_dss_host_key
./etc/bak/dropbear_rsa_host_key
./bin
./bin/busybox
./bin/usbnet-enable
./bin/dropbearmulti
./bin/usbnet-disable
./bin/usbnetwork
./bin/rsync
./bin/sftp-server
./bin/lsof
./bin/htop
./usbnetwork_install.log

Now we'll unmount (I.e., "eject") the Kindle from our computer, disconnect the USB connection to take it out of mass storage mode and enable usbnet mode.

Press [DEL] on your Kindle to bring up the search bar and do the following "searches":
```
;debugOn
~help # just for fun
~usbNetwork
;debugOff
```

The commands are not case sensitive. Usually you don't want to stay in debugging mode because it turns off various power savings features such as turning off WIFI is your Kindle is not connected to the USB. Also, it turns on verbose logging.

Now when you connect your Kindle to your computer via USB, it isn't recognized as a mass storage device but rather as a USB network device.

This is what dmesg says when I connect the Kindle in mass storage mode:

[138591.847428] usb 8-1: new high speed USB device using ehci_hcd and address 45
[138592.000857] usb 8-1: configuration #1 chosen from 1 choice
[138592.004480] scsi24 : SCSI emulation for USB Mass Storage devices
[138592.004541] usb-storage: device found at 45
[138592.004556] usb-storage: waiting for device to settle before scanning
[138596.996774] usb-storage: device scan complete
[138596.997900] scsi 24:0:0:0: Direct-Access     Kindle   Internal Storage 0100 PQ: 0 ANSI: 2
[138597.003881] sd 24:0:0:0: [sdc] 6410688 512-byte hardware sectors (3282 MB)
[138597.109966] sd 24:0:0:0: [sdc] Write Protect is off
[138597.109973] sd 24:0:0:0: [sdc] Mode Sense: 0f 00 00 00
[138597.109976] sd 24:0:0:0: [sdc] Assuming drive cache: write through
[138597.113952] sd 24:0:0:0: [sdc] 6410688 512-byte hardware sectors (3282 MB)
[138597.219787] sd 24:0:0:0: [sdc] Write Protect is off
[138597.219792] sd 24:0:0:0: [sdc] Mode Sense: 0f 00 00 00
[138597.219794] sd 24:0:0:0: [sdc] Assuming drive cache: write through
[138597.219799]  sdc: sdc1

And here's what dmesg says when I connect the Kindle in USB network mode:

[138741.453693] usb 8-1: new high speed USB device using ehci_hcd and address 48
[138741.604690] usb 8-1: configuration #1 chosen from 2 choices
[138741.610967] usb0: register 'cdc_ether' at usb-0000:00:1d.7-1, CDC Ethernet Device, ee:49:00:00:00:00

Note that with the usbnet hack, by default SSH only works over the USB host-to-host connection. SSH is configured not to ask for the root password so usbnet wisely disables SSH over WIFI for security reasons.

To safely turn SSH over WIFI on we'll want to harden our Kindle first a bit. Setup SSH authentication, change the default keys and passwords and then reconfigure usbnet to allow SSH over WIFI.

We can configure this stuff in mass storage mode by editing files in usbnet/etc under the Kindle root, or via SSH on the usb host-to-host network. BTW, the kindle root you see in mass storage mode is is mounted to /mnt/us on the Kindle.

Anyhow, after connecting the Kindle to our computer in usbnet mode we have a new device, usb0 which we will configure to suit the default usbnet setup:

$ sudo ifconfig usb0 192.168.2.1
$ ping 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.696 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.696/0.696/0.696/0.000 ms

Now let's login to our Kindle for the first time:

$ ssh 192.168.2.2
Welcome to Kindle!

#################################################
#  N O T I C E  *  N O T I C E  *  N O T I C E  #
#################################################
Rootfs is mounted read-only. Invoke mntroot rw to
switch back to a writable rootfs.
#################################################

[root@kindle root]# cat /proc/cpuinfo
Processor       : ARMv6-compatible processor rev 3 (v6l)
BogoMIPS        : 511.18
Features        : swp half thumb fastmult vfp edsp java
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x1
CPU part        : 0xb36
CPU revision    : 3
Cache type      : write-back
Cache clean     : cp15 c7 ops
Cache lockdown  : format C
Cache format    : Harvard
I size          : 16384
I assoc         : 4
I line length   : 32
I sets          : 128
D size          : 16384
D assoc         : 4
D line length   : 32
D sets          : 128

Hardware        : Amazon MX35 Luigi Board
Revision        : 35020
Serial          : "B008A0A0040298FC"
BoardId         : "SP1B000000000000"

[root@kindle root]# free
         total       used       free     shared    buffers     cached
Mem:        256536     151468     105068          0      15248 53372
-/+ buffers/cache:      82848     173688
Swap:            0          0          0

[root@kindle root]# mntroot rw
system: I mntroot:def:Making root filesystem writeable

[root@kindle root]# passwd root
Changing password for root
New password:
Retype password:

[root@kindle root]# cd /mnt/us
[root@kindle us]# ls
audible           documents         music             system
usbnet            linkjail
[root@kindle us]# cd usbnet/etc/
[root@kindle etc]# ls -l
-rwxr-xr-x    1 root     root          957 May 23 14:56 config
-rwxr-xr-x    1 root     root          458 May 23 01:54 dropbear_dss_host_key
-rwxr-xr-x    1 root     root          427 May 23 01:54 dropbear_rsa_host_key
-rwxr-xr-x    1 root     root          561 Oct 10  2010 htoprc
drwxr-xr-x    3 root     root         8192 May 22 20:59 terminfo

# setup my SSH key as an authorized key
[root@kindle etc]# echo ssh-rsa AAAAB3NzaC1yc2EAAAABIwAwAIEAvp+4FpjKlv1nsddevQtX8zMvQMkuJDwZSCHpFdm2IY20NmOhF0LY6dKRzQ+89pJ2MUYZYtotN1SmMk1ndUmHssQIRrmKKWdwnDzDUISTDB5iEQIg8JcPxwu6+uJnLrZvfNrx/fsMoRwRR3S9bHcKi9pxQT9T4Jbt+Gt6ewtuLAE= liraz@dev > authorized_keys

Note that with the usbnet hack, by default SSH doesn't ask for the root password so it disables SSH over WIFI for security reasons.

In summary here's what I did to enable SSH over WIFI safely:

added my SSH key to usbnet/etc/authorized_keys (a new file).

installed dropbear on my Ubuntu workstation (e.g,. apt-get install dropbear) and then recreated the dropbear host keys:

dropbearkey -t rsa -f rsa
dropbearkey -t dss -f dss

scp rsa 192.168.2.2:/mnt/us/usbnet/etc/dropbear_rsa_host_key
scp dss 192.168.2.2:/mnt/us/usbnet/etc/dropbear_dss_host_key

edit usbnet/etc/config to change K3_WIFI field from false to true
restart usbnet by toggling it off and back on with the hidden ~usbNetwork comand (from the search bar in ;debugOn mode).

Test that you can still log into SSH via the usb0 connection. That means you've configured everything correctly.

Now turn on Wifi and see if you can log in over WIFI. You can find out the Kindle's IP address by accessing the secret 711 network info screen:

Home > Menu > Settings >

    # ALT + U Q Q
    711

As long as your Kindle is plugged into USB (in your computer or the power charger), it will remain accessible via WIFI even if the screensaver is active. In debugging mode the WIFI stays on even when your Kindle is not plugged in.

As is typical for embedded ARM devices the WIFI chip is usually sleeping to conserve power which makes for a slightly jittery interactive SSH session. Not too bad though.

For extra convenience, I configured my local WIFI router to bind the Kindle always to the same IP address (e.g., 10.0.0.15).

Keep in mind that your Kindle filters out ICMP pings on the WIFI so it won't respond to a regular ping, but it will respond to arping:

$ sudo arping 10.0.0.15
ARPING 10.0.0.15
42 bytes from ee:19:00:00:00:00 (10.0.0.15): index=0 time=1.777 msec
42 bytes from ee:19:00:00:00:00 (10.0.0.15): index=1 time=54.230 msec

$ nc -vv 10.0.0.15 22
10.0.0.15 22 (ssh) open
SSH-2.0-dropbear_0.53.1

$ ssh 10.0.0.15
Welcome to Kindle!

#################################################
#  N O T I C E  *  N O T I C E  *  N O T I C E  #
#################################################
Rootfs is mounted read-only. Invoke mntroot rw to
switch back to a writable rootfs.
#################################################

[root@kindle root]# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:25 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2496 (2.4 KiB)  TX bytes:2496 (2.4 KiB)

usb0      Link encap:Ethernet  HWaddr EE:19:00:00:00:00
          inet addr:192.168.2.2  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:647 errors:0 dropped:0 overruns:0 frame:0
          TX packets:428 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:53670 (52.4 KiB)  TX bytes:56067 (54.7 KiB)

wlan0     Link encap:Ethernet  HWaddr 28:EF:01:83:A1:2C
          inet addr:10.0.0.15  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3079 errors:0 dropped:0 overruns:0 frame:0
          TX packets:727 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:333515 (325.6 KiB)  TX bytes:57404 (56.0 KiB)

[root@kindle root]# netstat -atn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address State
tcp        0      0 127.0.0.1:8784          0.0.0.0:* LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:* LISTEN
tcp        0      0 127.0.0.1:8022          0.0.0.0:* LISTEN
tcp        0      0 0.0.0.0:23              0.0.0.0:* LISTEN
tcp        0      0 0.0.0.0:40317           0.0.0.0:* LISTEN
tcp        0      0 10.0.0.15:22             10.0.0.10:41208 ESTABLISHED
tcp        0    496 10.0.0.15:22             10.0.0.10:41209 ESTABLISHED
tcp        0      0 192.168.2.2:22          192.168.2.1:48703 ESTABLISHED

[root@kindle root]# iptables --list
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:40317
ACCEPT     tcp  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere            state ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere            state ESTABLISHED
ACCEPT     all  --  localhost.localdomain  anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             localhost.localdomain

I transfered over a 50MB test file to test the transfer rate. With good connectivity I can get 1.5MB/s over the Wifi. The USB host-to-host is slightly faster at about 2MB/s, and the mass storage interface is fastest at 6MB/s.

Transfering Kindle ebooks over Wifi with SSH/SFTP

Example:

scp path/to/ebook.prc 10.0.0.15:/mnt/us/documents
ssh 10.0.0.15 dbus-send --system /default com.lab126.powerd.resuming int32:1

That last command triggers Amazon to refresh the book list. After I got sick of cut and pasting it into the CLI I made it into a tiny script:

cat > /usr/local/bin/hack-refresh << EOF
#!/bin/sh
dbus-send --system /default com.lab126.powerd.resuming int32:1
EOF

chmod +x /usr/local/bin/hack-refresh

Note that in addition to its native AZW format, Amazon also supports txt, mobi, prc, mp3 and PDF files.

Unfortunately, the Kindle doesn't support local HTML files natively but there's a really sweet open source project called calibre for converting between ebooks formats.

Before Calibre, I also had some success with mobi pocket creator, a free as in beer program I experimented with in my Windows XP VM.

Install a native terminal (kiterm)

Luigi Rizzo wrote a standalone Kindle terminal you can use from within the device. I used a slightly patched version that works full screen.

It hasn't been packaged into a Kindle *.bin file yet but installation was relatively straightforward thanks to a nice tutorial on TinyApps.

The short version:

extract the zip file to /mnt/us/kiterm

create an init script to launch it on startup:

cat>/etc/init.d/kiterm<<'EOF'
#! /bin/sh
# /etc/init.d/kiterm
case "$1" in
  start)
    echo "Starting kiterm "
    /mnt/us/kiterm/myts.arm &
    ;;
  stop)
    echo "Stopping kiterm "
    killall myts.arm
    ;;
  *)
    echo "Usage: /etc/init.d/kiterm
{start|stop}"
    exit 1
    ;;
esac
exit 0
EOF

ln -s /etc/init.d/kiterm /etc/rc5.d/S97kiterm

reboot the Kindle (Menu > Settings > Menu > Restart)

To access the Terminal you press Shift, let go and immediately press T.

The terminal configures various key bindings to make up for the all the missing keys missing from the Kindle's limited keyboard. I saved the most common ones to a text file on my kindle for reference:

[root@kindle root]# mntroot rw
system: I mntroot:def:Making root filesystem writeable
[root@kindle root]# cat>keys<<'EOF'
>     Ctrl = AA (aka Symbol)
>     Esc  = Left Next Page
>
>     .------------------------.     .----------------------.
>     | Key   Back  Back+Shift |     | Key   Alt  Alt+Shift |
>     +------------------------+     +----------------------+
>     |  Q      `        ~     |     |  Q     1      !      |
>     |  A     Tab   Back Tab  |     |  W     2      @      |
>     |  Z      <        >     |     |  E     3      #      |
>     |  U      -        _     |     |  R     4      $      |
>     |  I      =        +     |     |  T     5      %      |
>     |  O      [        {     |     |  Y     6      ^      |
>     |  P      ]        }     |     |  U     7      &      |
>     |  K      ;        :     |     |  I     8      *      |
>     |  L      '        "     |     |  O     9      (      |
>     | Del     \        |     |     |  P     0      )      |
>     |  .      ,        <     |     '----------------------'
>     | Sym     .        >     |
>     | Ret     /        ?     |
>     '------------------------'
> EOF
[root@kindle root]# mntroot ro
system: I mntroot:def:Making root filesystem read-only

Preventing Amazon from auto-updating your firmware

As far as I can tell the easiest and surest way to prevent Amazon from auto-updating your Kindle is to knock out the keys it uses to verify the signatures:

mv /etc/uks /etc/uks.disabled

Under the hood, the Kindle is programmed to get firmware updates automatically via the TODO service, which gives the Kindle a list of things to do including getting new books (or deleting existing books) and/or getting new firmware.

Some people in the community have gone as far as to change the URLs in the framework and pass them through a proxy server setup to selectively mirror Amazon's TODO requests.

# grep http /opt/amazon/ebook/config/framework.fiona.conf
BASE_WEBSITE_URL: http://www.amazon.com
CERT_SERVER_URL : https://fras-g7g.amazon.com/FrasProxy/
REGISTER_SERVER_URL : https://firs-g7g.amazon.com/FirsProxy/
TODO_SERVER=https://todo-g7g.amazon.com/FionaTodoListProxy/
CDE_SERVER=https://cde-g7g.amazon.com/FionaCDEServiceEngine/

Uninstalling hacks

All the hacks I've come across so far come with an installer and uninstaller *.bin files. Just in case, I copy the uninstaller for the hacks I install to my Kindle's root under "uninstallers". That way I can always roll back hacks later if I want:

[root@kindle uninstallers]# cd /mnt/us/uninstallers
[root@kindle uninstallers]# ls
update_jailbreak_0.7.N_k3w_uninstall.bin
update_usbnetwork_0.33.N_k3w_uninstall.bin

Stuff I still haven't figured out

How do I speed up the text-to-speech? Even at Amazon's fastest default rate the Kidnle's TTS voice isn't speaking as fast as I can read with my eyes. Make it gI want it to go faster!
How do I replace the TTS voice? I'm hoping the Polish hacker that got his Kindle to speak in Polish will share more details on his brilliant hack
How do I map all the dbus targets on the Kindle? I bet that would be useful in scripting the Kindle to new things.

Be nice. It's a fscking gift

Liraz Siri — Tue, 01 Nov 2011 01:24:30 +0000

Open source development is usually fun and rewarding. You get to work on whatever you like. No permission required. No "business justification". Here's this thing I've created, isn't it neat? There's a deep sense of satisfaction in making things. Especially when other people find them useful. It's also pretty awesome when people decide what you've made is interesting enough that they want to join in and help make it better. Successful projects often form into communities. Strangers from all over the world turned into enthusiastic users, co-developers. Friends.

The only parts that suck are that:

It is a bit more difficult to make a living purely from open source software. Giving stuff away generally doesn't pay very well.
Some people just don't get it.

For example, a while back someone who shall remained unnamed started e-mailing us privately with complaints that TKLBAM (TurnKey's Backup and Migration software) didn't work right for him. We eventually traced the problem back to a MySQL memory usage issue. It turns out that in some, thankfully rare situations MySQL consumes way too much memory when you restore a very particular kind of database from a mysqldump.

When the user complained this was "a fault of TKLBAM's design" I explained that it really didn't sound like a TKLBAM problem to me because:

If you peeled off TKLBAM and just used mysqldump / mysql command directly to backup / restore that kind of database you would run into exactly the same memory usage issue.
If Ubuntu issued a package update that fixed the bug, the issue would go away. Presto. No TKLBAM fix required.

Besides, even if this wasn't a rare edge case nobody else had run into there probably wasn't much I could do about it without debugging MySQL code - a daunting task.

The best I could do was add an item to my todo list to see if we could look for workarounds that would go into the next version. In the meantime I recommended that the user try using another solution.

Then I went on vacation. When I came back online I discovered an escalating series of e-mails from this user that eventually culminated in threats if we didn't drop everything to meet his demands. And this was no joke. This guy seemed to be dead serious!

Alon, who peaks into my TurnKey e-mail inbox when I'm not around tried calming the guy down:

Just so you know, Liraz has been working offline and on vacation for about a month, if not a little longer. He has not been ignoring you, he just hasn't read your emails.

I can understand your frustration, but even so keep in mind that TLKBAM is open source software, and released under the GPL!

I'm sure Liraz will reply to you once he returns online and finds the time, but even then understand that there is no obligation on his part to do so, except for common courtesy. Making threats is just disrespectful and wasteful.

Another demanding, entitled rant followed. When I finally came back I read through the whole series of e-mails, thought a little bit about what kind of confusion could lead to the (thankfully rare) behavior we were witnessing and put in my final response:

Sorry for the late reply and sorry for the bad experience you have had with TKLBAM.

As Alon said I've been offline for a while. As much as I'd like to help you in a friendly manner I'm getting the sneaking sensation from the demanding tone of your messages that you don't seem to understand how open source works.

The way I see it open source is basically a gift culture where people give the products of their labor away in a vague hope that some people (but probably not everyone) will find it useful. It's a gift, with everything that implies. There are no warranties, explicit or implied. There are no guarantees that it is fit for any purpose.

Even proprietary software you pay for is not guaranteed to fully satisfy you or to work flawlessly (it usually doesn't). The only way to really guarantee that technology works like you want is either to take pains to develop it yourself or pay someone else to develop it for you, in which case you can boss them around when they don't meet your expectations or schedule. For what it's worth I am prepared to offer you a full refund for the free software. :)

Seriously though I do appreciate the technical feedback but please remember that the open source license gives you permission to copy, distribute and improve TKLBAM yourself if you ever feel I am not responsive enough to your needs.

I don't know what I was expecting. A sudden moral epiphany? "Sorry I got carried away". I know I know, I probably shouldn't have bothered. Once a person gets so far out of whack it's unlikely they are interested in being sensible. But I'm a sucker for redemption. Anyhow, it certainly didn't help. A couple of additional e-mails with further demands and threats followed. Oh well, at least I tried.

The moral of the story: Come on, be nice. It's a fscking gift!