Blog

TurnKey Core 12.0 RC optimized builds

Alon Swartz — Mon, 16 Apr 2012 01:03:17 +0000

Last month we announced the release candidate for TurnKey Core 12.0 - the common base for all appliances, based on the rock solid Debian Squeeze (6.0.4).

It took a little longer than expected, but we've finally released all the optimized builds for TurnKey Core 12.0RC: ISO, VMDK, OVF, Amazon EC2, OpenStack, OpenVZ and Xen.

The optimized builds can be downloaded from the core appliance page, directly via the TurnKey channel in ProxmoxVE (OpenVZ), deployed in the Amazon EC2 cloud via the TurnKey Hub, or via one of the official TurnKey partners (soon).

Build specific release notes

Common (ISO)

See the announcement.

Amazon EC2

Deployment: The TurnKey Hub has been updated to support Core 12.0 deployment and management (Launch new server -> 12.0). Once the full library has been updated to TKL 12.0 it will become the default, and the current release will be moved to Legacy.

VM optimized (VMDK, OVF)

Open-VM-Tools: Previous VM optimized builds included the proprietry VMWare-Tools, but since VMWare have released a large portion of the code under the GPL, we've moved to open-vm-tools.
Swap warning: VMware products might display a warning that no swap space was detected. This is a false positive, as swap is configured in LVM.

OpenStack

Ramdisk required: TurnKey 12.0 requires the initrd to be registered for successful boot (exemplary import and registration code).

OpenVZ

Naming convention: We've updating the naming convention for openvz builds to support vanilla OpenVZ out of the box, and eliminate duplication for the Turnkey PVE channel. Thanks Jeremy!
Removed NTP daemon: The NTP daemon has been removed as the clock is managed by the host. Thanks Martin!
No more upstart hacks: Removed Ubuntu upstart hacks as they are not relevant in Debian.

Xen

Xen optimized kernel: Moved to the Xen optimized kernel provided by Debian (linux-image-xen-686).

As always, we need your help in testing the builds. If you come across any issues or have ideas how they can be improved, please post a comment.

New Hub feature: Server snapshots

Alon Swartz — Wed, 04 Apr 2012 13:56:02 +0000

I usually get excited when adding new features to the TurnKey Hub. Recent excitement included server monitoring, reserved instances, domain management, and the Hub API.

I'm very excited about todays annoucement, not only is it awesomely useful, it's also technically cool!

So what are snapshots?

I'm sure you can guess, but let me explain anyway.

Snapshots can be used with EBS-backed instances to create point-in-time snapshots of the root filesystem, which are persisted to Amazon S3 for storage durability. Snapshots are incremental, meaning that only changes since the last snapshot are saved, taking up less storage, time, and reducing costs (see below for technical details).

Snapshots ask Amazon's fiber-optic storage backplane to save your server's disk state while it's running without impacting performance.

Ok, but what can I do with them?

Server clones

Snapshots can be used as the basis for a new server, essentially creating a clone (the cloud server equivalent of a time machine crossed with a portal to a less obnoxious alternative dimension), for example:

You can clone a production server to create a staging enviroment for testing new features, hacking away, whatever, without the worry of hosing your production server (guess how I tested this new feature).
You can essentially upgrade your servers hardware if you need the extra horse power, memory or even disk space. Say you were testing an idea with a micro instance, and now its taking off. Firstly congrats, secondly just clone the micro's latest snapshot to a larger instance size and update the DNS record / re-associate the elastic IP.
Let you're imagination run wild!

EBS Volumes

Snapshots can be used as a starting point for a new EBS volume, for example:

You mistakenly deleted a file, hosed your database, or whatever bad thing that can happen. You create a volume from the snapshot of your choice, attach it to your instance (which is auto-mounted via ebsmount) and access the data you need.
Again, let you're imagination run wild!

Can I schedule automatic snapshots?

You sure can! You can schedule automatic zero-load server snapshots for hourly, daily, weekly and monthly frequency, or manually create one at anytime.

There is however a snapshot limit per Amazon account, per region, so when configuring automatic scheduled snapshots, snapshot retention is also configurable to prune old snapshots, keeping you within the limit and saving you money.

Sounds cool, what does it look like?

We've added 2 new fields to the server record:

And this is the snapshot dashboard:

Are there any limitations?

Snapshots only support EBS-backed instances, and not S3-backed instances. This is a technical limitation as snapshots are performed on the EBS-backed root volume, which S3-backed instance do not have.

Snapshots are saved to S3 storage, but they will not appear in your S3 buckets, nor can you access them using the standard S3 API. To access snapshot data you need to create an EBS volume or a server clone.

As mentioned above, there is a limit of the amount of snapshots each Amazon account can have, but you can request to increase your limit (specify you want the snapshots limit increased in the comments.)

Data consistency: Do not solely rely on snapshots for backups, as they may become inconsistent due to disk-buffering and locking. We use TKLBAM for our backups, and suggest you do the same.

Technical details - snapshots explained

I mentioned that snapshots are technically cool, and that they are incremental - let me try and explain what that means at how it works behind the scenes.

A snapshot of an EBS volume can be taken at anytime, which asks Amazon's fiber-optic storage backplane to save the data stored on the volume, at the block level, at that exact point-in-time, to S3 storage.

To improve performance and reduce storage space, Amazon will only copy the blocks of the volume that have changed since your last snapshot - hence incremental.

Now for the extra cool part, unlike regular incremental backup chains, you can delete any previous snapshot. Huh? What? Yep, snapshots are not chained, but are rather conceptually like a table-of-contents of pointers to saved data blocks.

When you delete a snapshot, only the data blocks that are solely used by that specific snapshot are deleted. Data blocks that are used by subsequent snapshots are not. In the below illustration, if SNAP-B is deleted, only SNAP-B:block-2 will be deleted from Amazon S3 as a newer version (SNAP-C:block-2) has already been saved.

Bottom line, take snapshots for a spin and let us know what you think.

TurnKey Core 12.0 RC based on Debian Squeeze

Alon Swartz — Thu, 01 Mar 2012 18:35:19 +0000

I'm pleased to announce a spanking brand new release candidate for TurnKey Core 12.0 - the common base for all appliances, based on the rock solid Debian Squeeze (6.0.4). The rumors were true! Hurrah! Hurrah!

This is an RC release, so take it for a spin and let us know what you think. If you come across any issues, please report them. If you have ideas on how to make it better, let us know!

Download RC: 138MB ISO (changelog, signature, manifest)

Did you say Debian?

Why yes, yes I did. Here's the back story...

In mid-2010 we released our first ever Debian appliance based on Lenny. In the announcement Liraz discussed whether Debian based appliances are worth the trouble as well as some notes on Ubuntu vs. Debian.

Back then we decided not to release the entire TurnKey library based on Lenny as Squeeze was around the corner, and we were spread quite thin.

Fast forward to a few weeks ago, Liraz and I were discussing the upcoming Ubuntu LTS release, which is scheduled for April. We were deliberating when would be the best time to begin the transition.

During the conversation we revisited the idea of supporting Debian, and decided it was time. We've been wanting to support Debian since TurnKey's inception, and it seems that a significant 59% of users want Debian-based appliances "a lot"!

Rolled up my sleeves

So, I rolled up my sleeves and got to work. It wasn't too long and I had a working TurnKey bootstrap image (102MB ISO, Meta) based on Squeeze. The most annoying part of that was dealing with the non-backwards compatible bootsplash. Turns out that was a good thing, as it forced me to do cleanup, and remove panel options that weren't actually doing anything. How nobody filed a bug on that is beyond me :)

Then I moved onto Core. Upgrading our Live Installer (di-live) was a little boring, but after fixing some bugs and seeing it work, not to mention setup LVM and install the entire OS in under a minute, it put a smile on my face.

After upgrading several key components, fixing bugs (thanks to everyone who submitted bug reports, and Jeremy for his excellent work triaging and keeping the bug tracker up to date), tweaks here and there, and testing, I was a happy camper.

To summarize, there were ups and downs but all in all it was good fun - ask my wife, I updated her on progress every evening whether she wanted to know or not.

But, there is still a long road ahead, and this is only the first milestone.

Changes

New and improved signature files: include detailed steps on how to verify image integrity, as well as md5 and sha1 checksums for convenience. Take a look.
Locale improvements: default locale is now set to en_US.UTF-8, updated configuration for compatibility with Squeeze. Freeing up disk space is now performed by localepurge.
Boot splash and loader: upgraded bootsplash for compatibility, removed unused panel options, and tweaked bootloader timeouts.
Live installer (di-live): upgraded for Squeeze compatibility and misc bugfixes.
Webmin: upgraded to latest upstream release and disabled inline upgrades (managed by APT).

All other changes, bugfixes and tweaks are available in the changelog.

As for the features, not much has changed except for the base distribution.

Long story short, try the RC and tell us what you think. Obviously we have immense respect for both Ubuntu and Debian, and we'd like to hear your views on where we should take it from here.

Rsync the entire TurnKey library from a mirror close to you in under 5 minutes!

Liraz Siri — Wed, 22 Feb 2012 05:51:39 +0000

Like TurnKey so much you want a local copy of all the appliances but too lazy to download individual appliance images from SourceForge by hand via browser?

I know exactly how you feel. Sloth is a virtue, and in the beginning was the command line.

So now you can use rsync or ftp to batch download the entire virtual appliance library in whatever build type you like best from a high-speed mirror near you. The way the net gods intended!

Thanks to generous donations of bandwidth and storage space from opensource friendly network samurais around the world TurnKey now has 16 high-speed rsync/ftp capable mirrors in 12 countries: China, Ireland, United Kingdom, France, Germany, Sweden, Japan, Belarus, Bulgaria, Denmark, Argentina, and Israel. And we're just getting started...

This means if your network is good enough you can now grab a copy of the entire TurnKey appliance library in any of the 6 supported build types (ISO, VMDK, OVF, Xen, OpenVZ and OpenStack) from a high-speed (e.g, 10-Gbps) mirror close to you in under 5 minutes.

A few minutes ago in a practice run I rsynced TurnKey images to my Amazon EC2 instance at an average 288 Mbps effective download rate over the network.

Wowsers! Up until I upgraded to a nice SSD a couple of weeks ago that was about as fast as I could copy files on my local hard drive.

If you find these network speeds hard to believe I invite you to log into your TurnKey Hub account and launch a small instance in Ireland. Now let's rsync the entire TurnKey appliance library in Xen format - all 8 GBs worth from the HEAnet mirror:

$ rsync rsync://ftp.heanet.ie/[snip]/turnkeylinux/

Welcome to the HEAnet mirror site, ftp.heanet.ie (http://ftp.heanet.ie/about)
-----------------------------------------------------------------------------

 NOTE: All connections and transfers are logged; if this is disagreeable,
 please disconnect now.

 * ftp.heanet.ie is located in Dublin, Ireland and operated by HEAnet, the
   Irish National Research and Education Network.

 * This is a four node cluster with 10 Gigabit access to the HEAnet backbone.

 * Please contact mirrors@heanet.ie with any operational queries.

 * You are connected to ftp-node2 (kokapetl)

-----------------------------------------------------------------------------

drwxr-xr-x         157 2012/02/12 17:05:27 .
drwxr-xr-x        4709 2012/02/08 18:00:08 iso
drwxr-xr-x        5686 2012/02/08 18:00:58 openstack
drwxr-xr-x        5434 2012/02/08 18:01:10 openvz
drwxr-xr-x        4930 2012/02/08 18:01:21 ovf
drwxr-xr-x        2759 2012/01/13 08:31:59 pve
drwxr-xr-x        5014 2012/02/08 18:00:44 vmdk
drwxr-xr-x        5266 2012/02/09 20:41:48 xen

$ time rsync -av -P rsync://ftp.heanet.ie/[snip]/turnkeylinux/xen ./

receiving incremental file list
xen/
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2
   299570436 100%   30.43MB/s    0:00:09 (xfer#1, to-check=83/85)
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2.sig
         490 100%    0.38kB/s    0:00:01 (xfer#2, to-check=82/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2
   191193216 100%   34.68MB/s    0:00:05 (xfer#3, to-check=81/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2.sig
         490 100%    0.48kB/s    0:00:00 (xfer#4, to-check=80/85)

[ .. snip .. ]

sent 1629 bytes  received 7993104501 bytes  36919658.80 bytes/sec
total size is 7992120707  speedup is 1.00

real    3m35.152s
user    0m29.290s
sys     0m30.220s

$

Speaking with the authority of someone that used to download shareware from a local BBS 20,000 times slower (I.e., 6MB/hour on a 14400 baud modem) I can unequivocally state that this is just friggin awesome. The future is now!

Shouts out to TurnKey mirror best buddies all over the globe:

Zhang from the USTC Linux User Group in China
James from Bytemark hosting in the UK
Arnoud from LIP6 in France, and Manuel from Ircam, also in France
Carsten from RWTH Aachen in Germany
Mattias from Umea Uni in Sweden
Mitry from Beltelecom MGTS in Belarus
Boian from IPACCT in Bulgaria
Georg from dotsrc.org in Denmark
Ariel from Cooperativa Telefonica in Argentina
Lior from the Israel Internet Association in Israel

Not yet a TurnKey mirror best buddy but thinking you might want to be? If you have the resources to provide a mirror for TurnKey in your country reach out to our global mirror commando team 24 hours a day in any real or fictional language!

Announcing TurnKey Xen optimized builds

Alon Swartz — Fri, 10 Feb 2012 08:45:17 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for the project.

Recently we announced TurnKey optimized builds in a number of new formats, which brings the supported list to: ISO, VMDK, OVF, Amazon EC2, OpenStack and OpenVZ.

I'm pleased to announce that we have just added Xen to the list of optimized builds. They are hot out of the build farm and available for immediate download.

You can get them from the Download link on the appliance pages.

Pre-seeding / Default passwords

The Xen images are mainly built for hosting providers who utilize the Xen Hypervisor.

Because Xen builds are used in headless deployments (without an interactive console), they include an inithook which preseeds default values and passwords.

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=turnkey

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=$MASTERPASS
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE
EOF

You will most likely want to have your provisioning system to override the defaults by creating /etc/inithooks.conf.

Note that inithooks.conf will be blanked out once its no longer needed for security. You should also make sure that inithooks.conf includes *ALL* of the variables, otherwise the inithook system will turn on interactivity.

If you cannot support preseeding, the alternative is to have the user execute turnkey-init on first login.

Muchas Gracias to Marc from GigaTux (an official TurnKey partner) for testing the Xen images and providing feedback!

Announcing TurnKey OpenStack optimized builds

Alon Swartz — Wed, 01 Feb 2012 11:51:51 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for us. Unfortunately there are too many players in the cloud software space for us to support every single one. It's much easier to put effort into making TurnKey work well with the winning horses.

TurnKey has been supported on the leading public cloud platform Amazon EC2 from early on, not to mention simplifying management and deployment via the Hub.

OpenStack is particularly interesting, because as it is most likely the future of open source clouds.

I originally got intrigued when I heard about NASA planning to open source Nebula in 2009, which has become the basis for Nova, the compute component in OpenStack. Since then, I've been following OpenStack development from a far and have been itching to develop support for TurnKey on the platform.

The time has finally arrived, and I'm pleased to announce TurnKey optimized builds are hot out of our build farm, and available for immediate download and deployment.

You can get them from the "Download -> More Builds" link on the appliance pages.

TurnKey OpenStack optimized builds

EBS auto-mounting support: we've updated our custom EBSmount mechanism for OpenStack, which automatically mounts EBS devices when attached.
Support for automating instance setup: via the user-data scripts mechanism.
Automatic APT configuration on boot: saves bandwidth costs by using the closest package archive for maximum performance.
SSH key support: instances that are launched with a key-pair will be configured accordingly.
SSH host key fingerprints displayed in system log: verification of server to prevent man-in-the-middle (mitm) attacks.
Randomly generated root password: is set on first boot, and displayed in the system log **.
Randomly generated mysql/postgres passwords: the MySQL root and/or PostgreSQL postgres passwords are set to to the same random password as root **.
Instance metadata python library and CLI: used internally, but useful for advanced users. (learn more).

** Because OpenStack builds are used in headless deployments (without a console), they include an inithook which preseeds default values, and random passwords:

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=$(mcookie | cut --bytes 1-8)

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=turnkey
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE

Depending on your use case, you can utilize user-data (note the security implications) to preseed during boot, or once the system has booted by executing turnkey-init.

Exemplary import of TurnKey Core on OpenStack

There are several ways of uploading an image into an OpenStack deployment, below is one way to get you started.


# cd /tmp
# tar -zxf turnkey-core-11.3-lucid-x86-openstack.tar.gz
# ls turnkey-core-11.3-lucid-x86
    turnkey-core-11.3-lucid-x86-initrd
    turnkey-core-11.3-lucid-x86-kernel
    turnkey-core-11.3-lucid-x86.img

# IMG=turnkey-core-11.3-lucid-x86

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=aki \
    disk_format=aki \
    name="$IMG-kernel" \
    < /tmp/$IMG/$IMG-kernel

Added new image with ID: 5

# KERNEL_ID=5

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=ami \
    disk_format=ami \
    kernel_id=$KERNEL_ID \
    name="$IMG" \
    < /tmp/$IMG/$IMG.img

Added new image with ID: 6

# glance -A $GLANCE_TOKEN index

ID  Name                                Disk Format  Container Format  Size
--  ----------------------------------  -----------  ----------------  ---------
6   turnkey-core-11.3-lucid-x86         ami          ami               688498688
5   turnkey-core-11.3-lucid-x86-kernel  aki          aki               4179712

# euca-describe-images

IMAGE   ami-00000006    turnkey-core-11.3-lucid-x86         available  public                  machine aki-00000005
IMAGE   aki-00000005    turnkey-core-11.3-lucid-x86-kernel  available  public                  kernel

Announcing TurnKey OpenVZ optimized builds (+ Proxmox VE channel)

Alon Swartz — Mon, 16 Jan 2012 14:30:41 +0000

OpenVZ and Proxmox VE has been a recurring topic of discussion on the forums, for which we have Jeremy to ~~blame~~ thank. He's done tons of research, testing, preaching, and then some.

What I love about Open Source is that if you have an itch, and the drive to scratch it yourself, you can.

That's exactly what Jeremy and Adrian did. They wanted OpenVZ optimized builds for their Proxmox VE deployments, so they developed a TKLPatch that would convert an ISO into an OpenVZ container. And if that wasn't enough, took the time to upload some of the builds to sourceforge so it would be easier for others to leverage their work.

Hats off to you guys, you rock!

TurnKey OpenVZ optimized builds

Based on Adrian's and Jeremy's work, we were able to add OpenVZ support to our build infrastructure in no time, and after some initial testing, triggered the whole appliance library to be built as optimized OpenVZ containers.

You can get them from the "Download -> More Builds" link on the appliance pages.

Pre-seeding / default passwords

Because OpenVZ builds are used in headless deployments (without a console), they include an inithook which preseeds default values and passwords (excluding the root password which is handled by the VZ CLI tools).

/usr/lib/inithooks/firstboot.d/29preseed

DB_PASS=turnkey
APP_PASS=turnkey
APP_EMAIL=admin@example.com
APP_DOMAIN=DEFAULT
HUB_APIKEY=SKIP
SEC_UPDATES=FORCE

Depending on your use case, you can preseed the values before the system is booted for the first time, or once the system has booted by executing turnkey-init.

It would be great if someone would add preseeding support to PVE...

TurnKey Proxmox VE channel

A while back the Proxmox folks came up with the idea of adding a TurnKey channel to PVE, to allow users to download TKL appliances in the same way their custom built appliances are downloaded.

It was a great idea, but unfortunately it never got off the ground.

As I mentioned above, the great thing about Open Source is that you can scratch your own itch, and I was curious how the channel mechanism worked - so I dived in. When I came up for air I had added minimal third party channel support and a TurnKey Linux channel (github).

What this basically means is you can now download and deploy any TurnKey appliance on your PVE server in a couple of clicks without leaving your browser.

I hope to see this integrated in the upcoming PVE 2.0 release [update: it's coming...]. If you're running PVE 1.9 then you can add the TurnKey channel as follows:

cd /usr/share/perl5/PVE
mv APLInfo.pm APLInfo.pm.bak
wget https://raw.github.com/turnkeylinux/pve-patches/master/PVE/APLInfo.pm

# update appliance list
pveam update

The DDoS spam bot from hell (a suburb of China)

Liraz Siri — Fri, 30 Dec 2011 14:50:32 +0000

Happy new year everyone,

I'm back online to put out a fire. My inbox was full of alerts that the CPU on the server that runs the site was maxing out.

Well boys and girls, it turns out www.turnkeylinux.org has been under an escalating distributed denial of service attack that started about two weeks ago. To the best of my knowledge the site continued operating normally. We use a ton of caching. Did any of you notice a slowdown?

Lucky for us the "attack" was braindead simple so it was easy to figure out what was happening and block the offending IPs. 32 nodes from 4 Chinese /16 network blocks which I sincerely hope aren't home to any TurnKey fans:

60.169.73.186
222.186.24.101
60.169.78.19
60.169.75.168
61.160.232.38
222.186.26.164
60.169.78.57
60.169.78.174
61.160.232.22
60.169.78.193
60.169.78.177
222.186.25.134
60.169.78.15
60.169.78.52
60.169.75.50
60.169.78.54
61.160.232.39
60.169.78.7
61.160.232.58
61.160.232.4
61.160.232.10
60.169.75.161
60.169.78.42

All using the same User Agent:

Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

Supposedly identifies as Firefox but from the logs it's transparent it isn't behaving like a real browser. For example, a real browser gets CSS and image files. This just crawls all over the site and POSTs a zillion times the kind of predictable crap our spam filter blocks half-asleep.

What does that sound like? Ah yes, a poorly programmed, incredibly persistent spam bot network from hell. None of the spam attempts went through our countermeasures but it still took up a ton of CPU time.

Being naturally inquisitive I investigated the offending IPs and it turns out most of them are running a remotely exploitable version of SSH (SSH-2.0-OpenSSH_4.3). I'm half tempted to run metasploit to get into these systems and clean away the spambot software as a public service but that's illegal and I'm a bit busy besides.

Wouldn't it be neat though if we had a net equivalent of the Justice League to deal with the kind of lowlife scum who commandeer hapless machines to run very low quality spam software?

Note that I tried doing the right thing and looked up the abuse contact for the network that was attacking us (and presumably thousands/millions of other sites) on WHOIS:

person:         Jinneng Wang
address:        17/F, Postal Building No.120 Changjiang
address:        Middle Road, Hefei, Anhui, China
country:        CN
phone:          +86-551-2659073
fax-no:         +86-551-2659287
e-mail:         wang@mail.hf.ah.cninfo.net
nic-hdl:        JW89-AP
mnt-by:         MAINT-NEW
changed:        wang@mail.hf.ah.cninfo.net 19990818
source:         APNIC

Then instead of sending off an angry e-mail into the void I actually picked up the phone, dialed the number, and listened to some funky Chinese elevator music until some guy (Mr. Jinneng Wang I presume?) who didn't speak English picked up and eventually hung up on me after an akward mutually incomprehensible exchange. Of course. How could it be any different?

I don't get it, what's the point of putting up an abuse contact in the WHOIS records if the person listed doesn't speak English? Just list the abuse contact in Mandarin and get it over with.

Sometimes I feel like a character in a Neal Stephenson novel.

Mapping AWS data centers for fastest connection

Alon Swartz — Thu, 29 Dec 2011 12:25:00 +0000

Yes, that's 'fastest', not closest.

Background

A while back I published a blog post entitled Finding the closest data center using GeoIP and indexing, which described how we automatically determine the AWS regional data center to be used for storing encrypted server backups.

We used the same solution to determine the preferred region when launching cloud servers in the Hub, as well as selecting the closest APT package archive for all TurnKey deployments.

New and improved

Since the original publication, Amazon built new regional data centers in Oregon, Sao Paulo and Tokyo, so the indexes needed to be updated.

While adding support for the new regions I decided to take it a step further and add some improvements.

Improvement #1: Automatic association (distance)

The method originally used to perform automatic association of countries/states to data centers was lacking some what and needed to be improved.

We are now using the Haversine formula, which is used to determine great-circle distances between two points on a sphere from their longitudes and latitudes.

Improvement #2: Incorporated world wide underwater cables (latency)

Originally we relied on user feedback of connection latency to tweak the indexes. This didn't scale very well, so we needed a way to make it easier.

Based on Gregs Cable map, we could mashup the automatic associations and tweak the index overrides based on expected latency.

It turns out that this was a crucial part of the equasion, as a user might be physically closer to data center X, but in reality the connection to data center Y is faster. For example, previously Australia was allocated to Singapore but has been moved to California as the pipe is much fatter (see the visual map below).

Improvement #3: Open source

We originally published the indexes, but have now open sourced the whole project on github in hope that others might find it useful, and make collaboration easier.

Putting it all together

The below screenshot plots countries/states to their associated AWS regional data centers, and overlays the world wide underwater cables for reference:

Want to zoom in? Toggle active and future cables? Check out the live mashup.

TurnKey 11.3 maintenance release - next stop Ubuntu 12.04!

Liraz Siri — Tue, 06 Dec 2011 07:34:59 +0000

Ho ho ho, happy holidays everyone! I know most of you are already shifting into holiday mode, so I'll keep it short and sweet.

We've just pushed out TurnKey 11.3 - the final maintenance release based on Ubuntu 10.04. The next release will be based on Ubuntu 12.04. We're already shifting into high gear for that. There will be surprises. Hopefully good ones!

Anyhow the new images we just pushed out from our CloudTask automation swarm include fixes for various bruises and scrapes, as well as the very latest security updates.

If you've already installed a previous version of TurnKey 11, you don't need to download anything because by default TurnKey is configured to automatically install all of the security updates over the network.

The maintenance release will mainly be of interest to new users and existing users doing new deployments. Especially those of you who are super impatient like mua and don't care to wait long minutes after deployment for the system to pull over a ton of security updates. This cuts down the time it takes to fully deploy Core in the cloud from 5 minutes to just 30 seconds.