news

Rsync the entire TurnKey library from a mirror close to you in under 5 minutes!

Liraz Siri — Wed, 22 Feb 2012 05:51:39 +0000

Like TurnKey so much you want a local copy of all the appliances but too lazy to download individual appliance images from SourceForge by hand via browser?

I know exactly how you feel. Sloth is a virtue, and in the beginning was the command line.

So now you can use rsync or ftp to batch download the entire virtual appliance library in whatever build type you like best from a high-speed mirror near you. The way the net gods intended!

Thanks to generous donations of bandwidth and storage space from opensource friendly network samurais around the world TurnKey now has 16 high-speed rsync/ftp capable mirrors in 12 countries: China, Ireland, United Kingdom, France, Germany, Sweden, Japan, Belarus, Bugaria, Denmark, Argentina, and Israel. And we're just getting started...

This means if your network is good enough you can now grab a copy of the entire TurnKey appliance library in any of the 6 supported build types (ISO, VMDK, OVF, Xen, OpenVZ and OpenStack) from a high-speed (e.g, 10-Gbps) mirror close to you in under 5 minutes.

A few minutes ago in a practice run I rsynced TurnKey images to my Amazon EC2 instance at an average 288 Mbps effective download rate over the network.

Wowsers! Up until I upgraded to a nice SSD a couple of weeks ago that was about as fast as I could copy files on my local hard drive.

If you find these network speeds hard to believe I invite you to log into your TurnKey Hub account and launch a small instance in Ireland. Now let's rsync the entire TurnKey appliance library in Xen format - all 8 GBs worth from the HEAnet mirror:

$ rsync rsync://ftp.heanet.ie/[snip]/turnkeylinux/

Welcome to the HEAnet mirror site, ftp.heanet.ie (http://ftp.heanet.ie/about)
-----------------------------------------------------------------------------

 NOTE: All connections and transfers are logged; if this is disagreeable,
 please disconnect now.

 * ftp.heanet.ie is located in Dublin, Ireland and operated by HEAnet, the
   Irish National Research and Education Network.

 * This is a four node cluster with 10 Gigabit access to the HEAnet backbone.

 * Please contact mirrors@heanet.ie with any operational queries.

 * You are connected to ftp-node2 (kokapetl)

-----------------------------------------------------------------------------

drwxr-xr-x         157 2012/02/12 17:05:27 .
drwxr-xr-x        4709 2012/02/08 18:00:08 iso
drwxr-xr-x        5686 2012/02/08 18:00:58 openstack
drwxr-xr-x        5434 2012/02/08 18:01:10 openvz
drwxr-xr-x        4930 2012/02/08 18:01:21 ovf
drwxr-xr-x        2759 2012/01/13 08:31:59 pve
drwxr-xr-x        5014 2012/02/08 18:00:44 vmdk
drwxr-xr-x        5266 2012/02/09 20:41:48 xen

$ time rsync -av -P rsync://ftp.heanet.ie/[snip]/turnkeylinux/xen ./

receiving incremental file list
xen/
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2
   299570436 100%   30.43MB/s    0:00:09 (xfer#1, to-check=83/85)
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2.sig
         490 100%    0.38kB/s    0:00:01 (xfer#2, to-check=82/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2
   191193216 100%   34.68MB/s    0:00:05 (xfer#3, to-check=81/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2.sig
         490 100%    0.48kB/s    0:00:00 (xfer#4, to-check=80/85)

[ .. snip .. ]

sent 1629 bytes  received 7993104501 bytes  36919658.80 bytes/sec
total size is 7992120707  speedup is 1.00

real    3m35.152s
user    0m29.290s
sys     0m30.220s

$

Speaking with the authority of someone that used to download shareware from a local BBS 20,000 times slower (I.e., 6MB/hour on a 14400 baud modem) I can unequivocally state that this is just friggin awesome. The future is now!

Shouts out to TurnKey mirror best buddies all over the globe:

Zhang from the USTC Linux User Group in China
James from Bytemark hosting in the UK
Arnoud from LIP6 in France, and Manuel from Ircam, also in France
Carsten from RWTH Aachen in Germany
Mattias from Umea Uni in Sweden
Mitry from Beltelecom MGTS in Belarus
Boian from IPACCT in Bulgaria
Georg from dotsrc.org in Denmark
Ariel from Cooperativa Telefonica in Argentina
Lior from the Israel Internet Association in Israel

Not yet a TurnKey mirror best buddy but thinking you might want to be? If you have the resources to provide a mirror for TurnKey in your country reach out to our global mirror commando team 24 hours a day in any real or fictional language!

Announcing TurnKey Xen optimized builds

Alon Swartz — Fri, 10 Feb 2012 08:45:17 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for the project.

Recently we announced TurnKey optimized builds in a number of new formats, which brings the supported list to: ISO, VMDK, OVF, Amazon EC2, OpenStack and OpenVZ.

I'm pleased to announce that we have just added Xen to the list of optimized builds. They are hot out of the build farm and available for immediate download.

You can get them from the Download link on the appliance pages.

Pre-seeding / Default passwords

The Xen images are mainly built for hosting providers who utilize the Xen Hypervisor.

Because Xen builds are used in headless deployments (without an interactive console), they include an inithook which preseeds default values and passwords.

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=turnkey

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=$MASTERPASS
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE
EOF

You will most likely want to have your provisioning system to override the defaults by creating /etc/inithooks.conf.

Note that inithooks.conf will be blanked out once its no longer needed for security. You should also make sure that inithooks.conf includes *ALL* of the variables, otherwise the inithook system will turn on interactivity.

If you cannot support preseeding, the alternative is to have the user execute turnkey-init on first login.

Muchas Gracias to Marc from GigaTux (an official TurnKey partner) for testing the Xen images and providing feedback!

Announcing TurnKey OpenStack optimized builds

Alon Swartz — Wed, 01 Feb 2012 11:51:51 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for us. Unfortunately there are too many players in the cloud software space for us to support every single one. It's much easier to put effort into making TurnKey work well with the winning horses.

TurnKey has been supported on the leading public cloud platform Amazon EC2 from early on, not to mention simplifying management and deployment via the Hub.

OpenStack is particularly interesting, because as it is most likely the future of open source clouds.

I originally got intrigued when I heard about NASA planning to open source Nebula in 2009, which has become the basis for Nova, the compute component in OpenStack. Since then, I've been following OpenStack development from a far and have been itching to develop support for TurnKey on the platform.

The time has finally arrived, and I'm pleased to announce TurnKey optimized builds are hot out of our build farm, and available for immediate download and deployment.

You can get them from the "Download -> More Builds" link on the appliance pages.

TurnKey OpenStack optimized builds

EBS auto-mounting support: we've updated our custom EBSmount mechanism for OpenStack, which automatically mounts EBS devices when attached.
Support for automating instance setup: via the user-data scripts mechanism.
Automatic APT configuration on boot: saves bandwidth costs by using the closest package archive for maximum performance.
SSH key support: instances that are launched with a key-pair will be configured accordingly.
SSH host key fingerprints displayed in system log: verification of server to prevent man-in-the-middle (mitm) attacks.
Randomly generated root password: is set on first boot, and displayed in the system log **.
Randomly generated mysql/postgres passwords: the MySQL root and/or PostgreSQL postgres passwords are set to to the same random password as root **.
Instance metadata python library and CLI: used internally, but useful for advanced users. (learn more).

** Because OpenStack builds are used in headless deployments (without a console), they include an inithook which preseeds default values, and random passwords:

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=$(mcookie | cut --bytes 1-8)

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=turnkey
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE

Depending on your use case, you can utilize user-data (note the security implications) to preseed during boot, or once the system has booted by executing turnkey-init.

Exemplary import of TurnKey Core on OpenStack

There are several ways of uploading an image into an OpenStack deployment, below is one way to get you started.


# cd /tmp
# tar -zxf turnkey-core-11.3-lucid-x86-openstack.tar.gz
# ls turnkey-core-11.3-lucid-x86
    turnkey-core-11.3-lucid-x86-initrd
    turnkey-core-11.3-lucid-x86-kernel
    turnkey-core-11.3-lucid-x86.img

# IMG=turnkey-core-11.3-lucid-x86

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=aki \
    disk_format=aki \
    name="$IMG-kernel" \
    < /tmp/$IMG/$IMG-kernel

Added new image with ID: 5

# KERNEL_ID=5

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=ami \
    disk_format=ami \
    kernel_id=$KERNEL_ID \
    name="$IMG" \
    < /tmp/$IMG/$IMG.img

Added new image with ID: 6

# glance -A $GLANCE_TOKEN index

ID  Name                                Disk Format  Container Format  Size
--  ----------------------------------  -----------  ----------------  ---------
6   turnkey-core-11.3-lucid-x86         ami          ami               688498688
5   turnkey-core-11.3-lucid-x86-kernel  aki          aki               4179712

# euca-describe-images

IMAGE   ami-00000006    turnkey-core-11.3-lucid-x86         available  public                  machine aki-00000005
IMAGE   aki-00000005    turnkey-core-11.3-lucid-x86-kernel  available  public                  kernel

Announcing TurnKey OpenVZ optimized builds (+ Proxmox VE channel)

Alon Swartz — Mon, 16 Jan 2012 14:30:41 +0000

OpenVZ and Proxmox VE has been a recurring topic of discussion on the forums, for which we have Jeremy to ~~blame~~ thank. He's done tons of research, testing, preaching, and then some.

What I love about Open Source is that if you have an itch, and the drive to scratch it yourself, you can.

That's exactly what Jeremy and Adrian did. They wanted OpenVZ optimized builds for their Proxmox VE deployments, so they developed a TKLPatch that would convert an ISO into an OpenVZ container. And if that wasn't enough, took the time to upload some of the builds to sourceforge so it would be easier for others to leverage their work.

Hats off to you guys, you rock!

TurnKey OpenVZ optimized builds

Based on Adrian's and Jeremy's work, we were able to add OpenVZ support to our build infrastructure in no time, and after some initial testing, triggered the whole appliance library to be built as optimized OpenVZ containers.

You can get them from the "Download -> More Builds" link on the appliance pages.

Pre-seeding / default passwords

Because OpenVZ builds are used in headless deployments (without a console), they include an inithook which preseeds default values and passwords (excluding the root password which is handled by the VZ CLI tools).

/usr/lib/inithooks/firstboot.d/29preseed

DB_PASS=turnkey
APP_PASS=turnkey
APP_EMAIL=admin@example.com
APP_DOMAIN=DEFAULT
HUB_APIKEY=SKIP
SEC_UPDATES=FORCE

Depending on your use case, you can preseed the values before the system is booted for the first time, or once the system has booted by executing turnkey-init.

It would be great if someone would add preseeding support to PVE...

TurnKey Proxmox VE channel

A while back the Proxmox folks came up with the idea of adding a TurnKey channel to PVE, to allow users to download TKL appliances in the same way their custom built appliances are downloaded.

It was a great idea, but unfortunately it never got off the ground.

As I mentioned above, the great thing about Open Source is that you can scratch your own itch, and I was curious how the channel mechanism worked - so I dived in. When I came up for air I had added minimal third party channel support and a TurnKey Linux channel (github).

What this basically means is you can now download and deploy any TurnKey appliance on your PVE server in a couple of clicks without leaving your browser.

I hope to see this integrated in the upcoming PVE 2.0 release. If you're running PVE 1.9 then you can add the TurnKey channel as follows:

cd /usr/share/perl5/PVE
mv APLInfo.pm APLInfo.pm.bak
wget https://raw.github.com/turnkeylinux/pve-patches/master/PVE/APLInfo.pm

# update appliance list
pveam update

TurnKey 11.3 maintenance release - next stop Ubuntu 12.04!

Liraz Siri — Tue, 06 Dec 2011 07:34:59 +0000

Ho ho ho, happy holidays everyone! I know most of you are already shifting into holiday mode, so I'll keep it short and sweet.

We've just pushed out TurnKey 11.3 - the final maintenance release based on Ubuntu 10.04. The next release will be based on Ubuntu 12.04. We're already shifting into high gear for that. There will be surprises. Hopefully good ones!

Anyhow the new images we just pushed out from our CloudTask automation swarm include fixes for various bruises and scrapes, as well as the very latest security updates.

If you've already installed a previous version of TurnKey 11, you don't need to download anything because by default TurnKey is configured to automatically install all of the security updates over the network.

The maintenance release will mainly be of interest to new users and existing users doing new deployments. Especially those of you who are super impatient like mua and don't care to wait long minutes after deployment for the system to pull over a ton of security updates. This cuts down the time it takes to fully deploy Core in the cloud from 5 minutes to just 30 seconds.

New Hub feature: Cloud server monitoring

Alon Swartz — Fri, 02 Dec 2011 02:15:11 +0000

Ladies and gentle geeks, I'm proud to announce we've just pushed out 100% free basic server monitoring to all TurnKey Hub accounts. This should make it easier to keep tabs on the health and performance of your cloud servers. Existing Hub users don't need to do anything to enjoy this new feature. It just works.

A better server dashboard

As you can see in the screenshot below, the server dashboard now includes thumbnail graphs of CPU utilization, disk IO and network traffic for the last hour:

Give me more!

Alright, so instead of the last hour, you want data on how your server was doing last night? Or last week? No problem. CloudWatch samples performance at 5 minute intervals, and stores up to two weeks worth of data.

Clicking on the thumbnail graph pops up a larger interactive graph that lets you zoom in and out, sample performance metrics ondifferent timescales (e.g., hourly, daily, weekly, etc.) and move back and forward in time:

No installation, monitoring agents required

You don't need to install or configure any monitoring agents, because the Hub pulls statistics directly from Amazon's CloudWatch API. CloudWatch in turn gets its data directly from the virtualized hardware layer running underneath your server's operating system.

The Hub now supports reserved instances - pay up to 50% less

Liraz Siri — Wed, 12 Oct 2011 08:05:02 +0000

In response to user demand on the forum we've added Hub support for Amazon EC2 reserved instances.

Reserved instances are an Amazon EC2 feature that allows you to reserve server capacity up to 3 years in advance by paying a low one-time fee.

In exchange the hourly usage rate is significantly reduced and server capacity is guaranteed to be available when you need it.

How much will I save?

For servers that are deployed full-time you can save up to 50% of costs. For example, running a small server continuously:

Cost	Unreserved	Reserved (1 year)	Reserved (3 years)
Up front	$0	$227	$350
Hourly rate	$0.085	$0.03	$0.03
Monthly cost	$61	$21	$21
1 year	$734 ($61 x 12)	$486 ($227 + $21 x 12)	$375 ($375/3 + $21 x 12)
3 years	$2,203	$1,460 ($486 x 3)	$1,127 ($375 + $21 x 12 x 3)
Total savings	$0	$247/year (35%)	$1,075/3 years (50%)

Similar relative cost savings are available for all server sizes.

Does reserving capacity obligate me to run a server full time?

No. At any time you can further reduce costs by shutting down unused capacity. Servers reserved for 3 years break even at 24% usage.

Which server types can I reserve?

Only EBS-backed TurnKey servers. S3-backed TurnKey servers cannot be reserved.

How exactly does it work?

Through the Hub you reserve a server instance of a specific size in a specific region. Amazon bills you a non-refundable one-time fee. The Hub applies the discounted hourly rate immediately to any matching server. You'll see this reflected in the server dashboard.

If you don't have a running server, the discounted rate will be applied to the next server you launch in the reserved size and region.

Are the reduced usage fees attached to a specific server?

You don't have to worry about that. Reserved instances are a property of the billing system, not your cloud servers. For example, if you shut down an old server and start a new one in the same region the reserved pricing will apply to the new server automatically.

For further details see Amazon's website

Introducing CloudTask - a cloud batch execution tool

Liraz Siri — Fri, 12 Aug 2011 13:17:48 +0000

The cloud. Isn't that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you'd forgive the cynics for thinking there's no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren't practical a few years back.

Like renting an armada of servers you could never afford to buy to parallelize a computational task and get results in an hour instead of days, weeks or even months, for exactly the same cost. Now that's kind of exciting if you can pull it off.

We came across a modest version of this problem for various routine TurnKey related maintenance tasks such as rebuilding appliances. It sure was nice to be able to fire up a server on-demand, run a batch job and be able upload new images to sourceforge at 100MB/s. Usually we would leave the batch job running overnight and terminate the server the next day or so. That wasn't so bad for non-frequent tasks, but we realized we could do better. On Amazon EC2 running 10 servers for 1 hour costs the same as running 1 server for 10 hours. That's the theory anyhow. In practice launching and controlling many servers by hand can be painful.

Obviously, a bit of clever automation would be just the ticket. I hate reinventing the wheel so I first tried surveying existing solutions, but I couldn't find any that fit our needs.

So I rolled up my sleeves, and about a month or so later cloudtask was born. It's kind of neat. If you've ever had to do this sort of thing by hand or put together an ugly mess of scripts you'll probably find cloudtask an easier and more reliable primitive to build on.

Here's the documentation.

Here's a tacky video demo I whipped up (best viewed full screen):

TurnKey 11.2, free micro instances, EBS backed cloud servers

Liraz Siri — Tue, 02 Aug 2011 12:37:11 +0000

TurnKey 11.2: micro instances, EBS support, built-in TurnKey DNS, security updates

We just updated the web site and the TurnKey Hub with the new TurnKey 11.2 maintenance release, which includes:

TurnKey Hub support for micro instances, Amazon's free tier and cloud servers backed by persistent network-attached storage volumes (AKA EBS backed instances).
Built-in support for TurnKey's new dynamic DNS service.
The latest security updates.

TurnKey Micro instances: 2 cents/hour or 0 cents/hour for a year with the free tier

We've added support for micro instances (613 MB RAM), Amazon EC2's smallest cloud server type which costs just 2 cents an hour to run, which is less than $15/month if you run a server 24x7. If that isn't close enough to free for you, Amazon is giving away a year's worth of micro instance usage to new users as part of their free tier program.

This means many of you will now be able to try out TurnKey in the cloud free for a year. Yay!

We would have added support for micro instances as soon as they came out except Amazon designed them to work differently from other instance types we already supported. In particular, we had to add support for EBS backed instances...

EBS backed instances: cloud servers that can be turned off any time

Up until now the TurnKey Hub only supported S3 backed instances. These are non-persistent cloud servers with temporary storage that is lost once you destroy the server. This means you can't just turn off an S3 backed instance to save usage fees when you are not using it, though you could work around this limitation by using TKLBAM to backup a cloud server before destroying it and later restoring its state into a new cloud server.

With the support we've added for EBS backed instances, this limitation has been removed. EBS is what Amazon calls its on-demand Network Attached Storage service. The catch is that the Hub has to pre-allocate a fixed size EBS volume for your cloud server to boot from. Unless you are in the free usage tier you'll have to pay an additional $0.10/GB per month in EBS storage fees for the convenience (e.g., 50GB EBS volume == extra $5/monthly). The ability to turn off servers when not in use may make up for this extra cost though.

Speaking of costs, the pricing structure on the TurnKey side is a bit different for EBS backed instances as Amazon doesn't allow vendors to add a 10% markup to hourly usage fees like we've been doing with S3 backed instances. So instead, we're probably going to be experimenting with a global fixed monthly fee for this feature. After the trial period ends (in a month or so). Currently there is no extra charge.

Note that this future extra monthly charge will not apply to micro instances.

A word of warning: EBS is not a backup replacement and EBS-backed instances still need to be backed up by TKLBAM. EBS volumes just provides data persistence. It's a network hard drive that lives in a specific Amazon datacenter. It is not a replacement for backups. For example, if the data on your EBS volume gets accidentally deleted or corrupted, without a backup system to restore from you will be out of luck. TKLBAM on the other hand provides true incremental backups, so good data can't be accidentally overwritten by bad. Also, TKLBAM uses S3, which is designed by default to provide 11 nines (99.999999999%) of storage reliability, much higher than EBS.

Security updates

As most of you know security updates are already installed automatically on first boot and nightly (by default). If you're using an older version of TurnKey this means you don't need to do anything to get the latest security fixes. But for new deployments pulling a large number of security updates over the network can take considerable time, so occasional maintenance releases that already include them are a good idea.

We're in the process of upgrading our development process so this sort of update will be easier to do in the future and can be done as frequently as necessary.

Important security notice: Your TurnKey system may no longer be receiving automatic security updates

Liraz Siri — Thu, 16 Jun 2011 17:37:17 +0000

I have some bad news and some good news. The bad news is that if your TurnKey installation is older than 2 weeks you may no longer be receiving security updates.

The good news is that you are reading this and there is a very easy fix. Either reboot your system, or log in and restart the cron service:

/etc/init.d/cron start

Until you start recron, security updates and other scheduler related services (e.g., daily backups) will not work.

What happened?

Ubuntu screwed up a recent security update. There was a nasty bug. When installed, the update breaks cron, the scheduling daemon TurnKey uses to auto-install security updates. Not good.

According to a routine report generated from the access logs on our security repository, there are currently thousands of TurnKey installations affected by this issue. Those systems are not getting automatic security updates. There's no immediate risk, but that could quickly change if a remote vulnerability is discovered in the time it takes whomever is responsible for the server to figure this out.

Make sure we can always reach you

There's moral in all of this: make sure we can always reach you somehow.

Sure, usually we don't need to get your attention regarding security issues because TurnKey is configured to auto-install updates, but as this incident shows, we can't rely on that always working.

This time we can't fix the issue on our side, since it effects the very auto-update mechanism that's usually used to fix security issues.

The best we can do is try to reach out to users and inform them that there is an issue that they need to manually intervene to resolve. Hopefully we can get through to anyone subscribed to this blog or the News and Security announcements newsletter, or that has a Hub account.

In any case, we'll soon find out from the logs on the security repository just how many of our users we can or can't reach.

Announcing TurnKey Hub v1.0 - now officially out of private beta

Alon Swartz — Wed, 30 Mar 2011 12:42:48 +0000

When we first announced the TurnKey Hub private beta about 9 months ago, we had limited capacity (invitation only) and a modest feature set. Since then we tested, bugfixed, removed bottlenecks and added features, constantly improving the Hub with the help and feedback from our excellent beta users. Thank you so much!

With the release of TurnKey 11 which was tightly integrated with TKLBAM and the Hub, the amount of Hub invitation requests exploded. We were prepared for this and managed to scale the Hub smoothly without any serious issues.

With several months of testing, feedback and bugfixes under our belt we are now confident enough to officially announce, a bit earlier than planned, that the Hub is out of private beta. As of today, the Hub is open to all, and new users will no longer be required to request an invitation.

Existing users can rest easy though. We will continue to carefully monitor the Hub's performance. There should be no interruptions to the service. Worse case scenario, if we start hitting unforseen capacity issues we will temprarily reintroduce the limit on new signups.

Review of notable changes since the initial release

TurnKey Backup and Migration

A few months into the private beta we announced support for TurnKey Backup and Migration (AKA TKLBAM), which amongst other uses makes previously difficult tasks such as testing your backups much easier.
In response to demand, we've added support for configurable backup retention. Users can specify how many full backups they would like to keep for any given server backup (set to unlimited by default).

TurnKey Cloud Servers

Support for TurnKey Linux 11 images (legacy images still available to ease migration).
Basic pre-launch configuration: No more having to fiddle with the default passwords after an instance launches. The Hub supports pre-seeding appliance configuration before launch. This makes up for not having console access that would usually be required for first boot configuration.
TKLBAM pre-initialization: No more having to cut and paste your Hub APIKEY to initialize TKLBAM. The Hub pre-initializes TKLBAM automatically when the instance is first launched.
Upgradeable Kernels: We've figured out how to make it easy to update the kernel via pv-grub.
Preset launch region automatically chosen by geo-location of user.

General stuff

Performance optimizations, improved stability and error handling.
Refined the look and feel with an update to the theme.
We now try harder to explain how the Hub works and what it's good for before and after you sign up. For example we've added nice visual tours of the Backup and migration and Cloud servers features.
We've added a pricing page answering frequently asked questions. Yes, the Hub is still free. You pay Amazon directly for the cloud resources you use.
Improved start page to get you going once you sign up. Once you setup your account, this transforms into a dashboard that provides a high level overview and quick access links.
New and improved notifications (growl style).
Removed invitation requirement and added support for OpenID signup and authentication.
Added functionality to change account email.
Full internationalization support (UTF-8).
APT archive geo-location API service for choosing the closest package archive.
Link to Privacy policy.

As usual, feedback is appreciated. If you don't have a TurnKey Hub account yet, go get one now or try out the demo. If you already have a TurnKey Hub account, go check out the new stuff.

The TurnKey Hub lives at: https://hub.turnkeylinux.org

TurnKey Linux 11 released (part one)

Liraz Siri — Thu, 27 Jan 2011 11:18:10 +0000

Ladies and gentlemen, part 1 of the TurnKey Linux 11 release is now officially out, including 45 new images based on Ubuntu 10.04.1. We pushed out the 11.0 release candidates 3 months ago, and with the help of the community have tested the images and resolved the few remaining issues.

Part 1 mostly refreshes the existing roster of appliances in the library. In the upcoming part 2 we'll release the new appliances the community has been helping us develop over the last year. This will roughly double the size of the library.

A handful of new appliances have also been squeezed in:

Joomla 1.6

Magento

StatusNet

PrestaShop

vTiger CRM

This was mostly a side effect of the original (misguided) plan to do one big massive release with over 80 appliances.

What's changed since the release candidates

VM optimized builds: are now available, in two exciting flavors...
1. Default: this is primary downloadable VM build. Works best with VirtualBox, low-end VMWare products (Player, Workstation, Server).
2. OVF: OVF is the new VM standard supported by VirtualBox and high-end VMWare products (e.g., ESX, vSphere).
We're also working on providing images optimized for other popular virtualization platforms such as Xen, UEC / Eucalyptus and OpenVZ.
Default passwords: You no longer have to keep track of any default passwords or change them after installation. TurnKey now helps you configure them on first boot, via the console.

For headless deployments without a console, it's possible to pre-seed answers to first boot configuration questions.
/etc under git: Automatic revision control of /etc using etckeeper, as suggested by Jeremiah Snapp. If a configuration change you made breaks something, just roll it back!
LVM snapshots: Fixed LVM snapshots by adding 10% unallocated disk space to default LVM configuration. This will make it easier to add support for atomic filesystem backups in upcoming versions of TKLBAM.
Amazon EC2 / TurnKey Hub related changes:
- Hub does TKL 11: TurnKey Hub now deploys TurnKey Linux 11 images by default, though support for older legacy images is still available to ease migration.
- Basic pre-launch configuration: No more having to fiddle with the default passwords after an instance launches. TurnKey Hub now supports pre-seeding appliance configuration before launch. This makes up for not having console access that would usually be required for first boot configuration.
- TKLBAM pre-initialization: No more having to cut and paste your Hub APIKEY to initialize TKLBAM. The TurnKey Hub pre-initializes TKLBAM automatically when the instance is first launched.
- Upgradeable Kernels: We've figured out how to make it easy update the kernel.
  
  Kernel upgrades were previously not supported on Amazon EC2, because each Amazon Machine Image (AMI) had to be associated with a specific Amazon Kernel Image (AKI). Now instead of associating the image to a specific kernel, we associate it with a special EC2 compatible bootloader (pv-grub), which can bootstrap whatever kernel is configured from within the system (e.g., security fix).
Updated website documentation
- All 45 appliance pages (e.g., TurnKey Core) now reflect the latest image versions.
- Documented appliance build types.
- Updated virtualbox installation tutorial, virtualization notes, automatic security updates pages.
Bug fixes: An assortment of fixes for issues reported by the community.

See the previous announcement of the release candidates for changes since the last maintenance release (2009.10-2, based on Ubuntu 8.04 LTS).

Many, many thanks to...

Everyone who helped test the release candidates and provided ideas and feedback.
The many rivers of upstream: Ubuntu, Debian and all of the wonderful open source communities who give love and write code for the software that goes into TurnKey.
JedMeister, Adrian Moya, Basil Kurian, and Rik Goldman - pillars of the TurnKey community who have inspired us with their dedication and generosity.
TurnKey enthusiasts everywhere. Without you, TurnKey's audience, there really wouldn't be a point!

What's next

Images optimized for Xen, Eucalyptus / UEC and OpenVZ
Part 2: double the size of the library
64-bit support
Debian squeeze based beta builds (we've decided to skip Lenny)
PostgreSQL support for TKLBAM
TurnKey Hub stuff
- Support for Amazon EBS backed root devices
- Support for micro instances. Amazon's 1-year free tier will now allow free evaluation of TurnKey on Amazon EC2.
- Support for larger instances (64-bit support required).
- Support for more hosting providers.

Announcing the winners of the first TurnKey development contest

Liraz Siri — Wed, 24 Nov 2010 15:17:55 +0000

Ladies and gentlemen, by now you are all shuddering in anticipation, dying to find out who won the very first TurnKey appliance development contest... in the history of the universe!

The stakes couldn't be higher. In truth, all three of our winners will lavish in (varying proportions of) unending fame and riches, but only one man will get the exotic, elusive embodiment of the Django pony goddess. Oh ye most blasphemous infidel will ask, why thee persists in worshiping a pony, thus pink and plush. Hush I say, tis not any earthly pony we worship, but the very symbol of the holy bond that ties all open source communities together. So say we all!

As you might recall from the summary of the contest published a few weeks ago, Basil Kurian and Adrian Moya were the two obvious candidates for the first place. We asked the community for feedback on who should win, and received hundreds of votes to the poll.

Before I announce the winners, let's give both of them a big round of virtual applause for participating in the contest and amazing us with their contributions. If it weren't for these two, the contest would surely have been a complete failure and we would have crossed off community contests from our list of good ideas. But thankfully these two heros came along, gave it their all in the true spirit of open source and now the upcoming release is going to nearly double the size of the TurnKey library. As soon as we have enough loot to give away (we're working on that) we'd like to try sponsoring more contests.

Alright, enough with the suspense. The winners:

Adrian Moya wins the gold!

First prize: $1500 and his very own pony.
Basil Kurian wins the silver!

Second prize: $800.
Rik Goldman wins the bronze!

Third prize: $100.

Cliche or not, everyone who participated in the contest is a winner in my book. We're going to be posting interviews with Adrian and Basil on the blog as soon as the release comes out, so everyone can get to know them better. Earlier we posted an interview with Rik Goldman.

Behind the scenes

Both Adrian and Basil did such fantastic work on the contest that it was hard to decide who won first place. Not just for us. Most of the community had trouble deciding as well. The most popular option in the poll was a tie, amongst both registered and anonymous users. So it was close, but Alon and I talked it over and decided it couldn't be a tie. It's a contest after all! In the real world no two quantities are exactly equal even if they sort of look that way from a distance. So if you measure accurately enough, one will have to be larger than the other, even if only by a little bit.

The problem is the closer it gets, the harder the measurement. There has to be a fair basis for deciding one way or the other.

We came up with a formula for assigning points to contributions:

Audience * Integration + Documentation

The rational behind this is that the value provided by the integration is multiplied by the size of the audience because you're providing that value over and over again to more people. The documentation adds value because it makes it easier for us to understand and adopt the TKLPatch, and it also makes it a better example for other budding community developers. But that value isn't multiplied over the entire audience, so it's an added rather than multiplied value.

When we ran the numbers through the formula Adrian got 101 points and Basil got 83.

We also looked at feedback from the community as reflected in the poll votes. To avoid bias we didn't count votes from India or Venezuela, our contestants home countries.

Non anonymous votes:

Tie (43%)
Adrian (38%)
Basil (19%)

Anonymous votes:

Tie (66%)
Adrian (24%)
Basil (10%)

As you can see most people found it difficult to decide and voted for a tie. Among the minority that decided, Adrian got roughly twice as many votes.

That's all folks. Congratulations to our winners!

New release candidates for TurnKey Linux 11.0 (part 1)

Liraz Siri — Sat, 23 Oct 2010 16:39:51 +0000

We've pushed out new RC (Release Candidates) builds for part 1 of the upcoming TurnKey Linux 11.0 release and we need your help testing them! See the appliance pages for download links.

The current crop of release candidates only include Ubuntu Lucid based ISO images for now. Debian Lenny based images will follow, as will builds specially optimized for the the full range of supported virtualization and hosting platforms (e.g., VM build, EC2 AMIs, ESX4, Xen, Eucalyptus, etc.).

In part 1 we've updated all the existing appliances except for Zimbra and Openbravo which we haven't got to yet. They'll be in part 2 along with all the fantastic new appliances developed during the community development contest.

We broke up work on the next release into two parts because with all the new appliances the size of the appliance libary is doubling and that was a bit too much to handle in one go.

Also, Alon is flying off to Orlando today for the Ubuntu Developer Summit and we wanted to have something out the door to show off to our friends in the Ubuntu community.

New versioning scheme

Instead of date based versions the new versioning scheme tracks TurnKey Core, regardless of the date or the underlying distribution.

In other words a TurnKey Linux 11.0 appliance based on Ubuntu Lucid and a TurnKey Linux 11.0 appliance based on Debian Lenny are the same version because they have the same Core features.

We're also hoping to get rid of some of the confusion with date based releases. For example, we pushed out the maintenance release 2009.10-2 in April 2010 but the version confused many people into thinking the appliances were obsolete, even though Hardy will be supported for another 3 years and security updates were installed on first boot and daily afterwards.

Now built-in: TKLBAM (TurnKey Linux Backup and Migration)

TKLBAM, which we announced a few weeks back as a separate component, is now tightly integrated with TurnKey out of the box. This includes a spiffy new Webmin module so users who dislike the command line can fully enjoy TKLBAM from the comfort of their web browser. Now you can backup and migrate fully working systems anywhere with literally just a few mouse clicks.

As an old school Unix guy who loves the command line, what I like the most about the new Webmin module is how it facilitates easy discovery of TKLBAM's functionality and ties everything together seamlessly. No need to memorize funny command line flags or decipher manual pages for configuration options. At a glance you can see what can be done. If you don't understand something you just click on the question marks for embedded help pop-ups.

Screenshot #1: backup menu

Screenshot #2: restore menu

New Core features

Besides TKLBAM integration, we've made a broad range of improvements to the Core features shared by all appliances. In fact one could argue more has changed between the last beta and the release candidate than between the previous release and the beta.

Basic configuration dialogs on first boot: Previously only ISO builds would prompt you, during installation, to configure the appliance (e.g., set passwords). Users of the pre-installed builds (e.g,. the virtual machine build for VMWare/VirtualBox) had to do basic configuration by hand.

With the new mechanism, the user experience is consistent for all build types. Also the first boot configuration mechanism is modular so appliance customizers can easily add their own hooks.
LVM (Logical Volume Management) support: instead of installing the appliance filesystem to a naked partition, we setup LVM by default. This makes it much easier to adjust storage capacity later. Also it adds basic support for filesystem snapshots, which future versions of TKLBAM may leverage to support new kinds of databases.
auto-apt-archive: uses GeoIP to automatically configure closest APT package archive for maximum network performance.
Command line convenience: a range of small improvements that make working with the command line a bit more comfortable:
1. smart, programmable bash shell completion: helps you get more done with fewer keystrokes.
2. support for $HOME/.bashrc.d hooks (see my blog post about shell hooks)
3. persistent environment variables (see $HOME/.bashrc.d/penv):
```
penv-set foo=bar
exit

# later...
echo $foo
```
Inverted webshell color scheme: real men use white on black for their command line shells. And they like it!
Display system info in the login motd: seen in the above screenshot.

Appliance specific changes

The latest and greatest: all appliances are now built using the newest software packages available with Long Term Support from Ubuntu. For software not available in the package management system (e.g., redmine), the latest stable software versions from upstream are used instead.
A range of performance optimizations: for example...
- All LAMP-based appliances now includes xcache for PHP opcode caching which improves PHP performance, as code only has to be recompiled by PHP into opcode if it changes, and not on every page load.
- All Rails based appliances now come with Ruby Enterprise which improves memory performance, traditionally one of Ruby's weak points.
Many small improvements and bug fixes
Exciting potential for new bugs: as yet unidentified.

Distribution level changes

Upgraded base distribution: from Ubuntu 8.04.3 to Ubuntu 10.04.1
- The good: newer packages based on a new Ubuntu LTS release which will be supported for 5 years.
- The bad: roughly 50MB in additional bloat for all appliances. This is mostly a consequence of bloated dependencies within the new Ubuntu version. Some of the dependencies are unnecessary for our use case (e.g., plymouth) but the only way to get rid of them is to fork the packages and we'd rather avoid that.
  
  Fortunately, the upcoming Debian Lenny based builds do not have this extra bloat so anyone who cares will have an alternative.
No more pinning: As discussed earlier we've discontinued use of complex package management configurations to achieve mixed hybrids of Ubuntu and Lucid.

In the works...

Debian Lenny based equivalent builds
PostgreSQL support for TKLBAM
Part 2 of the TurnKey with dozens of new appliances
64-bit support (it's about time!)

That's all folks! Any questions?

TKLPatch summer contest summary: let the judging begin!

Liraz Siri — Thu, 30 Sep 2010 17:56:01 +0000

It all started with a happy accident

I have a confession to make. This contest, which is directly fueling the largest expansion of the TurnKey library since the project started, is a happy accident. It wasn't something we planned. It wasn't on our summer todo list. It was just one of those unexpected, spontaneous ideas that light up the inside of your brain like a flash bulb, and demand you take action. Or else! (you won't get any sleep)

Back in June we had just launched the TurnKey Hub and were getting ready to focus all our energies on releasing TKLBAM. I logged into PayPal and noticed our donated beer budget had a sad little beer belly. It was just sitting there, giving me an accusing look. I felt guilty. Surely all those people who donated expected we would put these funds to better use. That's when it hit me. It was too much to buy beer, but not so much that we couldn't risk it all on a fun experiment...

I talked it over with Alon and on an impulse we decided to do a contest, but not just any contest. A wild and wet summer open source bonanza! With ponies!

What happened next took us both by surprise.

From the far corners of the globe... two superstars emerge

I figured if we were lucky we would attract about a dozen or so contributors who would create appliances for one or two of their favorite open source applications.

But what I do know. It turns out we got even luckier with two superstar developers who took different but complimentary approaches, and blew us away with over two dozen TKLPatches for an amazing range of excellent open source software, some of which we had never even heard of before!

From India, Basil Kurian went wide with about 20 relatively straightforward new appliances, developed at a furious pace that was hard to keep up with. Like a hungry bird of prey, Basil scanned the open source savanna from above, then swooped down for kills he could snatch back whole to his nest.

From Venezuela, Adrian Moya dug in deep, with about 10 relatively challenging appliances. His attention to detail rivals some of our best work to date. Adrian, like a tiger, stalked his prey, taking a slower, more deliberate approach. He went after larger targets, sunk his teeth in and tore them to pieces.

There were a few other contributions as well but the bulk of developments can be credited directly to these two admirable and highly skilled individuals.

Looking at the big picture, what they've accomplished is not only a tribute to their own impressive abilities but also to the power of the open source ecosystem they leveraged. I can't think of any other area of human enterprise where a loosely knit band of individuals is empowered (in their spare time no less) to help enable thousands of individuals and organizations all over the world to take advantage of so much technological innovation. That's what TurnKey means to Alon and I and we're delighted that more people are getting to share this experience!

Help! How do we figure out who should win?

It was probably best for the project that our two superstars took different complimentary approaches. But it complicates our decision as a direct apples to apples comparison isn't possible. That means instead of just counting the number of submissions we actually have to think this through, and we'd like the community to help us with that.

So what's at stake? You mean besides bragging rights and immortality in the TurnKey hall of fame?

Gold: $1500 (+ a make-believe pony with a suitcase full of broken dreams)
Silver: $800
Bronze: $100

Judging criteria

Your mission, should you choose to accept it, is to help us figure out who added the most total value to the project. That means the quantity of contributions multiplied by their quality. Yes, we realize quality is inherently subjective and open to endless interpretation. We've thrown in some rough metrics and have identified dimensions of evaluation so it isn't entirely arbitrary, but let's not kid ourselves - playing with grading systems and numbers won't save us from making subjective, qualitative judgments.

To make this a bit simpler we've attempted to summarize contributions in three key dimensions:

Anticipated interest from our audience: the more people will benefit from an appliance, the more valuable it is. Unfortunately, we don't have a TurnKey crystal ball (yet), so we can only make educated guesses. Mostly we sampled data from various public resources provided by Google (e.g., Google Trends, the number of search results for a given name, the page rank of a project's web site, etc.)

Grades: minimal (), modest (), large () and huge ()
Integration effort: an appliance saves users the trouble of finding the right components, integrating them into a working solution and testing it themselves. Hard appliance integrations are more valuable because they save users more trouble.

Examples of factors we took into account when we evaluated integration effort includes:
1. how well aligned the integration is with a well defined need
2. the number of integrated components
3. whether or not pre-installation (e.g., seeding databases) was required
4. approximate testing difficulty
5. added value such as custom glue scripts
Grades: simple (), involved (), challenging () and brilliant ()
Documentation: A pre-requisite to adding an appliance to the library is that we understand what it achieves, how it does it and why. Good documentation makes all of this easier for us, and also serves to educate the community and promote discussion. Truly awesome documentation goes beyond that and adds significant value to end-users by making it easier to get started (e.g., tutorials, HOWTOs, documentation resources, etc.)

Grades: minimal (), good (), excellent (), awesome ()

Summary of submissions competing for the gold

Adrian Moya
TKLPatch	Audience	Integration	Documentation
Asterisk: VoIP PBX
Magento: ecommerce
OpenLDAP
Alfresco: enterprise CMS
Web content filtering proxy: DansGuardian + ClamAV
Big Blue Button: web conferencing
Plone: Zope-based CMS
Bacula: network backup tool
Gitorious: Git collaborative repository hosting

Basil Kurian
TKLPatch	Audience	Integration	Documentation
Mono ASP.NET: IIS replacement
vTiger CRM
Cacti: web-based graphing tool
Prestashop: e-commerce / shopping cart
SugarCRM Community Edition
Tomato cart: ecommerce
Status.NET: micro blogging
eFront: eLearning software
OrangeHRM
Vanilla forum
Groupoffice: groupware
LEMP stack: Linux, Nginx, Mysql, PHP
EyeOS: web desktop
Foswiki: structured wiki fork of TWiki
Fedena: school/campus management system
Spree: ecommerce
Retrospectiva: agile project management
Typo: rails blogware

Notes:

This is not a complete listing of submissions, just those that are most likely to be included in the upcoming release batch.
I'm linking to the forum posts where the TKLPatch submissions are discussed for future reference and so anyone who wants can make up their own mind.

Third place?

While the vast bulk of contributions came from Basil and Adrian, there were a few sporadic contributions by other community members. Here too we're interested in feedback from the community on who should win third place, or alternatively whether we should just eliminate third place and merge it with second place. Or something else...

Candidates:

Bilal: contributed a barebones Citadel appliance, and a partially completed JBoss appliance.
JedMeister: contributed a partially finished KnowledgeTree patch, researched possible integration details for the TurnKey Core Client and provided encouragement, feedback and moral support throughout the contest to all involved.
Neil Wilson: contributed a patch fixing various issues with the TurnKey Core Lucid beta.
Rik Goldman: contributed an IEP-IPP appliance, that could make it easier for school to use open source software to track students with special needs. Before the contest Rik and his band of merry students were our most prolific community appliance developers (Ampache, LimeSurvey, Elgg). Rik also contributed excellent supplemental documentation on the TKLPatch development process.

So now what? That's where you come in!

Vote on who should win and share your thoughts with the community in the comments below. If for some reason you'd rather not share your thoughts publicly feel free to contact us in a private e-mail.