news

TurnKey Core 12.0 RC optimized builds

Alon Swartz — Mon, 16 Apr 2012 01:03:17 +0000

Last month we announced the release candidate for TurnKey Core 12.0 - the common base for all appliances, based on the rock solid Debian Squeeze (6.0.4).

It took a little longer than expected, but we've finally released all the optimized builds for TurnKey Core 12.0RC: ISO, VMDK, OVF, Amazon EC2, OpenStack, OpenVZ and Xen.

The optimized builds can be downloaded from the core appliance page, directly via the TurnKey channel in ProxmoxVE (OpenVZ), deployed in the Amazon EC2 cloud via the TurnKey Hub, or via one of the official TurnKey partners (soon).

Build specific release notes

Common (ISO)

See the announcement.

Amazon EC2

Deployment: The TurnKey Hub has been updated to support Core 12.0 deployment and management (Launch new server -> 12.0). Once the full library has been updated to TKL 12.0 it will become the default, and the current release will be moved to Legacy.

VM optimized (VMDK, OVF)

Open-VM-Tools: Previous VM optimized builds included the proprietry VMWare-Tools, but since VMWare have released a large portion of the code under the GPL, we've moved to open-vm-tools.
Swap warning: VMware products might display a warning that no swap space was detected. This is a false positive, as swap is configured in LVM.

OpenStack

Ramdisk required: TurnKey 12.0 requires the initrd to be registered for successful boot (exemplary import and registration code).

OpenVZ

Naming convention: We've updating the naming convention for openvz builds to support vanilla OpenVZ out of the box, and eliminate duplication for the Turnkey PVE channel. Thanks Jeremy!
Removed NTP daemon: The NTP daemon has been removed as the clock is managed by the host. Thanks Martin!
No more upstart hacks: Removed Ubuntu upstart hacks as they are not relevant in Debian.

Xen

Xen optimized kernel: Moved to the Xen optimized kernel provided by Debian (linux-image-xen-686).

As always, we need your help in testing the builds. If you come across any issues or have ideas how they can be improved, please post a comment.

New Hub feature: Server snapshots

Alon Swartz — Wed, 04 Apr 2012 13:56:02 +0000

I usually get excited when adding new features to the TurnKey Hub. Recent excitement included server monitoring, reserved instances, domain management, and the Hub API.

I'm very excited about todays annoucement, not only is it awesomely useful, it's also technically cool!

So what are snapshots?

I'm sure you can guess, but let me explain anyway.

Snapshots can be used with EBS-backed instances to create point-in-time snapshots of the root filesystem, which are persisted to Amazon S3 for storage durability. Snapshots are incremental, meaning that only changes since the last snapshot are saved, taking up less storage, time, and reducing costs (see below for technical details).

Snapshots ask Amazon's fiber-optic storage backplane to save your server's disk state while it's running without impacting performance.

Ok, but what can I do with them?

Server clones

Snapshots can be used as the basis for a new server, essentially creating a clone (the cloud server equivalent of a time machine crossed with a portal to a less obnoxious alternative dimension), for example:

You can clone a production server to create a staging enviroment for testing new features, hacking away, whatever, without the worry of hosing your production server (guess how I tested this new feature).
You can essentially upgrade your servers hardware if you need the extra horse power, memory or even disk space. Say you were testing an idea with a micro instance, and now its taking off. Firstly congrats, secondly just clone the micro's latest snapshot to a larger instance size and update the DNS record / re-associate the elastic IP.
Let you're imagination run wild!

EBS Volumes

Snapshots can be used as a starting point for a new EBS volume, for example:

You mistakenly deleted a file, hosed your database, or whatever bad thing that can happen. You create a volume from the snapshot of your choice, attach it to your instance (which is auto-mounted via ebsmount) and access the data you need.
Again, let you're imagination run wild!

Can I schedule automatic snapshots?

You sure can! You can schedule automatic zero-load server snapshots for hourly, daily, weekly and monthly frequency, or manually create one at anytime.

There is however a snapshot limit per Amazon account, per region, so when configuring automatic scheduled snapshots, snapshot retention is also configurable to prune old snapshots, keeping you within the limit and saving you money.

Sounds cool, what does it look like?

We've added 2 new fields to the server record:

And this is the snapshot dashboard:

Are there any limitations?

Snapshots only support EBS-backed instances, and not S3-backed instances. This is a technical limitation as snapshots are performed on the EBS-backed root volume, which S3-backed instance do not have.

Snapshots are saved to S3 storage, but they will not appear in your S3 buckets, nor can you access them using the standard S3 API. To access snapshot data you need to create an EBS volume or a server clone.

As mentioned above, there is a limit of the amount of snapshots each Amazon account can have, but you can request to increase your limit (specify you want the snapshots limit increased in the comments.)

Data consistency: Do not solely rely on snapshots for backups, as they may become inconsistent due to disk-buffering and locking. We use TKLBAM for our backups, and suggest you do the same.

Technical details - snapshots explained

I mentioned that snapshots are technically cool, and that they are incremental - let me try and explain what that means at how it works behind the scenes.

A snapshot of an EBS volume can be taken at anytime, which asks Amazon's fiber-optic storage backplane to save the data stored on the volume, at the block level, at that exact point-in-time, to S3 storage.

To improve performance and reduce storage space, Amazon will only copy the blocks of the volume that have changed since your last snapshot - hence incremental.

Now for the extra cool part, unlike regular incremental backup chains, you can delete any previous snapshot. Huh? What? Yep, snapshots are not chained, but are rather conceptually like a table-of-contents of pointers to saved data blocks.

When you delete a snapshot, only the data blocks that are solely used by that specific snapshot are deleted. Data blocks that are used by subsequent snapshots are not. In the below illustration, if SNAP-B is deleted, only SNAP-B:block-2 will be deleted from Amazon S3 as a newer version (SNAP-C:block-2) has already been saved.

Bottom line, take snapshots for a spin and let us know what you think.

TurnKey Core 12.0 RC based on Debian Squeeze

Alon Swartz — Thu, 01 Mar 2012 18:35:19 +0000

I'm pleased to announce a spanking brand new release candidate for TurnKey Core 12.0 - the common base for all appliances, based on the rock solid Debian Squeeze (6.0.4). The rumors were true! Hurrah! Hurrah!

This is an RC release, so take it for a spin and let us know what you think. If you come across any issues, please report them. If you have ideas on how to make it better, let us know!

Download RC: 138MB ISO (changelog, signature, manifest)

Did you say Debian?

Why yes, yes I did. Here's the back story...

In mid-2010 we released our first ever Debian appliance based on Lenny. In the announcement Liraz discussed whether Debian based appliances are worth the trouble as well as some notes on Ubuntu vs. Debian.

Back then we decided not to release the entire TurnKey library based on Lenny as Squeeze was around the corner, and we were spread quite thin.

Fast forward to a few weeks ago, Liraz and I were discussing the upcoming Ubuntu LTS release, which is scheduled for April. We were deliberating when would be the best time to begin the transition.

During the conversation we revisited the idea of supporting Debian, and decided it was time. We've been wanting to support Debian since TurnKey's inception, and it seems that a significant 59% of users want Debian-based appliances "a lot"!

Rolled up my sleeves

So, I rolled up my sleeves and got to work. It wasn't too long and I had a working TurnKey bootstrap image (102MB ISO, Meta) based on Squeeze. The most annoying part of that was dealing with the non-backwards compatible bootsplash. Turns out that was a good thing, as it forced me to do cleanup, and remove panel options that weren't actually doing anything. How nobody filed a bug on that is beyond me :)

Then I moved onto Core. Upgrading our Live Installer (di-live) was a little boring, but after fixing some bugs and seeing it work, not to mention setup LVM and install the entire OS in under a minute, it put a smile on my face.

After upgrading several key components, fixing bugs (thanks to everyone who submitted bug reports, and Jeremy for his excellent work triaging and keeping the bug tracker up to date), tweaks here and there, and testing, I was a happy camper.

To summarize, there were ups and downs but all in all it was good fun - ask my wife, I updated her on progress every evening whether she wanted to know or not.

But, there is still a long road ahead, and this is only the first milestone.

Changes

New and improved signature files: include detailed steps on how to verify image integrity, as well as md5 and sha1 checksums for convenience. Take a look.
Locale improvements: default locale is now set to en_US.UTF-8, updated configuration for compatibility with Squeeze. Freeing up disk space is now performed by localepurge.
Boot splash and loader: upgraded bootsplash for compatibility, removed unused panel options, and tweaked bootloader timeouts.
Live installer (di-live): upgraded for Squeeze compatibility and misc bugfixes.
Webmin: upgraded to latest upstream release and disabled inline upgrades (managed by APT).

All other changes, bugfixes and tweaks are available in the changelog.

As for the features, not much has changed except for the base distribution.

Long story short, try the RC and tell us what you think. Obviously we have immense respect for both Ubuntu and Debian, and we'd like to hear your views on where we should take it from here.

Rsync the entire TurnKey library from a mirror close to you in under 5 minutes!

Liraz Siri — Wed, 22 Feb 2012 05:51:39 +0000

Like TurnKey so much you want a local copy of all the appliances but too lazy to download individual appliance images from SourceForge by hand via browser?

I know exactly how you feel. Sloth is a virtue, and in the beginning was the command line.

So now you can use rsync or ftp to batch download the entire virtual appliance library in whatever build type you like best from a high-speed mirror near you. The way the net gods intended!

Thanks to generous donations of bandwidth and storage space from opensource friendly network samurais around the world TurnKey now has 16 high-speed rsync/ftp capable mirrors in 12 countries: China, Ireland, United Kingdom, France, Germany, Sweden, Japan, Belarus, Bulgaria, Denmark, Argentina, and Israel. And we're just getting started...

This means if your network is good enough you can now grab a copy of the entire TurnKey appliance library in any of the 6 supported build types (ISO, VMDK, OVF, Xen, OpenVZ and OpenStack) from a high-speed (e.g, 10-Gbps) mirror close to you in under 5 minutes.

A few minutes ago in a practice run I rsynced TurnKey images to my Amazon EC2 instance at an average 288 Mbps effective download rate over the network.

Wowsers! Up until I upgraded to a nice SSD a couple of weeks ago that was about as fast as I could copy files on my local hard drive.

If you find these network speeds hard to believe I invite you to log into your TurnKey Hub account and launch a small instance in Ireland. Now let's rsync the entire TurnKey appliance library in Xen format - all 8 GBs worth from the HEAnet mirror:

$ rsync rsync://ftp.heanet.ie/[snip]/turnkeylinux/

Welcome to the HEAnet mirror site, ftp.heanet.ie (http://ftp.heanet.ie/about)
-----------------------------------------------------------------------------

 NOTE: All connections and transfers are logged; if this is disagreeable,
 please disconnect now.

 * ftp.heanet.ie is located in Dublin, Ireland and operated by HEAnet, the
   Irish National Research and Education Network.

 * This is a four node cluster with 10 Gigabit access to the HEAnet backbone.

 * Please contact mirrors@heanet.ie with any operational queries.

 * You are connected to ftp-node2 (kokapetl)

-----------------------------------------------------------------------------

drwxr-xr-x         157 2012/02/12 17:05:27 .
drwxr-xr-x        4709 2012/02/08 18:00:08 iso
drwxr-xr-x        5686 2012/02/08 18:00:58 openstack
drwxr-xr-x        5434 2012/02/08 18:01:10 openvz
drwxr-xr-x        4930 2012/02/08 18:01:21 ovf
drwxr-xr-x        2759 2012/01/13 08:31:59 pve
drwxr-xr-x        5014 2012/02/08 18:00:44 vmdk
drwxr-xr-x        5266 2012/02/09 20:41:48 xen

$ time rsync -av -P rsync://ftp.heanet.ie/[snip]/turnkeylinux/xen ./

receiving incremental file list
xen/
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2
   299570436 100%   30.43MB/s    0:00:09 (xfer#1, to-check=83/85)
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2.sig
         490 100%    0.38kB/s    0:00:01 (xfer#2, to-check=82/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2
   191193216 100%   34.68MB/s    0:00:05 (xfer#3, to-check=81/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2.sig
         490 100%    0.48kB/s    0:00:00 (xfer#4, to-check=80/85)

[ .. snip .. ]

sent 1629 bytes  received 7993104501 bytes  36919658.80 bytes/sec
total size is 7992120707  speedup is 1.00

real    3m35.152s
user    0m29.290s
sys     0m30.220s

$

Speaking with the authority of someone that used to download shareware from a local BBS 20,000 times slower (I.e., 6MB/hour on a 14400 baud modem) I can unequivocally state that this is just friggin awesome. The future is now!

Shouts out to TurnKey mirror best buddies all over the globe:

Zhang from the USTC Linux User Group in China
James from Bytemark hosting in the UK
Arnoud from LIP6 in France, and Manuel from Ircam, also in France
Carsten from RWTH Aachen in Germany
Mattias from Umea Uni in Sweden
Mitry from Beltelecom MGTS in Belarus
Boian from IPACCT in Bulgaria
Georg from dotsrc.org in Denmark
Ariel from Cooperativa Telefonica in Argentina
Lior from the Israel Internet Association in Israel

Not yet a TurnKey mirror best buddy but thinking you might want to be? If you have the resources to provide a mirror for TurnKey in your country reach out to our global mirror commando team 24 hours a day in any real or fictional language!

Announcing TurnKey Xen optimized builds

Alon Swartz — Fri, 10 Feb 2012 08:45:17 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for the project.

Recently we announced TurnKey optimized builds in a number of new formats, which brings the supported list to: ISO, VMDK, OVF, Amazon EC2, OpenStack and OpenVZ.

I'm pleased to announce that we have just added Xen to the list of optimized builds. They are hot out of the build farm and available for immediate download.

You can get them from the Download link on the appliance pages.

Pre-seeding / Default passwords

The Xen images are mainly built for hosting providers who utilize the Xen Hypervisor.

Because Xen builds are used in headless deployments (without an interactive console), they include an inithook which preseeds default values and passwords.

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=turnkey

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=$MASTERPASS
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE
EOF

You will most likely want to have your provisioning system to override the defaults by creating /etc/inithooks.conf.

Note that inithooks.conf will be blanked out once its no longer needed for security. You should also make sure that inithooks.conf includes *ALL* of the variables, otherwise the inithook system will turn on interactivity.

If you cannot support preseeding, the alternative is to have the user execute turnkey-init on first login.

Muchas Gracias to Marc from GigaTux (an official TurnKey partner) for testing the Xen images and providing feedback!

Announcing TurnKey OpenStack optimized builds

Alon Swartz — Wed, 01 Feb 2012 11:51:51 +0000

As we mentioned before, making TurnKey easy to deploy on as many public and private clouds is an important goal for us. Unfortunately there are too many players in the cloud software space for us to support every single one. It's much easier to put effort into making TurnKey work well with the winning horses.

TurnKey has been supported on the leading public cloud platform Amazon EC2 from early on, not to mention simplifying management and deployment via the Hub.

OpenStack is particularly interesting, because as it is most likely the future of open source clouds.

I originally got intrigued when I heard about NASA planning to open source Nebula in 2009, which has become the basis for Nova, the compute component in OpenStack. Since then, I've been following OpenStack development from a far and have been itching to develop support for TurnKey on the platform.

The time has finally arrived, and I'm pleased to announce TurnKey optimized builds are hot out of our build farm, and available for immediate download and deployment.

You can get them from the "Download -> More Builds" link on the appliance pages.

TurnKey OpenStack optimized builds

EBS auto-mounting support: we've updated our custom EBSmount mechanism for OpenStack, which automatically mounts EBS devices when attached.
Support for automating instance setup: via the user-data scripts mechanism.
Automatic APT configuration on boot: saves bandwidth costs by using the closest package archive for maximum performance.
SSH key support: instances that are launched with a key-pair will be configured accordingly.
SSH host key fingerprints displayed in system log: verification of server to prevent man-in-the-middle (mitm) attacks.
Randomly generated root password: is set on first boot, and displayed in the system log **.
Randomly generated mysql/postgres passwords: the MySQL root and/or PostgreSQL postgres passwords are set to to the same random password as root **.
Instance metadata python library and CLI: used internally, but useful for advanced users. (learn more).

** Because OpenStack builds are used in headless deployments (without a console), they include an inithook which preseeds default values, and random passwords:

/usr/lib/inithooks/firstboot.d/29preseed

MASTERPASS=$(mcookie | cut --bytes 1-8)

cat>$INITHOOKS_CONF<<EOF
export ROOT_PASS=$MASTERPASS
export DB_PASS=$MASTERPASS
export APP_PASS=turnkey
export APP_EMAIL=admin@example.com
export APP_DOMAIN=DEFAULT
export HUB_APIKEY=SKIP
export SEC_UPDATES=FORCE

Depending on your use case, you can utilize user-data (note the security implications) to preseed during boot, or once the system has booted by executing turnkey-init.

Exemplary import of TurnKey Core on OpenStack

There are several ways of uploading an image into an OpenStack deployment, below is one way to get you started.


# cd /tmp
# tar -zxf turnkey-core-11.3-lucid-x86-openstack.tar.gz
# ls turnkey-core-11.3-lucid-x86
    turnkey-core-11.3-lucid-x86-initrd
    turnkey-core-11.3-lucid-x86-kernel
    turnkey-core-11.3-lucid-x86.img

# IMG=turnkey-core-11.3-lucid-x86

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=aki \
    disk_format=aki \
    name="$IMG-kernel" \
    < /tmp/$IMG/$IMG-kernel

Added new image with ID: 5

# KERNEL_ID=5

# glance add -A $GLANCE_TOKEN \
    is_public=true \
    container_format=ami \
    disk_format=ami \
    kernel_id=$KERNEL_ID \
    name="$IMG" \
    < /tmp/$IMG/$IMG.img

Added new image with ID: 6

# glance -A $GLANCE_TOKEN index

ID  Name                                Disk Format  Container Format  Size
--  ----------------------------------  -----------  ----------------  ---------
6   turnkey-core-11.3-lucid-x86         ami          ami               688498688
5   turnkey-core-11.3-lucid-x86-kernel  aki          aki               4179712

# euca-describe-images

IMAGE   ami-00000006    turnkey-core-11.3-lucid-x86         available  public                  machine aki-00000005
IMAGE   aki-00000005    turnkey-core-11.3-lucid-x86-kernel  available  public                  kernel

Announcing TurnKey OpenVZ optimized builds (+ Proxmox VE channel)

Alon Swartz — Mon, 16 Jan 2012 14:30:41 +0000

OpenVZ and Proxmox VE has been a recurring topic of discussion on the forums, for which we have Jeremy to ~~blame~~ thank. He's done tons of research, testing, preaching, and then some.

What I love about Open Source is that if you have an itch, and the drive to scratch it yourself, you can.

That's exactly what Jeremy and Adrian did. They wanted OpenVZ optimized builds for their Proxmox VE deployments, so they developed a TKLPatch that would convert an ISO into an OpenVZ container. And if that wasn't enough, took the time to upload some of the builds to sourceforge so it would be easier for others to leverage their work.

Hats off to you guys, you rock!

TurnKey OpenVZ optimized builds

Based on Adrian's and Jeremy's work, we were able to add OpenVZ support to our build infrastructure in no time, and after some initial testing, triggered the whole appliance library to be built as optimized OpenVZ containers.

You can get them from the "Download -> More Builds" link on the appliance pages.

Pre-seeding / default passwords

Because OpenVZ builds are used in headless deployments (without a console), they include an inithook which preseeds default values and passwords (excluding the root password which is handled by the VZ CLI tools).

/usr/lib/inithooks/firstboot.d/29preseed

DB_PASS=turnkey
APP_PASS=turnkey
APP_EMAIL=admin@example.com
APP_DOMAIN=DEFAULT
HUB_APIKEY=SKIP
SEC_UPDATES=FORCE

Depending on your use case, you can preseed the values before the system is booted for the first time, or once the system has booted by executing turnkey-init.

It would be great if someone would add preseeding support to PVE...

TurnKey Proxmox VE channel

A while back the Proxmox folks came up with the idea of adding a TurnKey channel to PVE, to allow users to download TKL appliances in the same way their custom built appliances are downloaded.

It was a great idea, but unfortunately it never got off the ground.

As I mentioned above, the great thing about Open Source is that you can scratch your own itch, and I was curious how the channel mechanism worked - so I dived in. When I came up for air I had added minimal third party channel support and a TurnKey Linux channel (github).

What this basically means is you can now download and deploy any TurnKey appliance on your PVE server in a couple of clicks without leaving your browser.

I hope to see this integrated in the upcoming PVE 2.0 release [update: it's coming...]. If you're running PVE 1.9 then you can add the TurnKey channel as follows:

cd /usr/share/perl5/PVE
mv APLInfo.pm APLInfo.pm.bak
wget https://raw.github.com/turnkeylinux/pve-patches/master/PVE/APLInfo.pm

# update appliance list
pveam update

TurnKey 11.3 maintenance release - next stop Ubuntu 12.04!

Liraz Siri — Tue, 06 Dec 2011 07:34:59 +0000

Ho ho ho, happy holidays everyone! I know most of you are already shifting into holiday mode, so I'll keep it short and sweet.

We've just pushed out TurnKey 11.3 - the final maintenance release based on Ubuntu 10.04. The next release will be based on Ubuntu 12.04. We're already shifting into high gear for that. There will be surprises. Hopefully good ones!

Anyhow the new images we just pushed out from our CloudTask automation swarm include fixes for various bruises and scrapes, as well as the very latest security updates.

If you've already installed a previous version of TurnKey 11, you don't need to download anything because by default TurnKey is configured to automatically install all of the security updates over the network.

The maintenance release will mainly be of interest to new users and existing users doing new deployments. Especially those of you who are super impatient like mua and don't care to wait long minutes after deployment for the system to pull over a ton of security updates. This cuts down the time it takes to fully deploy Core in the cloud from 5 minutes to just 30 seconds.

New Hub feature: Cloud server monitoring

Alon Swartz — Fri, 02 Dec 2011 02:15:11 +0000

Ladies and gentle geeks, I'm proud to announce we've just pushed out 100% free basic server monitoring to all TurnKey Hub accounts. This should make it easier to keep tabs on the health and performance of your cloud servers. Existing Hub users don't need to do anything to enjoy this new feature. It just works.

A better server dashboard

As you can see in the screenshot below, the server dashboard now includes thumbnail graphs of CPU utilization, disk IO and network traffic for the last hour:

Give me more!

Alright, so instead of the last hour, you want data on how your server was doing last night? Or last week? No problem. CloudWatch samples performance at 5 minute intervals, and stores up to two weeks worth of data.

Clicking on the thumbnail graph pops up a larger interactive graph that lets you zoom in and out, sample performance metrics ondifferent timescales (e.g., hourly, daily, weekly, etc.) and move back and forward in time:

No installation, monitoring agents required

You don't need to install or configure any monitoring agents, because the Hub pulls statistics directly from Amazon's CloudWatch API. CloudWatch in turn gets its data directly from the virtualized hardware layer running underneath your server's operating system.

The Hub now supports reserved instances - pay up to 50% less

Liraz Siri — Wed, 12 Oct 2011 08:05:02 +0000

In response to user demand on the forum we've added Hub support for Amazon EC2 reserved instances.

Reserved instances are an Amazon EC2 feature that allows you to reserve server capacity up to 3 years in advance by paying a low one-time fee.

In exchange the hourly usage rate is significantly reduced and server capacity is guaranteed to be available when you need it.

How much will I save?

For servers that are deployed full-time you can save up to 50% of costs. For example, running a small server continuously:

Cost	Unreserved	Reserved (1 year)	Reserved (3 years)
Up front	$0	$227	$350
Hourly rate	$0.085	$0.03	$0.03
Monthly cost	$61	$21	$21
1 year	$734 ($61 x 12)	$486 ($227 + $21 x 12)	$375 ($375/3 + $21 x 12)
3 years	$2,203	$1,460 ($486 x 3)	$1,127 ($375 + $21 x 12 x 3)
Total savings	$0	$247/year (35%)	$1,075/3 years (50%)

Similar relative cost savings are available for all server sizes.

Does reserving capacity obligate me to run a server full time?

No. At any time you can further reduce costs by shutting down unused capacity. Servers reserved for 3 years break even at 24% usage.

Which server types can I reserve?

Only EBS-backed TurnKey servers. S3-backed TurnKey servers cannot be reserved.

How exactly does it work?

Through the Hub you reserve a server instance of a specific size in a specific region. Amazon bills you a non-refundable one-time fee. The Hub applies the discounted hourly rate immediately to any matching server. You'll see this reflected in the server dashboard.

If you don't have a running server, the discounted rate will be applied to the next server you launch in the reserved size and region.

Are the reduced usage fees attached to a specific server?

You don't have to worry about that. Reserved instances are a property of the billing system, not your cloud servers. For example, if you shut down an old server and start a new one in the same region the reserved pricing will apply to the new server automatically.

For further details see Amazon's website

Introducing CloudTask - a cloud batch execution tool

Liraz Siri — Fri, 12 Aug 2011 13:17:48 +0000

The cloud. Isn't that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you'd forgive the cynics for thinking there's no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren't practical a few years back.

Like renting an armada of servers you could never afford to buy to parallelize a computational task and get results in an hour instead of days, weeks or even months, for exactly the same cost. Now that's kind of exciting if you can pull it off.

We came across a modest version of this problem for various routine TurnKey related maintenance tasks such as rebuilding appliances. It sure was nice to be able to fire up a server on-demand, run a batch job and be able upload new images to sourceforge at 100MB/s. Usually we would leave the batch job running overnight and terminate the server the next day or so. That wasn't so bad for non-frequent tasks, but we realized we could do better. On Amazon EC2 running 10 servers for 1 hour costs the same as running 1 server for 10 hours. That's the theory anyhow. In practice launching and controlling many servers by hand can be painful.

Obviously, a bit of clever automation would be just the ticket. I hate reinventing the wheel so I first tried surveying existing solutions, but I couldn't find any that fit our needs.

So I rolled up my sleeves, and about a month or so later cloudtask was born. It's kind of neat. If you've ever had to do this sort of thing by hand or put together an ugly mess of scripts you'll probably find cloudtask an easier and more reliable primitive to build on.

Here's the documentation.

Here's a tacky video demo I whipped up (best viewed full screen):

TurnKey 11.2, free micro instances, EBS backed cloud servers

Liraz Siri — Tue, 02 Aug 2011 12:37:11 +0000

TurnKey 11.2: micro instances, EBS support, built-in TurnKey DNS, security updates

We just updated the web site and the TurnKey Hub with the new TurnKey 11.2 maintenance release, which includes:

TurnKey Hub support for micro instances, Amazon's free tier and cloud servers backed by persistent network-attached storage volumes (AKA EBS backed instances).
Built-in support for TurnKey's new dynamic DNS service.
The latest security updates.

TurnKey Micro instances: 2 cents/hour or 0 cents/hour for a year with the free tier

We've added support for micro instances (613 MB RAM), Amazon EC2's smallest cloud server type which costs just 2 cents an hour to run, which is less than $15/month if you run a server 24x7. If that isn't close enough to free for you, Amazon is giving away a year's worth of micro instance usage to new users as part of their free tier program.

This means many of you will now be able to try out TurnKey in the cloud free for a year. Yay!

We would have added support for micro instances as soon as they came out except Amazon designed them to work differently from other instance types we already supported. In particular, we had to add support for EBS backed instances...

EBS backed instances: cloud servers that can be turned off any time

Up until now the TurnKey Hub only supported S3 backed instances. These are non-persistent cloud servers with temporary storage that is lost once you destroy the server. This means you can't just turn off an S3 backed instance to save usage fees when you are not using it, though you could work around this limitation by using TKLBAM to backup a cloud server before destroying it and later restoring its state into a new cloud server.

With the support we've added for EBS backed instances, this limitation has been removed. EBS is what Amazon calls its on-demand Network Attached Storage service. The catch is that the Hub has to pre-allocate a fixed size EBS volume for your cloud server to boot from. Unless you are in the free usage tier you'll have to pay an additional $0.10/GB per month in EBS storage fees for the convenience (e.g., 50GB EBS volume == extra $5/monthly). The ability to turn off servers when not in use may make up for this extra cost though.

Speaking of costs, the pricing structure on the TurnKey side is a bit different for EBS backed instances as Amazon doesn't allow vendors to add a 10% markup to hourly usage fees like we've been doing with S3 backed instances. So instead, we're probably going to be experimenting with a global fixed monthly fee for this feature. After the trial period ends (in a month or so). Currently there is no extra charge.

Note that this future extra monthly charge will not apply to micro instances.

A word of warning: EBS is not a backup replacement and EBS-backed instances still need to be backed up by TKLBAM. EBS volumes just provides data persistence. It's a network hard drive that lives in a specific Amazon datacenter. It is not a replacement for backups. For example, if the data on your EBS volume gets accidentally deleted or corrupted, without a backup system to restore from you will be out of luck. TKLBAM on the other hand provides true incremental backups, so good data can't be accidentally overwritten by bad. Also, TKLBAM uses S3, which is designed by default to provide 11 nines (99.999999999%) of storage reliability, much higher than EBS.

Security updates

As most of you know security updates are already installed automatically on first boot and nightly (by default). If you're using an older version of TurnKey this means you don't need to do anything to get the latest security fixes. But for new deployments pulling a large number of security updates over the network can take considerable time, so occasional maintenance releases that already include them are a good idea.

We're in the process of upgrading our development process so this sort of update will be easier to do in the future and can be done as frequently as necessary.

Important security notice: Your TurnKey system may no longer be receiving automatic security updates

Liraz Siri — Thu, 16 Jun 2011 17:37:17 +0000

I have some bad news and some good news. The bad news is that if your TurnKey installation is older than 2 weeks you may no longer be receiving security updates.

The good news is that you are reading this and there is a very easy fix. Either reboot your system, or log in and restart the cron service:

/etc/init.d/cron start

Until you start recron, security updates and other scheduler related services (e.g., daily backups) will not work.

What happened?

Ubuntu screwed up a recent security update. There was a nasty bug. When installed, the update breaks cron, the scheduling daemon TurnKey uses to auto-install security updates. Not good.

According to a routine report generated from the access logs on our security repository, there are currently thousands of TurnKey installations affected by this issue. Those systems are not getting automatic security updates. There's no immediate risk, but that could quickly change if a remote vulnerability is discovered in the time it takes whomever is responsible for the server to figure this out.

Make sure we can always reach you

There's moral in all of this: make sure we can always reach you somehow.

Sure, usually we don't need to get your attention regarding security issues because TurnKey is configured to auto-install updates, but as this incident shows, we can't rely on that always working.

This time we can't fix the issue on our side, since it effects the very auto-update mechanism that's usually used to fix security issues.

The best we can do is try to reach out to users and inform them that there is an issue that they need to manually intervene to resolve. Hopefully we can get through to anyone subscribed to this blog or the News and Security announcements newsletter, or that has a Hub account.

In any case, we'll soon find out from the logs on the security repository just how many of our users we can or can't reach.

Announcing TurnKey Hub v1.0 - now officially out of private beta

Alon Swartz — Wed, 30 Mar 2011 12:42:48 +0000

When we first announced the TurnKey Hub private beta about 9 months ago, we had limited capacity (invitation only) and a modest feature set. Since then we tested, bugfixed, removed bottlenecks and added features, constantly improving the Hub with the help and feedback from our excellent beta users. Thank you so much!

With the release of TurnKey 11 which was tightly integrated with TKLBAM and the Hub, the amount of Hub invitation requests exploded. We were prepared for this and managed to scale the Hub smoothly without any serious issues.

With several months of testing, feedback and bugfixes under our belt we are now confident enough to officially announce, a bit earlier than planned, that the Hub is out of private beta. As of today, the Hub is open to all, and new users will no longer be required to request an invitation.

Existing users can rest easy though. We will continue to carefully monitor the Hub's performance. There should be no interruptions to the service. Worse case scenario, if we start hitting unforseen capacity issues we will temprarily reintroduce the limit on new signups.

Review of notable changes since the initial release

TurnKey Backup and Migration

A few months into the private beta we announced support for TurnKey Backup and Migration (AKA TKLBAM), which amongst other uses makes previously difficult tasks such as testing your backups much easier.
In response to demand, we've added support for configurable backup retention. Users can specify how many full backups they would like to keep for any given server backup (set to unlimited by default).

TurnKey Cloud Servers

Support for TurnKey Linux 11 images (legacy images still available to ease migration).
Basic pre-launch configuration: No more having to fiddle with the default passwords after an instance launches. The Hub supports pre-seeding appliance configuration before launch. This makes up for not having console access that would usually be required for first boot configuration.
TKLBAM pre-initialization: No more having to cut and paste your Hub APIKEY to initialize TKLBAM. The Hub pre-initializes TKLBAM automatically when the instance is first launched.
Upgradeable Kernels: We've figured out how to make it easy to update the kernel via pv-grub.
Preset launch region automatically chosen by geo-location of user.

General stuff

Performance optimizations, improved stability and error handling.
Refined the look and feel with an update to the theme.
We now try harder to explain how the Hub works and what it's good for before and after you sign up. For example we've added nice visual tours of the Backup and migration and Cloud servers features.
We've added a pricing page answering frequently asked questions. Yes, the Hub is still free. You pay Amazon directly for the cloud resources you use.
Improved start page to get you going once you sign up. Once you setup your account, this transforms into a dashboard that provides a high level overview and quick access links.
New and improved notifications (growl style).
Removed invitation requirement and added support for OpenID signup and authentication.
Added functionality to change account email.
Full internationalization support (UTF-8).
APT archive geo-location API service for choosing the closest package archive.
Link to Privacy policy.

As usual, feedback is appreciated. If you don't have a TurnKey Hub account yet, go get one now or try out the demo. If you already have a TurnKey Hub account, go check out the new stuff.

The TurnKey Hub lives at: https://hub.turnkeylinux.org

TurnKey Linux 11 released (part one)

Liraz Siri — Thu, 27 Jan 2011 11:18:10 +0000

Ladies and gentlemen, part 1 of the TurnKey Linux 11 release is now officially out, including 45 new images based on Ubuntu 10.04.1. We pushed out the 11.0 release candidates 3 months ago, and with the help of the community have tested the images and resolved the few remaining issues.

Part 1 mostly refreshes the existing roster of appliances in the library. In the upcoming part 2 we'll release the new appliances the community has been helping us develop over the last year. This will roughly double the size of the library.

A handful of new appliances have also been squeezed in:

Joomla 1.6

Magento

StatusNet

PrestaShop

vTiger CRM

This was mostly a side effect of the original (misguided) plan to do one big massive release with over 80 appliances.

What's changed since the release candidates

VM optimized builds: are now available, in two exciting flavors...
1. Default: this is primary downloadable VM build. Works best with VirtualBox, low-end VMWare products (Player, Workstation, Server).
2. OVF: OVF is the new VM standard supported by VirtualBox and high-end VMWare products (e.g., ESX, vSphere).
We're also working on providing images optimized for other popular virtualization platforms such as Xen, UEC / Eucalyptus and OpenVZ.
Default passwords: You no longer have to keep track of any default passwords or change them after installation. TurnKey now helps you configure them on first boot, via the console.

For headless deployments without a console, it's possible to pre-seed answers to first boot configuration questions.
/etc under git: Automatic revision control of /etc using etckeeper, as suggested by Jeremiah Snapp. If a configuration change you made breaks something, just roll it back!
LVM snapshots: Fixed LVM snapshots by adding 10% unallocated disk space to default LVM configuration. This will make it easier to add support for atomic filesystem backups in upcoming versions of TKLBAM.
Amazon EC2 / TurnKey Hub related changes:
- Hub does TKL 11: TurnKey Hub now deploys TurnKey Linux 11 images by default, though support for older legacy images is still available to ease migration.
- Basic pre-launch configuration: No more having to fiddle with the default passwords after an instance launches. TurnKey Hub now supports pre-seeding appliance configuration before launch. This makes up for not having console access that would usually be required for first boot configuration.
- TKLBAM pre-initialization: No more having to cut and paste your Hub APIKEY to initialize TKLBAM. The TurnKey Hub pre-initializes TKLBAM automatically when the instance is first launched.
- Upgradeable Kernels: We've figured out how to make it easy update the kernel.
  
  Kernel upgrades were previously not supported on Amazon EC2, because each Amazon Machine Image (AMI) had to be associated with a specific Amazon Kernel Image (AKI). Now instead of associating the image to a specific kernel, we associate it with a special EC2 compatible bootloader (pv-grub), which can bootstrap whatever kernel is configured from within the system (e.g., security fix).
Updated website documentation
- All 45 appliance pages (e.g., TurnKey Core) now reflect the latest image versions.
- Documented appliance build types.
- Updated virtualbox installation tutorial, virtualization notes, automatic security updates pages.
Bug fixes: An assortment of fixes for issues reported by the community.

See the previous announcement of the release candidates for changes since the last maintenance release (2009.10-2, based on Ubuntu 8.04 LTS).

Many, many thanks to...

Everyone who helped test the release candidates and provided ideas and feedback.
The many rivers of upstream: Ubuntu, Debian and all of the wonderful open source communities who give love and write code for the software that goes into TurnKey.
JedMeister, Adrian Moya, Basil Kurian, and Rik Goldman - pillars of the TurnKey community who have inspired us with their dedication and generosity.
TurnKey enthusiasts everywhere. Without you, TurnKey's audience, there really wouldn't be a point!

What's next

Images optimized for Xen, Eucalyptus / UEC and OpenVZ
Part 2: double the size of the library
64-bit support
Debian squeeze based beta builds (we've decided to skip Lenny)
PostgreSQL support for TKLBAM
TurnKey Hub stuff
- Support for Amazon EBS backed root devices
- Support for micro instances. Amazon's 1-year free tier will now allow free evaluation of TurnKey on Amazon EC2.
- Support for larger instances (64-bit support required).
- Support for more hosting providers.