One

Eucalyptus has a long history of listening to our community. You asked for Eucalyptus to have a life of its own, and Eucalyptus Systems was born. You asked for a different license (GPL) and you got it. You asked for easier ways to install Eucalyptus and so we packaged Eucalyptus for the main linux distributions, we delivered UEC and we created FastStart.

We are still listening to our community. Currently we are working on the Eucalyptus 3 feature set and development process. Check our roadmap for more information
on the feature set, and stay tuned for more information about this
important release.

Our mantra is and always will be:  "Listen to our community and deliver quality software".

We always strive to get better at it. We are now in the process of redesigning the way we communicate with you. Currently we differentiate our users between customers (paying users) and community (intended here as the non-paying users). We have two web sites (www and open), two different sets of communication -- a support team,  an issue tracker, phone line 24/7 and direct emails for customers and a forum, IRC, launchpad and mailing list for community.

This setup is common enough, yet in our case it seems a bit arbitrary. Our engineers, our sales people, and our support team help community members on IRC and in the forum. Conversely we are all dedicated to help our customers. So we are in the process of removing this artificial division. Our support team is working to integrate the forum, mailing lists, and issue tracker. Our sales and marketing teams are working closely to understand the needs of our users and to provide them the information they want. Our professional services team is producing online training material. Our web team is working to tie all of the above together in a seamless and user-friendly way. And our engineering team is readying the best release of Eucalyptus yet.

Eucalyptus 3 will bring a lot of changes for our community: reconciling the various source branches into one, bringing a new plug-in architecture, and opening up the development process. Before Eucalyptus 3 is released, our two web sites will merge. We'll group information together following the cloud IT roles. Our webinars, forum posts, technical information, and documentation will be focused on delivering the best possible service to the specific IT role.

The new web site redesign, will put our community in the driver's seat. We will present updates and new design soon, and your feedback is necessary to carry on this ambitious plan.

One Eucalyptus Community.

I miss my '-g'

One thing I can say for sure: there is no time to get bored here at Eucalyptus. If you like big challenges and you have it to take them on, consider joining our team: I'm extremely pleased to say that Greg just did so.

If you are at Cloud Expo in Santa Clara or in the Bay area, come and meet Greg (or Rich or Marten or me for that matter :) at EucaSocial, or visit us at our booth (#615).

I have been knowing Greg by fame for a while, but in person for only few months, and I can say that the energy, and passion he brings to Eucalyptus are intoxicating. You may have already noticed the impulse on our community activities (from our #eucalyptus IRC channel, our mailing lists, our projects and our planet) and this is just the start!

The only downside is that I don't have anymore the exclusive '-g' at the end of emails. Welcome Greg:   I'm  thrilled to be able to work with and learn from you.

Half an hour for a Eucalyptus Dream

I hope you are enjoying the recently released Ubuntu 11.10 (Oneiric Ocelot). I recently did take it for a spin, and this blog is about my experience. Eucalyptus has been part of Ubuntu since Jaunty, and I wanted to see how well the latest available version of Eucalyptus integrates with Oneirc

And I am pleased to say that in less than 1/2 hour, you can have your Eucalyptus cloud running in Oneric! You don't believe me? Let's do it:

10:17: popped the CD for the server install into two desktops machines we had laying around:

I just did add the openssh server to the basic install;

10:31: install is done, machines rebooted. At this point I lost the monitor I was borrowing, so I switched to ssh from the comfort of my office in front of a cup of coffee. I did do a quick update and upgrade to ensure all the security patches were applied;


10:34: Time to get to the Eucalyptus install. There are nice instruction on help.ubuntu.com which I partly followed: my engineering background prevents me from following nicely written instructions ... Soo, I did
     apt-get install eucalyptus-cloud eucalyptus-cc eucalyptus-sc eucalyptus-walrus
On the second desktop I did install the node controller
     apt-get install eucalyptus-nc


10:43: got to the familiar webUI

and downloaded the admin credentials. The configure tab was missing some components. I assume the network I was using confused a bit the nice autoregistration mechanism of the UEC, so I switched to manual mode. I first de-register all components (I used the default cluster1 for the cluster name)
     euca_conf --deregister-cluster cluster1
     euca_conf --deregister-sc cluster1
and re-register with a cluster name I'm more familiar with
     euca_conf --register-cluster pippo 192.168.7.246
     euca_conf --register-sc pippo 192.168.7.246

10:46: I did register the node controller
     euca_conf --register-nodes 192.168.4.7
(yep, our network is not a /24) waited few seconds to allow the node to report and got

Phew, just under half an hour, as I promised you. In a future blog I will talk about the images I uploaded on the cloud and tested it: you can check the work we are doing on our projects.eucalyptus.com.

Planet Eucalyptus

You already know our planet, since quite a few visit it regularly and requested to add feeds (let us know if you want your feed on our planet). When you visit it, you will find a somewhat ordinary planet site, perhaps too plain if you will, but what I want to mention in this blog, is how we are handling it and how it is running 'in the cloud'.

In the previous drinking champagne blog, I mentioned we have quite a few application and services in our internal production cloud (of course powered by Eucalyptus) and planet is one of them. What we did with planet, is to make it very simple to deploy and customize it: the original work was done by Mark Atwood. In order to take advantage of the cloud we heavily relied on the meta-data service.

The meta-data service allows for instances started in the cloud (private and public clouds which follows the AWS API) to learn data pertinent to the instance itself (hence it is called meta-data). Public IP, ssh keys, storage information, instance IDs (EMIs, kernel and ramdisk) are examples of what can be retrieved. All these data are accessible at http://169.254.169.254/latest/meta-data, and are easily accessible from within the instance with a browser or most likely with wget or curl.

Amongst these data, the user is allowed to pass few kB of data to the instance. To do so one could use the euca2ools and in particular the euca-run-instances with the -f or -d option. The instance can then access this data at http://169.254.169.254/latest/user-data: cloud-init uses it to run scripts at boot time or otherwise customize the instance and the official Amazon Linux AMI uses a port of cloud-init. Cloud-init is not yet available to all distros and earlier version of Eucalyptus suffered a bug which prevented cloud-init to work properly (instance would get delayed at boot time), so we decided to use a much simpler rc.local script to allow for a subset of its functionality. You can find information on the rc.local script we use and of other information about images on projects.eucalyptus.com. With these starter images in our production cloud, we set to host planet in an instance.

Our first attempt to run a service in the cloud, emulated the boot from EBS capability (see our issue tracker in the cloud), but this time we changed completely tack. We pushed all the planet's configuration into a Walrus bucket, then we created a script to be used when starting an instance.

You can inspect the script on projects.eucalyptus.com under the Cloud Application Architect area. The script gets all the css, ini and others needed file from a walrus bucket, installs nginx and other needed packages, and set up a cron-job to re-read the configuration files at set intervals, thus allowing for the dynamic configuration of planet. When we changed the logo, it was a matter of uploading the new css and png files, and presto! Planet got a new skin.

While this setup seems complicated at first glance, it is fairly easy: consider that it's all done with a few line of a shell script. It also allows for a easy failure recovery since restarting planet is a matter of 2 euca2ools command. If the instance were to fail we can issue:

euca-run-instances -f planet.sh -k my-ssh-key emi-F3DF1488
euca-associate-address -i i-xxxxx 173.205.188.124

and since there is no persistent data on the instance, that's all we need to do. And to apply a security update, we simply spin up a new instance with the same script (the script we use upgrades to all latest packages at start up), disassociate the public IP from the old instance and associate it to the new instance. And of course we can terminate the old planet once the elastic IP has been moved.

To customize the planet we upload the new version of the specific file to the Walrus bucket, and to do so we use a version of s3curl modified to allow for different endpoint then S3. For example to add a new feed we first get the current planet.ini:

wget http://173.205.188.8:8773/services/Walrus/planet/planet.ini

modify it, to add the new feed, and we upload it back into the bucket

s3curl --id graziano --acl public-read --put planet.ini -- http://173.205.188.8:8773/services/Walrus/planet/planet.ini

and wait for the cron-job to execute. Easy, isn't it?

We have few more scripts which we use for our production services on projects.eucalyptus.com and if you have a similar script you want to share, let us know and we'll add it, or do a git-hub merge request. The recipes are ready to go, just add Eucalyptus to it.

FLOSS Podcast

Today, Aaron Newcomb and Simon Phipps of FLOSS weekly, graciously hosted me on their podcast. They asked me about Eucalyptus, covering the basics (what is cloud computing, what is Eucalyptus) and then quickly going deeper into the internals of Eucalyptus, our development processes, and our community engagement. Very interesting ground was covered, ranging from cathedral vs. bazaar to the age old question of vi vs. emacs.

Check out the podcast. If you have questions about Eucalyptus that were not covered, you can always ask us directly.

Drinking Champagne

Some call it eat your own dog food but we prefer the alternative (check the wikipedia article if you don't know), and so we want to give you the latest experience in running our services into Eucalyptus, or, in other terms, how we have been drinking our own Champagne.

Harvesting The Grapes

As with many enterprise companies, it is the responsibility of internal IT to deploy and maintain software tools, products and services that are a vital part of the company's day-to-day operations. The list of products and services is ever-growing and so the following is hopelessly incomplete, yet it can give you an idea of what we use:

Issue Tracker

this was one of our first service to deploy. We even have an article on how it came to be; 

Web Sites

we have 2 main web sites and internally we call them www and open and hopefully they are covering all your needs for private clouds; 

Downloads

as an open source company, the way to downloads Eucalyptus is very important to us, and we are always finding ways to improve it. We are in the process of spinning up a service on its own to handle this, so stay tune for the launch of downloads.eucalyptus.com; 

Starter Images

all of our 25,000 and counting Eucalyptus clouds need images to be turned into instances to be useful. We provide starter images which we are in the process of refreshing and making even more useful: we will be launching the EMIs service very soon with newer images, mechanism to customize them at run times and more documentation; 

Euca Projects

we recently launched Euca Projects: it is a young sites but it already has very interesting projects and documentation. We for sure used it when we needed to change the ECC logo; 

LDAP directory

we internally use an LDAP directory as the keystone for our identity management system. We always look for ways to improve our users experience and single-sign-on ranks pretty high in our list: we have plan to make LDAP the basis for our single-sign-on to all our services;

Internal wiki and more

we do have internal wiki, internal repositories and other services which are rely upon by the different departments. 

The First Glass 

In summary, we have important services with very different SLA associated with them (critical to nice to have it running), we have about 1 TB of very important data associate with it (with various level of backups and data retention policies), a handful of hostnames and IPs associate with the instances and different network and security requirements. We happen to hear about this new wonderful technology called private cloud early on, so we have all (or almost all) of the above running on our Eucalyptus-powered internal cloud, which unimaginatively we called it production cloud. We have various team working on the different services (the web-team, the support-team, engineering, sales, professional services ets ...) and Eucalyptus accommodate all of them nicely. To have fine control some team employs their own Cloud Application Architect (see cloud IT roles), while others rely on our Tech-team for the job, and the IT-team is our Cloud Administrator. The services are compartmentalized into their own security group and cloud account, they use their own Walrus buckets and EBS volumes, thus separating nicely the security concerns and the possibility of a rogue application bringing down all our infrastructure.

Refining The Process

Eucalyptus follow AWS semantics and offers the instances the Metadata Service which allow the instance to capture information about its creation, ownership and more, Amongst the data available to running instance, there is the user data: this data is passed at run-time (euca-run-instances time to be more precise) and it is up to the instance to use it. Our new instances (the EMIs projects mentioned above) will have a very simple mechanism which allows for them to execute any script passed as user-data. We are converting all of our services to be restarted using script to be passed as user-data: in a nutshell the script will download the specific configuration and data from Walrus buckets, and/or will be mounting their own EBS volume when needed, install and configured the wanted service. You can already see some of them on Euca Projects: the idea is very simple and it makes for a much simplified handling of our services. We already have planet, redmine (projects), databases and we will be adding the scripts for our web server soon.

Rebuilding the Engine while Racing

All of the above helped us tremendously early this week when a disk started to act up. One of our main server (the front-end for production cloud) had one disk starting to go bad, causing all sorts of issues. While Eucalyptus really helps in the IT menial tasks, it is clearly not a substitute for proper planning. Our server was still running on a single big disk (the project to update it keeps getting a lower priority over more fun activities) so this issue had the potential to have a huge impact and long downtime for a number of our services. Our IT-team got in overdrive and in the span of 1 day, planned the move: instances were moved off EBS volumes, Walrus backup were temporarily suspended, and the Eucalyptus Cloud Controller was shut down, while we were migrating the data to a secure location. Since we didn't have our SAN ready for action, we decided to go for a RAID5 temporary solution to give us more time to plan for the next big expansion. Externally, there was a less than a minute downtime for services using EBS (we needed to migrate the data to ephemeral and switch over). The instances with our precious services kept running while we serviced the front end machine and all was well when we restarted the Cloud Controller. Eucalyptus rediscovered the running instances, took ownership of them, and few minutes after, all the state was properly restored. We had another brief downtime when we moved the data back to EBS (for the services using them).

The Day After

The only casualty of this process was an instance that got accidentally terminated. It was a Cloud Operator error and we have taken steps to avoid it in the future (no more watching action movies when doing major upgrades!). In case you missed the announcement, Eucalyptus 3 is coming fast, and it will make it even easier to handle the situation like the one we encountered: HA is part of Eucalyptus 3, and moving a Eucalyptus component will be much easier to do while keeping the cloud running. Amongst the lessons learned, I would say remember to listen to the Cloud Architect when planning your deployment: having some services depending on a single disk is always a bad idea, not matter how many backups you have in your vault. Share with us your Eucalyptus or cloud experience

Eucalyptus Projects

We are proud to announce the availability of projects.eucalyptus.com. You have been asking how to participate more actively and this is our response: to create a single place where you can organize and coordinate the various efforts centered around Eucalyptus. The site is powered by redmine and the idea here is to host projects to allow interested parties to communicate and coordinate easily. Our community is very diverse, and so are the contributions we received. We decided then to create few top level projects around the roles involved with private clouds (for more info read more on Cloud IT roles): these top-level projects should allow to navigate quickly toward the wanted area of interest. So, based on the kind of contributions we have seen, these are 5 top level projects:

• Projects for Cloud Administrator customization of Eucalyptus, or documentation on particular Eucalyptus installations, scripts and code to help maintain and install;
• Projects for Cloud Application Architect projects related to images and instances running in Eucalyptus, for example howtos for creating images, or recipes to customized instances;
• Projects for Cloud Architect best practices documents, deployments guides;
• Projects for Cloud Developer projects for extending or modifying Eucalyptus, or for tools working with Eucalyptus;
• Projects for Teaching and Learning Eucalyptus documentation, course material, small class projects to help to teach cloud computing. of course these are just suggestions.

There is one project not relates to IT roles (as defined above), the Teaching and Learning Eucalyptus: we felt the need to give it its own area to it because of the prominence of Eucalyptus into Universities and schools as a teaching tool, and we hope to collect material to ease the creation of courses on private clouds and Eucalyptus in particular.

Figure 1 On the left you can see the my account page on open.eucalyptus.com, while on the right you can see the registration page for project.eucalyptus.com. Make sure you use the same ID and email. We will be moving to a single sign-on process and this step will ensure a smooth transition. 


Projects has been created for you, so if cannot find a project of your liking, contact us, propose a new project and we'll add it! To be started with a project is very easy:

• create an account on open.eucalyptus.com;
• create an account on projects with the same ID and email or let us know you want to be on projects and we'll do it for you (check Figure 1);
• find the project you are interested in, start filing bugs and contacting the people already working on it;
• if you have an interesting idea, contact us to create a new project for you. 

The site is very young, but we already have interesting projects available: most of these early projects are used internally by us (for example the projects site is hosted on a Eucalyptus cloud and we have scripts to regenerate it on demand, as well as planet Eucalyptus).

Eucalyptus 3 is here! Eucalyptus 3 is here! Eucalyptus 3 brings new exciting features (check our roadmap) which you have requested and we are sure quite a few projects will be based on these new features (HA, boot from EBS, EIAM and more). It will also bring a new development model: this is the first step as described in the blog. So get ready with your cool brand new Eucalyptus project.