Playing with Silvereye

Silvereye is a bird native to Australia (I have to admit I have never seen one live) but it is also a project that allows a very quick and easy installation of Eucalyptus. The project is fairly young, but it is already very usable: you can follow its development on github. While Faststart has a very small footprint (it fits on a 1GB thumb drive) and requires a CentOS base installation, Silvereye creates a CD iso image which can be used to install bare machines.

Tonight  I took the latest Silvereye for a spin, and here are my notes. Let me start with a single word: wow! I forked the latest version from github, moved the silvereye.sh script to a CentOS Virtual Machine (VM) I had on my laptop, and ran the script while I went to grab some coffee. When I came back, I had an iso file freshly baked and ready to go.

I booted a VM using the Silvereye CD (I will write in another blog my libvirt configuration for this experiment) and connected to it with a VNC viewer to be greeted with the boot menu.

Boot menu for Eucalyptus Front-End or Eucalyptus Node Controller.

The options are to install a Eucalyptus Front-End, a Eucalyptus Node Controller, a minimal CentOS without Eucalyptus, to boot into a rescue image or to boot from local hard disk. I selected the Front-End installation.

The process continued with the usual CentOS installation,

Default CentOS 6 installation.

and in few minutes the (virtual) machine was ready to be restarted into its new OS.

CentOS 6 is now installed and ready to go.

At the first root login, Silvereye kicked in and configured Eucalyptus. I did follow the few easy questions (it also allows you to reconfigure network, DNS and hostname if needed) and I ended up with a working Eucalyptus Front-End in no time!

Silvereye first question.

You will need the physical network layout of your machines: IP addresses used by the machines and IP addresses available for the instances,  network netmask, gateways, DNS server. Also make sure that when you configure the private network for the instances, there is no overlap with your real network.

Network configuration for Eucalyptus, and database initialization.

As a final touch, Silvereye created an Eucalyptus Machine Image (EMI) for me!  Of course I could have downloaded the one we provides, created my own one, or found some Amazon's compatible images, but I have to say that having Silvereye doing all the dirty work was simply gratifying.

Silvereye can create an EMI for you.

The end result was a kernel, ramdisk and image ready to use.

List of available images.

And here is where I stopped: I didn't have an NC ready to go, and my latest book was waiting for me. It took me about 20 minutes from the time I started the VM the first time with the Silvereye CD, to have a list of EMIs.

 

 

Euca @ UDS

In Greg's Where is Waldo game, one big event was missing: Eucalyptus is also at UDS. Harold, Brian and me are already in Oakland ready to rock and roll. We have exciting news for our session: you can check out Brian's blog for a sneak peak of  Eucalyptus 3.1 Alpha1. Your favorite on-premise IaaS now is available with HA, boot from EBS, and IAM (check the roadmap for more details).

Find us at the Eucalyptus session, or around the conference floor  to say hello or to get a t-shirt (limited supply). 

The One-Site Project

It's my great pleasure to announce our new web-site. The one-site project aimed to unified our two web sites. As previously mentioned, we started this project sometime ago to address the needs of our community. We believe that the diverse interests of our community are better represented by the cloud IT roles. And the new website has been designed to help our users to navigate following the interests, languages, and preferred communication channel of each role.

Navigation by Cloud IT Roles

First let me mentioned how the generic navigation of the site has been greatly improved. The main navigation is Learn, Eucalyptus Cloud, Participate, Services and Partners, and, if you already know where you want to go, in the footer of each page you will find direct links to subsections. The progression is obvious, starting from generic documents on cloud computing, on premise cloud, IT roles going to the specific of Eucalyptus Cloud, finally exposing how to reach our community, Services provided and last but not least diving into the rich and varied partner ecosystem.

Information customized on a per-role basis

Let's go back to the navigation by roles. In the Learn section there is a selection that allows each roles to explore what cloud means for them. Each role page has specific links and information related to the respective competences. Let me give you examples for some of the roles represented there:

• Cloud Administrators will find install and maintenance guides. In case they get into issues they can search our knowledge base for the specific error code, or engage other administrators to discuss about the different installation options or what is the best hypervisor for their situation; 
•  Developers can quickly reach the latest source code, looks at currently worked on issues, and start talking about the latest patch they have been working on. Detailed informations on Eucalyptus internal structure and API supported are also the bread and butter for Developers;
• Application Architects will find everything about our starter images, from what's inside and why, to how to recreate and modify them. They would want to use and modify the instances recipes and share their latest scripts to recreate their production environment. 

You will find the roles highlighted throughout the web site, so, for example, under GetEucalyptus you will find options for Managers (get the free trial), Architects (FastStart for a quick proof of concept), Administrators (get the latest release), Application Architect (get the starter images).

Find how to participate independently from your role

Participate and Resource Library are the two notable exceptions to the roles principle. While you will still find links to the cloud roles pages, these pages addresses our community as a single entity. Participate gathers selected blogs posts for all our users, as well as giving pointers to where to find online community. Resource Library allows you to find all the published information about Eucalyptus and cloud computing organized by type and date.

With the new web site, you will also find new redesigned forum. You will find the motivation behind it in Darren's blog, and more about its functioning in the Engage blog. Our new site is now available and ready to Engage you: let us know what you like and how we can improve it for you.

Engage!

We started the journey of One sometime ago. The original idea took shape and evolved into the one-site project and now into Engage. The sentiments and the ideas behind Engage have been articulated very clearly in Darren's blog. Since the time of Eucalyptus 1.5, we established forum within our web site, to allow our community to come together and discuss Eucalyptus and Cloud Computing. It has been my pleasure to help the best I could, participate in discussions, and meet new friends.

It didn't take too long before we realized how inadequate the forum and ultimate we were to help all our users. Limited search capabilities, repeating threads, difficult forum management, obscured error reporting, and a new field called on-premise cloud, got us into long per-post sessions. And yet all we could do was the idiomatic drop in the bucket. The whole process was wrong: we needed to change tack.

That's when Darren got into the picture. His experience with multiple support organizations, brought the needed fresh thinking. His efforts are now live with Engage, and these are the key points:

Intuitive user experience: type your question

• very intuitive user experience: just type in your question;
• the same intuitive experience for customers and non-customer: we are doing the necessary work to ensure that customers requests meets their SLA;
• our customers already love the competence and promptness of our support team. Our community shared the same love, perhaps without knowing: the support team is on IRC, on the forum, on emails, and Engage allows to more effectively track all open questions (wherever they are);
• modern forum tools: we can now tag an answer as 'best answer', turn it into an article, keep track of the unanswered active threads;
• single Knowledge Base! Perhaps the most important point: all the articles, questions, and soon issues/bugs form a common knowledge base that a simple and powerful search mines effectively. Every time you type in a question, you are searching the knowledge base!

Likely answers or topic offered while you type limits duplicate threads

Engage has two different basic tool: the Q&A and the Article. A Q&A can be seen as a forum post, everybody can create one and everyone can answer it, can be open ended or can have a 'best answer' to it. Customers can also flag it for faster escalation. Articles are the bulk of the knowledge base: we turn the most common, or important, Q&A into articles, where we can give more background information, have a common format, expose workaround or solutions.

Articles provides the bulk of the knowledge base

When posting a new question you will be asked for Topic, Name, Email Address, Subject, and Body. We divided the Topic based on the Cloud IT roles, following the navigation principle of our site. For a short amount of time, you will have to enter Name, and Email Address each time. This issue will be resolved once we deploy the single sign-on mechanism (based on OpenID) we are working on.

Topics are divided based on the Cloud IT roles

This new tool come with  a cost: we will not be able to migrate our old forum posts to it. The tools are too different and we simply don't have the man-power to make this migration manually. We already pre-populated the articles with the most note-worthy topic coming from the old forum, but we did probably miss some. Also your username has not been migrated from the old web site: once we implements the single sign-on, you will be able to utilize your preferred OpenID provider to login.

Please help us to migrate your unanswered or favorite post from the old forum: just go to engage and post it there. And if you encounter any issue, don't hesitate to contact us. We do believe you will love the simplicity and completeness of what you will find at Engage, and we will be happy to hear any comment you have to improve it.

4 of 20

The International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC) will commemorate its first 20 years with a special proceeding containing the best 20 papers. Rich Wolski, and Steve Fitzgerald, respectively founder and CTO, and VP of Technical Services at Eucalyptus Systems, will each have 2 papers published in that proceeding. Rich will have Forecasting Network Performance To Support Dynamic Scheduling Using The Network Weather Service (NWS) and Scheduling from the Perspective of the Application (AppLeS) while Steve will have Grid Information Services for Distributed Resource Sharing and Application experiences with the Globus toolkit. That is 20% of the papers!

I have been fortunate enough to have been working with them since the mid '90, back when a cloud on a whiteboard meant Internet, and Grid Computing was the pinnacle of Distribute Computing research. In addition to be excellent scientists, Rich and Steve have also been role models, teaching the new generations for more than 15 years. It's hardly a coincidence then, that Rich and Steve are now pushing the boundaries of  knowledge in Cloud Computing with Eucalyptus.

It's hard to overestimate the influence that Rich and Steve have within our organization. For example, in the latest Eucalyptus 3, I can find Rich's fingerprints in the overall architecture, in the attentions to the internal protocols, and Steve's touch in the identity integration (LDAP) and management (EIAM).  They are mentors and role model to all eucalyptoids, for their determination, insights, and skillful leadership each with his own personable yet efficient style.

Edit: check out the full list.

PyCon 2012

We (cpt_yesterday, gholms and obino) left Silicon Beach (aka Santa Barbara home of Eucalyptus) on Thursday and drove up to Santa Clara for my first PyCon.

Driving from Silicon Beach to Silicon Valley

I was a bit leery about dinner (just few week ago cpt_yesterday and I got food poisoned in Sunnyvale), but all went well and it was great to see Mitch.

Setting up the booth

We set up the booth on the morning, just before the keynotes. The booth has been a good meeting point throughout the conference to talk about Eucalyptus, get a bit of rest, or just grab one of our t-shirts. It's always rewarding to talk about how Eucalyptus is used in production or to receive thumbs up from people walking by.

Mitch and cpt_yesterday

This was my first PyCon, and been a Python n00b myself (my first python program is few weeks old), it was a bit intimidating to be around such a crowd. Everyone was very helpful and engaging and willing to tolerate newbie: a very nice experience.

  

gholms likes PyCon

We got a bit distracted from the conference by PG&E: they told us on Friday night that an urgent safety fix was needed and Saturday they will pull the power all around our HQ. Like all good DevOps we spent quite a bit of time to prepare for the outage, coordinating with the team back at HQ. Our core services runs on our public Eucalyptus cloud at CoreSite (read more about the ECC for more info on our setup) but we have an internal cloud we use at HQ, and we wanted to be sure all these internal services were shut down nicely.

The stars of the conference

And of course I needed a picture of the stars of PyCon 2012.

First steps with Eutester

In one of my previous post I mention our mantra: "Listen to our community and deliver quality software". To deliver quality software, it is necessary to have the QA process as a first class citizen. Our QA team created quite a spectacular infrastructure to test all sort of combinations and configurations automatically (distro, architectures, versions, hypervisors, networking, images etc ...) to guarantee our users, regression-free releases. They also managed to have fun in the process (check out Pigeons on a Euca).

Eutester is the latest brainchild of our QA team, and this blog is about my experience writing a test using it. Eutester is a framework to create automatic tests against a Eucalyptus installations (or any cloud following the AWS API for that matter).

 These are the ingredients I needed to bake my first test:

• a really annoying bug
• a Eucalyptus cloud to test it against
• the Eutester framework
• basic python knowledge helps although it is not necessary

Let's start with the main ingredient: a really annoying bug. I picked lp:737335, annoying and small enough for me to tackle. The issue reported is that the launch permissions as set by euca-modify-image-attribute are not respected in Eucalyptus 2.0. The bug happened also to have a proposed fix.

 Next: reproduce the issue on a Eucalyptus cloud of the desired version. This bug is easy to reproduce: I just need to upload an image, change the launch permission to remove all user, then try to launch it: if the instance starts the issue is present. Since I wanted to test the proposed patch, I did installed Eucalyptus 2.0.3, compiling it from source. Lester also tested it against the development branch of Eucalyptus 3.1 (check Andy's blog to work with the devel branch, and Greg's blog for more info on 3.next) and confirmed that only 2.0 is affected.

 I then downloaded Eutester, installed it locally on my client machine and started to hack the test. I used virtualenv to ensure a clean, reproducible environment to work with. My first attempt of the test successfully reproduced the issue, so I gingerly submitted it for integration. The QA team is still laughing at my python skills ... They kindly reworked the test and it is now sitting into the Eutester repository. With the new test added to a sequence in our QA system I am now guaranteed that all future version of Eucalyptus will be tested for regression against this issue.

 Onward to the patch for 2.0. I created a branch with the fix, and proposed it for merging. The QA system caught my sloppy programming at the first attempt (typo), but the second version was the win: QA passed with flying colors and the branch was eventually merged into the top of the Eucalyptus 2.0 branch.

Quite an accomplishment here: my first python script, my first Eutester test and no more lp:737335, just the perfect happy ending for this blog.