tag:blogger.com,1999:blog-69413341772312031552024-03-13T21:42:20.047-07:00parole e pensieriGrazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.comBlogger37125tag:blogger.com,1999:blog-6941334177231203155.post-4318653180528849052015-12-30T11:34:00.001-08:002016-01-06T21:51:32.926-08:00Life in a container: AppScale in Docker<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Docker and AppScaIe</span></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<b id="docs-internal-guid-669b427e-f190-07b4-06ef-36704806082e" style="font-weight: normal;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Docker hardly needs any introduction: it's been extremely popular as of late, for very good reasons. Based on a fairly old and tested container Linux technology, it makes it easy to create fresh environments to test and develop, or to isolate services within their own software stack, or to deploy a complex application stack.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-RxOxSKD_69s/VoQp7i2CBKI/AAAAAAAACIQ/cayih5XeQaA/s1600/images.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="109" src="http://4.bp.blogspot.com/-RxOxSKD_69s/VoQp7i2CBKI/AAAAAAAACIQ/cayih5XeQaA/s320/images.png" width="320" /></a></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">App Engine is another well-tested and appreciated technology, developed in 2008 and in full production at Google since 2011. An estimated 6 million App Engine applications are running in production. AppScale implements the App Engine model, allowing your application to be deployed on virtually any infrastructure. It seems only natural then to have Docker and AppScale working together.</span></div>
</b></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Developing and Testing App Engine Applications</span></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<b style="font-weight: normal;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The App Engine model makes application development easier, since the infrastructure components are already taken care of by AppScale or Google App Engine. For example, </span><a href="http://gobertelli.blogspot.com/2015/12/scale-in-appscale.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">scaling</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the application or the databases is already part of the model. The developer is then able to focus on the logic of the application, and rely on well known components servicing the APIs. The development cycle is the usual: develop in your favorite IDE, setup a test environment, demo/confirm the feature or bug fix, merge in master, rinse and repeat. There are of course a lot of different methodologies, but setting up a test environment is one of the phases that is always required (is anyone still developing with waterfall?!). </span></div>
<br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">AppScale can be deployed on a single node, thus simplifying the dev/test phases. The deployed system is a full environment, preserving the characteristics of the production environment (minus possibly the scale of nodes or data). Docker is one of the infrastructures supported by AppScale, and it is particularly suited for setting up multiple test environments. With AppScale 2.6.0 we officially released AppScale onto Docker Hub, so setting a new environment is as easy as</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.4000000000000001; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> docker pull appscale/appscale</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> docker run -t -i appscale/appscale /bin/bash</span><span style="background-color: #eeeeee; color: black; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></div>
<br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In few seconds you will have your own App Engine environment.</span></div>
</b></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Life in a Container</span></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<b style="font-weight: normal;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Typically Docker containers run one service, such as Nginx or Cassandra. It's a relatively recent push toward </span><a href="https://en.wikipedia.org/wiki/Microservices" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">microservices</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and it helps when integrating different systems since it isolates the software stack each service depends upon. Hence the base images in Docker don't have the usual 'boot' sequence, and many ancillary services aren't running since they are usually not needed to run the desired service. You typically don't need to have a cron or syslog daemon to run just Cassandra or Nginx.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-V26wMSeqJMk/VoQqDxdGPPI/AAAAAAAACIY/r1wDWhg8r58/s1600/Stack-of-Freight-Containers-at_Danicon-e1369757580664.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="http://2.bp.blogspot.com/-V26wMSeqJMk/VoQqDxdGPPI/AAAAAAAACIY/r1wDWhg8r58/s320/Stack-of-Freight-Containers-at_Danicon-e1369757580664.jpg" width="320" /></a></div>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
</h3>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">AppScale expects to have a normal instance running, i.e.: it expects quite a few of the usual services a Linux box has. For example, AppScale expects the ssh and syslog daemons to be running. This is typically done during the boot sequence, and virtually all Linux boxes have them running. So we created a simple script to start a dev/test environment easily; we call it FastStart, and when it detects a Docker container it starts the needed ancillary services. </span></div>
<br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From the AppScale container as started using the previous commands, FastStart can be invoked with the following command:</span></div>
<div dir="ltr" style="line-height: 1.4000000000000001; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #eeeeee; color: black; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> cd</span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ./appscale/scripts/fast-start.sh --no-demo-app</span><span style="background-color: #eeeeee; color: black; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="background-color: #eeeeee; color: black; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></div>
<br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The script creates a basic AppScalefile and starts AppScale. The AppScalefile is AppScale's configuration file and can be found and inspected in /root (the first </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cd</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> command ensures it will be put there). A sample application is also downloaded into /root. If you just want to test your container, you can deploy it and check it out with the following command:</span></div>
<div dir="ltr" style="line-height: 1.4000000000000001; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: #eeeeee; color: black; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> appscale deploy guestbook.tar.gz</span><span style="background-color: #eeeeee; color: black; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></div>
<br /><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The FastStart script works on any infrastructure AppScale supports, from Vagrant/VirtualBox, to AWS and GCE, and it can set up a single node instance on any of them. If the infrastructure supports the concept of public and private IP (say GCE or AWS), the script detects it and configures the system properly. For some infrastructures where detection cannot be easily done (for example Vagrant), modify the AppScalefile and change the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">login</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> directive to have the correct 'public' IP.</span></div>
</b></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dev/Test cycle made it easy</span></h3>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 14pt;">
<span style="font-weight: normal;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">AppScale's FastStart makes it trivial to generate a new development or testing environment. With Docker this process literally takes seconds, bringing a new level of convenience to the App Engine development cycle. </span><a href="http://www.appscale.com/get-started/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 14.6667px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Try it out</span></a><span style="font-family: Arial; font-size: 14.6667px; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
</span></h3>
<h3>
<span style="font-weight: normal;">
</span></h3>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-49556979928989120712015-12-03T13:08:00.000-08:002015-12-03T13:08:02.823-08:00'Scale' in AppScale<h2>
<span style="font-family: inherit;">App Engine</span></h2>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;"><i>A powerful platform to build web and mobile apps that scale automatically</i> is Google's punch line for <a href="https://cloud.google.com/appengine/">App Engine</a>. And <i>automatically</i> is the keyword: it's difficult to underestimate the power of a platform that allows any application to react to a different user induced load automatically, with no intervention from sysadmin, or developer. We loved that statement so much that we wanted everyone to be able to take advantage of autoscaling, and that's why <a href="https://github.com/AppScale/appscale">AppScale</a> was created.</span></div>
<div>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-T6PijZDx0NM/Vlvq_1_Y-_I/AAAAAAAACFc/DWM7ZnX-XFQ/s1600/Google_App_Engine_Logo%2B%2528Margaret%2BVan%2BSicklen%2527s%2Bconflicted%2Bcopy%2B2014-09-16%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-T6PijZDx0NM/Vlvq_1_Y-_I/AAAAAAAACFc/DWM7ZnX-XFQ/s1600/Google_App_Engine_Logo%2B%2528Margaret%2BVan%2BSicklen%2527s%2Bconflicted%2Bcopy%2B2014-09-16%2529.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">App Engine: A powerful platform to build web and mobile apps that scale automatically.</td></tr>
</tbody></table>
<br /></div>
<h3>
The Basics of Scaling</h3>
<div>
Google has extensive <a href="https://cloud.google.com/appengine/docs/scaling">documentation</a> on the scaling of App Engine applications. In the documentation you will find references to application <i>instances</i> and how <i>latency</i> is the main factor to understand how many instances are needed to satisfy a specific load. Since App Engine applications run on the Google platform, the promise of infinite resources at their disposal is as true as it can get. Limited perhaps by the customer wallet.</div>
<div>
<br /></div>
<div>
AppScale works in a very similar way: the application is allowed to scale up for as long as resources are available. While in Google instances determine the memory available in each instance, in AppScale we have a configuration option to achieve the same. Similarly, latency is used to determine if and how to scale the application up or down. What's different is how resources are acquired to allow the application to scale.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-atZ_JTL22WA/Vlvryrpt6SI/AAAAAAAACFs/u2n-0LZwLdg/s1600/scale.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="189" src="http://2.bp.blogspot.com/-atZ_JTL22WA/Vlvryrpt6SI/AAAAAAAACFs/u2n-0LZwLdg/s320/scale.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">App Engine applications scale automatically based on load, running in Google or AppScale. Users won't notice much: just that requests are served timely.</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br /></div>
<div>
<br /></div>
<div>
Within an AppScale deployment, some <i>nodes</i> are <i>AppServer</i> nodes, which means their CPUs and Memory will be dedicated to run application instances. Once the resources within the AppServers nodes are exhausted, if the underlying infrastructure allows it, new nodes can be acquired (up to the desired maximum) as new AppServers. AppScale supports this on Cloud environments, like GCE, AWS, OpenStack, HP Helion Eucalyptus, and there are some experimental work for vSphere.</div>
<div>
<br /></div>
<h3>
Scaling in AppScale</h3>
<div>
For autoscale to work properly, AppScale needs to be able to answer two questions: Does the application needs more resources? Do we have resources available to start a new instance? Whenever an application is uploaded within an AppScale deployment, the AppController (a component of AppScale) automatically creates a load balancer (we use <a href="http://www.haproxy.org/">haproxy</a>) configuration for it. This allows the application's instances to be added or removed with no service interruption. Periodically the AppController checks the application statistics within haproxy to see if the application is struggling. This allows AppScale to keep the application latency in check.</div>
<div>
<br /></div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-gYJ_ICCqNrY/Vlv7fE33ASI/AAAAAAAACF8/z4j2tL8Hm7s/s1600/arch.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="200" src="http://2.bp.blogspot.com/-gYJ_ICCqNrY/Vlv7fE33ASI/AAAAAAAACF8/z4j2tL8Hm7s/s200/arch.png" width="190" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">At a very high level, AppScale is similar to a usual three tier system. The front end acts as a load balancer and SSL termination, the middle tier, the AppServers, runs the application instances, and the lower tier is the Datastore.</td></tr>
</tbody></table>
Within each AppScale deployment, the Login node routinely checks the available resources on each AppServer node. When a new application instance is needed, the Login node needs to find an AppServer with enough memory for that application, and with enough CPU to spare. If there is no AppServer node meeting the requirements, a request is been sent to spawn a new virtual machine if the underlying infrastructure allows it.<br />
<br /></div>
<div>
<br /></div>
<div>
The biggest difference between Google and AppScale comes in the scaling of the datastore (the App Engine API to the integrated NoSQL database). AppScale implements the datastore API using <a href="http://www.datastax.com/">Cassandra</a>: the scaling we obtain has been extremely good, and we tested it in excess of 17,000 datastore transactions per second (equivalent to over a quarter million transactions in Cassandra for that specific workload). While Google service is limited only by the quota the user desires, the scaling of AppScale datastore implementation is manual. The main reason is that adding nodes to a running database will incur a re-balancing cost that at this time needs to be weighted and controller by an administrator.</div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-f5RtbB2jeIw/Vlv9Rg9Eu-I/AAAAAAAACGM/tfSFkxnVyeo/s1600/arch.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="155" src="http://2.bp.blogspot.com/-f5RtbB2jeIw/Vlv9Rg9Eu-I/AAAAAAAACGM/tfSFkxnVyeo/s320/arch.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The AppController monitor the application statistics, via a query to the load balancer information on the front end (1). The AppController can then inform the AppServers to start or stop an application instance if needed.</td></tr>
</tbody></table>
<br /></div>
<h3>
Tuning Scaling operations in 2.5.0</h3>
<div>
AppScale 2.4.0 and 2.5.0 bring some tuning to the scaling mechanism. In particular the hysteresis cycle has been introduced also for scaling instances within existing nodes. We observed that under certain loads, the scaling was a bit too aggressive, in particular if the application requires a long time to load (for example a complex Java application with a lot of dependencies).</div>
<div>
<br /></div>
<div>
We also increased the <i>cool-down</i> period for VMs started in a private cloud: we observed that in some private clouds environments the boot time can be long (depending on configuration), so we wanted to make sure we amortized the cost of starting a new instance, increasing the time to live. For the latter, we made sure we were well within the one hour mark, which is used by AWS as the unit of time to charge.</div>
<div>
<br />
<br />
For any question about AppScale find your preferred way to reach us at http://www.appscale.com/community.</div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-58851965168855053992013-11-21T18:56:00.001-08:002013-11-21T19:32:56.585-08:00A new Media Server (flash on Linux)I used to use an old <a href="http://www.thinkwiki.org/wiki/Category:T400s">T400s</a> (Core 2 Duo P9400) as my household media server. I used it as replication from my working files (ie using <a href="http://www.cis.upenn.edu/~bcpierce/unison/">unison</a> to synchronize my laptop home to it), as a backup repository (with 2 RAIDed USB drive as backup storage), as Music Server using <a href="http://www.samba.org/">samba</a> to export my music for <a href="http://www.sonos.com/">SONOS</a>, and again as Media Server using <a href="http://www.plexapp.com/">plex</a> to export movies to <a href="http://www.roku.com/">ROKU</a> (thanks <a href="https://plus.google.com/u/0/+DavidF/posts">David</a> for this suggestion!).<br />
<div>
<br /></div>
<div>
A reasonably simple setup, although it fell short of my needs. That media server is connected to my 1080p projector so I can watch <a href="http://www.motogp.com/">MotoGP</a>. The MotoGP web site has a very convenient subscription to watch the races whenever I want to. Usually I steer clear of any news for a few days after the race, till I have time to watch the race, and voila', MotoGP on demand with no spoilers. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><span style="margin-left: auto; margin-right: auto;"><a href="http://www.motogp.com/en/photos/2013/MotoGP+Laguna+Seca+RAC+5"><img border="0" height="180" src="http://3.bp.blogspot.com/-EzD76eJQwgc/Uo66h8L8VNI/AAAAAAAAAfs/MCFxV9a5ZQY/s320/__s1d0127_slideshow_169.jpg" width="320" /></a></span></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.motogp.com/en/photos/2013/MotoGP+Laguna+Seca+RAC+5">I just wanted to watch the race</a></td></tr>
</tbody></table>
<div>
The issue is that the venerable T400s didn't have enough horsepower to stream the races.</div>
<div>
<br /></div>
<h3>
Flash (and Linux?)</h3>
<div>
The stream coming from MotoGP uses <a href="http://en.wikipedia.org/wiki/Adobe_Flash">flash</a>, with quality up to HD (currently 720p). Till now I needed my wife's laptop with a quad core i7-3630QM with NVIDIA GT750M (nope she is not a gamer, just an architect) to be able to enjoy the races. Luckily my wife loves the races too, so no contention on her laptop during MotoGP season.</div>
<div>
<br /></div>
<div>
The late T400s started to be a bit noisy (the fan developed an annoying whine): the time had come to look for a substitute that would be able to stream the races. In case you wonder, yes, the Media Server had to run Linux. I started to sample all the Linux machine I had access to, to check what was the minimum configuration I would need.</div>
<div>
<br /></div>
<div>
There seem to have been some hardware acceleration (GPU assisted) available to Linux for <a href="http://blogs.adobe.com/penguinswf/2008/05/flash_uses_the_gpu.html">flash</a> some time ago, but to the <a href="http://askubuntu.com/questions/91932/why-doesnt-adobe-flash-player-11-x-have-hardware-acceleration">best of my knowledge</a>, the latest versions do not have it any longer. That is, CPU brute force is the answer. And looking at CPU usage during streaming, single core raw power is the key player.</div>
<div>
<br /></div>
<div>
As a side note, I do use firefox (well <a href="https://wiki.debian.org/Iceweasel">iceweasel</a>): Chrome seems to have better performance with <a href="https://support.google.com/chrome/answer/108086?hl=en-GB">pepper</a> for some, but during my testing I didn't notice any significant difference.</div>
<div>
<br /></div>
<h3>
The Search for the CPU</h3>
<div>
I was hoping to use some small and quiet machine like the <a href="http://www.intel.com/content/www/us/en/nuc/overview.html">NUC</a>, the very same ones used for the <a href="http://www.eucalyptus.com/cloudpack">Eucalyptus Backpack Availability Zones</a>. At about the same time I was looking at this, my working laptop (core i5-2520M, not enough to watch the races), started to have issues, and the replacement laptop I got (Dell <a href="http://wiki.gentoo.org/wiki/Dell_XPS_13_Ultrabook">XPS-13</a>) had exactly the same CPU as the most powerful of the NUC (core i7 3537U). Very convenient for my testings.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-q8xAaQqi2_w/Uo64j1IpfII/AAAAAAAAAfg/B9kZrE6ROfM/s1600/cloud-pack-front.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="http://3.bp.blogspot.com/-q8xAaQqi2_w/Uo64j1IpfII/AAAAAAAAAfg/B9kZrE6ROfM/s320/cloud-pack-front.jpg" width="240" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Eucalyptus Backpak AZ-2</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
Once the XPS-13 arrived, I did a quick test with Windows (!!), and then with Linux. Surprisingly, both OSes showed about the same performance watching the races: almost enough, but not quite. The video showed at times annoying micro-delays. Now I had a CPU I can reference as <i>almost</i> capable. Although the NUC and similar bricks were off the table, I could start cross reference various benchmark, and select CPUs with a healthy single core lead over the low power core i7.</div>
<div>
<br /></div>
<h3>
The new Media Server </h3>
<div>
After some search across various <i>tiny</i> and <i>small</i> form cases, CPU benchmarks, and with the constrain of a reasonable price, I ended up with a <a href="http://global.shuttle.com/main/productsDetail?productId=1669">DS61</a> case (USB 3.0 was a nice to have for faster backups), and a core i3-3245. The core i3 has enough <a href="http://cpuboss.com/cpus/Intel-Core-i7-3537U-vs-Intel-Core-i3-3245">of a lead</a> over the low power core i7, to give me comfort: only the HD 4000 (same GPU on both) was slightly lower clocked. </div>
<div>
<br /></div>
<div>
I recycled a 4G memory DIMM, and a small mSATA SSD drive (fast and silent) I already had, and there you have it: a brand new media server. Few more adjustments were done to make it more capable to stream MotoGP races, in particular:</div>
<div>
<ul>
<li>default to SNA for Intel GPU acceleration (without it, I observed some tearing)</li>
<li>switched to usb wireless card (SONOS didn't provide enough bandwidth to stream the races)</li>
<li>adjusted fancontrol to keep the fan from revving too high</li>
<li>Bluetooth keyboard to <i>couch</i> control the machine </li>
</ul>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-iEueFkC9gbk/Uo67SOZTJSI/AAAAAAAAAf0/EWT0eBw-L4w/s1600/DS61_03.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="http://3.bp.blogspot.com/-iEueFkC9gbk/Uo67SOZTJSI/AAAAAAAAAf0/EWT0eBw-L4w/s320/DS61_03.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Still fairly small, and fully capable</td></tr>
</tbody></table>
<h4>
Pros</h4>
<div>
<ul>
<li>small (fits on top of the bookshelf)</li>
<li>fits the requirements (I can watch the races now)</li>
<li>reasonably quiet under load (after fancontrol)</li>
</ul>
<h4>
Cons </h4>
</div>
<div>
<ul>
<li>desktop CPU (more watts than strictly needed I think)</li>
<li>no battery (the T400s had an incorporated UPS which was nice)</li>
</ul>
</div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-30504852691099412962013-03-04T07:16:00.000-08:002013-03-04T07:51:49.891-08:00Santa Barbara Cloud MeetupFew years back I heard Santa Barbara referred as <i>Silicon Beach. </i>The first <a href="http://www.meetup.com/Santa-Barbara-Cloud-Meetup/">Cloud Meetup</a> confirmed the rumor. The kick-off meetup, was co-organized between <a href="http://community.appscale.com/">AppScale</a> and <a href="http://www.eucalyptus.com/">Eucalyptus</a>, and it was held at AppScale HQ. We got a <a href="http://www.youtube.com/watch?v=-QQJKx5VTRY&feature=g-user-u">recording of the meetup</a> (a better quality one should be on the way), at least of the presentation parts: sorry no food or drinks in it. The first meeting was meant as a very quick introduction to both Eucalyptus and AppScale and where they respectively fit into the Cloud Computing landscape (hint one is <a href="http://www.eucalyptus.com/what-is-cloud-computing">IaaS while the other is PaaS</a>). In the spirit of any good developer meetup, some hacking was involved. After the presentation, laptops were fired up, demos were given, and more in-depth questions were asked.<br />
<div>
<br />
<br /></div>
<h3>
AppScale on Eucalyptus</h3>
<div>
My share of the hacking was done ahead of time. At Eucalyptus we maintain a <a href="http://www.eucalyptus.com/eucalyptus-cloud/community-cloud">CommunityCloud</a> (ECC). The idea behind the ECC is to allow our community access to the latest Eucalyptus without having to install it. It has been extensively used by library or tool developers, by University classes, by current Euca-user before upgrading their own cloud, and by just anyone interested in playing with a running cloud. I do use it sometimes to share big files (spins up an instances and <span style="font-family: Courier New, Courier, monospace;">scp</span> big files there), or to experiment with new images, or software. The draconian SLA (instances, volumes, and buckets will be terminated or deleted at regular intervals) is not an issue for my use cases, so I found it a lot more convenient then having to poke at my home firewall.</div>
<div>
<br /></div>
<div>
The ECC seems the perfect place to have an AppScale image for everyone to try, so ahead of the meetup we set to add a newer AppScale image. After some initial setbacks due to <span style="font-family: Courier New, Courier, monospace;">iptables</span> not willing to co-operate, we had the image uploaded and ready to go (<span style="font-family: Courier New, Courier, monospace;">emi-76294490</span>) with the latest appscale (1.7 at the time of this writing). Chris has a <a href="https://github.com/AppScale/appscale/wiki/AppScale-on-Cloud-as-Cluster">step by step write up</a>: just follow it and you will get it up and running in no time.<br />
<br />
Of course you are not limited to run AppScale on the ECC. If you want to run it on your Eucalyptus cloud, <a href="http://mdshaonimran.wordpress.com/2013/03/01/run-appscale-on-eucalyptus/">Shaon has a blog for you</a> explaining how to set it up.<br />
<br />
<br />
<h3>
Heads up: enabling DNS </h3>
</div>
<div>
Eucalyptus can be configured to <a href="http://www.eucalyptus.com/docs/3.2/ig/setting_up_dns.html">provide DNS</a> resolution for instances IP addresses and bucket names. Incidentally ECC is configured in such a way, so that describe instances will return hostnames instead of IP address, for example <span style="font-family: 'Courier New', Courier, monospace;">euca-173-205-188-102.eucalyptus.ecc.eucalyptus.com </span><span style="font-family: inherit;">instead of </span><span style="font-family: Courier New, Courier, monospace;">173.205.188.102</span><span style="font-family: inherit;">. Although this is a very useful feature, currently there are 2 bugs against it, <a href="https://eucalyptus.atlassian.net/browse/EUCA-1433">EUCA-1433</a> and <a href="https://eucalyptus.atlassian.net/browse/EUCA-1456">EUCA-1456</a>, which may create some difficulties. In particular AppScale relies on the AWS <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDMQFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FSplit-horizon_DNS&ei=s5cyUeXPGMmPrAHr_IDIDQ&usg=AFQjCNHmMCCyDN85kNIFzxf-0YiIw8t_mg">split-horizon</a> behavior observed within the instances. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The workaround here could be to either create a split-horizon DNS server for your cloud, or to ensure that the instances will associate the private IP to their public name.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">Ready for the next meetup?</span></h3>
</div>
<div>
We just finished our first meetup, but we are already planning for the next one! Topics are still up for grab, so make sure you join the group and propose your favorite one.</div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-63625291495568773072013-02-23T17:17:00.002-08:002013-02-25T09:58:43.277-08:003.2.1 and the road aheadIn my older posts, I have been talking about maintainability and how I see it as a <a href="http://gobertelli.blogspot.com/2012/12/its-maintainability-stupid.html">pillar for IaaS</a>. I have been talking about the homework the Cloud Admin <a href="http://gobertelli.blogspot.com/2013/01/will-my-internet-be-faster.html">needs to do</a> before deploying, and I have been talking about the <a href="http://gobertelli.blogspot.com/2013/01/maintainability-and-eucalyptus.html">invisible work done in Eucalyptus 3.2</a>. Of course maintainability is not the goal of a sprint, or of a single release, but not unlike security, a continued effort, a guiding principle. So, after only few weeks from the previous major release, welcome Eucalyptus 3.2.1.<br />
<br />
The <a href="http://www.eucalyptus.com/sites/all/modules/pubdlcnt/pubdlcnt.php?file=/docs/release-notes/eucalyptus-release-notes.pdf&nid=296">point release of Eucalyptus</a> brings very important fixes. I will let you go through the release notes yourself, but I 'm very pleased about the gratuitous ARP message, and the moving if <span style="font-family: Courier New, Courier, monospace;">iptables-preload</span>. <span style="font-family: inherit;">The former allow for a speedier network recovery after a Cluster Controller fail-over, and the latter makes Eucalyptus more compatibles with newer releases of linux distro. Of course there are a lot more fixes, and you can go and <a href="https://eucalyptus.atlassian.net/issues/?jql=%22Target%20Version%2Fs%22%20%3D%20%223.2.1%22">find your </a></span><a href="https://eucalyptus.atlassian.net/issues/?jql=%22Target%20Version%2Fs%22%20%3D%20%223.2.1%22">favorite</a><span style="font-family: inherit;"><a href="https://eucalyptus.atlassian.net/issues/?jql=%22Target%20Version%2Fs%22%20%3D%20%223.2.1%22"> one</a>.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">Ramping up QA</span></h3>
<div>
QA has always been the focus of our Engineering team. But as the perfectionists we are, we never rest on our laurels, and a tremendous effort has been put in extending the scope and speed of our QA. If you follow our <a href="http://www.eucalyptus.com/blog">blogs feed</a> you have already noticed a lot of the work done. And with the good works, comes a better list of Known Issues, and warnings for corner cases we don't cover yet (if you have an EMC SAN and you use 3.2.1, it's your turn to be the corner case). The last thing we want is for our users to be surprised from unexpected behavior.</div>
<div>
<br /></div>
<h3>
The Road Ahead</h3>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-s8iK_UeFCOw/USlo1QAsyeI/AAAAAAAAAPk/vOiRU9qf6BI/s1600/800px-Road_in_Norway-1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="203" src="http://4.bp.blogspot.com/-s8iK_UeFCOw/USlo1QAsyeI/AAAAAAAAAPk/vOiRU9qf6BI/s320/800px-Road_in_Norway-1.jpg" width="320" /></a></div>
<div>
<br /></div>
<div>
Aka Eucalyptus 3.3. The next major release of Eucalyptus will bring quite a few new features. You can check out the list of features <a href="https://github.com/eucalyptus/architecture/wiki/tag%3Arls-3.3">scoped out</a>. Of the major ones, we have Elastic Load Balancer, AutoScaling, CloudWatch, and Maintenance mode. </div>
<div>
<br /></div>
<div>
We just had the end of sprint 3 status review: check out <a href="http://vimeo.com/60253947">the demo yourself</a>. So far the road ahead of Eucalyptus 3.3 is nice and clear, and we'll expect it to land on your machines by the mid/late Q2.<br />
<br />
<b>Edited</b>: I don't know how to do math, since I considered a quarter to have 4 months ...</div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-75195766124738632112013-01-13T16:02:00.002-08:002013-01-13T18:31:01.316-08:00Will My Internet Be Faster?<div>
I have been beating the <a href="http://gobertelli.blogspot.com/2012/12/its-maintainability-stupid.html">Maintainability</a> drum lately, and <a href="http://gobertelli.blogspot.com/2013/01/maintainability-and-eucalyptus.html">highlighting what the latest Eucalyptus</a> did in that regard. I'm not done yet. This time I want to change the angle of approach, focusing more on the <i>Map Workload to Cloud Resources</i> step, using examples and some <a href="http://en.wikipedia.org/wiki/Back-of-the-envelope_calculation">back-of-the-envelope</a> calculations. </div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
Will my Internet be Faster?</h3>
<div>
Back in the day, I helped a user going through the installation of an ancient <a href="http://www.eucalyptus.com/">Eucalyptus</a> (I think it was 1.4), and after some hurdles (there was no <a href="http://www.eucalyptus.com/download/faststart">Faststart</a> back then), he finally got instances up and running on his two nodes home setup. Then he asked "Will my Internet connection be faster now?". </div>
<div>
<br /></div>
<div>
I think the question points more to how Cloud Computing has been a buzzword, and has been perceived as a panacea to all IT problems. But it is also a reminder of the need to understand the underlying hardware Infrastructure, in order to achieve the desired performance. An on-premise Cloud is able to go as fast as the underlying Infrastructure, and will have as much capacity as the hardware supporting it. There is a tremendous flexibility that the Cloud provides, yet there is also the risk to under-perform, if the Physical Infrastructure is not prepared for the workload. </div>
<div>
<br /></div>
<h3>
Case In Point: Cloud Storage</h3>
<div>
At the end of the day, all Cloud Resources map to physical resources: RAM, CPU, network, disks. I will now focus on the <a href="http://gobertelli.blogspot.com/2012/10/cloud-storage-types.html">Cloud Storage</a> story because of its importance (anyone cares about their data?), and because historically it is where there have been some interesting issues. In particular <a href="http://www.eucalyptus.com/eucalyptus-cloud/documentation/3.1">Eucalyptus 3.1.2</a> was needed because high disk load caused seemingly random failures.</div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-Ri4ozv0Tbcs/UPH8alkI8lI/AAAAAAAAAMM/a1ddFl0We50/s1600/back-envelope.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="151" src="http://3.bp.blogspot.com/-Ri4ozv0Tbcs/UPH8alkI8lI/AAAAAAAAAMM/a1ddFl0We50/s320/back-envelope.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Some back-of-the-envelope, hand-waiving calculation needed for this blog</td></tr>
</tbody></table>
</div>
<div>
From my <a href="http://www.urbandictionary.com/define.php?term=chicken%20scratch">chicken scratch</a> of the first figure, you should see how Ephemeral Storage resides on the NCs (Node Controller), while EBS (Elastic Block Storage) is handled by the SC (Storage Controller). Let's quickly re-harsh when the two are used:</div>
<div>
<ul>
<li>Ephemeral: both instance-store and bfEBS (boot from EBS instances) uses Ephemeral, although instance-store use Ephemeral Storage also for root, and swap;</li>
<li>EBS: any instance with an attach Volume uses EBS, and bfEBS uses Volume for root, and swap.</li>
</ul>
<div>
and which kind of contention there is:</div>
</div>
<div>
<ul>
<li>Ephemeral: any instance running on the <b>same</b> NC will compete for the same storage;</li>
<li>EBS: all instances using EBS within the same availability zone (Eucalyptus parlance for cluster) will access and use the <b>same</b> SC.</li>
</ul>
<div>
I used a <a href="https://docs.google.com/spreadsheet/ccc?key=0Asfyng5vi_hJdG1FenZGTE5TdzRjcW4zeWQzZVlSX2c">simple spreadsheet</a> to aid my examples. Feel free to copy it, play with it, enhance it, but please consider it a learning tool and not a real calculator: way too many variables have been left behind for the sake of simplicity.</div>
</div>
<div>
<br /></div>
<div>
In my examples I will measure the underlying storage speed with <a href="http://en.wikipedia.org/wiki/IOPS">IOPS</a>. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-SQIiul0RA3k/UPJoAP9GCiI/AAAAAAAAAOY/61Qmr3XXJcY/s1600/reference-values.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="http://2.bp.blogspot.com/-SQIiul0RA3k/UPJoAP9GCiI/AAAAAAAAAOY/61Qmr3XXJcY/s320/reference-values.png" width="289" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">IOPS values may vary dramatically. The above may be<br />
used only to have an indication of the expected performance. </td></tr>
</tbody></table>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
In the following examples, I will make the very unreasonable assumption that instances will access equally all their storage (both Ephemeral, and EBS), and that they will either use it 20% or 100% of the time. Moreover in the 20% case, an <a href="http://en.wikipedia.org/wiki/Oracle_machine">oracle</a> minimizes the concurrent disk access of all instances (ie if there are less than five running instances, they will not compete at all and see the full speed of the storage). </div>
<div>
<br /></div>
<div>
Thus one is a very light scenario, where the instances are mainly idles, while the other (100%) assumes the instances running benchmarks. Starting instances is a fairly disk intensive process, first because Eucalyptus needs to prepare the image to boot (which involve copying multi-GB files), and then because the OS will have to read the disk while booting. I added a column to the spreadsheet to show the impact of starting instances on the light workload.</div>
<h4>
</h4>
<h4>
Home setup</h4>
<div>
A small Cloud installation will most likely have the SC backed by local storage. Let's use an <a href="http://www.wmarow.com/strcalc/">IOPS calculator</a> to estimate the performance. Here I will use 2 Seagate Cheetah 15K rpm, and <a href="http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_0">RAID 0</a>, which gives about 343 IOPS (I will round it to 350). For the NC, I will assume 150 IOPS which should be a reasonably fast single disk (non SSD). </div>
<div>
<br /></div>
<div>
For a Home setup three NCs seems a good number to me. Each NC should have enough cores and RAM to allow more than ten instances running (12-24 cores, 12-24 GB RAM should do). If I run one instance-store, one bfEBS instance, and have one Volume per NC, the very unrealistic calculator gives</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-yvgCEsjoGjE/UPJsjJtXYPI/AAAAAAAAAOs/HU7_1aro4jU/s1600/small-light.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="140" src="http://4.bp.blogspot.com/-yvgCEsjoGjE/UPJsjJtXYPI/AAAAAAAAAOs/HU7_1aro4jU/s400/small-light.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Light load on the home setup: slowest<br />
storage is still comparable to a 5400RPM disk.</td></tr>
</tbody></table>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
Not bad for my Home setup. Even if the instances were to run <a href="http://www.iozone.org/">iozone</a> on all the disks, I can still see a performance of a slow 5400RPM disk. Now, let me create more load: four instance-store, four bfEBS, and have two Volume used per NC</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-2MjEwn-APFo/UPJtEZUSSsI/AAAAAAAAAO0/ITSCn2vPfkI/s1600/small-heavy.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="135" src="http://1.bp.blogspot.com/-2MjEwn-APFo/UPJtEZUSSsI/AAAAAAAAAO0/ITSCn2vPfkI/s400/small-heavy.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The home setup with a heavier load doesn't do thatwell: <br />
instances may see performance as slow as a floppy drive.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
That's a bit more interesting. If the instances are very light in disk usage, they will see the performance of a 7200RPM disk, but under heavier load, they will be using something barely faster then a floppy. Ouch!</div>
<div>
</div>
<h4>
A More Enterprisy setup</h4>
<div>
From the previous example, is fairly obvious why bigger installations tend to use a SAN for their SC storage back-end. For this example I will use a <a href="http://en.community.dell.com/support-forums/storage/f/3775/t/19339302.aspx">Dell Equallogic</a>. I will use a setup that gives a 5000 IOPS. Correspondingly, the number of NCs are increased to 10.</div>
<div>
<br /></div>
<div>
Let's start with a light load: one instance-store, one bfEBS, and one Volume per NC (similarly to the Home setup, although now there is a total of 20 running instances).</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-r02OnO3h1Zw/UPJt5cR2nNI/AAAAAAAAAPA/FuaYqjPF_GI/s1600/medium-light.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="135" src="http://4.bp.blogspot.com/-r02OnO3h1Zw/UPJt5cR2nNI/AAAAAAAAAPA/FuaYqjPF_GI/s400/medium-light.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A SAN backed cloud with a<br />
light load: pretty good all around.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
The results are pretty good with access to EBS around 250 IOPS under heavy load, and very fast access on the light load. Even Ephemeral compares well with a 3.5" desktop-class disk. </div>
<div>
<br /></div>
<div>
Now I will run more instances: four instance-store, four bfEBS, and have 4 Volumes per NC. . </div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-A4hi_UQp4zc/UPJus24_sRI/AAAAAAAAAPQ/o1qItcOE7dg/s1600/medium-high.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="136" src="http://4.bp.blogspot.com/-A4hi_UQp4zc/UPJus24_sRI/AAAAAAAAAPQ/o1qItcOE7dg/s400/medium-high.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A SAN backed cloud with an heavier load: EBS<br />
is now comparable to a 5400 RPM under heavy load.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Ephemeral still takes a beating: as in the Home setup case, there are eight different instances using the same local disk (bfEBS has access to Ephemeral too, and in my simplistic approach all disks are used at the same rate) . EBS slowed down quite a bit, and now it compares to a slow desktop-class disk. Although the instances should still have enough IOPS to access storage, perhaps it is time to start thinking about adding a second availability zones to this setup.</div>
<div>
<br /></div>
<h4>
Snapshots</h4>
<div>
The above examples didn't consider Snapshots at all. Snapshots allows to back-up Volumes, and to move them across availability zones (Volumes can be created from Snapshot in any availability zone). Snapshots resides on Walrus, which means that every time a Snapshot is created, a full copy of the Volume is taken on the SC, and sent to Walrus. If Snapshots are frequent on this Cloud, it is easy to see how the SC, Walrus, and the Network can become taxed serving them.</div>
<div>
<br /></div>
<h4>
Expectations</h4>
<div>
I would take all the above numbers as a best case scenario under their relative cases. A lot of variables have been ignored, starting from network, as well as others disk access. For example, Eucalyptus provides swap by default to instance-store, and the typical linux installation creates swap (ie bfEBS instances will most likely have swap), hence any instance running out or RAM, will start bogging down the respective disk. </div>
<div>
<br /></div>
<div>
There was also the assumption that not only the load is independent, but the instance co-operate to make sure they play nice with the disk. Finally in a production Cloud, a certain mix of operation is to be expected, thus, starting, terminating, creating volumes, creating snapshots, will increase the load of both Storage (Ephemeral and EBS) accordingly. </div>
<div>
<br /></div>
<div>
As I mention in my Maintainability post, having a proper workload example, will allow you to properly test and tune the cloud to satisfy your users.</div>
<div>
<br /></div>
<h3>
Making Internet Faster</h3>
<div>
In the above examples, I pulled off some back-of-the-envelope calculations which do not consider the software Infrastructure at all (ie they don't consider Eucalyptus overhead). Eucalyptus impact on the Physical Infrastructure has been constantly decreasing. Just to mention few of the improvements, before Eucalyptus 3, the NC would make straight copies multi-GB file, now it use device mapper to minimize the disk access to bare minimum, And the SC alongside with the SAN plugins, now has DASManager (Direct Access Storage, ie a partition or a disk), which allow to bypass the file system when dealing with Volumes. </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-y8ENVKUO6AI/UPIqgBEVirI/AAAAAAAAAOE/vMrFarQf0LE/s1600/speed.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="http://2.bp.blogspot.com/-y8ENVKUO6AI/UPIqgBEVirI/AAAAAAAAAOE/vMrFarQf0LE/s320/speed.jpg" width="320" /></a></div>
<div>
<br /></div>
<div>
There has been a nice performance boost with Eucalyptus 3, but there is still room for improvements, and no option has been left unexplored, from using distributed file systems as back-end, to employing <a href="http://en.wikipedia.org/wiki/Software-defined_networking">SDN</a>. Although Eucalyptus may not be able to make Internet faster yet, it is for sure trying hard.</div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-35045101819178731302013-01-01T16:48:00.001-08:002013-01-01T23:24:51.449-08:00Maintainability and EucalyptusI recently blogged about the importance of <a href="http://gobertelli.blogspot.com/2012/12/its-maintainability-stupid.html">Maintainability for on-premise Clouds</a>. Within the lists of steps to a successful on-premise Cloud deployment identified in the blog, Eucalyptus as IaaS software is heavily involved with the <i>Deploy</i> and <i>Maintain </i>part.<br />
<div>
<br /></div>
<h3>
Deploy</h3>
<div>
I already mentioned the work done to make Eucalyptus installation <a href="http://www.urbandictionary.com/define.php?term=easy%20peasy">easy peasy</a>, so let me summarize them here. Eucalyptus is packaged for the main Linux distritributions, so the installation is as easy as configuring the repository, and do a <span style="font-family: Courier New, Courier, monospace;">yum install</span> or <span style="font-family: Courier New, Courier, monospace;">apt-get install</span><span style="font-family: inherit;">. Configuring Eucalyptus is still a bit more complex that I would like to, and requires to register the components with each other, but the steps can easily be automated, as demonstrated by our <a href="http://www.eucalyptus.com/download/faststart">FastStart</a> installation. </span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Although there is always margin to improve, as distributed systems go, I dare to say that we are getting as easy as possible. Moreover, any good sysadmin already uses software to manage the infrastructure, so I see script-ability as the most important feature to allow easy progress with custom installations (ie Eucalyptus deploy recipes to use with <a href="http://ansible.cc/">ansible</a>, <a href="http://www.opscode.com/">chef</a>, and <a href="http://puppetlabs.com/">puppet</a>).</span></div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
Maintain</h3>
<div>
If you follow our development, you already know that <a href="http://www.eucalyptus.com/news/eucalyptus-drives-new-innovations-private-and-hybrid-cloud-software">Eucalyptus 3.2</a> got recently released. There are ample documents covering the release either in general (<a href="http://www.eucalyptus.com/blog/2012/11/27/eucalyptus-32-first-ease-use-release">Rich</a>, and <a href="http://www.eucalyptus.com/blog/2012/11/26/eucalyptus-32-harder-inside-softer-outside">Marten</a> blog) or for specific features (<a href="http://coderslike.us/2012/12/06/configuring-the-eucalyptus-user-console-with-a-reverse-proxy/">David</a>, <a href="http://cptyesterday.wordpress.com/2012/12/24/setting-up-rsyslog-and-logrotate-for-the-new-eucalyptus-console/">Andrew</a>, and <a href="http://eucatest.wordpress.com/2012/12/05/tcp-dumpster-monitoring-tcp-for-eucalyptus-user-console/">Kyo</a> blog), but if I wear my Cloud Admin hat, the part that didn't get enough coverage is the the amount of work that went into making Eucalyptus more maintainable.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-U1kA515KMcs/UOC_hO1tmKI/AAAAAAAAALY/HDDOPx_5yvA/s1600/Screenshot+from+2012-12-30+14:23:32.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="136" src="http://4.bp.blogspot.com/-U1kA515KMcs/UOC_hO1tmKI/AAAAAAAAALY/HDDOPx_5yvA/s320/Screenshot+from+2012-12-30+14:23:32.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Eucalyptus 3.2 fixed issues.</td></tr>
</tbody></table>
<div>
Eucalyptus 3.2 had <a href="https://eucalyptus.atlassian.net/issues/?filter=12803">350 fixed</a> bugs, and those are only the reported ones, since quite a few got fixed while restructuring parts of the code. Peek over the list, you will see the ones related to the new features but there is a large number of things done to make Eucalyptus more robust and hence maintainable. You don't believe me? Let me give you a sample:</div>
<div>
<ul>
<li>reworked the inner code paths of the <a href="http://lesterwade.wordpress.com/2012/11/14/ebs-architecture-edbp/">Storage Controller</a>, preventing now to accidentally configure the SC with am undesired backend;</li>
<li>added safety mechanism to the HA functioning which will prevent or greatly reduce the risk to have <a href="http://en.wikipedia.org/wiki/Split-brain_%28computing%29">split brain</a> Cloud Controllers;</li>
<li>more robust handling of <i>orphan</i> instances (the situation appears if the Node Controller is not able to timely relay its information all the way to the CLC)</li>
<li>plugged memory and database connections leaks (fairly annoying since they required restart of components under particular use cases).</li>
</ul>
<div>
Likely you got more excited about our awesome <a href="https://communitycloud.eucalyptus.com:8888/">new user console</a>, but it's the features like the above list that gives me the comfort of a solid Infrastructure. </div>
</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-3-NCv9sh1c8/UOOC9mn-SRI/AAAAAAAAALs/gl5HZk3TmKk/s1600/euca-console-dashboard.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="239" src="http://4.bp.blogspot.com/-3-NCv9sh1c8/UOOC9mn-SRI/AAAAAAAAALs/gl5HZk3TmKk/s320/euca-console-dashboard.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">User Console screenshot taken from <a href="http://coderslike.us/2012/11/11/installing-the-eucalyptus-console-from-source-and-packages/">David's blog</a></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<h3>
Are we there yet?</h3>
<div>
As I mentioned before, there is always room for improvements. The bulk of the work for 3.2 went into hardening the code, into covering all the corner cases, into improving QA coverage. I call all this work the <i>invisible</i> work since it is neither flashy, nor apparent at cursory inspection, yet it is the one that allows the Infrastructure to survive the test of time. </div>
<div>
<br /></div>
<div>
With most of the invisible work done, what is ahead of us is more easy to understand and categorize. From the list of the work <a href="https://github.com/eucalyptus/architecture/wiki/tag%3Arls-3.3">scoped for 3.3</a>, I see a lot of great new features: <a href="https://github.com/eucalyptus/architecture/wiki/autoscaling-3.3-spec">autoscaling</a>, <a href="https://github.com/eucalyptus/architecture/wiki/cloudwatch-3.3-spec">cloudwatch</a>, <a href="https://github.com/eucalyptus/architecture/wiki/elb-3.3-spec">elb</a> for example. This is still scoping work, so if you really like one feature, go and tell us, or up-vote it in our <a href="https://eucalyptus.atlassian.net/">issue tracker</a>. Yet, with all these new features, we don't loose focus on our Infrastructure roots. In particular the work for <a href="https://github.com/eucalyptus/architecture/wiki/maintenance-mode-3.3-spec">Maintenance Mode</a>, and <a href="https://github.com/eucalyptus/architecture/wiki/networking-3.3-spec">Networking</a>, alongside with a lot of other features that will make it much easier to deal with Cloud Resource, for example <a href="https://github.com/eucalyptus/architecture/wiki/vmtypes-3.3-spec">vmtypes</a> and <a href="https://github.com/eucalyptus/architecture/wiki/rsrc-tagging-3.3-design">tagging</a>.</div>
<div>
<br /></div>
<div>
So, are we there yet? As I mentioned in my previous blog, work on an infrastructure is done when the infrastructure is not in use anymore, so no, we are not there yet, but for sure we are having a great ride.</div>
<div>
<br /></div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-65143954997527909962012-12-05T00:45:00.001-08:002012-12-16T20:26:35.664-08:00It's Maintainability, StupidI have been playing with <a href="http://www.eucalyptus.com/blog/2012/10/26/installing-eucalyptus-31-using-faststart-312-under-30-minutes">FastStart</a> and <a href="http://www.eucalyptus.com/blog/2012/10/10/install-eucalyptus-31-using-silvereye-our-virtualized-sandbox-lab">Silvereye</a> for a while, as well as worked with the <a href="http://cptyesterday.wordpress.com/2012/09/20/single-system-test-cloud-take-2/">single-cloud script</a>. I also <a href="http://gobertelli.blogspot.com/2012/06/developer-cloud.html">wrote</a> about the setup I use for testing <a href="http://www.eucalyptus.com/">Eucalyptus</a> on a laptop. The latest tweets, and blogs clocked a Eucalyptus installation in less than 15 minutes. Quite impressive I have to say. Yet I think it's important not to lose prospective on the ultimate goal for an on-premise cloud.<br />
<br />
<h3>
On-Premise Cloud as Infrastructure</h3>
I see the installation of Eucalyptus (or any on-premise cloud for that matter) as a step during the deployment of a cloud, and neither the first one, nor the most important one. The on-premise clouds I am referring to, are <i><a href="http://www.eucalyptus.com/learn/what-is-cloud-computing">IaaS</a>,</i><i> </i>and right now I want to focus on the <i><b>I</b>nfrastructure</i> part . When I think about Infrastructure I think of satellites, power grid, aqueducts, Internet, and so on. There are quite a few characteristics of an Infrastructure, but one in particular comes to my mind to identify a successful one: <i>Maintainability</i>.<br />
<br />
<h3>
Why Maintainability?</h3>
An Infrastructure's purpose is to provide or support basics services: when infrastructure collapses, severe consequences are to be expected (think blackouts, blocked highway, etc...). An Infrastructure needs to be <i>dependable</i> (how can you build a house on unstable foundations?), to have a <i>long lifespan</i> (I can think of <i>temporary</i> only for Proof Of Concept installations), to sustain a very<i> different load</i> or use throughout its useful life (think about Internet, highways etc.. from their inception to what they are today), hence it needs to adapt to the load (<i>elasticity</i>), to isolate and/or limit the scope of failures (<i>resiliency</i>), to be functioning and accessible (<i>available</i>), to be inter-operable with different versions and/or similar minded infrastructure, to isolate operator access (<i>minimize human errors</i>). In my mind the above encapsulates the essence of <i>Maintainability. </i>I guess I'm taking the <a href="http://www.eucalyptus.com/learn/cloud-roles">Cloud Admin</a> side by considering reliability and availability as part of maintainability - since an Infrastructure that is neither reliable, nor available is not maintainable from the admin point of view.<br />
<br />
<h3>
Deploying On-Premise Cloud</h3>
<div>
So, how do we deploy an on premise cloud successfully? I see most of the difficulties in the planning, preparing, and forecasting. One key element is to understand where the workload and the cloud <i>will be</i> in the future, and to have a path to migrate both the underlying physical system and the cloud software incrementally. Very easy, right?<br />
<br />
<br /></div>
<div>
<h4>
Learn your Workload</h4>
<div>
In most cases an on-premise cloud is deployed for a specific application or set of applications, or departments, or group of users, or use-cases, so the workload is already implicit. Learn the needs to of the workload in term of compute, network, storage, and how parallel or spiky it is, and if possible at all, forecast the future workload.<br />
<br />
Capturing at this stage a sample of the workload, or to create an artificial load which mimic the real workload, is a boon for the successive stages. Although it not always easy or possible, having a model of the workload will be very helpful to validate the physical infrastructure, and the cloud deployment.<br />
<br /></div>
<h4>
Map Workload to Cloud Resources</h4>
<div>
Cloud Resources are basically three: compute (CPU and RAM used by the instances), <a href="http://gobertelli.blogspot.com/2012/10/cloud-storage-types.html">storage</a>, network (bandwidth, security/isolation, IP addresses). Understanding the workload needs in terms of Cloud Resources usage, will help to size the cloud appropriately. </div>
<h4>
<div style="font-weight: normal;">
Also, usage will most likely vary with the <a href="http://www.eucalyptus.com/learn/cloud-roles">Cloud Users</a> becoming more savvy: it's common to see a very heavy reliance on EBS (for instances and volumes) when starting using the cloud, and to move more toward instance-store once the applications become cloud-aware. </div>
<div style="font-weight: normal;">
<br /></div>
<div>
<br /></div>
</h4>
<h4>
Prepare your Physical Infrastructure</h4>
<div>
With the workload defined in term of Cloud Resources, we can start isolating the possible bottlenecks and prepare the physical infrastructure to successfully run the workload. Note that some Cloud Resources may end up using multiple physical resources at the same time: for example, boot from <a href="http://www.eucalyptus.com/blog/2012/11/14/ebs-architecture-edbp">EBS</a> instances may tax at the same time the Storage Controller (EBS service provider) and the network (to allow the Node Controller to boot the instance). </div>
<div>
<br /></div>
<div>
Also factor in the load incurred in fulfilling some operation: for example starting an instance may have the Node Controller fetch the Image from Walrus, copying it on its local disk, then starting the instance. Those operation may create contention on the network (instance traffic and image transfer), on Walrus (multiple NC asking for different images), or on the local disk (if the caching of the instances is done on the same disk where ephemeral storage resides).</div>
<div>
<br /></div>
<div>
The physical infrastructure plays also a very important role when thinking about scaling the cloud: if the storage cannot be easily expanded, if the network cannot be upgraded or reconfigured, growing the cloud to meet the forecast workload may be impossible without the need of a re-install or a long downtime<br />
<br />
<br />
<h4>
Deploy</h4>
</div>
<div>
Well, we finally got here: with the physical infrastructure properly sized, we can start installing each component, Cloud Controller, Walrus, Cluster Controller(s), Storage Controller(s), and Node Controller(s) on their respective hosts. And yes, this step can take a lot less than 1/2 hour, although on production installation, with multi-cluster, <a href="http://www.eucalyptus.com/blog/2012/04/05/high-availability-cloud-%E2%80%A6">HA</a>, and <a href="http://en.wikipedia.org/wiki/Storage_area_network">SANs</a>, it may take a bit longer.</div>
<div>
<br />
<br /></div>
<h4>
Maintain</h4>
<div>
Once the cloud is deployed, it truly becomes an infrastructure, and as such we need to ensure it stays up all the time, through upgrades (cloud software, host OS, router firmware, all should be up-gradable with hopefully no downtime for the cloud as a whole), failures (failure of a machine or a component should not impact the cloud, although it may impact an instance, a pending request, etc...), expansions (adding Node Controllers, clusters, storage, network), and load spikes (cloud should degrade gracefully and not collapse). </div>
</div>
<div>
<br />
Any of the above steps may happen after deployment (you may need to profile some problematic workload, or to deploy some new components), yet they all fall under the <i>Maintain</i> umbrella, since they are all needed to ensure that the Infrastructure fires on all cylinders and becomes <i>invisible</i>. Once the Infrastructure has been in place long enough, its usage will be taken for granted, and the only attentions will be received when there are deficiencies or problems (think about power grid and how black-outs or brown-outs get in the news). That is, when your cloud is not anymore maintainable, you will be in the news.<br />
<br /></div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-77102062239901632912012-10-03T14:01:00.000-07:002012-12-10T00:59:41.088-08:00Cloud Storage Types<i>Persistent Storage</i> sounds like a <a href="http://en.wikipedia.org/wiki/Tautology">tautology</a> to me. Been used as I am (was?) to Hard Disks, USB keys, DVDs, and all other possible way to store information, it seems that storage <b>is</b> persistent by definition, and only failures, or human errors can cause non-persistent and catastrophic behavior . Well, in <a href="http://www.eucalyptus.com/learn/what-is-cloud-computing">IaaS</a> terminology, Storage comes in different flavors and can also be <i>Ephemeral</i>.<br />
<br />
<h3>
Storage Types</h3>
<a href="http://www.eucalyptus.com/">Eucalyptus</a> follows the <a href="http://www.eucalyptus.com/learn/amazon-aws-compatibility">AWS API</a> and with these APIs comes 3 Storage types:<br />
<ul>
<li>Buckets -- objects store implemented by <a href="http://aws.amazon.com/s3/#functionality">S3</a> -- (provided by Walrus), </li>
<li>Elastic Block Storage (<a href="http://aws.amazon.com/ebs/">EBS</a>) Volumes (provided by the Storage Controller),</li>
<li><a href="http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/InstanceStorage.html">Ephemeral Instance Store</a> (provided by the Node Controller). </li>
</ul>
Our Storage Team, started a <a href="https://github.com/eucalyptus/eucalyptus/wiki/Storage">wiki</a> to dig into the technical aspect of the different storage types: stay tune on GitHub for more in depth technical dive on how they have been implemented.<br />
<br />
Of the above list, two are meant to be <i>persistent</i> (i.e. to persist across instance termination): Volumes and Buckets. Two provides the familiar <i>block interface</i> (i.e. they appear and are used as Hard Disks): Volumes and Ephemeral. One is designed to be <i>massively scalable</i>: Buckets. And one is meant to be <i>temporary</i>: Ephemeral.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-0WOozC42dsI/UMWkEWm3G2I/AAAAAAAAALE/O8pZbZtaP0Q/s1600/storage_types.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="273" src="http://4.bp.blogspot.com/-0WOozC42dsI/UMWkEWm3G2I/AAAAAAAAALE/O8pZbZtaP0Q/s320/storage_types.png" width="320" /></a></div>
<br />
<h3>
Instances and Storage</h3>
Instances by default are sitting on Ephemeral Storage. Uploaded images (EMIs) are the master copies, and all instances will start as a fresh copy of the that very image. All changes made to the instance (e.g. packages installed, configuration, application data) will disappear once the instance terminates. Notice that the termination of an instance can be voluntary (i.e. the Cloud User issue a terminate-instance command) or accidental (e.g. the hardware running the instance fail, or the software within the instance fails badly). This kind of instances are called <i>instance-store</i> instances<i>. </i><br />
<br />
Instances can also use Volumes for their root file system: they are aptly called <i>boot from EBS</i> instance. In this case, at instance creation, a Volume is cloned from a specific EMI <i>snapshot</i>. The instance will then have exclusive access to this Volume, throughout its lifetime, allowing for stopping and re-starting without loss of any changes made. The instance can be restarted wherever the Volume is available (i.e. EBS Storage is only available on a cluster basis, or availability zone), and its performance is driven by the Volume performance (e.g. network speed to a <a href="http://en.wikipedia.org/wiki/Storage_area_network">SAN</a>, or DAS serviced by the Storage Controller).<br />
<br />
The main different in Storage speed between instance-store and boot from EBS, is a trade between speed of the local disk (in the case of instance-store), and the speed of accessing a SAN (or DAS) across a network. Things gets more complicated when multiple instances competes for shared resource (e.g. common disks on a Node Controller, or network access to the SAN).<br />
<br />
<br />
<h3>
Cloud Admin</h3>
<div>
The <a href="http://www.eucalyptus.com/learn/cloud-roles">Cloud Admin</a>, although not a user of these Storage types, needs to have a clear deploy plan to provide enough Storage space for <i>each</i> type, limit contention on shared resources, and ensure that the performance and reliability meets the expected levels. An understanding of the specific load, will go a long way to size the cloud properly.</div>
<div>
<br /></div>
<div>
The deployment of Walrus and the Storage Controller, respectively providers of Buckets and Volumes, is key to ensure the right level of <a href="http://en.wikipedia.org/wiki/Nines_(engineering)">reliable Nines</a> of the Persistent Storage types. Walrus get/put interface, helps to ensure the scalabilty of the service, but a slow host (CPU is needed to decrypt uploaded images, and serve concurrent streams), or limited space (Walrus stores uploaded images and Buckets) can severely crippled the normal functioning of a cloud. The Storage Controller serve Volumes to a cluster, both for EBS attachment and boot from EBS: under-sizing the network between Storage Controller and Node Controllers, can slow down to a crawling halt each instance request to disk.<br />
<br />
Ephemeral is served by the Node Controller. Sizing the physical storage subsystem for the expected number and type of instances is needed to ensure the full load can be achieved. Also, with the current multi-cores CPUs, quite a few instances can run on the same Node Controller. Too many concurrent disk requests can easily overwhelm the Node Controller's host, causing instances to time out, or unpredictable and erratic behavior: the storage subsystem needs to be properly tested for the expected concurrent load.</div>
<div>
<br /></div>
<h3>
Cloud Application Architect</h3>
<div>
The <a href="http://www.eucalyptus.com/learn/cloud-roles">Application Architect</a> is reasonably isolated from the underlying hardware used to build the private cloud, insofar as the Cloud Admin has planned properly the Storage Types availability, performances and reliability. Thus the main decisions for the Architect is which Storage Type to use and when. </div>
<h4>
</h4>
<h4>
Persistent vs Ephemeral </h4>
<div>
When I started to <a href="http://gobertelli.blogspot.com/2011/09/drinking-champagne.html">drink champagne</a>, I went for what I was accustomed to, that is, very well know servers, well taken care of (ie very <i>persistent</i>). In short an environment where a server rebuild is an exceptional case. The ancient version of Eucalyptus we used then, didn't have boot from EBS, so we effectively implemented it using Volumes and <span style="font-family: Courier New, Courier, monospace;">chroot</span> environment. As a bonus, backups were as easy as to create a Snapshot (<span style="font-family: Courier New, Courier, monospace;">euca-create-snapshot</span>).</div>
<div>
<br /></div>
<div>
After few cloud moves and upgrades (both of software and hardware), I started to embrace the idea of the <a href="http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html">chaos monkey</a>, where no single instance is central to the <i>service</i>. Now I'm relying more and more on <a href="https://projects.eucalyptus.com/redmine/projects/recipes">scripts</a> to configure defaults images on the fly, and on Buckets to store the backups needed to recover the last good state. In the case of essential database I would still use a combination of Volumes and Bucket for availability and backups.</div>
<div>
<br /></div>
<div>
I think my experience is common, and I see how administrators coming from datacenter tend to start with the <i>Storage is Persistent</i> idea, looking for the comfort of <i>boot from EBS</i>. Administrators coming from the public cloud, are already familiar with the dynamic approach of the cloud, and are more comfortable with the idea that some Storage is Ephemeral, and plan accordingly for instances to be <i>disposable</i>. </div>
<div>
<br />
<b><span style="font-size: x-small;">Edited December 10, 2012</span></b><br />
Added links for the various storage types definitions, and made it clear that S3 provides Buckets. Added Cloud Storage Types properties picture.</div>
Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com5tag:blogger.com,1999:blog-6941334177231203155.post-80526016203880915862012-08-01T19:36:00.000-07:002012-08-02T11:18:17.228-07:00Customize Instance Libvirt Environment<a href="http://www.eucalyptus.com/downloads" target="_blank">Eucalyptus</a> supports a variety of hypervisors (<a href="http://www.linux-kvm.org/page/Main_Page">KVM</a>, <a href="http://www.vmware.com/">VMWare</a>, <a href="http://xen.org/">Xen</a>). <a href="http://libvirt.org/">Libvirt</a> is used to control instances when eucalyptus is configured to use KVM or Xen. Simply put, Eucalyptus generates a <i><a href="http://libvirt.org/formatdomain.html">domain file</a></i> (aptly called <i>libvirt.xml</i>) to start the instance: the domain file can be found in a working directory of a running instance.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-mqDQgRD08uY/UBmbJGnanSI/AAAAAAAAAKI/QJFwMit0LH8/s1600/libvirt_blog.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://4.bp.blogspot.com/-mqDQgRD08uY/UBmbJGnanSI/AAAAAAAAAKI/QJFwMit0LH8/s320/libvirt_blog.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Eucalyptus generates the domain file (<i>libvirt.xml</i>) in<br />
response to the user action <i>euca-rum-instances</i>. <i>Libvirt</i><br />
will then instruct the hypervisors to execute it.</td></tr>
</tbody></table>
<br />
Older version of Eucalyptus (up to 2.0.3) used an helper perl script (<i>gen_libvirt_xml</i> or <i>gen_kvm_libvirt_xml</i>) to generate the domain file. Changing the hypervisor behavior was a matter of modifying the helper script.<br />
<br />
Eucalyptus 3 brings a greater flexibility to customize the domain file. The Node Controller produces a stub xml file with all the instance-related information (the file, called <i>instance.xml</i>, can be found in the instance working directory). Then, using an <a href="http://www.w3schools.com/xsl/">XSL Transformation</a> on <i>instance.xml</i>, Eucalyptus generates the domain file (<i>libvirt.xml</i>)<i> </i>used to start the instance. The XSL filter can be found on the Node Controller at <i>/etc/eucalyptus/libvirt.xsl</i>. At this point a couple of examples will clarify the process, and how it can be customized. To simplify the debugging and creation of the new filter, we suggest employing a command line XSLT processor during the development of the new <i>libvirt.xsl</i> (the examples below will use <a href="http://xmlsoft.org/xslt/xsltproc2.html" style="font-style: italic;">xsltproc</a>).<br />
<br />
<pre style="background-image: URL(http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <?xml version="1.0" encoding="UTF-8"?>
<instance>
<hypervisor type="kvm" capability="hw" bitness="64"/>
<backing>
<root type="image"/>
</backing>
<name>i-37F04164</name>
<uuid>2b94d8ea-5438-4aa9-b69d-5c48781e62b7</uuid>
<reservation>r-DBAD4219</reservation>
<user>4XVKCF4WDM4NAIXBORW99</user>
<dnsName></dnsName>
<privateDnsName></privateDnsName>
<instancePath>/instances/work/4XVKCF4WDM4NAIXBORW99/i-37F04164</instancePath>
<consoleLogPath>/instances/work/4XVKCF4WDM4NAIXBORW99/i-37F04164/console.log</consoleLogPath>
<userData></userData>
<launchIndex>1</launchIndex>
<cores>1</cores>
<memoryKB>1048576</memoryKB>
<key isKeyInjected="false" sshKey="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPAQQZD644Jep3HWbRfv2TZxRKYSfXI6omWZV/JKnyOxJAkYS9ZxTPCeWeg/J0mguaXHVQapUYWnZkRRfJ2CAv4Yss8ya2mG9Itc3l113C1Rjiyk1YFZcDzxikauJX/r25+M32r1CUbxOnK90z16HUdOFBe78ebe/uA9P+FWCdo/qItF8VfBnKsTqrTi4pe2DP5fJnrtrsJA9vPNh+jrWcUCjN5byknGR/wgiQ0CeySeec0k7TIKKi8aIMcvEezUX0laY1kCC7WblT6HIRH6K+5VmXFmsMpdgENnakwvVIwX9MlT6scAtVRyTOCY1qz5YyK1U7pcxWs8bGyKhTQYvp 345590850920@eucalyptus.admin"/>
<os platform="linux" virtioRoot="true" virtioDisk="true" virtioNetwork="false"/>
<disks>
<diskPath targetDeviceType="disk" targetDeviceName="sda" targetDeviceNameVirtio="vda" targetDeviceBusVirtio="virtio" targetDeviceBus="scsi" sourceType="block">/dev/mapper/euca-4XVKCF4WDM4NAIXBORW99-i-37F04164-prt-15360none-5085ae2c</diskPath>
</disks>
<nics>
<nic bridgeDeviceName="eucabr558" mac="D0:0D:37:F0:41:64"/>
</nics>
</instance>
</code></pre>
<div style="text-align: center;">
<span style="font-size: x-small;">This is an example of <i>instance.xml</i> extracted from one of</span></div>
<div style="text-align: center;">
<span style="font-size: x-small;">our QA </span><span style="font-size: x-small;">run. Notice the comprehensive instance information,</span></div>
<div style="text-align: center;">
<span style="font-size: x-small;">available: </span><span style="font-size: x-small;">only a small subset will make it into <i>libivrt.xml</i>, but </span></div>
<div style="text-align: center;">
<span style="font-size: x-small;">all can be used </span><span style="font-size: x-small;">in the XSL Transformation.</span></div>
<h3>
<span style="font-weight: normal;"><br /></span></h3>
<h3>
<span style="font-weight: normal;">Example: Using Huge Pages</span></h3>
<div>
<a href="http://www.linux-kvm.com/content/get-performance-boost-backing-your-kvm-guest-hugetlbfs">Huge pages can boost KVM performance</a>. Once the Node Controller has been modified to use huge pages, the domain files needs to be modified as well, to ensure that the hypervisor will take advantage of this feature. This is as simple as adding a static stanza to the domain file, and can be easily achieved with the following addition to <i>libvirt.xsl</i></div>
<div>
<i><br /></i></div>
<pre style="background-image: URL(http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <xsl:value-of select="/instance/name"/>
</name>
<description>Eucalyptus instance <xsl:value-of select="/instance/name"/></description>
+ <memoryBacking>
+ <hugepages/>
+ </memoryBacking>
<os>
<xsl:choose>
<xsl:when test="/instance/os/@platform = 'linux' and /instance/backing/root/@type = 'image'">
</code></pre>
<div>
<div style="text-align: center;">
<span style="font-size: x-small;">This diff shows how we configured huge pages within the domain file.</span></div>
</div>
<h3>
<span style="font-weight: normal;">Example: Legacy Images</span></h3>
<div>
RHEL 6 <a href="http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html-single/Virtualization/index.html">does not support anymore</a> the <i>SCSI</i> driver: this can be a problem if you are using old images with no VIRTIO driver. When VIRTIO is disabled (<i>eucalyptus.conf</i> variables USE_VIRTIO_ROOT, and USE_VIRTIO_DISK), the generated domain file uses the <i>SCSI</i> bus, which results in a non-compatible image for your RHEL 6 Node Controller (the image will not be able to find its own root device). </div>
<b><br /></b>
<br />
<br />
<pre style="background-image: URL(http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">
< <os platform="linux" virtioRoot="true" virtioDisk="true" virtioNetwork="false"/>
---
> <os platform="linux" virtioRoot="false" virtioDisk="false" virtioNetwork="false"/> </code></pre>
<div style="text-align: center;">
<span style="font-size: x-small;">The diff of <i>instance.xml</i> generated when VIRTIO is disabled</span></div>
<div style="text-align: left;">
<span style="font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The output of<br />
<br /></div>
<div style="text-align: center;">
<span style="font-family: Courier New, Courier, monospace;">xsltproc /etc/eucalyptus/libvirt.xsl /tmp/instance.xml</span></div>
<div style="text-align: left;">
<span style="font-family: inherit;"><br /></span><br />
<span style="font-family: inherit;">confirms that the generated <i>libvirt.xml</i> uses the deprecated driver.</span><br />
<span style="font-family: inherit;"><br /></span><br />
<pre style="background-image: URL(http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <disk device="disk" type="block">
<source dev="/dev/mapper/euca-4XVKCF4WDM4NAIXBORW99-i-37F04164-prt-15360none-5085ae2c"/>
<target dev="sda" bus="scsi"/>
</disk>
</code></pre>
<div style="text-align: center;">
<span style="font-family: inherit; font-size: x-small;">When disabling VIRTIO, Eucalyptus defaults to SCSI.</span></div>
<div style="text-align: left;">
<span style="font-family: inherit; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: inherit; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
With a simple modification of <i>libvirt.xsl</i>, we can add support for these legacy images:</div>
<div style="text-align: left;">
<br /></div>
<pre style="background-image: URL(http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> graziano@x220t:~/Prog/Eucalyptus/xsl$ diff libvirt.xsl /etc/eucalyptus/libvirt.xsl
116,117c116,117
< <cmdline>root=/dev/hda1 console=ttyS0</cmdline>
< <root>/dev/hda1</root>
---
> <cmdline>root=/dev/sda1 console=ttyS0</cmdline>
> <root>/dev/sda1</root>
213,217c213
< <xsl:call-template name="string-replace-all">
< <xsl:with-param name="text" select="@targetDeviceName"/>
< <xsl:with-param name="replace" select="'sd'"/>
< <xsl:with-param name="by" select="'hd'"/>
< </xsl:call-template>
---
> <xsl:value-of select="@targetDeviceName"/>
220c216
< <xsl:value-of select="'ide'"/>
---
> <xsl:value-of select="@targetDeviceBus"/>
</code></pre>
<div style="text-align: left;">
<div style="text-align: center;">
<span style="font-size: x-small;">The diff with the modified <i>libvirt.xsl</i> shows how some hard coded variable</span></div>
<div style="text-align: center;">
<span style="font-size: x-small;">have been changed (sda1 </span><span style="font-size: x-small;">becomes hda1), and how we hard code the default bus</span></div>
<div style="text-align: center;">
<span style="font-size: x-small;">to be IDE instead of </span><span style="font-size: x-small;">using what comes in </span><i style="font-size: small;">instance.xml</i><span style="font-size: x-small;">. Finally we need to</span></div>
<div style="text-align: center;">
<span style="font-size: x-small;">change every </span><span style="font-size: x-small;">reference to block device starting with </span><i style="font-size: small;">sd</i><span style="font-size: x-small;"> to </span><i style="font-size: small;">hd</i><span style="font-size: x-small;">.</span></div>
<div style="text-align: left;">
<span style="font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: inherit;">Possibly not the best XSLT to handle the task (if you have better, please let me know), but it does the job.</span></div>
<div style="text-align: left;">
<br />
<h3>
<span style="font-weight: normal;">Customization And Drawbacks</span></h3>
<div>
The information contained in <i> instance.xml</i> is fairly complete, and allows a wide range of customization. For example, there could be specific rules based on the user ID, or rules to add another NIC, or rules to make some PCI device available to the instances. All of this customization may require modifications within the image itself: for example, an extra NIC would require the image to be aware of it and to configured it correctly.</div>
<div>
<br />
Eucalyptus 3 added also <i>hooks</i> for the Node Controller: in <i>/etc/eucalyptus/nc-hooks</i> you will find an example on how to add scripts to tailor your cloud to your specific needs. Hooks gets invoked at specific time during the instance staging (<i>post-init</i>, <i>pre-boot</i>, <i>pre-adopt</i>, and <i>pre-clean</i>). Hooks and XSL Transformation allow a complete control of what is passed to the hypervisor, and the environment your instances will find.</div>
<div>
<br /></div>
<div>
One of the attractive promise of the cloud, and in particular of the hybrid cloud, is to be able to run the same images everywhere: on your cloud, on your friend's cloud, on the public cloud. A heavily customized environment, may bind your images to the specific cloud you have, thus nullifying the benefit of running everywhere. As usual, with great power comes great responsibility, and the budding Cloud Administrator should be fully aware of all possible consequences.<br />
<br />
<h4>
[Edited: Aug 2, 2012]</h4>
</div>
<div>
It looks like blogspot is not the best way to discuss code: one cannot put xml in the comments. <a href="http://lesterwade.wordpress.com/">Lester</a> had another great example. Disabling the hosts pagecache for EBS volumes can be done with the following in <i>libvirt.xsl</i>:</div>
<div>
<br />
</div>
<pre style="background-image: URL( http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <xsl:when test="/instance/hypervisor/@type='kvm' and ( /instance/os/@platform='windows' or /instance/os/@virtioRoot = 'true')">
<xsl:attribute name="bus">virtio</xsl:attribute>
<xsl:attribute name="cache">none</xsl:attribute>
<xsl:attribute name="dev">
<xsl:call-template name="string-replace-all">
<xsl:with-param name="text" select="@targetDeviceName"/>
<xsl:with-param name="replace" select="'sd'"/>
<xsl:with-param name="by" select="'vd'"/>
</xsl:call-template>
</xsl:attribute>
</code></pre>
<br />
Which is then rendered as:
<br />
<br />
<pre style="background-image: URL( http://2.bp.blogspot.com/_z5ltvMQPaa8/SjJXr_U2YBI/AAAAAAAAAAM/46OqEP32CJ8/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <disk device="disk" type="block">
<source dev="/dev/mapper/euca-3IRYEXXJ6OHXZXXYAFDOG-i-82CD4318-prt-04096none-11df8dda"/>
<target bus="virtio" cache="none" dev="vdb"/>
</code></pre>
<div>
<br /></div>
<div>
<br /></div>
</div>
</div>
</div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-67173509144076852312012-07-11T16:09:00.000-07:002012-07-11T16:09:00.786-07:00Wind of Change: Eucalyptus 3.1 is hereIt's been few weeks that Eucalyptus 3.1 have been (highly) <a href="http://www.eucalyptus.com/blog/2012/06/04/high-availability-in-the-dna-of-eucalyptus" target="_blank">available</a>, and <a href="http://www.eucalyptus.com/eucalyptus-cloud/security/advisories" target="_blank">securely</a> powering quite a few installations for the delight of Cloud Administrators, and all Cloud population. Eucalyptus 3.1 brings quite a few changes, the most obvious within the code itself (more in the <a href="http://www.eucalyptus.com/sites/all/modules/pubdlcnt/pubdlcnt.php?file=/docs/latest/rn.pdf&nid=296" target="_blank">Release Notes</a>, <a href="http://www.eucalyptus.com/blog/2012/06/27/eucalyptus-31-one-source-many-destinations" target="_blank">Rich's blog</a>, <a href="http://gregdekspeaks.wordpress.com/2012/06/27/today-is-the-day/" target="_blank">Greg's blog</a>, <a href="http://blog.devzero.com/2012/06/27/less-obvious-changes-in-eucalyptus-3-1/" target="_blank">Garrett's blog</a>), but a lot is happening without.<br />
<div>
<br /></div>
<div>
<div>
Some of the behind the scenes work, is targeted to support QA (more in <a href="http://testingclouds.wordpress.com/2012/07/09/bridging-the-gap-manual-to-automated-testing-6/" target="_blank">Vic's blog</a>, <a href="http://eucatest.wordpress.com/2012/06/23/quality-flow-in-eucalyptus/" target="_blank">Kyo's blog</a>), and dev-test. Our IT team, responsible for the happiness of QA and dev teams, is the <a href="http://www.blogger.com/blogger.g?blogID=6941334177231203155#editor/target=post;postID=4497996105521374370;onPublishedMenu=template;onClosedMenu=template;postNum=17;src=postname" target="_blank">first customer of Eucalyptus</a>, and it is always on top of the automation game (check <a href="http://cptyesterday.wordpress.com/" target="_blank">Andrew's blog</a>, <a href="http://hspencer77.wordpress.com/" target="_blank"> Harold's blog</a>, the <a href="https://projects.eucalyptus.com/redmine/projects/recipes" target="_blank">recipes project</a>). Eucalyptus 3.1 opened up quite a few fun configurations and possibilities, and I'm looking forward the next release of <a href="http://www.blogger.com/blogger.g?blogID=6941334177231203155#editor/target=post;postID=1674295274109689788;onPublishedMenu=template;onClosedMenu=template;postNum=2;src=postname" target="_blank">Silvereye</a>, or the next crazy setup they can devise.<br />
<br /></div>
<div>
Perhaps the most visible changes, are related to the way we interact. </div>
<div>
<br /></div>
</div>
<div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-bpp0mhzjNfE/T_3SzZeo3cI/AAAAAAAAAJc/EeJTscI4uZg/s1600/github-logo.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="126" src="http://1.bp.blogspot.com/-bpp0mhzjNfE/T_3SzZeo3cI/AAAAAAAAAJc/EeJTscI4uZg/s320/github-logo.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Find Eucalyptus code in GitHub</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
First and foremost the code. We moved all our code development to git and <a href="https://github.com/eucalyptus/eucalyptus" target="_blank">GitHub</a> where you can watch it, fork it, or otherwise enjoy it in any way you like it. All the old branches and releases are tagged there, if archaeology is your cup of tea. <a href="http://eucatest.wordpress.com/2012/07/01/eucalyptus-3-2-coming-soon-2/" target="_blank">Eucalyptus 3.2</a> is coming fast: make sure you got Eucalyptus in your watch list.</div>
<div>
<br /></div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-aZ731tqtc54/T_3TtviN4uI/AAAAAAAAAJk/99wLsNdQgb8/s1600/jira.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="192" src="http://1.bp.blogspot.com/-aZ731tqtc54/T_3TtviN4uI/AAAAAAAAAJk/99wLsNdQgb8/s320/jira.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Bug, Feature Requests, and Release Management all together</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Issues, bug reports, features requests are the lifeblood of any project. We are bringing all of them closer to our development team, and to the release process: you can now find all of the above in <a href="http://eucalyptus.atlassian.net/" target="_blank">one place</a>. A part from the usual operation (create, search, comments, follow-up, watch issues), you can get <a href="https://eucalyptus.atlassian.net/browse/EUCA#selectedTab=com.atlassian.jira.plugin.system.project%3Aissues-panel" target="_blank">high level stats</a>, <a href="https://eucalyptus.atlassian.net/browse/EUCA#selectedTab=com.atlassian.jira.plugin.system.project%3Aroadmap-panel" target="_blank">check the Road Map</a>, <a href="https://eucalyptus.atlassian.net/secure/VersionBoard.jspa" target="_blank">follow the progress</a> on the next release and more. Explore it, and let us know how you like it.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-vbzhUhhF45Y/T_3idWcj2eI/AAAAAAAAAJw/Nxf7p-SRnk0/s1600/distro.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="84" src="http://3.bp.blogspot.com/-vbzhUhhF45Y/T_3idWcj2eI/AAAAAAAAAJw/Nxf7p-SRnk0/s320/distro.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Eucalyptus project pages for Debian, Ubuntu, Fedora </td></tr>
</tbody></table>
<div>
<br /></div>
<div>
We also have pages representing our progress and status for each distro: we are in <a href="http://alioth.debian.org/projects/pkg-eucalyptus/" target="_blank">alioth</a> for Debian, we have a <a href="http://fedoraproject.org/wiki/Eucalyptus" target="_blank">wiki page</a> for Fedora, and of course we are on <a href="http://fedoraproject.org/wiki/Eucalyptus" target="_blank">Launchpad</a>. The latter has served has faithfully as bug tracker, and code repository for a long time, and now we'll keep using it to interact with Ubuntu. And of course you can always join, or check interesting <a href="http://projects.eucalyptus.com/" target="_blank">Eucalyptus related projects.</a></div>
<div>
<br />
<br />
As usual, for all questions, comments, or social interactions you can find us on the <a href="http://engage.eucalyptus.com/" target="_blank">forum</a>, <a href="http://lists.eucalyptus.com/cgi-bin/mailman/listinfo/community" target="_blank">mailing list</a>, or <a href="http://webchat.freenode.net/?channels=eucalyptus" target="_blank">IRC</a>. If you liked this Wind of Change, just holler.<br />
<br />
</div>
</div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-68567382117930753742012-06-06T14:12:00.004-07:002012-06-06T14:12:57.208-07:00A Developer CloudThe title of this blog says it all: in this blog I will detail how I created my own private <a href="http://www.eucalyptus.com/" target="_blank">Eucalyptus</a> cloud on my laptop, using <a href="http://en.wikipedia.org/wiki/Virtual_machine" target="_blank">VMs</a>, <a href="http://en.wikipedia.org/wiki/Network_bridge" target="_blank">bridge</a> and <a href="http://en.wikipedia.org/wiki/Iptables" target="_blank">iptables</a>, and of course <a href="http://gobertelli.blogspot.com/2012/05/playing-with-silvereye.html" target="_blank">Silvereye</a>. The instructions may be specific of my laptop, which runs <a href="http://www.debian.org/releases/sid/" target="_blank">Debian Sid</a>. The relevant specs of the laptop (a Lenovo x220t to be precise) are: 8GB of RAM, CPU is INTEL i5-2520M, and a 160GB SSD. Let me repeat: this is a <i>developer cloud setup</i>, which means that it will do the testing I need to do, but it doesn't run any production environment, nor is meant to. <a href="http://blog.dustinkirkland.com/" target="_blank">Dustin</a> created <a href="https://blueprints.launchpad.net/ubuntu/+spec/server-maverick-uec-liveusb" target="_blank">Cloud on a Stick</a> and this work was inspired by Dustin's excellent work with <a href="https://help.ubuntu.com/community/UEC" target="_blank">UEC</a>.<br />
<br />
The idea is simple: create 2 VMs (respectively front-end and node controller), attach their <a href="http://en.wikipedia.org/wiki/Network_interface_controller" target="_blank">NIC</a> to a local bridge, use <a href="http://en.wikipedia.org/wiki/NAT" target="_blank">NAT</a> for when the VMs need external connectivity, install <a href="http://gregdekspeaks.wordpress.com/2012/05/23/eucalyptus-3-1-beta-is-now-on-github/" target="_blank">Eucalyptus 3-devel</a> on them, and play with it. Few reasons for the above setup:<br />
<ul>
<li>2 VMs because I want to use the MANAGED or MANAGED-NOVLAN networking modes (there is no HTML documentation yet, so get the <a href="http://www.eucalyptus.com/eucalyptus-cloud/documentation" target="_blank">manuals</a> for more information on 3.1 networks modes) to take full advantage of security groups and elastic IPs;</li>
<li>a local bridge (with no physical device attached to it) because I keep changing networks between wireless and wired (and no network for my coffee shop breaks), and because most of the time I don't need the cloud to talk to the outside world, so NATting is sufficient for me.</li>
</ul>
<div>
Let's start setting up the virtualization bits. I use <a href="http://www.linux-kvm.org/page/Main_Page" target="_blank">KVM</a> on my laptop, and I will take advantage of the <i>nested virtuatlization</i> capabilities of the CPU I have. The <span style="font-family: 'Courier New', Courier, monospace;">kvm_intel</span><span style="font-family: inherit;"> module do not enable it by default, so I added a </span><span style="font-family: 'Courier New', Courier, monospace;">x220t.conf</span><span style="font-family: inherit;"> file to </span><span style="font-family: 'Courier New', Courier, monospace;">/etc/modprobe.d</span><span style="font-family: inherit;"> with the following</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; text-align: center;"> options kvm_intel nested=1</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; text-align: center;"><br /></span></div>
<div>
<span style="text-align: center;"><span style="font-family: inherit;">and reloaded the module (</span><span style="font-family: 'Courier New', Courier, monospace;">rmmod kvm_intel</span><span style="font-family: inherit;"> and then </span><span style="font-family: 'Courier New', Courier, monospace;">modprobe kvm_intel</span><span style="font-family: inherit;">). My old laptop had an AMD Turion and nested virtualization was on by default: if you are attempting to replicate this YMMV.</span></span></div>
<div>
<span style="text-align: center;"><span style="font-family: inherit;"><br /></span></span></div>
<div>
<span style="text-align: center;"><span style="font-family: inherit;">Next, let's setup the bridge to nowhere. I could have used the default bridge setup by </span><span style="font-family: 'Courier New', Courier, monospace;"><a href="http://libvirt.org/" target="_blank">libvirt</a></span><span style="font-family: inherit;"> (called </span><span style="font-family: 'Courier New', Courier, monospace;">virbr0</span><span style="font-family: inherit;">), but I found it easier to setup my own own. I followed the <a href="http://wiki.debian.org/QEMU#Host_and_guests_on_same_network" target="_blank">QEMU Debian wiki</a> to setup the tap devices I needed. Note how only </span><span style="font-family: 'Courier New', Courier, monospace;">tap0</span><span style="font-family: inherit;"> and </span><span style="font-family: 'Courier New', Courier, monospace;">tap1</span><span style="font-family: inherit;"> are enslaved to the bridge</span><span style="font-family: inherit;">. The complete entry (excerpt from </span></span><span style="font-family: 'Courier New', Courier, monospace;">/etc/network/interfaces</span>) <span style="font-family: inherit; text-align: center;">is</span></div>
<div>
<span style="text-align: center;"><span style="font-family: inherit;"><br /></span></span></div>
<div>
<span style="text-align: center;"></span><br />
<div style="text-align: left;">
<span style="text-align: center;"><span style="font-family: 'Courier New', Courier, monospace;"> auto br0</span></span></div>
<span style="text-align: center;">
</span><br />
<div style="text-align: left;">
<span style="text-align: center;"><span style="font-family: 'Courier New', Courier, monospace;"> iface br0 inet static</span></span></div>
<span style="text-align: center;">
</span><br />
<div style="text-align: left;">
<span style="text-align: center;"><span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>address 172.17.0.1</span></span></div>
<span style="text-align: center;">
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>netmask 255.255.255.0</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>pre-up ip tuntap add dev tap0 mode tap user graziano</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>pre-up ip link set tap0 up</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>pre-up ip tuntap add dev tap1 mode tap user graziano</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>pre-up ip link set tap1 up</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>bridge_ports tap0 tap1</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>bridge_stp off</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>bridge_maxwait 0</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>bridge_fd 0</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>post-down ip link set tap0 down</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>post-down ip tuntap del dev tap0 mode tap</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>post-down ip link set tap1 down</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>post-down ip tuntap del dev tap1 mode tap</span></div>
<div style="font-family: inherit; text-align: left;">
<br /></div>
<div style="font-family: inherit; text-align: left;">
When I need my VMs to reach the outside world, I ensure that ip forward is enabled</div>
<div style="font-family: inherit; text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;">echo 1 > /proc/sys/net/ipv4/ip_forward</span></div>
<div style="text-align: left;">
<div style="font-family: inherit;">
<span style="font-family: inherit;"><br /></span></div>
</div>
<div style="font-family: inherit; text-align: left;">
and I use the simplest NAT rules I found:</div>
<div style="font-family: inherit; text-align: left;">
<br /></div>
<div style="text-align: left;">
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> /sbin/iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> /sbin/iptables -A FORWARD -i wlan0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> /sbin/iptables -A FORWARD -i br0 -o wlan0 -j ACCEPT</span></div>
</div>
<div style="font-family: inherit; text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: inherit;">The above works when I'm connected with the wireless card; I substitute </span><span style="font-family: 'Courier New', Courier, monospace;">wlan0</span><span style="font-family: inherit;"> with </span><span style="font-family: 'Courier New', Courier, monospace;">eth0</span><span style="font-family: inherit;"> otherwise. I didn't add any rule to the default network configuration since I don't allow the cloud to be online all the time.</span></div>
<div style="text-align: left;">
<span style="font-family: inherit;"><br /></span></div>
<div style="text-align: left;">
I need the last few pieces, before I can start the instances. I already have the Silvereye iso, so I need to create a file big enough to hold the Eucalyptus component. I don't have too much space on the disk, so I settled for 10GB and 15GB respectively for the front-end (FE) and node controller (NC). I did use the following command</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;">dd if=/dev/zero of=nc.img count=1 bs=1G seek=10</span></div>
<div style="text-align: left;">
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div style="text-align: left;">
to create both <span style="font-family: 'Courier New', Courier, monospace;">fe.img </span>(for the front-end) and <span style="font-family: 'Courier New', Courier, monospace;">nc.img</span> (for the node controller).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Last piece is to create the libvirt xml configuration for the 2 instances. Here is the front-end one:</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><domain type='kvm'></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <name>frontend</name></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <memory unit="GiB">2</memory></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <description>Front End</description></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <cpu match='exact'></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <model>core2duo</model></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <feature policy='require' name='vmx'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </cpu></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <os></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <type arch="x86_64">hvm</type></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <boot dev='cdrom'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </os></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <features></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <acpi/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </features></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <clock sync="localtime"/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <devices></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <emulator>/usr/bin/kvm</emulator></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <disk type='file' device='disk'></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <source file='/home/graziano/Prog/ImagesAndISOs/fe.img'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <target dev='vda' bus='virtio'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </disk></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <disk type='file' device='cdrom'></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <source file='/home/graziano/silvereye.1337754931.857206.iso'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <target dev='hdc'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </disk></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <interface type='ethernet'></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <target dev="tap0" /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <mac address='24:42:53:21:52:45'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </interface></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> <graphics type='vnc' port='-1'/></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> </devices></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"></domain></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">note how the flag <i>vmx</i> has been forced. <i>vmx</i> is the flag indicating hardware virtualization for INTEL processors: AMD's one is <i>svn</i>. The NC configuration file is very similar: the network device is </span><span style="font-family: 'Courier New', Courier, monospace;">tap1</span><span style="font-family: inherit;">, I changed the MAC address, VM name, and the file backing the disk . I didn't even need to comment out the </span><span style="font-family: 'Courier New', Courier, monospace;"><boot> </span><span style="font-family: inherit;">flag after installation, since Silvereye boots from local disk by default.</span></div>
<div>
<br /></div>
<div>
<span style="font-family: inherit;">My machine was already setup to run instance with my username (you can check the connection to libvirt with </span><span style="font-family: 'Courier New', Courier, monospace;">virsh list</span><span style="font-family: inherit;">), so I can start the FE with </span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;"> </span><span style="font-family: 'Courier New', Courier, monospace;">virsh create frontend.xml</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">and similarly the NC using </span><span style="font-family: 'Courier New', Courier, monospace;">node.xml</span><span style="font-family: inherit;">. The first time I booted them, I followed the steps in my <a href="http://gobertelli.blogspot.com/2012/05/playing-with-silvereye.html" target="_blank">previous blog on Silvereye</a> (</span><span style="font-family: 'Courier New', Courier, monospace;">virsh vncdisplay</span> is your friend when you have multiple VMs on <a href="http://en.wikipedia.org/wiki/Virtual_Network_Computing" target="_blank">VNC</a>)<span style="font-family: inherit;">. I did ensure that the VMs were configured to use static IPs assignments (I used </span><span style="font-family: 'Courier New', Courier, monospace;">172.17.0.2</span><span style="font-family: inherit;"> for the frontend and </span><span style="font-family: 'Courier New', Courier, monospace;">172.17.0.3</span><span style="font-family: inherit;"> for the NC). Because I wanted to change the network configuration and hostname ahead of time I didn't run the Silvereye script at the first login, but I run them afterwards (the scripts are in</span><span style="font-family: 'Courier New', Courier, monospace;"> /usr/local/sbin</span><span style="font-family: inherit;"> in the installed VM).</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Few extra tips. I did install the NC few times: make sure you remove the </span><span style="font-family: 'Courier New', Courier, monospace;">/root/.ssh/known_hosts</span><span style="font-family: inherit;"> when re-registering the NC if you keep the same IP between installs. If you want to play with different image sizes make sure you have enough disk space on the NC (few re-installs were needed for me to settle on a proper image size/NC disk size and I wish I could use a much bigger disk for it). After the first time installation, I prefer not to use VNC to connect to the VMs, so I added logic for the console (just before the VNC line)</span><br />
<span style="font-family: inherit;"><br /></span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;"> <serial type='pty'></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> <target port='0'/></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> </serial></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> <console type='pty'></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> <target type='serial' port='0'/></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> </console></span><br />
<br />
<span style="font-family: inherit;"><br /></span><br />
<span style="font-family: inherit;">and in the instances I add to add </span><span style="font-family: 'Courier New', Courier, monospace;">console=ttyS0</span><span style="font-family: inherit;"> to the boot line (in </span><span style="font-family: 'Courier New', Courier, monospace;">/etc/default/grub</span><span style="font-family: inherit;"> for Debian and in </span><span style="font-family: 'Courier New', Courier, monospace;">/boot/grub/grub.conf</span><span style="font-family: inherit;"> for CentOS). After a VM reboot I can login with</span><span style="font-family: 'Courier New', Courier, monospace;"> </span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> virsh console frontend</span><br />
<br />
<span style="font-family: inherit;">Finally, these are not <i> instances</i>: if you modify them, delete things or make them unusable, well, they are gone. The good news is that they are very easy to re-create them.</span></div>
</div>
</span></div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com5tag:blogger.com,1999:blog-6941334177231203155.post-16742952741096897882012-05-22T22:58:00.000-07:002012-05-22T22:58:14.590-07:00Playing with SilvereyeSilvereye is a <a href="http://en.wikipedia.org/wiki/Silvereye" target="_blank">bird native to Australia</a> (I have to admit I have never seen one live) but it is also a <a href="https://projects.eucalyptus.com/redmine/projects/silvereye" target="_blank">project</a> that allows a very quick and easy installation of <a href="http://www.eucalyptus.com/" target="_blank">Eucalyptus</a>. The project is fairly young, but it is already very usable: you can follow its development on <a href="https://github.com/eucalyptus/silvereye" target="_blank">github</a>. While <a href="https://github.com/eucalyptus/faststart" target="_blank">Faststart</a> has a very small footprint (it fits on a 1GB thumb drive) and requires a CentOS base installation, Silvereye creates a CD iso image which can be used to install bare machines.<br />
<br />
Tonight I took the latest Silvereye for a spin, and here are my notes. Let me start with a single word: wow! I <a href="https://github.com/obino/silvereye" target="_blank">forked</a> the latest version from github, moved the <span style="font-family: 'Courier New', Courier, monospace;">silvereye.sh</span><span style="font-family: inherit;"> </span>script to a CentOS <a href="http://en.wikipedia.org/wiki/Virtual_machine" target="_blank">Virtual Machine</a> (VM) I had on my laptop, and ran the script while I went to grab some coffee. When I came back, I had an iso file freshly baked and ready to go.<br />
<br />
I booted a VM using the Silvereye CD (I will write in another blog my libvirt configuration for this experiment) and connected to it with a VNC viewer to be greeted with the boot menu.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-T6alO6OZcgs/T7xw_JAwnbI/AAAAAAAAAIg/l6loaT29Vas/s1600/boot.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="155" src="http://1.bp.blogspot.com/-T6alO6OZcgs/T7xw_JAwnbI/AAAAAAAAAIg/l6loaT29Vas/s320/boot.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Boot menu for Eucalyptus Front-End or Eucalyptus Node Controller.</td></tr>
</tbody></table>
The options are to install a Eucalyptus Front-End, a Eucalyptus Node Controller, a minimal CentOS without Eucalyptus, to boot into a rescue image or to boot from local hard disk. I selected the Front-End installation.<div>
<br /></div>
<div>
The process continued with the usual CentOS installation,<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-_Kq-QIC8Bo4/T7xyO0D6R3I/AAAAAAAAAIo/3SKlfCoTl2s/s1600/install-2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="155" src="http://2.bp.blogspot.com/-_Kq-QIC8Bo4/T7xyO0D6R3I/AAAAAAAAAIo/3SKlfCoTl2s/s320/install-2.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Default CentOS 6 installation.</td></tr>
</tbody></table>
and in few minutes the (virtual) machine was ready to be restarted into its new OS.<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-yb--0A82PPg/T7xylAQWGCI/AAAAAAAAAIw/OWsfXMXHIm0/s1600/install-done.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="155" src="http://3.bp.blogspot.com/-yb--0A82PPg/T7xylAQWGCI/AAAAAAAAAIw/OWsfXMXHIm0/s320/install-done.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">CentOS 6 is now installed and ready to go.</td></tr>
</tbody></table>
At the first root login, Silvereye kicked in and configured Eucalyptus. I did follow the few easy questions (it also allows you to reconfigure network, DNS and hostname if needed) and I ended up with a working Eucalyptus Front-End in no time!<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-IVG-uwAwSQI/T7xzNMFFQwI/AAAAAAAAAI4/lGY4A3Ow3ws/s1600/euca-1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="162" src="http://2.bp.blogspot.com/-IVG-uwAwSQI/T7xzNMFFQwI/AAAAAAAAAI4/lGY4A3Ow3ws/s320/euca-1.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Silvereye first question.</td></tr>
</tbody></table>
You will need the physical network layout of your machines: IP addresses used by the machines and IP addresses available for the instances, network netmask, gateways, DNS server. Also make sure that when you configure the private network for the instances, there is no overlap with your real network.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-55Jei5K08aY/T7xznJqzv6I/AAAAAAAAAJA/-fiaIazAs6Y/s1600/euca-2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><span style="color: black; font-size: x-small;"><img border="0" height="162" src="http://2.bp.blogspot.com/-55Jei5K08aY/T7xznJqzv6I/AAAAAAAAAJA/-fiaIazAs6Y/s320/euca-2.png" width="320" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Network configuration for Eucalyptus, and database <span style="font-size: x-small;">initializatio</span>n.</td></tr>
</tbody></table>
<div>
As a final touch, Silvereye created an Eucalyptus Machine Image (EMI) for me! Of course I could have downloaded the one we <a href="http://emis.eucalyptus.com/" target="_blank">provides</a>, <a href="https://engage.eucalyptus.com/customer/portal/articles/288687-creating-a-debian-6-0-%22squeeze%22-machine-image-for-eucalyptus" target="_blank">created my own one</a>, or found some Amazon's compatible images, but I have to say that having Silvereye doing all the dirty work was simply gratifying.</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-3fZMenItKNU/T7x1kYauLsI/AAAAAAAAAJI/teW6wZjdhYo/s1600/euca-emi.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="162" src="http://1.bp.blogspot.com/-3fZMenItKNU/T7x1kYauLsI/AAAAAAAAAJI/teW6wZjdhYo/s320/euca-emi.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Silvereye can create an EMI for you.</td></tr>
</tbody></table>
The end result was a kernel, ramdisk and image ready to use.<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-QG40uhHBwKI/T7x14T8XYdI/AAAAAAAAAJQ/qDdW1_pG0kg/s1600/euca-list-images.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="162" src="http://4.bp.blogspot.com/-QG40uhHBwKI/T7x14T8XYdI/AAAAAAAAAJQ/qDdW1_pG0kg/s320/euca-list-images.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">List of available images.</td></tr>
</tbody></table>
And here is where I stopped: I didn't have an NC ready to go, and my latest book was waiting for me. It took me about 20 minutes from the time I started the VM the first time with the Silvereye CD, to have a list of EMIs.<br /><div>
<br /><div>
<br /><div>
<div>
<br /><div>
</div>
</div>
</div>
</div>
</div>
</div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-42853281115793900952012-05-06T22:41:00.000-07:002012-05-08T07:53:17.748-07:00Euca @ UDSIn Greg's <a href="http://gregdekspeaks.wordpress.com/2012/04/27/wheres-euca-in-early-may/" target="_blank">Where is Waldo</a> game, one big event was missing: Eucalyptus is also at <a href="http://uds.ubuntu.com/event/" target="_blank">UDS</a>. <a href="http://hspencer77.wordpress.com/" target="_blank">Harold</a>, <a href="http://www.omgubuntu.co.uk/author/iamfuzz/" target="_blank">Brian</a> and me are already in Oakland ready to rock and roll. We have exciting news for our session: you can check out <a href="http://www.omgubuntu.co.uk/2012/05/eucalyptus-3-1-alpha-available-for-ubuntu-12-04-cloud/" target="_blank">Brian's blog</a> for a sneak peak of Eucalyptus 3.1 Alpha1. Your favorite <a href="http://www.eucalyptus.com/learn/what-is-cloud-computing/on-premise-clouds" target="_blank">on-premise</a> <a href="http://www.eucalyptus.com/learn/what-is-cloud-computing/iaas" target="_blank">IaaS</a> now is available with HA, boot from EBS, and IAM (check the <a href="http://www.eucalyptus.com/eucalyptus-cloud/iaas/roadmap/3.0" target="_blank">roadmap</a> for more details).<br />
<div>
<br /></div>
<div>
Find us at the <a href="http://summit.ubuntu.com/uds-q/meeting/20231/servercloud-q-eucalyptus/" target="_blank">Eucalyptus session</a>, or around the conference floor to say hello or to get a t-shirt (limited supply). </div>
<div>
<br /></div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-90505051740947653552012-04-04T09:20:00.001-07:002012-05-12T13:33:19.182-07:00The One-Site ProjectIt's my great pleasure to announce our new <a href="http://www.eucalyptus.com/" target="_blank">web-site</a>. The one-site project aimed to unified our two web sites. As <a href="http://gobertelli.blogspot.com/2011/11/one.html" target="_blank">previously mentioned</a>, we started this project sometime ago
to address the needs of our community. We believe that the diverse
interests of our community are better represented by the <a href="http://www.eucalyptus.com/learn/cloud-roles" target="_blank">cloud IT roles</a>. And the new website has been designed to help our users to
navigate following the interests, languages, and preferred communication
channel of each role.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-ZDrB0-4fwbw/T3tgs3AvtqI/AAAAAAAAAHM/kzksjNJcn6I/s1600/roles.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="208" src="http://2.bp.blogspot.com/-ZDrB0-4fwbw/T3tgs3AvtqI/AAAAAAAAAHM/kzksjNJcn6I/s320/roles.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Navigation by Cloud IT Roles</b></td></tr>
</tbody></table>
<br />
First let me mentioned how the generic navigation of the site has been
greatly improved. The main navigation is <i>Learn</i>, <i>Eucalyptus Cloud</i>, <i>Participate</i>, <i>Services</i> and <i>Partners</i>, and, if you already know where
you want to go, in the footer of each page you will find direct links to
subsections. The progression is obvious, starting from generic documents
on cloud computing, on premise cloud, IT roles going to the specific of
Eucalyptus Cloud, finally exposing how to reach our community, Services
provided and last but not least diving into the rich and varied partner
ecosystem.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-S7T1hVTZMUc/T3tmT0i0THI/AAAAAAAAAHU/BRIzOUbV0QE/s1600/admin.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="208" src="http://3.bp.blogspot.com/-S7T1hVTZMUc/T3tmT0i0THI/AAAAAAAAAHU/BRIzOUbV0QE/s320/admin.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Information customized on a per-role basis</b></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Let's go back to the navigation by roles. In the <i>Learn</i> section there is a
selection that allows each roles to explore what cloud
means for them. Each role page has specific links and information related
to the respective competences. Let me give you examples for some of the
roles represented there:<br />
<ul>
<li><a href="http://www.eucalyptus.com/learn/cloud-roles/administrators" target="_blank">Cloud Administrators</a> will find install and maintenance guides. In case
they get into issues they can search our knowledge base for the specific
error code, or engage other administrators to discuss about the different
installation options or what is the best hypervisor for their situation; </li>
<li> <a href="http://www.eucalyptus.com/learn/cloud-roles/developers" target="_blank">Developers</a> can quickly reach the latest source code, looks at currently
worked on issues, and start talking about the latest patch they have been
working on. Detailed informations on Eucalyptus internal structure and API
supported are also the bread and butter for Developers;</li>
<li><a href="http://www.eucalyptus.com/learn/cloud-roles/application-architects" target="_blank">Application Architects</a> will find everything about our starter images, from
what's inside and why, to how to recreate and modify them. They would want
to use and modify the instances recipes and share their latest scripts to
recreate their production environment. </li>
</ul>
You will find the roles highlighted throughout the web site, so, for example, under <a href="http://www.eucalyptus.com/eucalyptus-cloud/get" target="_blank"><i>GetEucalyptus</i></a> you will find options for Managers (<a href="http://www.eucalyptus.com/eucalyptus-cloud/free-trial" target="_blank">get the free trial</a>),
Architects (<a href="http://www.eucalyptus.com/download/faststart" target="_blank">FastStart</a> for a quick proof of concept), Administrators (<a href="http://www.eucalyptus.com/download/eucalyptus" target="_blank">get the latest release</a>), Application Architect (<a href="http://www.eucalyptus.com/download/images" target="_blank">get the starter images</a>).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-tFbcY8VIq7Y/T3tsbDDWSsI/AAAAAAAAAHc/gCVspT5Vh5w/s1600/participate.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="208" src="http://4.bp.blogspot.com/-tFbcY8VIq7Y/T3tsbDDWSsI/AAAAAAAAAHc/gCVspT5Vh5w/s320/participate.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Find how to participate independently from your role</b></td></tr>
</tbody></table>
<br />
<a href="http://www.eucalyptus.com/participate" target="_blank"><i>Participate</i></a> and <a href="http://www.eucalyptus.com/resources" target="_blank"><i>Resource Library</i></a> are the two notable exceptions to the
roles principle. While you will still find links to the cloud roles pages,
these pages addresses our community as a single entity. <i>Participate</i>
gathers selected blogs posts for all our users, as well as giving pointers
to where to find online community. <i>Resource Library</i> allows you to find
all the published information about Eucalyptus and cloud computing
organized by type and date.<br />
<br />
With the new web site, you will also find new redesigned forum. You will find the motivation behind it in<a href="http://dstagnaro.wordpress.com/2012/03/16/one-customer-one-community/" target="_blank"> Darren's blog</a>, and more about its functioning in the <a href="http://gobertelli.blogspot.com/2012/04/engage.html" target="_blank">Engage blog</a>. Our new site is now available and ready to Engage you: <a href="mailto:feedback@eucalyptus.com" target="_blank">let us know</a> what you like and how we can improve it
for you.Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-36812787446514896062012-04-04T09:19:00.000-07:002012-05-08T15:54:16.912-07:00Engage!We started the journey of <a href="http://gobertelli.blogspot.com/2011/11/one.html" target="_blank">One</a> sometime ago. The original idea took shape and evolved into <a href="http://gobertelli.blogspot.com/2012/04/onesite.html" target="_blank">the one-site project</a> and now into <a href="http://engage.eucalyptus.com/" target="_blank">Engage</a>. The sentiments and the ideas behind Engage have been articulated very clearly in <a href="http://dstagnaro.wordpress.com/2012/03/16/one-customer-one-community/" target="_blank">Darren's blog</a>. Since the time of Eucalyptus 1.5, we established forum within our web site, to allow our community to come together and discuss Eucalyptus and Cloud Computing. It has been my pleasure to help the best I could, participate in discussions, and meet new friends.<br />
<div>
<br /></div>
<div>
It didn't take too long before we realized how inadequate the forum and ultimate we were to help all our users. Limited search capabilities, repeating threads, difficult forum management, obscured error reporting, and a new field called on-premise cloud, got us into long per-post sessions. And yet all we could do was the idiomatic drop in the bucket. The whole process was wrong: we needed to change tack.<br />
<br />
That's when Darren got into the picture. His experience with multiple support organizations, brought the needed fresh thinking. His efforts are now live with Engage, and these are the key points:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-eoXQIbIqNVQ/T3qA-Ytr7TI/AAAAAAAAAGs/rfM77UkOcDA/s1600/Engage.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="51" src="http://3.bp.blogspot.com/-eoXQIbIqNVQ/T3qA-Ytr7TI/AAAAAAAAAGs/rfM77UkOcDA/s320/Engage.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Intuitive user experience: type your question</b></td></tr>
</tbody></table>
<br />
<ul>
<li>very intuitive user experience: just type in your question;</li>
<li>the same intuitive experience for customers and non-customer: we are doing the necessary work to ensure that customers requests meets their SLA;</li>
<li>our customers already love the competence and promptness of our support team. Our community shared the same love, perhaps without knowing: the support team is on IRC, on the forum, on emails, and Engage allows to more effectively track all open questions (wherever they are);</li>
<li>modern forum tools: we can now tag an answer as 'best answer', turn it into an article, keep track of the unanswered active threads;</li>
<li>single Knowledge Base! Perhaps the most important point: all the articles, questions, and soon issues/bugs form a common knowledge base that a simple and powerful search mines effectively. Every time you type in a question, you are searching the knowledge base!</li>
</ul>
<div>
<br /></div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-Op3YwMzLhn4/T3xxx3Y8XBI/AAAAAAAAAH0/lvyc-pZORUo/s1600/Screen+shot+2012-04-04+at+8.55.36+AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="191" src="http://4.bp.blogspot.com/-Op3YwMzLhn4/T3xxx3Y8XBI/AAAAAAAAAH0/lvyc-pZORUo/s320/Screen+shot+2012-04-04+at+8.55.36+AM.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Likely answers or topic offered while you type limits duplicate threads</b><br />
<b><br /></b><br />
<b><br /></b></td></tr>
</tbody></table>
</div>
<div>
Engage has two different basic tool: the <i>Q&A </i>and the <i>Article</i>. A Q&A can be seen as a forum post, everybody can create one and everyone can answer it, can be open ended or can have a 'best answer' to it. Customers can also flag it for faster escalation. Articles are the bulk of the knowledge base: we turn the most common, or important, Q&A into articles, where we can give more background information, have a common format, expose workaround or solutions.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-EnhGtC5lm4I/T3xxc7S-Q9I/AAAAAAAAAHs/jG9oPKU8kqk/s1600/Screen+shot+2012-04-04+at+8.57.41+AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://4.bp.blogspot.com/-EnhGtC5lm4I/T3xxc7S-Q9I/AAAAAAAAAHs/jG9oPKU8kqk/s320/Screen+shot+2012-04-04+at+8.57.41+AM.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Articles provides the bulk of the knowledge base</b></td></tr>
</tbody></table>
</div>
<div>
<div style="text-align: -webkit-auto;">
<br /></div>
<br />
When posting a <a href="https://engage.eucalyptus.com/customer/portal/questions/new" target="_blank">new question</a> you will be asked for <i>Topic</i>, <i>Name</i>, <i>Email Address</i>, <i>Subject</i>, and <i>Body</i>. We divided the Topic based on the <a href="http://open.eucalyptus.com/learn/cloud-it-roles" target="_blank">Cloud IT roles</a>, following the navigation principle of our <a href="http://www.eucalyptus.com/" target="_blank">site</a>. For a short amount of time, you will have to enter Name, and Email Address each time. This issue will be resolved once we deploy the <a href="http://en.wikipedia.org/wiki/Single_sign-on" target="_blank">single sign-on</a> mechanism (based on <a href="http://en.wikipedia.org/wiki/OpenID" target="_blank">OpenID</a>) we are working on.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-ux6oRYyWlWY/T3qHX2AAbYI/AAAAAAAAAHE/1iZ53dAPHE4/s1600/Topic.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-ux6oRYyWlWY/T3qHX2AAbYI/AAAAAAAAAHE/1iZ53dAPHE4/s1600/Topic.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Topics are divided based on the Cloud IT roles</b></td></tr>
</tbody></table>
<br /></div>
<div>
This new tool come with a cost: we will not be able to migrate our old forum posts to it. The tools are too different and we simply don't have the man-power to make this migration manually. We already pre-populated the articles with the most note-worthy topic coming from the old forum, but we did probably miss some. Also your username has not been migrated from the <a href="http://open.eucalyptus.com/" target="_blank">old web site</a>: once we implements the single sign-on, you will be able to utilize your preferred OpenID provider to login.</div>
<div>
<br /></div>
<div>
Please help us to migrate your unanswered or favorite post from the old forum: just go to engage and post it there. And if you encounter any issue, don't hesitate to <a href="mailto:support@eucalyptus.com" target="_blank">contact us</a>. We do believe you will love the simplicity and completeness of what you will find at Engage, and we will be happy to hear any comment you have to improve it.</div>
<div>
<br /></div>
</div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-51685289367487537772012-03-28T13:51:00.002-07:002012-04-09T17:26:42.981-07:004 of 20<a href="http://www.hpdc.org/" target="_blank">The International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC)</a> will commemorate its first 20 years with a special proceeding containing the best 20 papers. <a href="http://www.eucalyptus.com/blogs/rich" target="_blank">Rich Wolski</a>, and Steve Fitzgerald, respectively founder and CTO, and VP of Technical Services at Eucalyptus Systems, will each have 2 papers published in that proceeding. Rich will have <a href="http://mechatronics.ece.usu.edu/ece7750/ece7750_2005/notes_slides/w02b-delay-compensation/ZhenPresent/Forecasting%20network%20performance%20to%20support%20dynamic%20scheduling.pdf">Forecasting Network Performance To Support Dynamic Scheduling Using The Network Weather Service</a> (NWS) and <a href="http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=014AB6892EBEF55C63804232CA53D76D?doi=10.1.1.40.5946&rep=rep1&type=pdf">Scheduling from the Perspective of the Application</a> (AppLeS) while Steve will have <a href="http://dl.acm.org/citation.cfm?id=876489" target="_blank">Grid Information Services for Distributed Resource Sharing</a> and <a href="http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=709959" target="_blank">Application experiences with the Globus toolkit</a>. That is 20% of the papers!<br />
<br />
I have been fortunate enough to have been working with them since the mid '90, back when a cloud on a whiteboard meant Internet, and <a href="http://en.wikipedia.org/wiki/Grid_computing" target="_blank">Grid Computing</a> was the pinnacle of <a href="http://en.wikipedia.org/wiki/Distributed_computing" target="_blank">Distribute Computing</a> research. In addition to be excellent scientists, Rich and Steve have also been role models, teaching the new generations for more than 15 years. It's hardly a coincidence then, that Rich and Steve are now pushing the boundaries of knowledge in <a href="http://en.wikipedia.org/wiki/Cloud_computing" target="_blank">Cloud Computing</a> with Eucalyptus.<br />
<br />
It's hard to overestimate the influence that Rich and Steve have within our organization. For example, in the latest <a href="http://open.eucalyptus.com/participate/roadmaps/eucalyptus_3.0" target="_blank">Eucalyptus 3</a>, I can find Rich's fingerprints in the overall architecture, in the attentions to the internal protocols, and Steve's touch in the identity integration (LDAP) and management (EIAM). They are mentors and role model to all eucalyptoids, for their determination, insights, and skillful leadership each with his own personable yet efficient style.<br />
<br />
<b>Edit:</b> check out the <a href="http://hpdc.org/best.php" target="_blank">full list</a>.Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com1tag:blogger.com,1999:blog-6941334177231203155.post-28091585895890846092012-03-11T14:05:00.001-07:002012-03-13T16:45:03.964-07:00PyCon 2012We (<a href="http://cptyesterday.wordpress.com/" target="_blank">cpt_yesterday</a>, <a href="http://fedoraproject.org/wiki/User:Gholms" target="_blank">gholms</a> and <a href="http://gobertelli.blogspot.com/" target="_blank">obino</a>) left Silicon Beach (aka Santa Barbara home of <a href="http://www.eucalyptus.com/" target="_blank">Eucalyptus</a>) on Thursday and drove up to Santa Clara for my first <a href="https://us.pycon.org/2012/" target="_blank">PyCon</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-cuXmbnEi6d8/T1z3iTLjT2I/AAAAAAAAAE4/SGHU90JSLM8/s1600/IMG_20120308_115505.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://2.bp.blogspot.com/-cuXmbnEi6d8/T1z3iTLjT2I/AAAAAAAAAE4/SGHU90JSLM8/s320/IMG_20120308_115505.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Driving from Silicon Beach to Silicon Valley</td></tr>
</tbody></table>
<br />
I was a bit leery about dinner (just few week ago cpt_yesterday and I got food poisoned in Sunnyvale), but all went well and it was great to see <a href="http://www.elastician.com/" target="_blank">Mitch</a>.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-YqtqNa0dCyM/T1z5RQd5xBI/AAAAAAAAAFA/uLAN8NLEzH4/s1600/IMG_20120309_082753.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://1.bp.blogspot.com/-YqtqNa0dCyM/T1z5RQd5xBI/AAAAAAAAAFA/uLAN8NLEzH4/s320/IMG_20120309_082753.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Setting up the booth</td></tr>
</tbody></table>
<br />
<div>
We set up the booth on the morning, just before the keynotes. The booth has been a good meeting point throughout the conference to talk about Eucalyptus, get a bit of rest, or just grab one of our t-shirts. It's always rewarding to talk about how Eucalyptus is used in production or to receive thumbs up from people walking by.<br />
<div>
<br />
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-Lf65J8K4qPo/T1z5Wq7TExI/AAAAAAAAAFI/2j8lk09614o/s1600/IMG_20120310_103448.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://1.bp.blogspot.com/-Lf65J8K4qPo/T1z5Wq7TExI/AAAAAAAAAFI/2j8lk09614o/s320/IMG_20120310_103448.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mitch and cpt_yesterday</td></tr>
</tbody></table>
</div>
<div>
This was my first PyCon, and been a Python n00b myself (my first python program is <a href="http://gobertelli.blogspot.com/2012/03/first-steps-with-eutester.html" target="_blank">few weeks old</a>), it was a bit intimidating to be around such a crowd. Everyone was very helpful and engaging and willing to tolerate newbie: a very nice experience.</div>
<div>
</div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-rJWqar4Jm_k/T1z5bmaOUGI/AAAAAAAAAFQ/yPeZ8tHonzs/s1600/IMG_20120309_104027.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://1.bp.blogspot.com/-rJWqar4Jm_k/T1z5bmaOUGI/AAAAAAAAAFQ/yPeZ8tHonzs/s320/IMG_20120309_104027.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">gholms likes PyCon</td></tr>
</tbody></table>
</div>
<div>
We got a bit distracted from the conference by PG&E: they told us on Friday night that an urgent safety fix was needed and Saturday they will pull the power all around our HQ. Like all good <a href="http://en.wikipedia.org/wiki/DevOps" target="_blank">DevOps</a> we spent quite a bit of time to prepare for the outage, coordinating with the team back at HQ. Our core services runs on our public Eucalyptus cloud at <a href="http://www.coresite.com/Resources-Cloud_Community.php" target="_blank">CoreSite</a> (read more about the <a href="http://open.eucalyptus.com/try/community-cloud" target="_blank">ECC</a> for more info on our setup) but we have an internal cloud we use at HQ, and we wanted to be sure all these internal services were shut down nicely.</div>
<div>
<br /></div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-rZPYN5YaUqs/T1z5eT4dspI/AAAAAAAAAFY/gv1xZPJ7AKo/s1600/IMG_20120310_115241.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://1.bp.blogspot.com/-rZPYN5YaUqs/T1z5eT4dspI/AAAAAAAAAFY/gv1xZPJ7AKo/s320/IMG_20120310_115241.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The stars of the conference</td></tr>
</tbody></table>
<br />
And of course I needed a picture of the <a href="http://pycon.blogspot.com/2011/12/pycon-us-2012-i-got-something-special.html" target="_blank">stars</a> of PyCon 2012.</div>
</div>
</div>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com1tag:blogger.com,1999:blog-6941334177231203155.post-38995399870615932332012-03-05T11:00:00.000-08:002012-03-05T11:30:22.134-08:00First steps with Eutester<br />
In one of my previous post I mention our mantra: <i>"Listen to our community
and deliver quality software"</i>. To deliver quality software, it is
necessary to have the QA process as a first class citizen. Our QA team
created quite a spectacular infrastructure to test all sort of
combinations and configurations automatically (distro, architectures, versions,
hypervisors, networking, images etc ...) to guarantee our users,
regression-free releases. They also managed to have fun in the process (check
out <a href="http://eucatest.wordpress.com/2012/02/10/pigeons-on-a-euca-eucalyptus-cloud-monitoring-mobile-app-via-twitter/" target="_blank">Pigeons on a Euca</a>).
<br />
<br />
<a href="https://projects.eucalyptus.com/redmine/projects/eutester" target="_blank">Eutester</a> is the latest brainchild of our QA team, and
this blog is about my experience writing a test using it. Eutester is a framework to
create automatic tests against a Eucalyptus installations (or any cloud
following the AWS API for that matter).<br />
<br />
These are the ingredients I needed to bake my first test:
<br />
<ul>
<li style="text-align: justify;">a really annoying bug</li>
<li style="text-align: justify;">a Eucalyptus cloud to test it against</li>
<li style="text-align: justify;">the Eutester framework</li>
<li style="text-align: justify;">basic python knowledge helps although it is not necessary</li>
</ul>
Let's start with the main ingredient: a really annoying bug. I picked <a href="https://bugs.launchpad.net/eucalyptus/+bug/737335" target="_blank">lp:737335</a>, annoying and small enough for me to tackle. The
issue reported is that the launch permissions as set by <span style="font-family: 'Courier New', Courier, monospace;">euca-modify-image-attribute</span> are not respected in
Eucalyptus 2.0. The bug happened also to have a <a href="https://launchpadlibrarian.net/76849719/launch_permission_2.0.patch" target="_blank">proposed fix</a>.<br />
<br />
Next: reproduce the issue on a Eucalyptus cloud of the desired version.
This bug is easy to reproduce: I just need to upload an image, change the
launch permission to remove all user, then try to launch it: if the instance
starts the issue is present. Since I wanted to test the proposed patch, I did
installed Eucalyptus 2.0.3, compiling it from source. <a href="http://lesterwade.wordpress.com/" target="_blank">Lester</a> also
tested it against the
development branch of Eucalyptus 3.1 (check <a href="http://agrimmsreality.blogspot.com/2012/01/building-eucalyptus-3-devel.html" target="_blank">Andy's blog</a> to work with the devel branch, and <a href="http://gregdekspeaks.wordpress.com/2012/03/05/coming-soon-eucalyptus-3-next/" target="_blank">Greg's blog</a> for more info on 3.next) and
confirmed that only 2.0 is affected.<br />
<br />
I then downloaded <a href="https://github.com/eucalyptus/eutester" target="_blank">Eutester</a>, installed it locally on my client machine and
started to hack the test. I used <span style="font-family: 'Courier New', Courier, monospace;"><a href="http://www.virtualenv.org/en/latest/index.html" target="_blank">virtualenv</a></span> to ensure a clean, reproducible
environment to work with. My first attempt of the test successfully reproduced
the issue, so I gingerly <a href="https://github.com/eucalyptus/eutester/issues/3" target="_blank">submitted</a> it for integration. The QA team is
still laughing at my python skills ... They kindly reworked the test and it is
now sitting into the <a href="https://github.com/eucalyptus/eutester/blob/master/testcases/unstable/image_attribute.py" target="_blank">Eutester repository</a>. With the new test added to a sequence
in our QA system I am now guaranteed that all future version of Eucalyptus will
be tested for regression against this issue.<br />
<br />
Onward to the patch for 2.0. I <a href="https://code.launchpad.net/~graziano.obertelli/eucalyptus/2.0.0-fix-737335" target="_blank">created a branch</a> with the fix, and <a href="https://code.launchpad.net/~graziano.obertelli/eucalyptus/2.0.0-fix-737335/+merge/85715" target="_blank">proposed it for merging</a>. The QA system caught my sloppy
programming at the first attempt (typo), but the second version was the win: QA
passed with flying colors and the branch was eventually merged into the top of
the Eucalyptus 2.0 branch.<br />
<br />
Quite an accomplishment here: my first python script, my first Eutester test and no more <a href="https://bugs.launchpad.net/eucalyptus/+bug/737335" target="_blank">lp:737335</a>, just the perfect happy ending for this blog.Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-13316465957551894442012-02-21T14:19:00.000-08:002012-02-21T15:53:32.301-08:00FOSDEM 2012After having heard about <a href="http://fosdem.org/" target="_blank">FOSDEM</a> for long time, and this year I had the privilege to attend it. This may be the reason of the <a href="http://www.wunderground.com/blog/weatherhistorian/comment.html?entrynum=62" target="_blank">extreme cold spell</a> that hit Europe at that time. I reached <a href="https://launchpad.net/~brian-thomason" target="_blank">Brian Thomason</a> in Hamburg where he presented <a href="http://www.hbt.de/va-details-anmeld.html?&cHash=b93a0c3e1a216bbbdd45e60235f6db7f&tx_ttnews[tt_news]=133" target="_blank">Eucalyptus</a>. They loved him, and I have to agree with them: the presentation was very interactive and Brian was very charming. We then reached Brussels by train, on the only non-heated car: needless to say I was ready for a hot bath once we reached the hotel.<br />
<br />
FOSDEM was as good as I read about, and more. My interests brought me most of the time at the Cloud and Virtualization track, although I did walked into a Google Summer of Code talk, CentOS and Debian talks, and a survey on how to have HA with MySQL. I like the HA talk (search for Ivan Zoratti's HA reloaded): there is no free lunch when deploying HA, and the talk highlights how decisions need to be taken early on about what kind of HA one wants and she is willing to pay for.<br />
<br />
The Cloud and Virtualization track was very well attended. It was good to see old friends and shake hands, from Xen's Lars, to Dave and James from the Ubuntu server team, Thierry and Rick and more. Very good content too. I did enjoy quite a bit the libguestfs talk, since lately I have been doing <a href="https://projects.eucalyptus.com/redmine/projects/images" target="_blank">some images work</a>, and I already have been using zerofree, but I learned that there are a lot more tools I can use to simplify my life.<br />
<br />
<br />Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-20163246398522695342012-01-25T22:22:00.000-08:002012-03-23T14:58:14.757-07:00AnalogiesI have been reminded that "<i>We build roads so everyone can drive the car</i>" this past weekend at <a href="http://www.socallinuxexpo.org/scale10x" target="_blank">SCALE 10x</a>. <a href="http://dendrobates.rustedhalo.com/" target="_blank">Rick Clark</a> mentioned it during a <a href="http://www.socallinuxexpo.org/scale10x/presentations/panel-open-source-cloud-computing" target="_blank">cloud computing panel</a> toward the end of the conference, <a href="http://gregdekspeaks.wordpress.com/" target="_blank">Greg</a> was there too, but sadly didn't wear the penguin hat given to the keynote speakers (his talk is <a href="http://www.socallinuxexpo.org/scale10x/presentations/keynote-amazon-and-future-open-cloud" target="_blank">online</a> as well as most of the others presentations).<br />
<br />
I do like the comparison of IaaS to road infrastructure in its simplicity and obviousness: building IaaS may not be glamorous, yet it's a fundamental piece of the cloud puzzle. I wonder where the analogy is for the API: is the AWS API like the signs and lights in the streets to direct and control the traffic?<br />
<br />
Thanks to all who stopped by at our <a href="http://www.facebook.com/media/set/?set=a.336998882987642.78741.164828240204708&type=3" target="_blank">booth</a> to check with us, having some technical discussion or just to chill. It's always gratifying to put a face to an email address or to an IRC handle. And if weren't around SCALE, you can meet us at our <a href="http://www.meetup.com/Eucalyptus-Bay-Area/" target="_blank">Bay Area Eucalyptus Cloud</a> meetup.Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-66988744500344301502011-11-22T09:23:00.001-08:002011-12-06T15:44:11.703-08:00OneEucalyptus has a long history of listening to our community. You asked for Eucalyptus to have a life of its own, and Eucalyptus Systems was born. You asked for a different license (GPL) and you got it. You asked for easier ways to install Eucalyptus and so we packaged Eucalyptus for the main <a href="http://open.eucalyptus.com/downloads">linux distributions</a>, we delivered <a href="https://help.ubuntu.com/community/UEC">UEC</a> and we created <a href="http://open.eucalyptus.com/try/faststart">FastStart</a>.<br />
<br />
We are still listening to our community. Currently we are working on the Eucalyptus 3 feature set and development process. Check our <a href="http://open.eucalyptus.com/participate/roadmaps/eucalyptus_3.0">roadmap</a> for more information<br />
on the feature set, and stay tuned for more information about this<br />
important release.<br />
<br />
Our mantra is and always will be: "<i>Listen to our community and deliver quality software</i>".<br />
<br />
We always strive to get better at it. We are now in the process of redesigning the way we communicate with you. Currently we differentiate our users between <i>customers</i> (paying users) and <i>community</i> (intended here as the non-paying users). We have two web sites (<a href="http://www.eucalyptus.com/">www</a> and <a href="http://open.eucalyptus.com/">open</a>), two different sets of communication -- a support team, an <a href="http://support.eucalyptus.com/">issue tracker</a>, phone line 24/7 and direct emails for <i>customers</i> and a forum, IRC, launchpad and mailing list for <i>community</i>.<br />
<br />
This setup is common enough, yet in our case it seems a bit arbitrary. Our engineers, our sales people, and our support team help community members on IRC and in the forum. Conversely we are all dedicated to help our customers. So we are in the process of removing this artificial division. Our support team is working to integrate the forum, mailing lists, and issue tracker. Our sales and marketing teams are working closely to understand the needs of our users and to provide them the information they want. Our professional services team is producing online training material. Our web team is working to tie all of the above together in a seamless and user-friendly way. And our engineering team is readying the best release of Eucalyptus yet.<br />
<br />
Eucalyptus 3 will bring a lot of changes for our community: reconciling the various source branches into one, bringing a new plug-in architecture, and opening up the development process. Before Eucalyptus 3 is released, our two web sites will merge. We'll group information together following the <a href="http://open.eucalyptus.com/learn/cloud-it-roles">cloud IT roles</a>. Our webinars, forum posts, technical information, and documentation will be focused on delivering the best possible service to the specific IT role.<br />
<br />
The new web site redesign, will put our community in the driver's seat. We will present updates and new design soon, and your feedback is necessary to carry on this ambitious plan.<br />
<br />
One Eucalyptus Community.Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-8633970197387452682011-11-08T11:36:00.000-08:002011-11-08T11:40:12.692-08:00I miss my '-g'One thing I can say for sure: there is no time to get bored here at Eucalyptus. If you like big challenges and you have it to take them on, consider <a href="http://www.eucalyptus.com/careers">joining our team</a>: I'm extremely pleased to say that <a href="http://www.eucalyptus.com/news/greg-dekoenigsberg-joins-eucalyptus-vice-president-community">Greg</a> just did so.<br />
<br />
If you are at <a href="http://cloudcomputingexpo.com/">Cloud Expo</a> in Santa Clara or in the Bay area, come and meet Greg (or Rich or Marten or me for that matter :) at <a href="http://go.eucalyptus.com/Euca-Social-2011-Invitation.html">EucaSocial</a>, or visit us at our booth (#615).<br />
<br />
I have been knowing Greg by fame for a while, but in person for only few months, and I can say that the energy, and passion he brings to Eucalyptus are intoxicating. You may have already noticed the impulse on our community activities (from our #eucalyptus IRC channel, our <a href="http://lists.eucalyptus.com/">mailing lists</a>, our <a href="http://projects.eucalyptus.com/">projects</a> and our <a href="http://planet.eucalyptus.com/">planet</a>) and this is just the start!<br />
<br />
The only downside is that I don't have anymore the exclusive '-g' at the end of emails. Welcome Greg: I'm thrilled to be able to work with and learn from you.<br />
<br />Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-73506197984007569742011-10-26T14:12:00.000-07:002011-10-26T14:24:58.710-07:00Half an hour for a Eucalyptus DreamI hope you are enjoying the recently released <a href="https://wiki.ubuntu.com/OneiricOcelot/ReleaseNotes">Ubuntu 11.10</a> (Oneiric Ocelot). I recently did take it for a spin, and this blog is about my experience. Eucalyptus has been part of Ubuntu since Jaunty, and I wanted to see how well the latest available version of Eucalyptus integrates with Oneirc<br />
<br />
And I am pleased to say that in less than 1/2 hour, you can have your Eucalyptus cloud running in Oneric! You don't believe me? Let's do it:<br />
<br />
<strong>10:17</strong>: popped the CD for the <a href="http://www.ubuntu.com/download/server/download">server install</a> into two desktops machines we had laying around:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-7ksFjXl18-U/Tqh4JrPsV_I/AAAAAAAAADs/YRPMZTqr4UM/s1600/oneiric-server.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="239" src="http://1.bp.blogspot.com/-7ksFjXl18-U/Tqh4JrPsV_I/AAAAAAAAADs/YRPMZTqr4UM/s320/oneiric-server.jpg" width="320" /></a></div><br />
I just did add the openssh server to the basic install;<br />
<br />
<strong>10:31</strong>: install is done, machines rebooted. At this point I lost the monitor I was borrowing, so I switched to ssh from the comfort of my office in front of a cup of coffee. I did do a quick update and upgrade to ensure all the security patches were applied;<br />
<strong><br />
</strong><br />
<strong>10:34</strong>: Time to get to the Eucalyptus install. There are nice instruction on <a href="https://help.ubuntu.com/community/UEC/PackageInstall">help.ubuntu.com</a> which I partly followed: my engineering background prevents me from following nicely written instructions ... Soo, I did<br />
apt-get install eucalyptus-cloud eucalyptus-cc eucalyptus-sc eucalyptus-walrus<br />
On the second desktop I did install the node controller<br />
apt-get install eucalyptus-nc<br />
<strong><br />
</strong><br />
<strong> 10:43</strong>: got to the familiar webUI<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-k3FUcI_yrBM/Tqh4yGpAh3I/AAAAAAAAAD0/t8XsJyf9i_s/s1600/Screenshot+-+10262011+-+01%253A04%253A38+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="195" src="http://1.bp.blogspot.com/-k3FUcI_yrBM/Tqh4yGpAh3I/AAAAAAAAAD0/t8XsJyf9i_s/s320/Screenshot+-+10262011+-+01%253A04%253A38+PM.png" width="320" /></a></div><br />
and downloaded the admin credentials. The configure tab was missing some components. I assume the network I was using confused a bit the nice autoregistration mechanism of the UEC, so I switched to manual mode. I first de-register all components (I used the default <em>cluster1</em> for the cluster name)<br />
euca_conf --deregister-cluster cluster1<br />
euca_conf --deregister-sc cluster1<br />
and re-register with a cluster name I'm more familiar with<br />
euca_conf --register-cluster pippo 192.168.7.246<br />
euca_conf --register-sc pippo 192.168.7.246<br />
<br />
<strong>10:46</strong>: I did register the node controller<br />
euca_conf --register-nodes 192.168.4.7<br />
(yep, our network is not a /24) waited few seconds to allow the node to report and got <br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-HvxDrpyx4gQ/Tqh49mY-2XI/AAAAAAAAAD8/4XqiBj_-cTw/s1600/Screenshot+-+10262011+-+01%253A55%253A24+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="189" src="http://1.bp.blogspot.com/-HvxDrpyx4gQ/Tqh49mY-2XI/AAAAAAAAAD8/4XqiBj_-cTw/s320/Screenshot+-+10262011+-+01%253A55%253A24+PM.png" width="320" /></a></div><br />
Phew, just under half an hour, as I promised you. In a future blog I will talk about the images I uploaded on the cloud and tested it: you can check the work we are doing on our <a href="https://projects.eucalyptus.com/redmine/projects/images">projects.eucalyptus.com</a>.Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0tag:blogger.com,1999:blog-6941334177231203155.post-68362214978894934342011-10-17T15:19:00.000-07:002011-10-28T00:12:19.210-07:00Planet EucalyptusYou already know our <href="http: planet.eucalyptus.com"="">planet, since quite a few visit it regularly and requested to add feeds (<a href="http://open.eucalyptus.com/contact">let us know</a> if you want your feed on our planet). When you visit it, you will find a somewhat ordinary planet site, perhaps too plain if you will, but what I want to mention in this blog, is how we are handling it and how it is running 'in the cloud'.<br />
<br />
In the previous <a href="http://open.eucalyptus.com/blog/2011/09/19/drinking-champagne">drinking champagne</a> blog, I mentioned we have quite a few application and services in our internal production cloud (of course powered by <a href="http://open.eucalyptus.com/downloads">Eucalyptus</a>) and planet is one of them. What we did with planet, is to make it very simple to deploy and customize it: the original work was done by <a href="http://mark.atwood.name/">Mark Atwood</a>. In order to take advantage of the cloud we heavily relied on the meta-data service.<br />
<br />
The <a href="http://open.eucalyptus.com/participate/wiki/accessing-instance-metadata">meta-data service</a> allows for instances started in the cloud (private and public clouds which follows the AWS API) to learn data pertinent to the instance itself (hence it is called meta-data). Public IP, ssh keys, storage information, instance IDs (EMIs, kernel and ramdisk) are examples of what can be retrieved. All these data are accessible at http://169.254.169.254/latest/meta-data, and are easily accessible from within the instance with a browser or most likely with wget or curl.<br />
<br />
Amongst these data, the user is allowed to pass few kB of data to the instance. To do so one could use the <a href="http://open.eucalyptus.com/wiki/Euca2oolsGuide">euca2ools</a> and in particular the euca-run-instances with the -f or -d option. The instance can then access this data at http://169.254.169.254/latest/user-data: <a href="https://help.ubuntu.com/community/CloudInit">cloud-init</a> uses it to run scripts at boot time or otherwise customize the instance and the official <a href="http://aws.amazon.com/amazon-linux-ami/">Amazon Linux AMI</a> uses a port of cloud-init. Cloud-init is not yet available to all distros and earlier version of Eucalyptus suffered a bug which prevented cloud-init to work properly (instance would get delayed at boot time), so we decided to use a much simpler rc.local script to allow for a subset of its functionality. You can find information on the <a href="https://projects.eucalyptus.com/redmine/projects/recipes/repository/entry/rc.local">rc.local</a> script we use and of other information about images on <a href="http://projects.eucalyptus.com/">projects.eucalyptus.com</a>. With these starter images in our production cloud, we set to host planet in an instance. <br />
<br />
Our first attempt to run a service in the cloud, emulated the boot from EBS capability (see our <a href="http://open.eucalyptus.com/learn/first-service-in-the-cloud">issue tracker</a> in the cloud), but this time we changed completely tack. We pushed all the planet's configuration into a Walrus bucket, then we created a script to be used when starting an instance. <br />
<br />
You can inspect the script on <a href="https://projects.eucalyptus.com/redmine/projects/recipes/wiki/Wiki">projects.eucalyptus.com</a> under the Cloud Application Architect area. The script gets all the css, ini and others needed file from a walrus bucket, installs nginx and other needed packages, and set up a cron-job to re-read the configuration files at set intervals, thus allowing for the dynamic configuration of planet. When we changed the logo, it was a matter of uploading the new css and png files, and presto! Planet got a new skin. <br />
<br />
While this setup seems complicated at first glance, it is fairly easy: consider that it's all done with a few line of a shell script. It also allows for a easy failure recovery since restarting planet is a matter of 2 euca2ools command. If the instance were to fail we can issue:<br />
</href="http:><br />
<blockquote><href="http: planet.eucalyptus.com"=""> euca-run-instances -f planet.sh -k my-ssh-key emi-F3DF1488</href="http:><br />
<href="http: planet.eucalyptus.com"=""> euca-associate-address -i i-xxxxx 173.205.188.124</href="http:><br />
<href="http: planet.eucalyptus.com"=""></href="http:></blockquote><href="http: planet.eucalyptus.com"=""> <br />
and since there is no persistent data on the instance, that's all we need to do. And to apply a security update, we simply spin up a new instance with the same script (the script we use upgrades to all latest packages at start up), disassociate the public IP from the old instance and associate it to the new instance. And of course we can terminate the old planet once the elastic IP has been moved.<br />
<br />
To customize the planet we upload the new version of the specific file to the Walrus bucket, and to do so we use a version of <a href="http://173.205.188.8:8773/services/Walrus/s3curl/s3curl-euca.pl">s3curl</a> modified to allow for different endpoint then S3. For example to add a new feed we first get the current planet.ini:<br />
</href="http:><br />
<blockquote><href="http: planet.eucalyptus.com"=""> wget http://173.205.188.8:8773/services/Walrus/planet/planet.ini</href="http:><br />
<href="http: planet.eucalyptus.com"=""></href="http:></blockquote><href="http: planet.eucalyptus.com"=""> <br />
modify it, to add the new feed, and we upload it back into the bucket</href="http:><br />
<br />
<br />
<blockquote>s3curl --id graziano --acl public-read --put planet.ini -- http://173.205.188.8:8773/services/Walrus/planet/planet.ini</blockquote><br />
<blockquote><href="http: planet.eucalyptus.com"=""></href="http:></blockquote><href="http: planet.eucalyptus.com"=""> <br />
and wait for the cron-job to execute. Easy, isn't it?<br />
<br />
We have few more scripts which we use for our production services on projects.eucalyptus.com and if you have a similar script you want to share, let us know and we'll add it, or do a git-hub merge request. The recipes are ready to go, just add Eucalyptus to it.</href="http:>Grazianohttp://www.blogger.com/profile/08661090801060351957noreply@blogger.com0