Chris Boos on Automation

Kevin Timmons

A friend of mine - Kevin Timmons - won the Bintan Triathlon last weekend in his category 12 (!!!) minutes ahead of everyone else. Look at these results and bow your head. Congratulations Kevin and keep up the fun.

Cloud Computing was a big topic at this year´s EMC World in Orlando. I think it is a given that virtualization is a pre requirement for any kind of Cloud concept that can be implemented today and thus EMC is playing a vital role in the Cloud space with VMware. Following the “keynote” on Cloud Computing and virtualization on Tuesday showed quite well what EMC expects. First of all I want to mention, that I thought it a pretty neat idea to turn a keynote into a panel discussion, because that demonstrates the impact Cloud Computing has on IT - it touches every part of IT and thus every part of a major vendor like EMC gets involved.

EMC World - Cloud Keynote Discussion

The discussion clearly showed that EMC is thinking of what can be done with the Cloud today rather than proposing the overall concept and waiting for it to be technically possible. For EMC Clouds have to tackle the space of legacy applications rather than requiring the users to rewrite all their software. In my opinion this is the absolutely logical step and therefore I liked the content showing how different concepts at EMC support making today´s applications “Cloud ready”. The biggest step into this direction is VMware´s latest release of vSphere that enables outtasking of compute resources on demand while turning the hardware available internally into a resource pool that can be allocated flexibly and automatically. This is supported by resource and system management software as well as storage. It is all done by adding management capacities and predefined behavior to the virtualization capabilities already available and by bringing other components closer to the virtual world by adding direct control over hardware though interfaces to the management program driving and allocating the virtual resources. This concept creates a resource pool out of all involved components (storage, network, compute) that can be allocated dynamically. The feature that tops up the concept is the ability to externalize such services if peak loads require additional resources.

By simulating the environments we are using today and bringing this simulation into such an dynamic space a pre Cloud becomes reality very quickly. This is what can realistically be done today and this is what makes Cloud concepts available to legacy application, short term project requirements as well as test and integration environments.

For my taste the fact that the Cloud concept would require the reprogramming of all software was a little overdone. Yes, I too believe that there is no way dynamic parallelization of computation cannot be reached unless you write programs for such a kind of super dynamic scheduler (like Google does).But this is where computing is headed in the long run. To try and reimplement everything on the spot is absolutely unrealistic and therefore the concepts of bringing at least some of the benefits of Cloud concepts to today´s applications and architects is great. But to say that reimplementation can be avoided in a very long term perspective is just incorrect. I think we should have learned something from the immense cost generated by maintaining the big monolithic legacy apps we do rely on today (If you want something, you make something new because changing the old think to look new will create more cost through maintenance in the long run).

Last but not least the EMC team emphasized many times, that the VMware approach created much less dependencies for customers than giving their applications into the proprietary domains of Google App Engine (where your program only runs with the Google API) or Amazon EC2 (where the virtual machine itself is hard to retrieve once deployed). This is a valid point. And despite the hype created around Amazon EC2 or Google App Engine this addresses the fear of many business users. On the other hand one should state too that EMC as well is building features into their “Cloud OS” that make a customer “want” to use EMC hardware and other EMC preferred services. All in all EMC is doing a good job of opening up the specs and standards for these kind of dependencies enabling other providers to step into the world of VMware and be just as well integrated.

In the long run I am sure standards for Cloud machine images, templates and Cloud programming interfaces will evolve. I think this will be an evolutionary process rather than the job of a standardization committee, because the Cloud idea spreads so quickly and many many different concepts are being tried out every day. Survival of the fittest is not the worst thing to happen here.

As one should save the best for last I can say now that EMC management and engineers obviously understand the need for more effective automation technologies. The discussion returned to the point that such very dynamic infrastructures and environments simply can no longer be managed manually and that the current toolsets available in system and resource management will have to take major steps towards actually automating the maintenance process fully. So in the eyes of EMC and VMWare management and engineering the operational auto pilot discussed in this blog many times and actually implemented in the aAE (arago Automation Engine) is not just a good way for cost cutting or freeing up resources for innovation and change, but becomes an absolute necessity in a dynamic environment where the speed of change is too high to be reflected in human experience. Thus I conclude that the idea of preserving these experiences within an automation engine as described before is the best way to protect investment into these experiences.

You will definitely find more information on Chuck´s Blog and for a more day to day recap you can look at Len´s Blog

I am sitting in the blogger´s lounge at EMCWorld 2009 – a really cool idea from ZDNet. After half a day of the conference I must say I am really impressed. In his keynote Joe Tucci (CEO EMC) talked about the challenges presented by the current economic downturn and EMC´s reaction to them. Technologically they concentrate on the areas of:

1. Storage
   storage virtualization and the trend towards SSD
2. information management
   where they are moving from a content management platform or system towards an information management framework with multiple repositories
3. security
   though virtualization and HA solutions on the one side and a group of solutions around identity management and security on the other side
4. clouds
   strategy for bringing the dynamic and flexible aspects of a cloud infrastructure to legacy applications while keeping security, reliability and control at the level they are today and promoting automation as a key point in making dynamic infrastructures possible on a large scale

Paul Maritz (CEO vmware) elaborated on the latter point by giving an actual demonstration of vSphere – vmware´s brand new “cloud operating system”. Even thou I think the term cloud OS is used a little prematurely, the concepts of delivering a dynamic management solution with the virtualization solution is obviously well designed and a great next step. This “cloud OS” will automatically manage resources from a service perspective – including automated provisioning and SLA tracking. This resource management does not only include computing power as before, but now also extends towards storage (dynamically moving storage, WOW) as well as automated HA, user based environment templates and the possibility to externalize resources on demand. To me this approach and the actual availability of the solution shows how a technology driven company can harvest the fruits of a clean and diligent design process even in turbulent times while at the same time making a big contribution to its customers cost reduction scenarios. 

Joe Tucci also made a very strong commitment towards EMC remaining a technology company and their strategy within the economic downturn. To him this means

• getting closer to the customers,
• securing talent,
•  no cuts in R&D budgets,
• increase in cash reserves,
• opportunistic  M & A as was as
• strategic investment.

To me this sounds like a viable strategy of a well positioned company.
Client virtualization and automated operating were put out as the next “hot things” they will be dealing with.

And I can tell you that I found some people to talk to about operating auto pilots and automation beyond dynamic provisioning really quickly. I will be going to an engineering round table this afternoon and I will surely keep you posted.

EMC World 2009

Not just because I am a roller coaster fan (especially the Dueling Dragons in Universal Park), but mainly because I am interested in technology and innovation, you will find me at EMCWorld 2009 in Orlando (18. May - 21 May 2009). If you are around and want to strike up a chat on Clouds or automated IT operation please contact me or send me a tweet.

Last week we had an interesting lunch break with an eloquent cloud. We used this environment and especially the good atmosphere to discuss the current situation of cloud computing from a business angle. There is no recording of the actual discussion, but I have translated the slides I used as a guideline and want to share them with you.

The conclusion was very clear: Clouds have a fabulous business case, but internal restrictions (psychologically, regulatory and in company procedures) currently restrict using Cloud computing.

It was also very clear that these restrictions are likely to disintegrate over time.

At this year´s JAVA development conference (JAX 09) I had the honor and pleasure of giving the keynote presentation for the track on IT in the financial industry. As this touches many of the concepts we deal with in this blog, I want to share this presentation with you and maybe have a good discussion about:

1. Why clouds are cool, but average administrators hate them.
2. Why SOA can be a heaven or hell.
3. Why no one should think that writing code is a creative process.

Enjoy the presentation.

  Chris

UK-based start-up Phorm probably considered its software the hottest stuff in online advertisements since Google adwords. Now, Phorm finds itself at the center of a blamestorm where it gets bashed by EU regulators and web platforms such as Amazon or Wikipedia for violating privacy rights.

So what’s it all about? Phorm intends to cooperate with ISPs to monitor customer web usage. A test trial with BT has already been implemented. Based on the user profiles created, Phorm will display matching advertisements.

A short side note on the technology used: Phorm effectively creates the “mother of all cookies” by placing their cookie containing a unique user identification (UID) on every website you visit. Normally, a website can set cookies only for itself, not for other websites. Here’s the trick: The ISPs reroutes all incoming requests to a Phorm server that pretends to be the desired domain (i.e. www.amazon.com). Using this fake, Phorm is able to set its cookie and reroute again to the real desired web server.

So Phorm manipulates existing web protocols for cookie handling in a fashion that is more than dubious. “Big brother is watching You surfing” comes to mind immediately. Let’s keep bashing them, everyone.

On the other hand we should be aware that tacking users and adapting content to their profiles is nothing new. In fact, it’s a major trend in web 2.0 to create adaptive web sites for unique user experiences. We at syngenio have created EBIT 2.0, which adds user adaptive advertisement to online banking applications (without using cookies and without web tracking users).

Amazon itself constantly profiles its customers. Just the same thing as Phorm does, only limited to the www.amazon.com domain. The data is used to create the famous recommendations. But Amazon has also used it to exclude existing customers from price reductions targeted at attracting new customers. Not so nice.

So Amazon protesting against Phorm has little to do with user privacy. It’s about who owns a customer and the value hidden in customer data. Right now, it’s up to the individual web sites to make the most of it. (And many still have a long way to go … I can tell from my own discussions with the financial industry about their own web sites…) Phorm tries to bring in the ISPs. It’s as simple as that.

Which brings back an old question: Should cookies be considered harmful? It’s been so long since I last discussed that, I can’t even remember what happened to all the cookie paranoia from the early days of the web. I guess we just traded it in for nice features based on cookies. Well, what do you think? Responses welcome …

Chris Boos is attenting the IT Automation Summit on BrightTALK.com on 7th of April 14:00 CET. All webcasts will be recorded and are available for download afterwards.

Warning: This post contains just as much sarcasm as it contains serious content.

In many discussions I have founds that grasping the concept of automation is alien to most people´s mindset. Are you one of them? Do you really prefer to work your butt off doing seriously dull stuff than sitting in an arm chair with a cocktail? Or if you are not that lazy, do you really prefer mind numbing repetitive tasks to trying out thrilling new things or finding an elegant solution to a tricky problem? (Well, if you answered yes to any of these, please go and visit some soap opera or sitcom blog instead and never ask yourself why your life is soooo boring that you need to tune into life somewhere else….)

So you are still reading? Glad to meet you. I do believe that most great inventions were made because we are a lazy kind of animal. The only thing that can get us out of our laziness is something stimulating to our brains. Everything else we try to get rid of. Usually we start out with the low hanging fruits and move on to more complex problems from there. E.g. inventing the wheel meant getting rid of the need to carry everything on our backs, using many people to transport a heavy item or it meant using fewer people or animals to do the same job (ancient form of cost cutting and let us not talk about the invention of sliced bread here). An example of a more complex problem would be managing a Web Portal with 1.2 Million transactions a day that is connected to three different ERP Systems using two different SOA approaches and so on…

Corporate Culture without Automation

Are we back to the point where you say “that cannot or should not be automated”? Yes it can, and yes it should be automated, because once you know how to handle the everyday hick-ups of even this complex IT environment you become very bored with it. Well you might say, if that really is automated then the job of administrating this stuff will be gone – so what? So were the jobs of the people who used to carry the bricks to the pyramids when they all of a sudden started using wheels and carts. And guess what…. Since then the population and average wealth of people has increased greatly. And one more interesting piece of information… The people who started using the wheels right away got much richer or at least had much more fun that those “traditionalists” who said carrying bricks is supposed to be done manually. Why is that? Well because management liked to get things done quicker and cheaper… Sound familiar? Well, management has much fancier titles today than “just” pharaoh.
Well back to serious business, I guess you get the point – progress in IT administration is on its way and stopping it is not an option – especially not in the current economic situation.
So what do all these great inventions that really took work off our backs do? They conserve knowledge collected by hard work and experience and apply and reapply them. 

So conserving knowledge on how complex IT environments are managed is what we set out to dowhen developing the arago Automation Engine (aAE). Looking back at our operations we have done quite well. We are now able to handle roughly 68% of all issues coming up during the day automatically and only deal with the interesting ones manually. This is also why our administrators actually have an interesting job compared to the ones who do the same thing over and over and over again – just to keep busy.

So what do we do? We take a model of the IT environment and collect all the tiny steps necessary to keep this environment up and running at all times. These tiny steps are then generalized, so they can be applied and reapplied as needed. The big invention behind this is the algorithm that actually analyzes incoming issues and finds out which of the tiny administrative steps need to be combined in order to resolve these issues. So automated IT operation is the conservation of IT experience and knowledge as well as a fairly smart machine (not quite as cool as the wheel, but getting there) which knows how and when to apply these experiences.

PS: Downloaded and actually licensed that cartoon from www.CartoonStock.com… Really love it.

The history of delivering IT Services is certainly an evolutionary process. This is not even considering the huge evolution that has taken place in the technology available to deliver such services. The evolution in IT delivery or IT operation is more or less an evolution of tools. It began with the host operating systems where much of the software that came with the computer was only used to manage the machine itself. Skipping many steps, these tools went through the various stages of network and system management to business service management or business transaction management tools. The latter’s claim to fame is actually achieving what business service management set out to do – making IT manageable from a business point of view.

Automation Auto Pilot

Speaking abstractly all these tools are automation tools. They automate several steps of work that an IT operator, administrator or delivery manager previously had to perform manually. But they are still just tools. They make life easier for the one who is doing the job, but would you call an industrial hammer an automation tool? Therefore I think it is time to take a look into the fish tank of (IT-)tools and approaches available today and show how evolution points towards engines (not so much the tools) that actually decide what to do and then take the action autonomously – only asking for permission, reassurance or assistance if required by process or if no solution is available to them. Such an engine could be called an automation auto pilot and is sitting on top of all the tools available to IT experts today.

We have been developing and using such an engine for more than ten years now and have achieved very good results in quality improvement, availability of documentation as part of compliance and cost cutting. But why do I most strongly believe that this is not an exotic idea, but the logical next step?

If we focus on the two dimensions IT management tool that can takes actions automatically or facilitate taking complex actions on a complex IT and application landscape, we end up with a trigger axis and an approach axis. The trigger axis describes under what conditions an action or tool invocation is triggered. The approach axis describes what kind of action will be taken and how flexible these actions can be taking the trigger conditions into account.

At the left of the trigger axis (x) we place “scheduled”, in the middle “event triggered” and at the right automated. This means that a tool positioned to the far left of the trigger axis will take action at a predefined time. Tools placed in the middle will take action if certain events occur and tools to the far right will take action as they become necessary. On the approach axis we placed “standardized” at the bottom, “rationalized” in the middle and “dynamic” at the top. This means that tools that perform predefined actions without reacting to any information gathered while executing (e.g. cron scripts), would be placed on the bottom, tools following a predefined process but building branches into the process that take current conditions into account would be placed in the middle and tools that combine the best process to be taken for the given situation out of a pool of possible actions are placed on top.

Tool Classification Dimensions

Placing the tools and concepts currently on the market onto these axes will show a clear evolutionary development from a scheduled standardized batch process to an engine that combines possible actions to a solution as the situation requires. The auto pilot function that I was talking about earlier is such a tool that would be placed up and to the right on our chart of automation evolution.

In the chart presented below, the placement of “hot” topics such as data center automation, work load automation and even run book automation are much more “old school” in their approaches and are therefore placed accordingly. Our auto pilot engine clearly takes up the “new approach” position – with a very notable difference – we have been running a successful business on this model for a long time. Thus this is not a fancy idea, but a valid approach and current trends in management software are pointing to exactly this approach.

Automation Auto Pilot as Trend

Maybe this “sorting of the tools” article has helped a little to place other thoughts on automation published here. It will certainly be necessary when we look at why dynamic automation becomes more and more unavoidable as complexity and change rate increase. E.g. following the current discussions on cloud computing from the Atlanta cloud camp organized by John Willis or even the dynamically evolving enterprise clouds as described by Mark Masterson, an automation auto pilot is the only way to keep track of an IT landscape that is fully distributed and dynamic. Just solving the problem of distributed computing and dynamic resources from an OS point of view by creating good cloud managers or VMs does not solve the problem of keeping business applications alive and available with proper execution quality and correct business results. If any of you have ever configured e.g. the Tivoli Correlation Engine in an Enterprise console successfully you know how much work that is. Putting your environment in a cloud would essentially mean you would have to review all correlation rues every time the cloud manager changes your environment. Not possible you say - well that was only the correlation engine. No other system management, IT service management or business service management tool or visualization was even touched. So you see, something will have to be done in order to keep the actual delivery of business services up and running when moving to a fully dynamic environment – this something is an autonomous automation engine or an automation auto pilot.