Chris Boos on Automation

Talking and thinking about automation so much can be like sitting in a forest and not seeing the wood from the trees. Only recently I have discovered that some of the ideas we are already applying at arago are a step beyond what is generally called IT automation. Therefore I want to give a clear picture of what is possible compared to what is widely known to be top of the pops automation technology.

Automation Tools for Stressed out Admins

As you have seen in previous posts, there are many buzzwords describing automation technology and they are all more or less cool and more or less useful approaches towards making the process of maintaining ever more dynamic IT and application landscapes. But this is not where it stops. All of these tools just take a part of the work process, wrap it into a nice user interface and hopefully standardized configuration. None of these tools actually does the maintenance work, takes the necessary decisions or finds solutions to upcoming problems or adequate answers to imminent questions. But that is what we want, isn´t it? So what we want is something to automatically use all these automation tools to “just do the job”, something to automate the automation.
The best way to explain these different automation tools and their application in IT is comparing IT maintenance to flying an aircraft. In order to keep the plane up and running there are many different tools and technologies that automate the actual flying process. There are also many systems that automate the task of executing all the commands that originate from the flight support systems. These commands are transported to the actual aircraft mechanics where changes in the wing positions, thrust, flaps etc are executed.  All these systems themselves automate manual tasks. A pilot flying a modern aircraft no longer has to manually move parts but uses automation tools to do the job for him. Still he is flying the plane. For tedious standard situations our pilot has another tool, called the autopilot. This system does the job of the pilot in such standard situations. The autopilot uses all the other automation tools aboard the aircraft to perform the task of keeping the plane up in the skies. Theoretically the autopilot would only call for assistance when it cannot cope with the situation at hand and that is when the pilot has to step in.

Autopilot Approach to IT Maintenance

It is exactly the same in IT. With all the automation tools around, you should use an auto pilot that can handle all kinds of standard situations when managing incidents, problems, changes, IT capacity and overall availability. At the core of this auto pilot for IT operations is an autopilot engine. A large set of possible actions is stored within this engine. The job of the engine is to combine and recombine these possible actions to resolve any upcoming issues automatically. Only when it encounters a situation it cannot resolve after applying possible actions should it contact the IT experts and ask for their assistance.
This approach changes an IT expert from someone who has all kinds of good automation tools at his fingertips but is constantly battered and chased by important and urgent issues to an expert who is contacted only when his expertise is required.

003366;">This auto pilot approach minimizes the probability of human error (which is constantly high in IT operations, as there is always more than one task that needs attention in a normal environment), guarantees short reaction time, relieves the IT experts of tedious standard tasks and give them time to concentrate on important and interesting issues. An autopilot in IT operations pushes the job of an IT expert up the value chain and improves service quality at the same time.

Cloud Computing was a big topic at this year´s EMC World in Orlando. I think it is a given that virtualization is a pre requirement for any kind of Cloud concept that can be implemented today and thus EMC is playing a vital role in the Cloud space with VMware. Following the “keynote” on Cloud Computing and virtualization on Tuesday showed quite well what EMC expects. First of all I want to mention, that I thought it a pretty neat idea to turn a keynote into a panel discussion, because that demonstrates the impact Cloud Computing has on IT – it touches every part of IT and thus every part of a major vendor like EMC gets involved.

EMC World - Cloud Keynote Discussion

The discussion clearly showed that EMC is thinking of what can be done with the Cloud today rather than proposing the overall concept and waiting for it to be technically possible. For EMC Clouds have to tackle the space of legacy applications rather than requiring the users to rewrite all their software. In my opinion this is the absolutely logical step and therefore I liked the content showing how different concepts at EMC support making today´s applications “Cloud ready”. The biggest step into this direction is VMware´s latest release of vSphere that enables outtasking of compute resources on demand while turning the hardware available internally into a resource pool that can be allocated flexibly and automatically. This is supported by resource and system management software as well as storage. It is all done by adding management capacities and predefined behavior to the virtualization capabilities already available and by bringing other components closer to the virtual world by adding direct control over hardware though interfaces to the management program driving and allocating the virtual resources. This concept creates a resource pool out of all involved components (storage, network, compute) that can be allocated dynamically. The feature that tops up the concept is the ability to externalize such services if peak loads require additional resources.

By simulating the environments we are using today and bringing this simulation into such an dynamic space a pre Cloud becomes reality very quickly. This is what can realistically be done today and this is what makes Cloud concepts available to legacy application, short term project requirements as well as test and integration environments.

For my taste the fact that the Cloud concept would require the reprogramming of all software was a little overdone. Yes, I too believe that there is no way dynamic parallelization of computation cannot be reached unless you write programs for such a kind of super dynamic scheduler (like Google does).But this is where computing is headed in the long run. To try and reimplement everything on the spot is absolutely unrealistic and therefore the concepts of bringing at least some of the benefits of Cloud concepts to today´s applications and architects is great. But to say that reimplementation can be avoided in a very long term perspective is just incorrect. I think we should have learned something from the immense cost generated by maintaining the big monolithic legacy apps we do rely on today (If you want something, you make something new because changing the old think to look new will create more cost through maintenance in the long run).

Last but not least the EMC team emphasized many times, that the VMware approach created much less dependencies for customers than giving their applications into the proprietary domains of Google App Engine (where your program only runs with the Google API) or Amazon EC2 (where the virtual machine itself is hard to retrieve once deployed). This is a valid point. And despite the hype created around Amazon EC2 or Google App Engine this addresses the fear of many business users. On the other hand one should state too that EMC as well is building features into their “Cloud OS” that make a customer “want” to use EMC hardware and other EMC preferred services. All in all EMC is doing a good job of opening up the specs and standards for these kind of dependencies enabling other providers to step into the world of VMware and be just as well integrated.

In the long run I am sure standards for Cloud machine images, templates and Cloud programming interfaces will evolve. I think this will be an evolutionary process rather than the job of a standardization committee, because the Cloud idea spreads so quickly and many many different concepts are being tried out every day. Survival of the fittest is not the worst thing to happen here.

As one should save the best for last I can say now that EMC management and engineers obviously understand the need for more effective automation technologies. The discussion returned to the point that such very dynamic infrastructures and environments simply can no longer be managed manually and that the current toolsets available in system and resource management will have to take major steps towards actually automating the maintenance process fully. So in the eyes of EMC and VMWare management and engineering the operational auto pilot discussed in this blog many times and actually implemented in the aAE (arago Automation Engine) is not just a good way for cost cutting or freeing up resources for innovation and change, but becomes an absolute necessity in a dynamic environment where the speed of change is too high to be reflected in human experience. Thus I conclude that the idea of preserving these experiences within an automation engine as described before is the best way to protect investment into these experiences.

You will definitely find more information on Chuck´s Blog and for a more day to day recap you can look at Len´s Blog

I am sitting in the blogger´s lounge at EMCWorld 2009 – a really cool idea from ZDNet. After half a day of the conference I must say I am really impressed. In his keynote Joe Tucci (CEO EMC) talked about the challenges presented by the current economic downturn and EMC´s reaction to them. Technologically they concentrate on the areas of:

1. 000080;">Storage
   storage virtualization and the trend towards SSD
2. 000080;">information management
   where they are moving from a content management platform or system towards an information management framework with multiple repositories
3. 000080;">security
   though virtualization and HA solutions on the one side and a group of solutions around identity management and security on the other side
4. 000080;">clouds
   strategy for bringing the dynamic and flexible aspects of a cloud infrastructure to legacy applications while keeping security, reliability and control at the level they are today and promoting automation as a key point in making dynamic infrastructures possible on a large scale

Paul Maritz (CEO vmware) elaborated on the latter point by giving an actual demonstration of vSphere – vmware´s brand new “cloud operating system”. Even thou I think the term cloud OS is used a little prematurely, the concepts of delivering a dynamic management solution with the virtualization solution is obviously well designed and a great next step. This “cloud OS” will automatically manage resources from a service perspective – including automated provisioning and SLA tracking. This resource management does not only include computing power as before, but now also extends towards storage (dynamically moving storage, WOW) as well as automated HA, user based environment templates and the possibility to externalize resources on demand. To me this approach and the actual availability of the solution shows how a technology driven company can harvest the fruits of a clean and diligent design process even in turbulent times while at the same time making a big contribution to its customers cost reduction scenarios. 

Joe Tucci also made a very strong commitment towards EMC remaining a technology company and their strategy within the economic downturn. To him this means

• getting closer to the customers,
• securing talent,
•  no cuts in R&D budgets,
• increase in cash reserves,
• opportunistic  M & A as was as
• strategic investment.

To me this sounds like a viable strategy of a well positioned company.
Client virtualization and automated operating were put out as the next “hot things” they will be dealing with.

And I can tell you that I found some people to talk to about operating auto pilots and automation beyond dynamic provisioning really quickly. I will be going to an engineering round table this afternoon and I will surely keep you posted.

EMC World 2009

Not just because I am a roller coaster fan (especially the Dueling Dragons in Universal Park), but mainly because I am interested in technology and innovation, you will find me at EMCWorld 2009 in Orlando (18. May – 21 May 2009). If you are around and want to strike up a chat on Clouds or automated IT operation please contact me or send me a tweet.

Last week we had an interesting lunch break with an eloquent cloud. We used this environment and especially the good atmosphere to discuss the current situation of cloud computing from a business angle. There is no recording of the actual discussion, but I have translated the slides I used as a guideline and want to share them with you.

The conclusion was very clear: Clouds have a fabulous business case, but internal restrictions (psychologically, regulatory and in company procedures) currently restrict using Cloud computing.

It was also very clear that these restrictions are likely to disintegrate over time.

At this year´s JAVA development conference (JAX 09) I had the honor and pleasure of giving the keynote presentation for the track on IT in the financial industry. As this touches many of the concepts we deal with in this blog, I want to share this presentation with you and maybe have a good discussion about:

1. Why clouds are cool, but average administrators hate them.
2. Why SOA can be a heaven or hell.
3. Why no one should think that writing code is a creative process.

Enjoy the presentation.

  Chris

000080;">Warning: This post contains just as much sarcasm as it contains serious content.

In many discussions I have founds that grasping the concept of automation is alien to most people´s mindset. Are you one of them? Do you really prefer to work your butt off doing seriously dull stuff than sitting in an arm chair with a cocktail? Or if you are not that lazy, do you really prefer mind numbing repetitive tasks to trying out thrilling new things or finding an elegant solution to a tricky problem? (Well, if you answered yes to any of these, please go and visit some soap opera or sitcom blog instead and never ask yourself why your life is soooo boring that you need to tune into life somewhere else….)

So you are still reading? 000080;">Glad to meet you. I do believe that most great inventions were made because we are a lazy kind of animal. The only thing that can get us out of our laziness is something stimulating to our brains. Everything else we try to get rid of. Usually we start out with the low hanging fruits and move on to more complex problems from there. E.g. inventing the wheel meant getting rid of the need to carry everything on our backs, using many people to transport a heavy item or it meant using fewer people or animals to do the same job (ancient form of cost cutting and let us not talk about the invention of sliced bread here). An example of a more complex problem would be managing a Web Portal with 1.2 Million transactions a day that is connected to three different ERP Systems using two different SOA approaches and so on…

Corporate Culture without Automation

Are we back to the point where you say “that cannot or should not be automated”? Yes it can, and yes it should be automated, because once you know how to handle the everyday hick-ups of even this complex IT environment you become very bored with it. Well you might say, if that really is automated then the job of administrating this stuff will be gone – so what? So were the jobs of the people who used to carry the bricks to the pyramids when they all of a sudden started using wheels and carts. And guess what…. Since then the population and average wealth of people has increased greatly. And one more interesting piece of information… The people who started using the wheels right away got much richer or at least had much more fun that those “traditionalists” who said carrying bricks is supposed to be done manually. Why is that? Well because management liked to get things done quicker and cheaper… Sound familiar? Well, management has much fancier titles today than “just” pharaoh.
Well back to serious business, I guess you get the point – progress in IT administration is on its way and stopping it is not an option – especially not in the current economic situation.
So what do all these great inventions that really took work off our backs do? They conserve knowledge collected by hard work and experience and apply and reapply them. 

So conserving knowledge on how complex IT environments are managed is what we set out to dowhen developing the arago Automation Engine (ff0000;">aAE). Looking back at our operations we have done quite well. We are now able to handle roughly 68% of all issues coming up during the day automatically and only deal with the interesting ones manually. This is also why our administrators actually have an interesting job compared to the ones who do the same thing over and over and over again – just to keep busy.

So what do we do? We take a model of the IT environment and collect all the tiny steps necessary to keep this environment up and running at all times. These tiny steps are then generalized, so they can be applied and reapplied as needed. The big invention behind this is the algorithm that actually analyzes incoming issues and finds out which of the tiny administrative steps need to be combined in order to resolve these issues. So automated IT operation is the conservation of IT experience and knowledge as well as a fairly smart machine (not quite as cool as the wheel, but getting there) which knows how and when to apply these experiences.

PS: Downloaded and actually licensed that cartoon from www.CartoonStock.com… Really love it.

IBM PULSE is definitely the event for ITSM. Compared to other major vendors IBM encourages critics to join in and really tries to get discussion going on this conference. This makes it an event I definitely left with a lot of lessons learned.

This said and me being an attendee of PULSE 2008 in Orlando, I want to share some personal insight comparing these two exceptional events. In 2009 the economic crises and its palpable and coming impact on the IT industry are definitely looming over the event and have influenced it quite a bit. The most obvious change is the choice in examples in ALL IBM and external keynotes. All the examples came from the industries we expect to be least influenced by the economic crises. I.e. case studies these days focus on utilities (especially energy and energy grid), food and medical services.

PULSE 2009 Magic on the Stage

The most stunning change however is the conference bringing IBM Tivoli back to its roots. The hot topic this year is dynamic infrastructure (nice article by Timothy Prickett Morgan). Even though this is a good marketing entry to get people interested in IBM´s approaches to cloud computing and most likely will positively influence sales of Tivoli Provisioning Manager TPM it also makes sure that even in keynotes talk about events, monitoring and system management are frequent and quite technical. Last year the hot topic was to hammer home the key drivers for modern ITSM solutions from an IBM point of view (visibility, control, automation) which was a fairly new and definitely innovative message. These are still around but now serve as a driver to promoting basic services. I, being a “fairly” technical person, enjoyed that shift in topic, because even though there is a lot of marketing and sales feeling around any conference of this kind, IBM PULSE 2009 managed to actually show most of the things speakers were on about rather than just brainwashing attendees to adopt new language and terminology.

Also coming from Germany, where serious issues concerning the job market are never addressed directly, I was positively surprised, that cost cutting, job loss and a growing competition between IT experts was directly addressed by keynote and session speakers. Actually discussing these topics with the affected people seems much less like sticking your head in the sand than anything that is going on back home.

PULSE 2009 Live Demo

Besides all the talk about events, event management, new versions and many many technical sessions for actual learning there were a lot of interesting announcements. Most of them went around the topic of cloud computing and/or dynamic infrastructure. For IBM the first step in cloud computing is mostly IT on demand (as John Willis and Michael Coté said in their first day wrap up video posted as a special edition of the “IT Management Podcast” that term was not bad either) and is now called dynamic infrastructure. I will come back to my interpretation of IBM´s positioning in the cloud market in a separate post shortly.

So the current situation brought forth some very honest words on the IT market, a lot of back to the roots talk about infrastructure, events, process management and managing your IT properly. Pulling the whole techie talk back up into an IT management and business perspective was quite hard for the keynote speakers. Luckily IBM has Doug McClure around to tie the whole technical perspective up to the business view not just on a tool level. Doug seemed to be a little torn on the possibility of actually getting BSM done in a large scale enterprise in the last couple of month and published quite a bit on BTM as an alternative – articles I really enjoyed, because they were really straight forward. But in his PULSE 2009 reviews (overall, day 1, day 2 and BSM special) he is back to connecting all the IT gadgets in ITSM to the real world of profit and service quality.

So my overall recap of IBM PULSE 2009 in the shadow of the economic crises is:

1.       Dynamic infrastructure can prevail as it really offers IT on demand and IBM is actually able to deliver and made that quite clear on the conference.

2.       The strategy is much back to the roots, talking a lot about system management, event consolidation and the like with BSM pulling these topics back onto a business level.

3.       IBM strategists are obviously very closely following economic development, which can be seen from all the examples picked from utilities, food and medical services.

4.       Automation is a key topic in all tracks (event consolidation, dynamic IT, service management, …) and getting rid of swivel chair interfaces via automation solutions is actually in the crosshairs now.

I wish you all a happy new year. This may sound hollow as the upcoming year is starting out with immeasurable uncertainties. A recession is unavoidable as the economic mechanisms are working their way through the different economic sectors and into everyday life. Given the origin of this recession – the financial industry with capital being one of three pillars of our economic system– even systematic change may be in store. The greatest problem is decisions being held back due to these uncertainties thereby creating an even greater economical impact. Thus what we definitely are feeling as a crisis is a powerful well of change. This well will flood through economy, society and of course technology. We will need strong decision makers and innovators – real entrepreneurs – to embrace change and make use of its power to tackle some of the grand challenges built up during the last 50 years.

For those of us promoting new technologies the willingness to embrace change is often the biggest obstacle in putting these new technologies to use. Think about the argument of how cloud computing cannot be a good thing because it changes the relationship between our data and our computations we are so much used to. Or think about bringing the concept of automatic system operation to the administrators who will no longer be just operators but turn into system experts. All these high tech concepts require a dramatically changed way of approaching everyday problems and those of us implementing these new technologies know that inventing the technology is less than 50% of the way. The biggest challenge is attracting enough interest in all players the new technology touches, in order to make them embrace the required change to effectively make use of the new technology. The current situation may prove to be one of the most potent accelerators for technological change possible. So to all of you – those who invent, implement, decide upon or just make use of new technologies – make wise, well thought of and brave decisions embracing change. You will be the ones who will contribute towards a speedy way out of the current uncertain situation.

After giving you so much leeway ( ) by posting a few personal stories from the past summer to past autumn we are all back to business and I want to share some of the reading and thinking that I have done during the quiet time between Christmas and New Year´s Eve in the articles coming up this week. I will start out with a little catching up on the “clouds are bad discussion” started by Richard Stallman with an interview given to the Guardian in September 2008. I do believe there was a good deal of stubbornness and corporate mistrust behind condemning the cloud concept as you will read. I will then continue with a post on integrating the concept of automation – rather than just tools – into IT operation processes and tool infrastructure. After you have read Roland´s post on “Automating What?” in November you may be interested in how the concept of automation is integrated into everyday IT service management and how our concept of e.g. an automated incident management is incorporated into a working IT environment. Following this post I will try to show a landscape of technology and tools and the way the ongoing development is focusing in on automation as a concept. This process was started when tools were used to ease the manual process of maintaining system functionality (e.g. system management tools) and continued by the automation tools that enable complex changes to be performed by entering a simple command (e.g. change automation or run book automation tools). The process is now at a point where actually decisions are taken by the automation software (e.g. what hardware is used to do what tasks by which is decided by workload automation tools) and will finally come to tools that make use of all the experience of system administrators in order to automatically decide how to keep systems alive. Thus automating incident- problem- capacity- and availability management. This kind of tool is what we have been using and developing for quite some time now ( see the aAE) and the post will show how this kind of tool integrates with the whole landscape of tasks and tools involved in IT service management.

Yes I know, is that special season, where everyone is looking for some charity money and as usual most causes are more annoying than worthy. Most “welfare money” given as foreign aid does not do much good – as many publications and analysis show. Well, maybe it at least makes most doners feel great. If you really want to help someone to become independent and not just help to keep them in their place, supporting entrepreneurial approaches is the only way to go in a world governed by markets.

Some years ago I found kiva.org a micro finance portal on the internet. I have “lent” and “donated” money all of this time now and have found that 90% of all business financed grow and even report back to their original financiers. So far 100% of all money has been repaid. Personally I do believe – and this should be common knowledge at least after Muhammad Yunus won the 2006 Nobel Prize for setting up Grameen Bank as a micro finance bank. With kiva.org anyone can join in on making the world a better place and pushing entrepreneurs where they are needed most. Is it not much better to fund a fishing-net for a local fisherman than to pour your donations into some anonymous pot where projects like the Aswan Dam are created?

If you feel like supporting entrepreneurs (not much money is required to buy a frying pan in Rawanda) take a look at my lender page at kiva.org and join…