Chris Boos on Automation

Talking and thinking about automation so much can be like sitting in a forest and not seeing the wood from the trees. Only recently I have discovered that some of the ideas we are already applying at arago are a step beyond what is generally called IT automation. Therefore I want to give a clear picture of what is possible compared to what is widely known to be top of the pops automation technology.

Automation Tools for Stressed out Admins

As you have seen in previous posts, there are many buzzwords describing automation technology and they are all more or less cool and more or less useful approaches towards making the process of maintaining ever more dynamic IT and application landscapes. But this is not where it stops. All of these tools just take a part of the work process, wrap it into a nice user interface and hopefully standardized configuration. None of these tools actually does the maintenance work, takes the necessary decisions or finds solutions to upcoming problems or adequate answers to imminent questions. But that is what we want, isn´t it? So what we want is something to automatically use all these automation tools to “just do the job”, something to automate the automation.
The best way to explain these different automation tools and their application in IT is comparing IT maintenance to flying an aircraft. In order to keep the plane up and running there are many different tools and technologies that automate the actual flying process. There are also many systems that automate the task of executing all the commands that originate from the flight support systems. These commands are transported to the actual aircraft mechanics where changes in the wing positions, thrust, flaps etc are executed.  All these systems themselves automate manual tasks. A pilot flying a modern aircraft no longer has to manually move parts but uses automation tools to do the job for him. Still he is flying the plane. For tedious standard situations our pilot has another tool, called the autopilot. This system does the job of the pilot in such standard situations. The autopilot uses all the other automation tools aboard the aircraft to perform the task of keeping the plane up in the skies. Theoretically the autopilot would only call for assistance when it cannot cope with the situation at hand and that is when the pilot has to step in.

Autopilot Approach to IT Maintenance

It is exactly the same in IT. With all the automation tools around, you should use an auto pilot that can handle all kinds of standard situations when managing incidents, problems, changes, IT capacity and overall availability. At the core of this auto pilot for IT operations is an autopilot engine. A large set of possible actions is stored within this engine. The job of the engine is to combine and recombine these possible actions to resolve any upcoming issues automatically. Only when it encounters a situation it cannot resolve after applying possible actions should it contact the IT experts and ask for their assistance.
This approach changes an IT expert from someone who has all kinds of good automation tools at his fingertips but is constantly battered and chased by important and urgent issues to an expert who is contacted only when his expertise is required.

003366;">This auto pilot approach minimizes the probability of human error (which is constantly high in IT operations, as there is always more than one task that needs attention in a normal environment), guarantees short reaction time, relieves the IT experts of tedious standard tasks and give them time to concentrate on important and interesting issues. An autopilot in IT operations pushes the job of an IT expert up the value chain and improves service quality at the same time.

000080;">Warning: This post contains just as much sarcasm as it contains serious content.

In many discussions I have founds that grasping the concept of automation is alien to most people´s mindset. Are you one of them? Do you really prefer to work your butt off doing seriously dull stuff than sitting in an arm chair with a cocktail? Or if you are not that lazy, do you really prefer mind numbing repetitive tasks to trying out thrilling new things or finding an elegant solution to a tricky problem? (Well, if you answered yes to any of these, please go and visit some soap opera or sitcom blog instead and never ask yourself why your life is soooo boring that you need to tune into life somewhere else….)

So you are still reading? 000080;">Glad to meet you. I do believe that most great inventions were made because we are a lazy kind of animal. The only thing that can get us out of our laziness is something stimulating to our brains. Everything else we try to get rid of. Usually we start out with the low hanging fruits and move on to more complex problems from there. E.g. inventing the wheel meant getting rid of the need to carry everything on our backs, using many people to transport a heavy item or it meant using fewer people or animals to do the same job (ancient form of cost cutting and let us not talk about the invention of sliced bread here). An example of a more complex problem would be managing a Web Portal with 1.2 Million transactions a day that is connected to three different ERP Systems using two different SOA approaches and so on…

Corporate Culture without Automation

Are we back to the point where you say “that cannot or should not be automated”? Yes it can, and yes it should be automated, because once you know how to handle the everyday hick-ups of even this complex IT environment you become very bored with it. Well you might say, if that really is automated then the job of administrating this stuff will be gone – so what? So were the jobs of the people who used to carry the bricks to the pyramids when they all of a sudden started using wheels and carts. And guess what…. Since then the population and average wealth of people has increased greatly. And one more interesting piece of information… The people who started using the wheels right away got much richer or at least had much more fun that those “traditionalists” who said carrying bricks is supposed to be done manually. Why is that? Well because management liked to get things done quicker and cheaper… Sound familiar? Well, management has much fancier titles today than “just” pharaoh.
Well back to serious business, I guess you get the point – progress in IT administration is on its way and stopping it is not an option – especially not in the current economic situation.
So what do all these great inventions that really took work off our backs do? They conserve knowledge collected by hard work and experience and apply and reapply them. 

So conserving knowledge on how complex IT environments are managed is what we set out to dowhen developing the arago Automation Engine (ff0000;">aAE). Looking back at our operations we have done quite well. We are now able to handle roughly 68% of all issues coming up during the day automatically and only deal with the interesting ones manually. This is also why our administrators actually have an interesting job compared to the ones who do the same thing over and over and over again – just to keep busy.

So what do we do? We take a model of the IT environment and collect all the tiny steps necessary to keep this environment up and running at all times. These tiny steps are then generalized, so they can be applied and reapplied as needed. The big invention behind this is the algorithm that actually analyzes incoming issues and finds out which of the tiny administrative steps need to be combined in order to resolve these issues. So automated IT operation is the conservation of IT experience and knowledge as well as a fairly smart machine (not quite as cool as the wheel, but getting there) which knows how and when to apply these experiences.

PS: Downloaded and actually licensed that cartoon from www.CartoonStock.com… Really love it.

I wish you all a happy new year. This may sound hollow as the upcoming year is starting out with immeasurable uncertainties. A recession is unavoidable as the economic mechanisms are working their way through the different economic sectors and into everyday life. Given the origin of this recession – the financial industry with capital being one of three pillars of our economic system– even systematic change may be in store. The greatest problem is decisions being held back due to these uncertainties thereby creating an even greater economical impact. Thus what we definitely are feeling as a crisis is a powerful well of change. This well will flood through economy, society and of course technology. We will need strong decision makers and innovators – real entrepreneurs – to embrace change and make use of its power to tackle some of the grand challenges built up during the last 50 years.

For those of us promoting new technologies the willingness to embrace change is often the biggest obstacle in putting these new technologies to use. Think about the argument of how cloud computing cannot be a good thing because it changes the relationship between our data and our computations we are so much used to. Or think about bringing the concept of automatic system operation to the administrators who will no longer be just operators but turn into system experts. All these high tech concepts require a dramatically changed way of approaching everyday problems and those of us implementing these new technologies know that inventing the technology is less than 50% of the way. The biggest challenge is attracting enough interest in all players the new technology touches, in order to make them embrace the required change to effectively make use of the new technology. The current situation may prove to be one of the most potent accelerators for technological change possible. So to all of you – those who invent, implement, decide upon or just make use of new technologies – make wise, well thought of and brave decisions embracing change. You will be the ones who will contribute towards a speedy way out of the current uncertain situation.

After giving you so much leeway ( ) by posting a few personal stories from the past summer to past autumn we are all back to business and I want to share some of the reading and thinking that I have done during the quiet time between Christmas and New Year´s Eve in the articles coming up this week. I will start out with a little catching up on the “clouds are bad discussion” started by Richard Stallman with an interview given to the Guardian in September 2008. I do believe there was a good deal of stubbornness and corporate mistrust behind condemning the cloud concept as you will read. I will then continue with a post on integrating the concept of automation – rather than just tools – into IT operation processes and tool infrastructure. After you have read Roland´s post on “Automating What?” in November you may be interested in how the concept of automation is integrated into everyday IT service management and how our concept of e.g. an automated incident management is incorporated into a working IT environment. Following this post I will try to show a landscape of technology and tools and the way the ongoing development is focusing in on automation as a concept. This process was started when tools were used to ease the manual process of maintaining system functionality (e.g. system management tools) and continued by the automation tools that enable complex changes to be performed by entering a simple command (e.g. change automation or run book automation tools). The process is now at a point where actually decisions are taken by the automation software (e.g. what hardware is used to do what tasks by which is decided by workload automation tools) and will finally come to tools that make use of all the experience of system administrators in order to automatically decide how to keep systems alive. Thus automating incident- problem- capacity- and availability management. This kind of tool is what we have been using and developing for quite some time now ( see the aAE) and the post will show how this kind of tool integrates with the whole landscape of tasks and tools involved in IT service management.

Surfing our intranet I was totally surprised to find one of our administrators – Thomas “thommy” Neuderth – writing about his first contact with automation. I am really happy that one of the best IT experts I had the pleasure of working with has found himself having “no fear of being automated away” and rather interprets automation as a good way to actually live the life of an “IT expert” instead of being an “IT nanny”.

The automation of a simple task like archiving logfiles obviously convinced a “real techie” that there is more than just a little upside to using an automation engine. Of course the implementation of automation actually forced quite a bit of rethinking the common ways of administration and “thommy” describes the skepticism the first contact and the actual adoption of change in a down to earth way. If you are interested, you may read the whole document here.

Well, what a very basic question… Should we be discussing automation engines, when we should not have trust in them automatically taking action? Surely not, and obviously we are discussing automation engines.

So why do I hear so much about the lack of trust towards automated actions? It may be a stunning change in the field of system administration, that some entity takes automatic action where normally a system administrator would have typed in a couple of commands up to now. And change always induces fear and prejudice. Questions like “do you really trust the engine to restart this business critical service?” are not really uncommon. Well why should the machine not do that? After all the only action a system administrator would have taken is to restart the whole machine instead of just the service?

This simple every day example shows the real problem: Trust

We seem to have a problem when faced with the necessity to trust a machine or some lower level of reactive “intelligence”. Maybe this is just due to the many science fiction books we have read on robots and machines gone mad. In the end we are the ones who gave the engine the rule set by which it acts.

Actually we trust in automation every day we step into a lift. Much more than that, we rely on hard wired automation when we breathe or when our heart beats. I think none of us would be too happy about the idea of having to think and act out every breath and heartbeat consciously and willingly. Not much difference in automated actions in IT administration – and just like you can hold your breath automated actions can be overridden at any time.

This sounds very logical, doesn´t it? But logic is not the drink for “unsinkable rubber ducks” (the term true believer nowadays it too closely connected to politics – and besides much less enjoyable). So a good argument usually does not help much. In order to get on with automation either management uses force or try to employ man´s oldest habit – laziness (maybe we could get entangled in a discussion on greed or laziness being around first). And do not get me wrong, great things like the wheel were invented because of laziness. And on the way, we build trust towards automation in a non intrusive way – i.e. everyone involved can discover for himself that automation helps and is not evil. So this is how it is done:

1. Setup the automation engine in full
2. Disable all automated commands and redirect them to a trouble ticket or service management tool.
3. Have administrators use this tool and hence make them see what the engine would have done.
4. After a while people will start to copy and past the commands form the trouble ticket or service management tool into the various command lines.
5. This is the time to enable automatic command execution. The connection to the service management or trouble ticket system stays as it is. So the commands executed are not in any way “block boxed”.
6. There will not be mistrust and all the discussions, bad feelings and politics attached to it.

As discussed before, automation in IT operations definitely has a strong social impact. It is a question of how IT professionals deal with the change that will make the difference in the end.

As I spent most of last week at an American University, I obviously had quite some discussions on how automation impacts the lives of IT administrators. There seems to be a lot of personal discomfort (understandably). Unfortunately these personal issues get mixed up with the technical ones. Many people have asked me questions like “do you trust the machine to stop a service, restart a machine or even allocate resources dynamically?” Well, yes I do. I have trusted my system for quite some time to allocate memory and disk space for me and so have you and we are trusting computer programs to land planes, control elevators and life support systems in an ER. So why – WHY – should we not trust a machine to do something radical like rebooting a server?

In my opinion a machine has two major advantages over a human administrator in standard situations. First it never executes radical commands due to “gut feeling” (like boot feels good) and second it documents the path it took to reach to conclusion that executing specific commands is a good idea. So you do have documentation (hello to all you SOX consultants out there) and if there really is an error you know where to look and you will be able to change you rule set accordingly.

Ok, so maybe we can solve the problem of trust through logical argument. Unfortunately some people are very much resistant to logic. So another approach we sometimes take is to do a dry run. That means, we install the automation engine and disable all execution and redirect the execution command to document everything it would do into a trouble-ticket. As soon as administrators start pasting commands out of the tickets you know it is time to enable the real automation.

But let us get down to the actual administrators and the consequences all that automation has on them. There is this geek shirt “Go away, or I will replace you with a very small shell script”. By the way, the guy in the picture is actually one of our administrators – one of the guys who really DO automation. I think the shirt was done to scare off users. But nowadays this is actually what will happen to administrators who do not want to be part of this changing world. In my vision of the future there will only be two kinds of administrative staff close to a data center: Real IT experts (the Gurus) and janitors. The experts are today´s administrators who want to get rid of all the boring – I have done that about 10.000 times – tasks and deal with the exciting stuff instead. Well the others …..

To get it straight: I actually do not think that there will be fewer jobs in IT administration in the future, mainly because IT is an ever growing plant. I do think that there will be a lot less “boring” and unqualified work in IT – as we have seen in all other industries. Before.

So, is that really a bad thing? More exciting tasks, more real results, more happy administrators? I don´t think so… Let´s get it on guys

I know it. So does does Dave Bartlett @ IBM. <SHORT_PEEK>

I wish Dave had given more information about their showcase.

But wait, doesn’t this showcase sound familiar to me? Yes, I knew it. The “Microsoft Home of the future” <READ_HERE>. Interesting. These ‘home’ thing is a good idea to create a personal relationship between the customers/partners and the vendors technology and research.

Roland

Often automating IT is handled as an obscure Art. Maybe some regard it as the black magic of the 21^st century. When I don’t understand things, I tend to divide and then conquer them, so in this case why black and why magic? Maybe black, because automation is regarded as something evil by quite a few IT people. Good techies could lose their jobs or at least their “God” status, when automation actually works. And maybe magic, because automation is clear to us viewed on a single system – i.e. things you didn’t want to do manually are put into a script and voila the system does them automatically – but in a large IT environment, all of a sudden things seem to happen by themselves.

But let me tell you, IT automation is neither black, nor magic. It is not magic, because after all it can be broken down just to that simple script example above. So if you divide the automation of a large IT environment you will in the end arrive at one – or maybe more – scripts being executed under certain conditions. So the question – I guess we will be talking about that in a little while – is which script or scripts to execute under what condition. And automation is not black, because “people losing their jobs or their current status” is nothing evil but the way our world works. Change is the driving force of everything and anybody trying to position himself against the power of change will definitely loose in the long term. So I would recommend embracing the ideas of automation rather than putting it down there with devils and demons – and by the way we do have enough of the latter around in IT anyway.

Automation in IT is about the last part on the IT landscape not under constant renewal – and this will come to an end soon.

It is about time technology becomes part of the change process…

IT has gained tremendous influence in everyday life and “we” – the IT crowd – are proud of “our” technology changing the world. But to put it out bluntly, IT itself has not changed that much and it is about time to pick up the pace.

Sure marketing has improved very much but most good concepts in IT have a very long life (e.g. virtual machines). On the other hand many IT trends are focused on an oscillating movement between polar architectural concepts e.g. host vs. PC vs. Client Server vs. Web vs. Web Services, vs. Web 2.0 and so on. Still the way the technical community handles IT did not change much. You may say that there have been a lot of changes all the while and that is certainly true. But these changes were mostly focused on development and creating cool interfaces. How about the concepts, techniques and social patterns applying to the operational environment? How about those of us who keep “IT” running? Not much change detected here….

Either we change or we are changed (away?)…

Do you know anyone (not from an IT profession) who is really happy with the way the IT around him works? Do you know people who believe their system administrator to be the greatest guy since Frank Sinatra? No, or why else would we need “Sysadmin appreciation day”? It seems that the people in and around IT operation are simply detached from the rest of the world. As IT becomes an integral part of “the rest of the world” just this world will not accept to depend on totally alienated concepts and people.

In IT it is still en vogue to be “god” of the system. Would anyone in any other industry accept the fact that the guy who runs the assembly line claims to be “god” of the place? No way and if there were such a guy – well in a positive environment he would be set up for counseling and in a bald world he would simply be sacked. So let´s face it, IT has worked its way into everyday life and cannot be alien anymore. Change is in the air. And if user complaints cannot do the job, controllers certainly will. Cost cutting initiatives have reduced IT operating budgets drastically. Still, just maintaining the status quo swallows up roughly 70% of all available IT budgets.

On the other hand, the guys I am just writing about are being bored to tears with everyday work and keep their sanity by building fancy tools just for themselves. Still for that rare occasion, when sh*t really hits the fan we are all happy these people are around, people who really know their way and understand the system. What an incredible squandering of talent, creativity and know how….

… a glimpse at the future

So finally we have arrived at a point where the way things are handled will be changed. IT operation no longer is a question of finding a cool new tool to sit in front of, but it is a question of having IT maintenance controlled by tools and processes – just like the work of so many other productive and creative people is controlled by machine driven processes.

In all other industries this is called automation and it is catching up with us. So if we want to be the gurus, we have to be part of the change process, we have to drive it and we have to find the technologies to enable the automation of IT operation and maintenance – after we are content with the fact that change is a good (and unavoidable) development.