Chris Boos on Automation

You hear the story over and over again: Business asks IT to design and build yet another custom application. IT resources are limited, so the IT department subcontracts the job to some external software company. Meanwhile, the IT department itself takes care of the existing applications. You think that’s just the way it is? Think again.
What makes the difference between internal and external IT personnel? That your internal staff knows your company better, I’d say. They know more about the way things are run at your place. There are always lots of things that business will forget to write down in the specification – simply because they are taken for granted. Fortunately, your internal IT staff probably considers them as basic standards as well – so they will build them into the application anyway. Not true for external software companies. So take some of your own IT staff, mix them with some externals of the latest technological flavours, stir – and you got yourself a winning team.
Which leaves you with the question how to maintain and improve your existing applications with some internal staff set aside to get that new development project right. Actually, there is an obvious way to do that – just the same way you always did it: subcontract to some external software company. Not any software company, of course. One that specializes in application management. By application management I mean that special flavour of outsourcing, where the outsourcer takes care of a custom application, its maintenance, incident and problem management at a fixed rate. You probably wonder if that’s going to work. After all, your custom applications are not easy to maintain (and no wonder that is – if you let external software companies develop them without internals to guide …). Believe me: It can be done. More than that: you would be surprised how short the time to break even is. Of course, the outsourcer needs to review the application and build up knowledge. And this leads to a fresh view on the software, one focused on maintenance efficiency. Nine times out of ten, this is the first time your application has been scrutinized for maintenance efficiency. Including but not limited to the identification of potentials for automation in incident and problem management. Which is why I am telling you about it in this blog. In my next entry, I will share my thoughts on how to select a matching application management provider. Of course, you can always shortcut the waiting time and contact me at info@syngenio.de

Elmar Borgmeier

Everyone knows the big four auditors (Deloitte, Ernst&Young, KPMG, PWC) for a long time. In recent times, while our favourite topic becomes more important from week to week, there are the IT Service management big four aka BMC, CA, HP and IBM (alphabetical order!).

As Erin Joyce on enterprisestorageforum.com describes <here> EMC is integrating two solutions they acquired in recent years (Smarts in 2005 and Voyence 2007). Voyence Control is a solution for Automated Configuration and Change Management, while EMC Smarts is a solution for fault isolation and root cause analysis. In combination with the in 2006 acquired nLayers solution for Application Discovery, which is now labelled as EMCs ADM (Application Discovery Manager), it looks like they added ITSM and BSM (Business Service Management) to their interesting product portfolio.

So if I were Cisco and had 45bn$ to spend, I definitly would take a deep look into EMC. I guess together with Cisco, we would see the Big Five.

Roland

This spring I took part in a course of horsemanship where Uwe Jourdain did his very best to teach riders who spend most of their time riding underneath the sky and our horses the art of circus riding. I really enjoyed the course and learned a lot from Uwe.

So I was glad to get involved when Mr. Jourdain asked me, if I would be willing to support a group of young girls who practice a very interesting form of riding. Just with a rope around the horse´s neck they perform all the “tricks” most of us struggle hard to get done with saddle and bridle. The “Brave Summer Riders” as these girls call themselves are coached by Uwe Jourdain and are supported by their parents. I really enjoy seeing such close partnership between riders and their horses and this is why I gladly support these brave riders (not just in summer). Tonight I received the first pictures from the team´s captain Christin Turck and wanted to share them with you right away.

I started out this weekend with a fierce discussion on graph theory on Saturday. Surely I gave more ground than I normally would have, as I spent Friday night at an Irish birthday party. Well, this discussion is about how far it went for work this weekend.

The rest of the weekend I had great fun with my family. We spent our weekend in Feuchtwangen (Franken – that special part of Bavaria) and Augsburg. Besides tasting the great local food (and beer) we had the pleasure of taking a very broad cultural trip.

Saturday evening we saw Shakespeare´s “As you like it” in Feuchtwangen. As we attend the theatre festival (Kreuzgangspiele) in this beautiful city almost every year, our expectations were high and they were met to the fullest. The cast, directing and the translation were really great. The people organizing and playing this performance really brought honor to that famous quote from “As you like it”:

All the world's a stage,
And all the men and women merely players;
They have their exits and their entrances,
And one man in his time plays many parts,
His acts being seven ages.

On Sunday we jumped from the ironic but highly decorated world of Shakespeare back to the childhood days. We visited the “Augsburger Puppenkiste” (probably Europe´s most famous puppet playing company) and saw one of the classical fairytales “der gestiefelte Kater”. The play was done absolutely beautiful and gives truth to the saying that you have lost your childhood when you start seeing the puppet strings while watching the play.

So you can see. Besides the great feeling of meeting up with the family we did work on our education and on staying young and educated.

Not just for all of us who have to deal with day-to-day operation of IT the topic of automation seems to be of great interest. Naturally the interest of people maintaining systems and services becomes the interest of vendors. I had the pleasure of attending the HP BTO Talk in Frankfurt and was glad to find out, that automation itself is the main focus of HP´s system management efforts.

For the first time since HP acquired OpsWare in 2007 I was actually able to see the platform in a customer environment. Swisscom attended the event and demonstrated their efforts in network automation. More impressive was the presentation of Mr. Rossa from Wien IT, who was able to show how standard changes and standard procedures in provisioning were captured into the automation suite.

I have seen more complex provisioning environments but in the HP presentation on the OpsWare platform I could get a glimpse at the visualization and reporting offered behind the scene. Coming up from the network layer they really found a very intuitive way to show what is actually available and going on in an IT infrastructure.

The strategic presentation offered by Mr. Winkler from HP put forward automation as the key to the HP software strategy. I consider this corporate understanding to be a major advantage in market development – much more than all the thousand features us techies like to talk about every day. So in my opinion HP´s view of the future is absolutely correct:

Good IT operation is, when you see nothing of it

I was a little amazed to see that the actual automation of operational tasks as well as tasks dealing with incidents and problems are still in a fairly basic state. All the cute things we have been talking about in this blog are still in the vision only. Simple rules and actions can be applied but that is all. Compared to the field of automated deployment, standard changes and predefined tasks the automated reaction to upcoming problems is not in an advanced stage. Even though there obviously is a really fancy interface for cross platform command execution. This interface could actually be hooked up to an automation engine like aAE and voila, commands would go out to the world. I actually think we will give this a try.

All in all I have to say that the visualization is impressive and the strategic alignment of the softwarestack is convincing. I will keep a close eye on the things happening around there – even though integration all the new acquisitions may still take some time.

Well, what a very basic question… Should we be discussing automation engines, when we should not have trust in them automatically taking action? Surely not, and obviously we are discussing automation engines.

So why do I hear so much about the lack of trust towards automated actions? It may be a stunning change in the field of system administration, that some entity takes automatic action where normally a system administrator would have typed in a couple of commands up to now. And change always induces fear and prejudice. Questions like “do you really trust the engine to restart this business critical service?” are not really uncommon. Well why should the machine not do that? After all the only action a system administrator would have taken is to restart the whole machine instead of just the service?

This simple every day example shows the real problem: Trust

We seem to have a problem when faced with the necessity to trust a machine or some lower level of reactive “intelligence”. Maybe this is just due to the many science fiction books we have read on robots and machines gone mad. In the end we are the ones who gave the engine the rule set by which it acts.

Actually we trust in automation every day we step into a lift. Much more than that, we rely on hard wired automation when we breathe or when our heart beats. I think none of us would be too happy about the idea of having to think and act out every breath and heartbeat consciously and willingly. Not much difference in automated actions in IT administration - and just like you can hold your breath automated actions can be overridden at any time.

This sounds very logical, doesn´t it? But logic is not the drink for “unsinkable rubber ducks” (the term true believer nowadays it too closely connected to politics - and besides much less enjoyable). So a good argument usually does not help much. In order to get on with automation either management uses force or try to employ man´s oldest habit - laziness (maybe we could get entangled in a discussion on greed or laziness being around first). And do not get me wrong, great things like the wheel were invented because of laziness. And on the way, we build trust towards automation in a non intrusive way - i.e. everyone involved can discover for himself that automation helps and is not evil. So this is how it is done:

1. Setup the automation engine in full
2. Disable all automated commands and redirect them to a trouble ticket or service management tool.
3. Have administrators use this tool and hence make them see what the engine would have done.
4. After a while people will start to copy and past the commands form the trouble ticket or service management tool into the various command lines.
5. This is the time to enable automatic command execution. The connection to the service management or trouble ticket system stays as it is. So the commands executed are not in any way “block boxed”.
6. There will not be mistrust and all the discussions, bad feelings and politics attached to it.

The worlds-famous retiree Bill Gates annouced some weeks ago, that in the future Microsoft will have many millions of servers in their datacenters, which first of all is a very impressive proposition. Nicolas carr has some thoughts on Bills speech <HERE>.

If you break down the numbers - Microsoft claims to add between 10.000 to 20.000 servers to their infrastructure each month - and read datacenter newsletters like <THIS> and <THAT>, you soon will realize, that this definitely will become true rather sooner than later. Sure, MS has to because we might be right before a huge shift of paradigm in the software industy - moving every type of application and data from your destop straight to some datacenter somewhere (hi Google) . Thus making Microsofts main business modell looking kind of old-fashioned.

If you bring things together and look at “Live Mesh” and “Office Live” you will understand, where the world will be in 3 to 5 years from now.

And, I’ll promise you: Microsoft is paying attention to the enterprise market much better than Google and Apple together.

Roland

Good music surely is a matter of taste. BUT some things are just such classics that I do not hesitate to declare them “simply good”. I consider myself a lucky guy. Last week I had the pleasure of seeing Mark Knopfler in Cologne, Bon Jovi in Frankfurt and Alanis Morissette in Frankfurt as well. All three concerts were a real experience and even though there were quite some problems with sound quality I would definitely go again to all the events.

The Mark Knopfler concert was really very cozy; drinks served at your seat and ALL of the old Dire Straits classics. My favorite “the telegraph lane” was played in an XXL version and I really loved it.

Bon Jovi was the concert with the worst sound – Frankfurt stadium with the roof closed seems to be a really bummer for music – but with the best atmosphere. 40.000 people really had a good time. Next to me – right in the middle of the crowd – was a 70-year-old who knew all the lyrics and on the other side a teenager (in love with John). It was a really amazing party there.

Last but not least Alanis Morissette was performing in Frankfurt´s old opera house. What a location for rocky music. She really had a hard time to get the crowed off their seats (yes really everybody was seated) but it eventually happened and the Frankfurt Opera staff had some very new experiences. She played some new and very good songs.

We are all moved by compliance issues. Mainly storage vendors, consultants and auditors are having a feast. For most corporations introducing the new rules is quite a drain on resources. Besides this, changes in the working processes are the main cause for discomfort in the workforce and management of the entities affected by the rules.

Automation actually solves one big problem compliance poses on IT operation. However it may also make an old one reappear.

So let us take a look at the good news first. One demand often posed by auditors and clearly stated in all new compliance rule sets is, that all actions and the reasoning behind taking the actions should be well documented and archived. In a normal working environment this usually means getting on the case of everybody and forcing them to type explanations of what they did into some documentation system after the system has behaved like big brother and logged the technical parts of the doing. This can become tedious and does not have much positive effect on day-to-day business. So most explanations in these systems look like ‘fixed the ABC problem’ and the reasoning part is lost forever. This is where an automation engine really helps. An automation engine will document each action it takes, archive the data and the rules that have caused the action to be taken and reveal the planned next steps and all related actions and problems. So there is one big relief for everybody working on or auditing IT operations. Great, isn’t it?

The second topic is the way roles and rights are managed along compliance rule sets. In the dark ages, there was a super user (many administrators are still worshippers of this creed). According to the new rules one administrator can have the rights to perform manipulations on exactly the entities he is attached to. A database administrator for example should only be able to talk to his database and if he needs some different system settings, because his database requires more semaphores he will have to create a change request to the OS administrators. At least that is how it works in theory or whenever administrators want to slow each other down dramatically. I think the intention of the new rules is clear and unarguable: One human should only be able to have influence on the direct area he is dedicated to. Everything else can produce unpredictable risks and should thus be avoided. All fine and good and most corporations (at least the larger ones) have implemented ‘the admin silo view’ by using simple mechanisms like ’sudo’ or more complicated rights management systems.  Upon inserting an automation engine in this environment any administrator who can create a rule that is reusable could lead to command executions outside the rule author’s area of competence.

Well one would argue that is exactly what we want. We want to reuse the expert knowledge of someone who solved a problem in different environments. Auditors probably would say ‘no this is exactly what we do not want’….. A big dilemma?

I do not really think so. And I do think that we really want the knowledge to be distributed and here is why:

1. The ones who are writing rules are experts. Like the export we call in, when we really cannot find the cause of or remedy for a problem.
2. The guy who wrote the rule will always be identifiable from the engines point of view and that was the original intent of the compliance rules (make sure we know what was done by whom and where).
3. One could restrict rule attachment by group signatures and the like (additional parameter in the IT model) to create peace and quiet, but should one really dismiss the power of implicit rules if every action and its originator is well documented? (Maybe someone really into the field of compliance could answer this question for me???).

So all in all automation may cause some auditors or process consultants some headaches, but heck - this is what they are paid for, isn´t it? On the other hand an automation engine produced well formed documentation and reasoning for the auditors, which is something that any kind of silo restriction on the human workforce cannot guarantee.

As discussed before, automation in IT operations definitely has a strong social impact. It is a question of how IT professionals deal with the change that will make the difference in the end.

As I spent most of last week at an American University, I obviously had quite some discussions on how automation impacts the lives of IT administrators. There seems to be a lot of personal discomfort (understandably). Unfortunately these personal issues get mixed up with the technical ones. Many people have asked me questions like “do you trust the machine to stop a service, restart a machine or even allocate resources dynamically?” Well, yes I do. I have trusted my system for quite some time to allocate memory and disk space for me and so have you and we are trusting computer programs to land planes, control elevators and life support systems in an ER. So why – WHY – should we not trust a machine to do something radical like rebooting a server?

In my opinion a machine has two major advantages over a human administrator in standard situations. First it never executes radical commands due to “gut feeling” (like boot feels good) and second it documents the path it took to reach to conclusion that executing specific commands is a good idea. So you do have documentation (hello to all you SOX consultants out there) and if there really is an error you know where to look and you will be able to change you rule set accordingly.

Ok, so maybe we can solve the problem of trust through logical argument. Unfortunately some people are very much resistant to logic. So another approach we sometimes take is to do a dry run. That means, we install the automation engine and disable all execution and redirect the execution command to document everything it would do into a trouble-ticket. As soon as administrators start pasting commands out of the tickets you know it is time to enable the real automation.

But let us get down to the actual administrators and the consequences all that automation has on them. There is this geek shirt “Go away, or I will replace you with a very small shell script”. By the way, the guy in the picture is actually one of our administrators - one of the guys who really DO automation. I think the shirt was done to scare off users. But nowadays this is actually what will happen to administrators who do not want to be part of this changing world. In my vision of the future there will only be two kinds of administrative staff close to a data center: Real IT experts (the Gurus) and janitors. The experts are today´s administrators who want to get rid of all the boring – I have done that about 10.000 times – tasks and deal with the exciting stuff instead. Well the others …..

To get it straight: I actually do not think that there will be fewer jobs in IT administration in the future, mainly because IT is an ever growing plant. I do think that there will be a lot less “boring” and unqualified work in IT – as we have seen in all other industries. Before.

So, is that really a bad thing? More exciting tasks, more real results, more happy administrators? I don´t think so… Let´s get it on guys