First of all, before I really start, I hope that you all had a great start in 2009. Mine was actually pretty mixed. The good side was, how my year really started and what I saw when I looked out the window at January 1st (yes, I was on vacation skiing and this was how the view was almost each and every morning):
But honestly, this is not the only reason, why I wrote this post. There is another one which is much, much more serious:
Unfortunately there are still plenty of customers playing Russian Roulette with their network. This term was actually used by one of our security engineers – who was kind of upset to say the least – who had to work December 31st and January 1st because of customers still not having rolled out MS08-067 – and not just one! We ran to our limits with regards to support capacity in EMEA.
Just to remind you: This is the Out of Band security update we released back on October 23rd and which then was pretty soon attacked by Conficker.A. But it seems that a lot of customer did not care back then – they were not attacked, so why bother? In the last days of 2008 Conficker.B broke out and even though it was not spread too widely, the customers who were hit (or still are hit) are hit very, very badly. Account Lockouts all over the place, admin passwords that were grabbed (often the Domain Admins) etc – and we had some really upset engineers as they had to work instead of having off because some customers were not up to their duty (and this is what it is for me!).
And this is not the end of the story:
Let me add a final comment: The story above is not a Microsoft-only story. The same processes and technologies around patch management have to be applied to each and every component of your environment. Back after the Blaster times, we start to tell the consumer to apply three things to their PC to protect it:
Guess what: If you would have applied 2 and 3 to your network, you would not have been hit by this problem.
OMG! View is amazing....
I might just forget about work when i see this type of view everyday... :)
To be honest: We got no other choice than to play Russian Roulette with our Systems - and the Systems of our Customers.
Everytime I patch my System at work there are problems I'm facing when trying to get back to work. So I'm in delay with patches about half a year.
Our customers about 1 year or more. With the last update of our Software we shipped a Link to the official MS Security Rollup for MS Win2k SP4 - we were facing following problems afterwards:
1) Customers were just not sane what to do and how this would affect the system. Well no problem so far, tech-support resolved it.
2) System hung in boot-loops not only single systems, but the most. We had to work on this for weeks, til every System ran normal again.
3) Some Systems even crash(ed) unreasonable, we're still working on configurations here - and still means: Our update shipped middle of Novembre.
I see the relevance of security patches, and am kinda paranoid myself, so I try to hold my systems up to date - BUT: It's nearly impossible. I can't tell our customers "there's a new security patch, we faced no problems installing it, but most probably you will." And I can't tell them "There'll be no problems" since they'll make me responsible for upcoming errors.
We are not able to apply patches by time since we got no guarantee they work properly and we are in debt to get systems secure - so much for Russian Roulette - thanks a lot for calling us dumbasses.
Oh and for "Windows NT", it's a good tip to get more actual systems, but it's also a question of the possibility for the companys. We needed to cancle NT support for our software partal with the last update, since some libs are using the .NET Framework 2 SP1, the same customers which want - and NEED security called to ask if there was no way using our software on NT Boxes, since the got no time oder money to replace them - or they just don't trust 2k or XP Systems, since they are facing problems after problems on single 2k or XP boxes.
Now tell me what to tell them?
"Sure thing, there will be no problem if you change und it's your problem if you don't what do I care?" Sounds like we're gonna lose them...
Sorry, but what is about:
1. Writing a Package Management System that can cleanly resolve dependencies?
In May Debian/Ubuntu had a security hole with a similar severity (http://www.ubuntu.com/usn/usn-612-2) and all everbody had to do was to type "apt-get install openssh-client openssh-server" and the problem was fixed. Not only was it fixed, you could be 100% sure that it won't have any site effects on other services you where running on your box.
I'm using Windows since the days of memmaker and 620 KB of free RAM and applying Microsoft Hotfixes and Service Packs still feels like shutting your eyes while driving a car on a motorway.
Yes, you can do some advance checking. Yes, you don't have to close your eyes very long. But if something hits you in-between you are in real trouble.
And once you have experienced things like a "simple" service pack installation on your central Exchange Server end up roasting the device driver for your RAID controller (ever tried to uninstall a service pack from an six disk RAID-Array you can't access?). You tend to get very, very cautionary in applying any updates at all.
let me try to challenge your statements: YOu say "you just have to type...". Well, quite some customers that were (or still are) hit by Conflicker are not too big. So, the problem is not technically rolling out the update. You do not even have to touch any server but use technologies like Windows Upate Server to roll it out. People are simply ignoring the risk. So, this is not a technolgoy problem, this is a process problem.
When it comes to side-effects: I would challenge the statement that you can be sure not having any side effects. This is a change in code and it always might have side-effects. Sometimes you have side-effects because somebody uses a feature not the way they should or they use an undocumented API or, or, or. The December update was sucessfully installed approx. 480 Mio times and we just had very few cases - mostly with content question.
I think the last major problem with a security update dates back to summer 2007... which was the TCP/IP Update. Since then, we just hear a lot of rumors that somebody heard from somebody who heard from somebody that there is a problem but looking at our cases we have a different picutre.
Last but not least: You are comparing a Service Pack (for Exchange) with a security update. A Service Pack defintiely has much more impact than a security update.
But let me be clear: Touching a running system is always a risk (no matter which OS) but it has to be balanced with the risk or not touching it. If we go Out of Band, our assessment shows that there is a big risk for our customers being exploited. So, as a rule of thumb: If we go OOB, deploy it as soon as possible.
Just my 2 cents
sorry, I read your comment later... One thing has to be very, very clear: I did not call anybody a dumbass but we knwo that not patching is playing Russian Roulette with your network.
Your feedback shows fundamental problems in the industry:
- Companies do not properly maintain their network. I am not talking about Microsoft products only. You probably saw my posts earlier on the Secunia PSI. The problem is much, much bigger
- A lot of customers not even have a patch management process
- There is a lot of rumors out there what works and what not. If you have a proper process you at least take a concious decision which risk is bigger: Deploying or Not Deploying
- With regards to NT: It is absolutely careless to me if we still ahve companies selling products today with NT embedded! I do not drive a car today with the safety features from 1920. And this is what we are talking of - at least
So, I know that it is not an easy task but not maintaining the network is just ignoring a huge risk
Okay, I'm sorry for the "dumbasses" I see your point, just the "Russian Roulette" made me kinda angry, because I feel like a dumbass, if someone tells me that I'm accepting a risk like that without thinking about it.
2. It was no critic specific to MS, i just picked MS products since I'm working with them and am facing these Problems with MS products.
3. I totally agree with you but this doesn't make every customer agree with you, and that's the ppl that I have to serve. Even if everything inside me tells me I can't help them if they don't accept my help or are willing to do the simplest things to get their systems secure and/or up to date.
4. Just a few words to NT: Many customers think "Never touch a running system" and NT works fine for them. They fear to be confronted with problems they - or we can't solve when changing their system. Even now since the update I spoke of, many ppl are NOT willing to change their Boxes to XP Boxes even if we tell them it would be best.
So, question again: What should - or could - I do? I've got to wait til the last decides to change. If he won't I have to accept that he "drives a 1920s vehicle". I've gotta hope that the end users lose the fear of changing their system. But I can't do anything than wait and tell them "when applying this patch there are no problems known to me, but this doesn't mean there are none".
I see your points.
I would like to add two things.
Your point 3: This is exactly what I am talking of. Nobody (or at least not too many people) drive a car without maintaining it - as it is dangerous. But they do with their computer systems. This is the reason why I did this post and I share your frustration (and trust me: our engineers do this as well). But at the end of the day it is all about awareness and making patching easy and stable.
Your point 4: I see this point as well. Today, NT to me is one of the biggest security risks. Not because it was a bad product when we initially launched it but because it is outdated. The criminals changed heavily since we initially designed NT - and so has the technolgoy. In certain industry this "close the eyes" is live threatening as we are talking of live saving systems!
No, i will not make an update. I will be in the news.
Roger - can a PC patched with MS08-067 saftey patch still be attacked and infected by Conficker or any mutants in a company network?
Fortunately, I do not have such problems. The only notebook I use is an iBook G4.
Well, then you do not have the Conficker problem but let's not open the can of worms with regards to Apple vulnerbailities and the willingness of Apple to understand the necessity of AV-software on Apple - I would get too sarcastic openign that.
Conficker.B has several ways to spread. One of them is the MS08-067 vulnerabiltiy but there are others. If you click on the link above on Conficker you will find the entry in our encyplopedia where you find all the ways it spread
Hope this helps
Thank you, Roger,
I am just a simple user who never had any problems with viruses, worms etc using the Mac. No reason to get sarcastic or upset and I do not want to open the stupid Mac-PC discussion.
One reason for my concern - our company has just shut down the whole network and I just wanted to ask if patched PCs could also catch the worm. You answered the question - yes - only one unpatched PC in a network is enough to spread the infection in the network. Really frightening though! Kind regards
I did not get upset at all :)
But yes, it is very frightening and I do not understand the motivation behind it and whether it is pure vandalism.
Ende Oktober hat Microsoft eine außerplanmäßige Sicherheits-Aktualisierung veröffentlicht. Siehe dazu
Look, your company distributed software that has a security problem. So it has to bear the consequences and deliver the support. I would expect Microsoft to go to customers and help them to install the software as it is their fault, your company delivered broken software.
Why do customers have to roll out patches? Why do companies and European citizens have to invest their man-hours to fix what you broke! How are the compensated?
"Microsoft recommends that customers apply the update immediately."
Oh, nice attitude, go an fix your stuff as we said in bullettin message PBFX #1038478. As if it was our fault!
Who was fired and slain by your company for letting it happen? Did your company express its regret for delivering defect software? No, you rather insult your customers:
"Account Lockouts all over the place, admin passwords that were grabbed (often the Domain Admins) etc – and we had some really upset engineers as they had to work instead of having off because some customers were not up to their duty (and this is what it is for me!)."
Oh yes, you delivered defect software! You were paid for support! So don't complain! Go and fix the mess.
Andy, you must realize that Microsoft does indeed "help them to install the software". It is called Windows Update.