All too often people are prepared to blame the wrong things when dealing with the painful area of security issues and compromises. They are embarrassing, costly, painful, and severely unpleasant experiences, and a culture of recrimination, and fault finding soon arises in our attempts to understand what happened, and why. My perspective on this is very clear, having done my share of post-mortems on customer site. The blame culture is unhelpful, and responsibility should be borne by several areas of the IT function; however a few people in the organisation seldom get caught up in security issues and in my view are amongst the most responsible. The security and the infrastructure architects!
When we have a security incident, the blame usually goes first to the vendor of the vulnerability in question, unless of course it was a configuration exploit, in which case the vendor also gets it for making it too easy to configure it incorrectly, or the engineers get blamed for a “sub-standard translation of the architectural vision into an implemented infrastructure”. Vendors are nice to blame as they are external to the org, and its not a personal" attack. Two areas seldom get blamed in the post-mortem. One, the firewalls or perimeter device which allowed the vulnerability to pass through it (because it is never the security devices fault for having weak security, the application gets blamed), and the other is the architect who masterminded an infrastructure which got hacked in the first place. Both of these points are controversial, and we’ll deal with dumb firewalls in another posting.
You’re an Architect – OK answer one skill testing question
My favourite question to ask to test the understanding of the security/infrastructure architects big picture awareness is to ask them about patching. The question usually goes like this – how do you feel about patching, and its importance and relevance to IT security? I would expect several different answers depending on the level of awareness, vision, and at what point in the infrastructure hierarchy and career the candidate is at.
The Build Technicians’ (Rack Monkey J ) View
A newcomer to our industry who is working their way up the IT hierarchy, and is currently an entry level helpdesk engineer, or server build engineer would respond with something to the extent of: “ITS PAINFUL, TIMECONSUMING, and KILLS MY WEEKENDS, but its critical to our security”. This is an understandable position from someone who is on the front line of the vulnerability arms race, and quite justified from their viewpoint. He or she is paid to do that update management function, and suffers that pain. They dislike genetically the second Tuesday of every month, and know that Quarterly update cycles of their database products mean no weekend for them. Maybe to this level of experience anything to make the process faster would help, and tools like WSUS, Systems Center configurations manager, and/or other management tools would be the next step that should be taken, to reduce the time spent on such tasks. This is an entirely practical and acceptable position at this point in your career.
The Engineers View
With a few to several more years experience, product knowledge, and Technet sessions, the technician becomes an “engineer” with lots of exams passed, certifications, project experience, and product mastery. At this point Microsoft technical people tend to specialise in a given area of technology, for example messaging, DBA, or security, and start to question orders from above, made by people with a perceived incomplete understanding of the technology as opposed to theirs, but who they respect as more senior and “closer to the ”business”. To the engineer, they look at mitigation through layers of defence, the use of advanced security devices like IPS/IDS, central policy, audit, and integration of some cross product security features and infrastructures as the more relevant security platforms from which to build upon. They will for example, understand the role of ISA to protect Exchange, and see that there is a better way to do it than sticking a front-end in a DMZ. Engineers will usually tell you that “group policy” is the best security tool that Microsoft has, and will look at advanced lockdown mechanisms, and best practice whitepapers, and translate them into security implementations.
Engineers will also apply their deep knowledge in the product areas of their expertise to lock down and secure the systems as much as possible and will lead design streams in projects that carry those tasks. To them, best practice analyzers, the automation of patch management (made possible through infrastructure standardisation, and update test process, etc), and best practice whitepapers are the main weapons in their arsenal. They would answer the question of patch management as a necessary evil, and point to several upcoming technologies that would help alleviate the exploits going forward, and they shouldn’t be too religious about platform, or vendor; understanding that all require updating, and the infrastructure should facilitate and empower that. In short, their opinion is balanced, tempered by experience, and is trying to solve the problem at an engineering level. Entirely reasonable in this phase of their career.
And the Architect? First define what you mean by Architect....
After many more years of experience, and countless project meetings, (and increasingly less technology ones), the engineer should progress to become an “architect” which is a very misunderstood term. Many companies use the word architect as designator of rank. For example, a higher form of engineer, or a higher rank of IT person (the pinnacle of a technical role for example). Usually they enhance the current job title by adding the word architect to it. For example, Architectural Consultant (a higher grade of consultant) or Architectural Technical Specialist (a higher rank pre-sales engineer grade in the same job family). In which case, the word architect loses its value, or its meaning. Like for example the word “senior” when applied to consultant, or specialist, Senior is meaningless, it is a simple modifier of the standard role definition of consultant. Maybe with loosely defined higher expectations, but largely the same class of role. These are the people who I consider “rank architects” meaning, a higher paid, form of a current consultant, or IT engineer role, but they aren’t true “architects” in the sense of the word it was meant by the industry. I was a solutions architect for my first IT job, man did I ever lie well at the interview !
In the real world it is impossible to promote a building landscape engineer to become an “architect”, no matter how good they are as a landscape engineer. Architects have different skills, different points of view, and are trained differently, it’s not a rank. The studies required are different, and the qualifications are different. I would expect a rank “architect” to respond the same way as an engineer to the question of patch management, as they would answer on technology terms, though maybe less precisely as they did when they did technology for a living. Architects in the real world are paid to transform ! To envision, and to create a structure or entity that doesn’t exist, and is fit for purpose, and breaks new ground. Barcelona (where the excellent IT Forum conference was just held) is an incredibly beautiful place, with the buildings (in my view) far outshining the Mediterranean. Its
architects constantly push the limits, and do better buildings, with more beauty and functionality than just simple designs would mandate.
In IT, architects are supposed to be paid to envision the type of foundation required for an (infra)structure, and commission the engineers to work on the implementation of the vision into reality, and transform the infrastructure into what the business requires. They are intimately close to the business customer, and spend a large part of their time serving and communicating with them. This is a very different role, with a different point of view. I would expect such a position to tell me that patch management shouldn’t matter as much as it does today. Because the the system is built assuming failures or vulnerabilities in it, and that the entire structure of the design wouldn’t permit a configuration exploit to occur, because its caught in three other places (defence in depth in the best sense – not the “salesey buy my product because I can’t think of another reason” way). Just like architects in high tectonic risk zones assume earthquakes in their design, and don’t blame the manufacturer of the planet earth for rattling the ground every once in a while. We know the earth has tectonic vulnerabilities, and buildings defy gravity so there is an issue there. So we build a structure that takes this into consideration.
Demand More, and question the experts
For too long our IT Architects have neglected their transformational role, and responsibility. Either by choice, or by lack of organisational empowerment (most probable). They still deliver “engineer” level infrastructures and vision which assume the same 30-40 year old design principles that lead to specific structural security flaws. IPv4 is still the standard for networking, people still assume VLAN segmentation works, people still think DMZs are a good idea, they still expect patches to be a primary defence line (thus rendering them susceptible to zero day exploits), firewalls are still relatively dumb, and people still put encrypted traffic through security barriers without inspecting the payload, to name but a few. Kevin Sangwell, one of Microsoft’s Infrastructure Architects says it brilliantly: “some architects are afraid of breaking new ground: we know this way works and we’re tight on time so we do it this way again and again.”
This questioning or envelope pushing doesn’t occur nearly enough in most IT infrastructures, where the IT architect hides behind seniority or lack of proximity to the day to day operations when there is a security issue. Or they blame the vendor for the issue, or the IPS for not having a signature, with the classic excuses heard in post mortems. Next time, ask your architects where their vision is for transformation ? Ask them why we are dependent on patching, when the human body isn’t ! Ask them what the plan is for the transformational technologies like IPv6, NA(P/C), IdM, Dynamic Systems, etc that will secure the infrastructure holistically, rather than rely on the signatron 10,000 IPS which will keep the same architecture but just add an ever more clever tool to do some small part of a big thing. What are the client side countermeasures, and where was the code review of the application ?
Ask them where and how they used to have to patch when they were “technicians” all those years back, and when they tell you, “that was a long time ago”, and they didn’t use to have to patch, they will usually smile and say “times have changed” in a reminiscent kind of way, answer back with: “then why hasn’t our architecture ?”
It’s high time we took collective responsibility for security, not just Microsoft as a vendor or the industry as a whole, but the administrators, designers, and architects of modern systems who consume the technologies produced by the industry. We all know there are better ways to do things, and that times have changed. Vendors are working hard to set right design decisions proved wrong by history and to improve the security in their products, but we need to do the same in our deployments as well ! It’s time we work together to understand the new vision for holistic security, and architect it, design it, and implement it. I for one, believe that we will need all ranks of IT to do this, and architects should lead the charge, not hide behind legacy not enough time” / too high risk excuses. It’s no good the major powers in the IT industry adopting, changing, and improving security if the features, and architectures enabled by them aren’t used or deployed by customers. Architects should lead that vision, its their job to, and I like reminding them of that once in a while J
fred@microsoft.com for comments