September 2007
Escalation Engineer
By Greg Lirette

During the past seven years I have been an escalation engineer (EE) first for Microsoft for over six of my nine year Microsoft career, and more recently Citrix Systems. At Microsoft, I worked on the operating systems and at Citrix, my focus was Presentation Server. The EE position is the highest level technical support position given for a software development company. Support Engineers and technical leads escalate extremely difficult issues to the escalation team when the skills required to work the issue far exceed the skills generally required, even of the most senior support engineers.

Generally when a support issue is escalated, the possibility exists that the resolution may be a hot fix or patched code for the current product release. The goal of the EE is to find the best resolution for the issue. The customer is often very frustrated and the business stakes are extremely high.

The EE position requires the ability to do programming in the languages and development environments that the given product is built in. The ability to debug is essential, however often the newly hired employee is not able to debug. The skill is usually acquired through mentoring and practice. The EE must be able to properly read any logging file the given product creates and be able to extend those logging capabilities by using instrumented code. Reading network captures and other low level troubleshooting is required.

People often assume that the necessary skills required to become an EE are technical and development related. This has some truth to it but equally important is customer service and business skills. The position requires constant communication with developers, support engineers, sales, consulting, managers and numerous customer contacts. The EE must take ownership of the given issue and commit to a resolution to the customer even when he has no idea yet what the fix will be.

There are generally two types of people that get hired as an EE. One type that may get hired off the streets without having moved up the ranks of support, is a low level developer such as one that develops device drivers. The challenges that this type of new hire face is that they often lack the customer exposure and product knowledge. Looking at a problem at a low level is often great if you can define it in those terms. Often the problem is not clearly articulated in a way that the skills of a low level developer can debug and get to the root cause. Learning the product and troubleshooting is a must! Be sure you focus on your breath of knowledge rather than your depth. The other type is a support engineer or technical lead whom wants to advance but still remain technical. In this case the largest challenges are usually the programming and deep troubleshooting aspects of the position.

Regardless of your background, I cannot stress enough how important it is to become familiar with using the debugger WinDbg. This is both a kernel and user mode symbolic debugger. Get familiar with call stacks and how to set and modify breakpoints. Equally important to knowing the debugger, become familiar with network captures using tools like the Microsoft Network Monitor.

For non developers, I suggest learning C and become familiar with the concepts of C++. It is important that you also be able to read assembly and understand what it is doing in the context of disassembling code. You should feel comfortable looking at multiple processes in the debugger and debugging hung systems.

The culture in this position is typically very laid back with your team members. Team members often turn to each other for advice and many issues are worked in cooperation with your team. Teams that escalate to your team typically take great care in presenting the issue correctly so that they do not waste your valuable time. Your advice to support is taken very seriously and support is typically highly valued then advice. When you file a formal bug to development, you take great care in preparing your report so that the issue is well defined and you may often make suggested code changes.

Many companies want to know your skill set and recruiters often make you sound better than what you are. The people interviewing you have seen some of the world's most difficult issues. Trying to wow the interviewers may work to your disadvantage unless you have some really good horror stories. In the interview and on the job, it is important to focus on how well you can find information and resolve them. A downfall is that you generally focus on one or more products so over time your skills may suffer. If you can get hired into this position and have the drive, I highly recommend it!
If you would like to submit an article providing an insider's view of your tech or engineering job, please send your article to us at ITtrenches@dice.com.

Comments on this article? Share your feedback on our discussion forum, Dice Discussions.

*Please note, you must be a registered job seeker in order to submit your question to Dice Discussions.