Neil Carpenter's Blog

Incident Response: The Importance of Anti-Virus

neilcar — Mon, 23 Nov 2009 17:17:00 GMT

Heading home from the CSS Security Global Summit on Friday, I got stuck in Cincinnati’s airport. While walking through baggage claim, I saw this displayed on the arrivals board:

(I didn’t have a proper camera with me so, if that’s hard to read, it’s a Symantec AntiVirus Auto-Protect notification of a trojan horse found. Symantec was unable to quarantine it due to an access denied.)

Now, I suspect that this was a false positive; however, I’m just a guy walking by their wall of monitors. Even if it is a false positive, it was first detected at 4:26PM…and it was 8:30pm when I was walking by. (CORRECTION: It occurred to me, when I looked back at the picture, that this was detected on THURSDAY at 4:26PM. This was, apparently, displayed for over a day.) Even if there is no risk, I think it would have been wise for somebody to react to this faster just so that you don’t have smart-alec security guys walking by taking pictures…

This reminded me of a common thread that I’ve seen in security incidents lately: You can’t ignore your anti-virus software. Your AV software is whispering softly in your ear, telling you that something bad has happened – if you ignore it, you may go on (for months, sometimes) not knowing that something bad has happened.

What about if it’s a false positive? Do something about it. You paid for AV software and it’s an important factor in early detection of security incidents, so don’t just accept that it’s generating noise. If there’s not a clear resolution, contact your AV vendor and get them to help figure it out.

The same thing is true of errors -- “Unable to scan file X because of reason Y”, for example. Either this is a valid error that should be solved or it’s noise which shouldn’t be bubbling up to you as an error. In some cases, this is an indication of an attacker who is circumventing your AV software. If you can’t easily fix these, talk to your AV vendor and get their help in resolving them.

Another important thing, in regards to AV vendors, is their ability to react quickly in an emergency. Make sure that you know how to contact them and get samples to them if you have something rampant in your environment that isn’t caught by their current signatures. Also, be sure to understand how the vendor would deliver a new signature and how you would deploy it as quickly as possible. Include disaster scenarios such as “What if my network was completely down?” and “What if my enterprise had no Internet connectivity?” in your planning.

In general, make sure that you have an excellent relationship with your AV vendor and that they are a company that you can rely on, especially in a crisis.

AV isn’t a perfect solution but, then again, neither is any other single part of your security strategy. In some cases, it may be the difference between responding quickly to an incident and not responding until significant damage has been done.

I’ve often discussed these concepts with customers; however, two specific incidents that I’ve seen recently have made me think more about this topic.

In the first incident, a company was first penetrated by a remote-access trojan that their AV software didn’t detect; however, when I looked at each of the machines that this was installed on, I saw that the AV software had logged failures to scan various files on every machine that was compromised. (There might be a high volume of these errors in the enterprise; however, I doubt it since none of these machines had errors except where they correlated with specific malware being dropped on the system.) Additionally, some of the other tools did cause AV detections – for example, PWDump was detected on one of the compromised workstations. The attackers went on to use a different password hash dumping tool that was not detected…

Had these issues been remediated immediately, the company could have stopped the incident before it really started. Instead, the company didn’t discover the incident until months later from unrelated symptoms.

The second incident was large enough to take the company entirely off the Internet and turn off most of their Windows machines as a precaution. The company had an outbreak of a piece of malware similar to Conficker (in that it impersonated users to spread to machines that were otherwise not vulnerable) but different enough that its AV vendor didn’t have signatures that covered it. It was 50+ hours between when the initial infections were discovered and when signatures that were considered to be totally effective were deployed.

In this case, the AV vendor wasn’t a ‘trusted advisor’ to the company. The company was unable to effectively deliver on the promise of AV software to contain and remove malware at the time that it counted most to them.

SQL Injection Hijinks

neilcar — Fri, 31 Oct 2008 23:07:00 GMT

or Why I Keep Harping On Blacklisting

Summary:

An incident reveals attempts to get around blacklisting by manipulating behavior in ASP, illustrating the weakness of blacklist approaches.

A new version of UrlScan is shipping today with a change specifically to address this.

Discussion:

I was working with a colleague on an incident last week that looked like a garden-variety SQL injection drive-by except for something interesting.

While looking through the IIS logs from the affected server, I saw this:

abc=120364DEC%LARE%20@S%20VAR%CHAR(4000)%3BS%ET%20@S%...

As I looked at this, "DEC%LARE", "VAR%CHAR", and "BS%ET" immediately stood out to me. Obviously, the percent sign is usually used to escape something in a URL (like the %20's in there, which are spaces); however, this naked percent sign thrown in there didn't seem to have any purpose and should have caused SQL to not execute the code in question.

When I see somebody do something like this, it's usually for a purpose so I took another look at it. I realized that, if ASP silently stripped that percent sign out of there, then this would be an efficient way to bypass a lot of blacklist-based filters.

I wrote a quick test ASP page(1) and found that my guess was right on -- ASP drops a percent sign from the query string if it isn't followed by two valid hex characters(0-9, A-F) when it actually interprets it via Request.QueryString. This means that any filter that inspects raw headers using Request.ServerVariables is going to miss "DEC%LARE" if it is looking for "DECLARE" but, on the other hand, the ASP app that actually consumes that string using Request.QueryString("abc") is going to get it without the percent sign.

Conclusion:

As this incident illustrates, a blacklist approach to SQL injection only works for as long as nobody finds a way around your blacklist. As soon as somebody finds a way around it (and experience suggests that attackers are motivated to do so), the value of your blacklist is zero.

The right approach is to fix the actual vulnerability in the code using parameterized queries. See the articles below for more information and examples.

http://msdn.microsoft.com/en-us/library/cc676512.aspx

SQL Injection Mitigation- Using Parameterized Queries

SQL Injection Mitigation- Using Parameterized Queries part 2 (types and recordsets)

The IIS team is releasing an update to UrlScan today that includes changes to address this in their filtering product. Of course, as I've said over and and over, no filter-based approach is going to be perfect, but UrlScan is still an excellent defense-in-depth tool and a way to mitigate SQL injection vulns in the short term while your developers fix them.

For more information on the UrlScan update, Wade Hilmo has all the details:

http://blogs.iis.net/wadeh/archive/2008/10/31/urlscan-3-1.aspx

(1)

<HTML>
QUERY_STRING = <%= Request.ServerVariables("QUERY_STRING") %> <BR>
test =<%= Request.QueryString("test") %> <BR>
</HTML>

PASSGEN

neilcar — Wed, 22 Oct 2008 21:52:02 GMT

Occasionally, I see a security incident where one of the things that went wrong was that all of the customer's machines have the same password for the built-in administrator's account. Whenever this happens, I suggest the PASSGEN tool that was included with the book "Protect Your Windows Network" by Steve Riley and Jesper Johansson. Obviously, most people don't want to run to the bookstore in the middle of a security incident but, fortunately, it was available on their website.

Unfortunately, the website disappeared recently and I had to scramble around to find it. If you're looking for PASSGEN (and you should be if you have the same password for local admin across a number of machines), you can find it in two places:

Err

neilcar — Wed, 13 Aug 2008 05:10:00 GMT

I might be the last person to know this but one of my favorite internal Microsoft tools is now external. Err.exe is a command-line tool that looks up error codes and spits out possible matches from various header files. This is invaluable when you're reading through a log and run across something like "Failed, err 0x80070003" -- just run err and you'll find out what this possibly means:

C:\Users\Neilcar\Downloads\Err>err 0x80070003
# for hex 0x80070003 / decimal -2147024893 :
COR_E_DIRECTORYNOTFOUND corerror.h
# MessageText:
# The specified path couldn't be found.
# 1 matches found for "0x80070003"

The download for err is available at http://www.microsoft.com/downloads/details.aspx?familyid=be596899-7bb8-4208-b7fc-09e02a13696c&displaylang=en. Download it and extract it to somewhere in %PATH% so that it's only a command prompt away.

Input Validation Is Not The Answer

neilcar — Thu, 07 Aug 2008 21:11:00 GMT

I just sent a piece of e-mail to my team about input validation and SQL injection and it occurred to me that I've been meaning to get into this here, too:

If you're trying to solve a SQL injection problem, input validation is NOT the answer!

There, I've said it. I keep seeing blog posts, forum posts, e-mail, etc, that say "Oh, you got hax0red by SQL injection, you should have been doing input validation". I'm sorry, but y'all are wrong, wrong, wrong, wrong. Let me copy-and-paste my e-mail to explain why:

Your customer is failing to stop SQL injection because they don’t understand the problem (or, by extension, the solution).

It sounds like the customer is trying to do input validation. What input validation does is to check input coming from an untrusted user to make sure that it doesn’t contain any blacklisted characters/phrases. Depending on the implementation, it either replaces items on the blacklist with something innocuous or it blocks the input entirely.

This is the wrong way to stop SQL injection, period. Input validation is sometimes useful as part of a defense-in-depth strategy but that’s it. There are several major problems with input validation:
It only works for as long as you’re smarter than your attackers because you have to anticipate every potential attack.
It doesn’t solve your real problem, which is that SQL can potentially execute something in input you get from your untrusted user.
You can end up with a lot of false positives if you’re not careful — if you’re blocking “exec”, what happens when one of your users has the title “Executive Assistant”?
To use an analogy, using input validation to stop SQL injection is like using anti-virus software to stop malware. It might work, it might not, but you’d be far better off if you actually resolved the vulnerability instead of just trying to mitigate it.

How, then, should the customer fix their vulnerabilities? Parameterizing queries is the single best step. Instead of simply mitigating the vulnerability, this actually resolves it. At a high level, I think of parameterized queries as DEP for SQL — it separates the executable code from the data and prevents anything in the data from executing.

Bala Neeumalla wrote a great MSDN article on how to code in ASP to prevent SQL injection (http://msdn.microsoft.com/en-us/library/cc676512.aspx) that covers this in detail. His article should be considered definitive. I wrote a few blog entries (http://blogs.technet.com/neilcar/archive/2008/05/21/sql-injection-mitigation-using-parameterized-queries.aspx, http://blogs.technet.com/neilcar/archive/2008/05/23/sql-injection-mitigation-using-parameterized-queries-part-2-types-and-recordsets.aspx) that have additional examples that might be helpful to the customer.

Michael Howard also wrote a great blog entry regarding SQL injection and the SDL (http://blogs.msdn.com/sdl/archive/2008/05/15/giving-sql-injection-the-respect-it-deserves.aspx). Besides using parameterized queries, there are two additional steps that can be taken to further protect an application:
Use properly designed stored procedures
Use SQL Execute-only permission so that the application can only execute the stored procedures and cannot execute other statements.

This doesn't mean that input validation isn't useful and it doesn't mean it isn't appropriate mitigation in some cases. It's still not the way to prevent SQL injection.

And I'm not just talking about ASP, either. The same thing holds true for ASP.Net, the same thing holds true for Cold Fusion (look up CFQueryParam), Java, etc, etc. Wherever you query, there shall ye parameterize.

Forefront Server Security Management Console, Templates, and Revisions

neilcar — Fri, 11 Jul 2008 23:57:00 GMT

Sometimes, working in support, you come across a best practice or a bit of knowledge that is well-known to some people...but that bit of knowledge has never actually been documented. Today was one of those days.

While working in an environment with multiple Exchange Server 2003 servers running Antigen 9.1 Hotfix Rollup 3, we had to reinstall Antigen on one of the servers. We installed Antigen 9.1 and tested to make sure that mail was flowing after the install (it was). We then configured Antigen, including re-installing the FSSMC agent, redeploying the template for this server, and disabling Antigen performance counters.

At that point, things went off the rails. When we opened the Antigen admin console, it told us "You have 4 days left on your evaluation". Confused, we tried various things, including rebooting the box; however, every time, the console mocked us with it's eval message.

Since we were working in a maintenance window and we had run out of time, we made a decision to disable Antigen temporarily and investigate further. We took the template to a non-production Antigen 9.1 server and applied it. After applying it, we opened the admin console and we were greeted with "You have 4 days left on your evaluation".

At this point, we knew we were onto something. After working with our sustained engineering team to investigate further, we found out the root cause was something that is an FSSMC best practice even though I don't know that it's been written down before:

You should never apply a template created with a later version of Antigen or Forefront Server to an earlier version.

In this case, the template had been created in Antigen 9.1 Hotfix Rollup 3. As long as we applied it to servers running rollup 3 (or later), everything was A-OK; however, when we applied it to an Antigen 9.1 with no hotfix rollup on it, we ran into trouble.

The trouble, in this case, is that the schema for scanjobs was changed to add some additional options into the scanjobs. The template includes this new information but, once it's applied, the older code doesn't know how to handle it. This resulted in memory corruption which caused the false eval notice.

The takeaway here is that, if you're running a mixture of patch levels for your Antigen or Forefront Server servers, you have to be sure that the templates you are deploying with FSSMC were created in the earliest patch level you have in production. This will mean that you can't take advantage of any settings that are added in later patch levels but it also means that you won't run into issues like the one we wrestled with today.

Alternately, you could create templates for each patch level but I think that would end up being more difficult to manage.

Does This Make Me A Fanboy?

neilcar — Fri, 11 Jul 2008 06:21:00 GMT

I upgraded my iPhone to the 2.0 firmware today and I've been playing with the app store all day. It's pretty neat stuff.

Since I'm on a conference call tonight but I'm only here in an advisory/observational way, I put my phone on mute and kept playing with the app store. I downloaded the PhoneSaber application, which lets you pretend your phone is a light saber from Star Wars. As you move the phone around, it makes light saber sounds.

So, I amused myself with my virtual light saber for a few moments before somebody said "What is all that noise? I think we have a bad connection!"

I stopped what I was doing and thought for a long, hard moment before sending her an instant message: "Ummm, did it sound like a light saber?"

"Yes."

Antigen 9.1 Hotfix Rollup 3 and Performance Monitor

neilcar — Wed, 09 Jul 2008 22:56:30 GMT

While investigating an issue where mail was queuing in the Exchange Information Store, we discovered an issue that affects customers running Antigen 9.1 Hotfix Rollup 3 when there are performance monitoring tools such as Perfmon, Perfwiz, and the MOM client running. This issue will manifest itself as mail queuing (and never un-queueing), particularly immediately after the store is restarted. In this particular instance, we were seeing this happen when we failed from one cluster node to another. This could also occur in non-cluster environments and it could occur if scanjobs are restarted for other reasons (such as scan timeouts).

Additionally, you may see entries in ProgamLog.txt similar to the following:

"ERROR: scanjobs.cpp::ConfigScanJobFile(): AddNewScanJob() Failed 0x80030021"
"ERROR: scanjobs.cpp::CheckScanJobs(): ConfigScanJobFile() failed. hr[0x80030021]"

"ERROR: Unexpected, RetrieveScanJobIdentifier could not find the index"
"ERROR: Problems retrieving ScanJob identifier from RegisterMonitor"
"ERROR: antigenvsapi.cpp::VSAPINavigatorThread(): RegisterMonitor() returned 8000ffff"

You may also see instances where you open the Antigen Administrator console and scanjobs are not visible.

The root cause of this is a regression in the Antigen performance counters DLL that results in Antigen services being unable to access the configuration information for scanjobs; thus, when the server is in this state, scanning processes cannot be started and the admin console cannot access scanjob configuration information.

These symptoms will not occur in all instances.

Recommendations:

If a server is having this issue, you should be able to resolve the immediate issue by stopping all applications that are performing performance monitoring and restarting Exchange services.

If you are running services/applications that gather performance data on your Exchange Server with Antigen 9.1 Hotfix Rollup 3, you can mitigate this in the short-term by disabling Antigen performance counters. The following steps will disable those counters:

At c:\program files\microsoft antigen for exchange\
Enter command: antigenpmsetup -uninstall
You will also have to restart any application that loads performance counters. Rebooting the server will accomplish this; however, short of that, you can run 'tlist -m antigenpmdll.dll' to get a list. (Tlist is part of the debuggers package.)

This will be resolved in Rollup 4 when it is released. After Rollup 4 is available, we recommend re-enabling Antigen performance counters by running 'antigenpmsetup -install'.

SQL Storm: Possible ASP.Net

neilcar — Thu, 05 Jun 2008 00:13:59 GMT

I’ve had an unconfirmed report that the SQL Storm attacks are now also affecting ASP.Net pages, specifically with a URL of http://www.chliyi.com/m.js (this appears to be offline currently but I wouldn't suggest browsing there...) being injected into those pages. My team hasn’t worked on any incidents yet so I can’t confirm that it is the same issue; however, it certainly looks very similar.

This is a good time for me to remind everybody that Microsoft does provide no-cost support in the case of a security incident. If you’ve been affected, you can call 1-866-PCSAFETY in the United States & Canada. Outside of that area, refer to http://support.microsoft.com/common/international.aspx?rdpath=4 to find the right contact information.

(Thanks to Erwin Geirnaert for the heads-up.)

SQL Injection: Trends & Guidance

neilcar — Fri, 30 May 2008 19:17:11 GMT

I've been working with the SWI team to write a comprehensive overview of the SQL Storm attacks with guidance for IT administrators, developers, and end users. That article is posted at sql-injection-attack.aspx.

For developers, specifically, Bala Neerumalla has written an excellent overview of SQL injection and classic ASP code for MSDN at cc676512.aspx. This is well worth a read for any developer who has legacy ASP code running -- it covers a variety of scenarios and how to resolve them.