Folks it just occured to me that I haven't formally introduced you to a colleague of mine, Tim Rains.Tim Rains is also a tech-lead on the PSS Security team and is an avid C++ coder (un-like me who despises the language). In fact Tim has a long and distinguished track record of writing a number of useful utilities over the years (some even more well known than my Autodump+ vbscript! <G>) many of which are used every day by PSS and some of which are used every day by PSS Security.
He has recently released a new tool to the web - Promqry (we've gotta work on his tool name creativity). You can read more about it here:http://www.entmag.com/news/article.asp?EditorialsID=6557
Tim also maintains his own blog located here that I highly recommend checking out:http://blogs.msdn.com/tim_rains/
In the future I'm going to try and get the other tech-leads on the PSS Security team to publish informative posts like the ones I have done on recent interesting hacking cases we've been involved in so that I do not become a single point of failure in the sharing process. :) Maybe I can convince them to start a dedicated PSS Security blog that anyone from the team can post to . . . hmmmm.
Tim is currently in the process of taking WOLF (Windows Online Forensics - our live response toolkit that we use to collect data from customers systems) to the next level with numerous improvements that only moving to compiled code can give you (it will no longer be a batch file).
As a finaly FYI before you ask - no, WOLF is not available for public download for many reasons. One of the better reasons is that we redistribute numerous 3rd party tools (with permission of course) and per the terms of our licensing agreement we are allowed to send WOLF to customers on an as-needed basis but we are not allowed to post WOLF for public download. As we continue to improve the data collection piece of our incident response process this may change in the future but right now we are not allowed to distribute WOLF broadly or post it for public download - sorry.
I just became aware of your blog by way of the unestimable Susan Bradley and I congratulate you on one of the more interesting Seurity blogs I've read recently as well as your encouragement of others to post similarly.
I have a question about WOLF which I hope you may be able to answer publicly but if not it's understandable...
I understand WOLF to be a tool to detect the existence of malware through various means.
Susan and I have been discussing a situation where it's uncertain whether the Administrator Account had been compromised on a machine with RDP enabled and whether WOLF would be an appropriate tool for that situation.
IMHO WOLF would not be able to detect a number of possible compromises which simply modify existing security rather than attempt to create ways to circumvent security... such as
- Verify access to an alternative account with Administrative privileges for later use
- Remove evidence related to a break-in, while use of a valid account in an invalid way would be difficult to detect.
- Install a rootkit. Since online forensics depends on data reported by the WinOS, how can you detect something the WinOS may not be able to report?
<If> sufficient auditing and the ability to inspect appropriate logs were possible the first two scenarios <mught> be identified, but in the first instance that might also require good human memory and/or alternative logs for comparison which could be <very> difficult to analyze.
Also, you may find a PPT presentation I created for the local UGs in my area interesting which is available on my website suggesting a better approach to password construction by not relying on possibly faulty rules but educating the User on what to avoid in a practical manner.
Ah Susan - the goddess of SBS. :)
So this is a lot to comment on via a comment and most of this would be addressed in some sort of IR training my team would provide if we provided IR training to the outside world (I'm pushing for this).
Okay to start with I think you have a misunderstanding of WOLF and its use. WOLF doesn't 'detect' anything - WOLF is a data collection agent that's actually fairly stupid - it doesn't have any AI or built-in expert system / detection etc. It just collects stuff I tell it to collect, no more, no less.
Its up to the *analyst* to do the analysis and draw conclusions about the scenarios you describe above.
1. Alternate admin accounts - these are trivial to spot using a variety of tools, many of which are automated by WOLF. We dump the accounts, their privilege level, their properties, the password policy, the event logs and all sorts of stuff that would allow us to determine whether an account with admin rights was abused. We sometimes can tell you WHICH account was abused, other times we can't. We draw these conclusions based on the evidence in the security event log if its there, or we do it emperically based on conclusions we draw from other sources. For example we may be able to see that some malware was dropped at X time and Y account was last logged in to at that time (based on the properties of the account stored in the SAM, not on event log data).
The only challenge we would face here is with rootkits that can modify the token of an account on the fly making it an admin without the account actually being in the admin group.
As for rootkits - *any* live response toolkit that is not able to detect the presence of *all* of the well known rootkits is useless. I assure you our IR toolkit has multiple tools that we've developed specifically for the purpose of detecting the presence of well known rootkits (and the way they work even allows us to detect rootkits that aren't well known either).
Thx for your further explanation.
I totally agree with you and have said that evaluating "The Big Picture" is a significant part in evaluating a situation since I've been trying to explain that it's awfully hard to prove that something hasn't happened because first it's hard to prove the absence of something that may not leave a trace and second because data is only as good as what you have available to you and its integrity.
Although because of "The Big Picture" I believe the issue Susan and I were discussing was evaluated properly, it nevertheless seems to be based "On a Preponderance of Circumstantial Evidence" and it's really not possible to "Prove Without a Doubt" because the absence of a Smoking Gun doesn't necessarily mean that the crime wasn't committed.
So, for that reason Susan and I still have a slight difference of opinion... If this wasn't a relatively insignificant box, the Cracker wasn't someone the Potential Victim knew and the cracking attempts weren't terminated by the SysAdmin, could this situation with more unknowns have warranted SB1386 notifications (notifications of all persons that their personal information stored on the box may have been acquired by an unauthorized person)?
In a high stakes situation, I don't know if simply the lack of evidence of a successful intrusion might be sufficient to say it hadn't happened. I also wonder if this question might not become more common in the future in ordinary situations as cracking over the Internet by anonymous or unidentifiable individuals becomes even more common than what is happening now.
WOLF sounds really cool, especially if there are rules to quickly correlate the data you've collected which would vastly speed up the analysis. Hope to see it one day if/when MS permits someone like me to view.
Appreciate your thoughts, opinion and insight,
It's always hard, if not impossible to prove a negative. Especially if you're like me and aren't an expert. Robert would be able to say a system hasn't been compromised far more conclusively than me, but even then there would still be a slight chance that he could of missed something for whatever reason. In the end I think you would have to look at how strong the evidence that suggests it has been compromised is, as well as your ability to find evidence that refutes it, and then determine if it requires increased monitoring.
I'm interested in this question also since even skiddies will attempt to get passwords covertly by key loggers or capturing traffic instead of trying to brute force a login. That makes it much harder to both detect and investigate. The Tao of Network Security Monitoring briefly touched on this...
So here is a random stream of thoughts, first for Tony: I'm having a hard time following but it sounds like you and Susan responded to an intrusion, but did NOT open up a case with the PSS Security team? :) I'll have to have a chat with her and remind her about what we do. :) It sounds like what you're saying is that you couldn't find any evidence of a compromise but that you weren't satisfied with this conclusion so you may have rebuilt the server based on the value of the box. I think this was a wise decision - let me tell you a bit about our process.
First we have WOLF - the data collection agent - it runs for 20 minutes to a few hours on a given machine and it collects tons of data which are placed in a .CAB file. We setup a secure workspace (HTTPS, strong 15 char password etc.) for our customers to then upload the .CAB file to. We then analyze the contents of the .cab file back here using a viewer we lovingly call Wolf Hound. The hound has many !analyze (to steal a command from the debuggers) type wizards which do the common tasks like create an 'autostart' report of all the commonly hooked autostart entry points (ASEPs). It also rebuilds the service control manager for us so we can see all the services, who they run as, the path to them etc. It also has a 'date view' to show all activity on a given date (file system activity and event log activity sorted chronologically and intermingled). It also has a kick-ass string search ability that is faster than anything I've ever seen (takes seconds to parse XXMb of data). It's an analyst tool useful to people who know what to look for, where to look and who look at output several times a day. I could give the tool to 5 people and have them all analyze the same .CAB file and get 5 different conclusions.
That said there are two kinds of cases we get:
1. Those where we are able to prove a compromise occurred (via the presence of clues in the volatile data like files on disk, registry entry remnants, user accounts, event log entries, rootkits, etc.)
2. Those where we are not able to find any evidence of an intrusion or rootkits.
Sadly scenario 1 is probably 80% of our escalations - we are very good at finding signs of an intrusion (sometimes its blatantly obvious to even the most casual observer, sometimes the miscreants are quite clever and use rootkit technology or hide in plain site through obfuscation techniques). But we also get some false positive from overly paranoid admins. Can we prove something DIDN'T happen? Of course not - you can never prove the absence of something in this field (well that's not true - I can prove event log entries have been deleted using a tool which would be proving the absence of an entry but that's the rare exceptoin to the rule) but you can prove the presence of something somewhat easily (its all about having the right tools for the job).
> Tim Rains is also [...] and is an avid C++
> coder (un-like me who despises the language).
Actually it is possible to be an avid C++ coder and despise the language concurrently. That's not much different from being an avid computer forensic expert while despising the fact that the need exists for computer forensice experts.
LOL - you are wise beyond your years my friend. :)
Actually the case Susan I were discussing was evaluated by PSS, so as expected Susan followed protocol.
I agree that at least 80% of the situations I look at are similarly fairly easy to arrive at a conclusion but I chalk these situations up to Hacker impudence/insolence, the idea that they believe their victims have to be more stupid than they ever could be so why hide themselves?
Those are the ones I don't worry too much about, if the crime is bad enough they're the ones that will be sent away easily.
The ones I am more concerned about are the ones we don't catch or can't identify and I would like to see Windows hardened sufficiently that we can have a high assurance we <know> what is happening... Minimally that would have to include
NetMon running all the time. If this is too resource intensive, then a lightweight way to do this should be created.
Default logging audit settings sufficient to capture all IDS related info.
Permissions set to make it <very> difficult to modify any important logfiles, which are preferably stored in a database, not a flat file. This goes for App logs, not just Windows logging.
If IPSec isn't enforced today, then at least some kind of alternate method should be used regularly to verify identity, especially when a suspected attack in under way (reverse ping? lookup?).
You could probably come up with a longer list than me of what Windows is <not> doing today which should be to capture relevant information and decrease the amount of guessing that goes on today.
The object is to remove unknowns. I can't be satisfied that the situation as it exists today is because nothing can be done to tighten it up.
I'm also concerned that an approach based on WOLF assumes that the Intruder is going to install malware or make adjustments hiding activity. If the Intruder "owns" the Administrator (or other valid) account, a lot can be done within that security context that looks very natural but could be very damaging to the business.
The best concealment could be operating in plain view as though the Intruder had nothing to hide if his actions look perfectly natural.
I can imagine that a lot of people asked about WOLF: when you talk of something cool, you make people interested in that, they get curious, and they would like to see it...
But even if people can't use WOLF, I would like to tell them that IMHO there's a lot of things out there (available for the masses) to start doing forensic analysis with:
Harlan Carvey was putting down a list of the best tools for this task:
There are also several CD-based projects.
"FIRE" is one of those, and it looked like a nice project, I've used it in the past - but it has been quiet and not really updated for a while now: http://fire.dmzs.com/
It is aimed at both linux AND windows forensics. IMHO the linux part is including more things than the windows counterpart, but it still can be handy.
There's an italian Incident Response project (http://www.iritaly.org) that has been using FIRE as a base (for their version 1 forensics CD), and later they changed it for Knoppix (http://www.knopper.net/knoppix/index-en.html) instead:
...even if, unfortunately for everybody who's not italian, NOT EVEN the README file is provided in english! :-(
This CD is also linux-based, but (same as FIRE) provides also a collection of "trusted" win32 binaries to perform analysis on a machine where you can't trust the executables you have.
I was actually thinking that something based on WindowsPE (http://www.microsoft.com/licensing/programs/sa/support/winpe.mspx) would also be nice to have - something similar to "ERD Commander" (http://www.winternals.com/products/repairandrecovery/index.asp?pid=ap#erdcommander2005)...
...often what works for system recovery also works for forensics! ;-)
For example, a lot of other scripts that are useful to dump system configurations are also used by Microsoft PSS when troubleshooting "standard"(=not security related) issues, and those ARE a public download:
Again - these are meant for troubleshooting, but they can (and do) help for forensics too, IMHO.
Tim hardly needs an introduction - he's a real hero!!