This is a very useful tool that is available on codeplex - http://www.codeplex.com/ExportCrawlLog
It aims to give administrators the ability to export the Crawl log (formerly known as the Gatherer log) and is installed as a stsadm extension. Although you can see the crawl log in SSP Administration by going to Search Administration, Crawl log, there is no simple way of exporting it. This extension will give you the ability to export the crawl log and manipulate the output by host and/or date and time.
How do I install it?
Download the .zip archive form the location above and extract it. Run the deploySTSADMExportCrawlLog.cmd command. This may fail and report problems "adding assembly to the cache". If this is the case first use explorer to open up C:\WINDOWS\assembly. Next go to the ..\bin\Release folder and drag and drop the STSADMExportCrawlLog.dll into C:\WINDOWS\assembly. Check that under the ..\Common Files\Microsoft Shared\web server extensions\12\CONFIG folder there is a stsadmcommands.STSADMExportCrawlLog.xml file present.
How do I use it?
Start a command prompt and path to the BIN folder. You will now be able to use the extension in the following way :
stsadm -o ExportCrawlLog -t (d|s|c|p) -site <portal url> [-outfile filename] [-history] [-s startdatetime] [-e enddatetime] [-from #] [-thru #] [-cat (Portal_Content|ProfileImport)] [-csid #] [-msgid #] [-u url pattern] [-hostname hostname] [-mt (s|w|e)] where -t = type: d=details, s=summary, c=content sources and start addresses where -mt = Message type to include in output (applies to -t d only, if -mt is used only one message type can be specified, when -mt is omitted all types are emitted): s=success, w=warnings, e=errors where -csid = the # of the content source, which you can spy in the query string of the url of the Content Source edit page.
Here's an example :
You can see the end of the export above. The command I used is also shown, this gave me a detailed export of a crawl I did on a site on www.microsoft.com (the warnings are expected). Obviously you can use the -outfile switch to save the log for further analysis.
The Enterprise Search Team have an article describing the extension in more detail, see it at http://blogs.msdn.com/enterprisesearch/archive/2008/05/26/introducing-the-exportcrawllog-stsadm-command-extension.aspx