Found yourself in the position of wanting to find out what is going on on the server when you are crawling ? How about cranking up the logging level ? Nice but if you do it for all , you will rapidly get big log files that will not be so easy to analyze …. Then how about "verbose -ing " the crawl...