I've been gathering performance counter data all week on a server and found a really long disk latency issue where disk writes were taking longer than 1 second (should be less than 15ms). The odd part is that the server was very responsive and the typical symptoms just weren't there. I even wrote my own tool to measure disk write times which showed less than 1ms responses.
A long story short, the cause was really a calculation drift issue on certain AMD processor computers. The tell tell sign of this issue is when your ping responses are negative. Like this:
Reply from 220.127.116.11: bytes=32 time=-2695ms TTL=53 Reply from 18.104.22.168: bytes=32 time=-2697ms TTL=52 Reply from 22.214.171.124: bytes=32 time=-2759ms TTL=52 Reply from 126.96.36.199: bytes=32 time=-2694ms TTL=53
According to the knowledge base article 938448, the fix is simple:
Add the /usepmtimer parameter to the Boot.ini file, and then restart the server. The /usepmtimer parameter is automatically added to the Boot.ini file when you install the latest AMD PowerNow! Technology driver from AMD. The updated driver itself does not resolve this problem. However, the installation process makes the necessary changes to the Boot.ini file to resolve this problem.
Here is the KB article link:
A Windows Server 2003-based server may experience time-stamp counter drift if the server uses dual-core AMD Opteron processors or multiprocessor AMD Opteron processors http://support.microsoft.com/kb/938448/en-us
Anyway, I just wanted to let everyone know that this is a very *real* issue and if you apply the boot.ini setting, then it will save you a whole lot of time and effort. Believe me I wasted a week chasing ghosts on this until one of the guys I was working with casually mentioned this issue.
If during your troubleshooting, you run in to some counters (Disk Read/Writes) that just don't seem correct