Just a quick note out there on a specific issue we’ve been seeing lately. It is the really the only issue with Windows 2012 server hanging or becoming unresponsive that we’ve had up almost a year into the release of this OS.
On Windows 2012 running on HP servers that use the October 15th 2012 version of HpCISSs2.sys, those servers will run into an issue were we have an IO stall accessing the disk. This stall happens when we hit the HpCISSs2.sys and will never recover. The first time it happens the servers don’t actually hang but that is when the traffic jam starts to backup. Meaning – someone can have a car accident block the highway 5 miles down the road. You wouldn’t stop moving right away but eventually you’re going to be stopped in the resulting line of traffic.
Some of the symptoms you may see is that the server will show you a grey screen instead of the logon option. You may not be able to RDP to the server but if you test ports such as 3389 135 445 with port query they will be open. Remote WMI queries may fail. Ping will work. File shares will not be accessible.
For those familiar with the Windows debugger the driver version info is:
14: kd> lmvmhpcisss2
start end module name
Loaded symbol image file: HpCISSs2.sys
Image name: HpCISSs2.sys
Timestamp: Mon Oct 15 14:09:41 2012 (507C5165)
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
From Microsoft diagnostics reports (MPS or MSDT) we see the file as:
STORAGE (SCSI) HPCISSS2.SYS Hewlett-Packard Company 220.127.116.11 10/15/2012 153 KB (156,992 bytes) Smart Array P410i Controller
Although MSinfo32 may show a different date, the version number in these cases has been 18.104.22.168:
Driver c:\windows\system32\drivers\hpcisss2.sys (22.214.171.124, 153.31 KB (156,992 bytes), 7/2/2013 12:28 AM)
Customers with this issue have followed what HP has documented here for a separate issue involving HpCISSs2.sys on 2012.
If you have that version of the driver, are running Windows 2012 and are running into this behavior please contact HP and open a ticket with them to investigate this further.
To get conclusive data that confirms this is the issue, a Kernel Memory dump is the best route. Since Servers are typically setup for a Kernel dump by default the only change that would be required is to setup the system to generate that dump on an NMI signal:
How to generate a complete crash dump file or a kernel crash dump file by using an NMI on a Windows-based system
Not having to get a Full memory dump and only needing a Kernel dump greatly reduces the resulting size of the dump file, which is becoming problematic when generating dumps on systems with a large amount of RAM.
This has been an issue with HP Servers since Server 2003 with that hpciss2 driver. That and storport don't play well together unless they are at specific versions.
There have been issues in the past with that driver, yes. I cannot tell from the data that we have if the cause here is due to similar code issues with that driver or if we're getting stuck down another code path. Unfortunately, once we hit that driver the debugging goes into a black box.
I believe there might be a small hickup in the article.
We had the hang problem on our HP Windows Server 2012 hosts with upgraded firmware/driver versions and version 126.96.36.199 of the Smart Array driver (February 2013 version in SPP-2013.02).
In fact we had to downgrade to the October 2012 version 188.8.131.52 which SOLVES the problem. More recent versions have the hang problem as documented in this HP article.
All the Server 2012 hang cases we have seen have been with the October 15th 2012 version of that file.
we are experiencing the exact issue on our HP Blade server all of sudden. hopefully the downgrade of the driver will fix this problem. Thanks for the article!
I face some similar issue to this . See below
I got HP server 2012 storage box (HP Store Easy 1430) running which is member sever in domain.
This server hanged two times last two months and when this happens AD users can't log-in there computes either, Not sure this because of storage box.
I can't shutdown or restart the server2012 storage box ((HP Store Easy 1430)and only option it to power off.
After powering on back the server2012 storage box everything comes back to normal.
You guys reckon that I should downgrade the hpciss2 driver in HP Store Easy 1430?
Worked for me. All systems stable after downgrade! thanks!!
HpCISSs2.sys new version - 184.108.40.206 with SPP2014020 was released on February 18th 2014.
DL360p Gen8 hereSymptoms: Disk Management - Connecting to Virtual Disk Serviceas per this and following topichttp://social.technet.microsoft.com/Forums/en-US/47156c37-9cde-49c2-b97a-a668d71ef663/disk-management-hangs-cant-connect-to-virtual-disk-management-service?forum=winserveressentialsi upgraded HpCISSs2.sys to 220.127.116.11After restart still Connecting to Virtual Disk Service problem