August, 2013

  • Missing System Writer Case Explained

    I worked on a case the other day where all I had was a procmon log and event logs to troubleshoot a problem where the System Writer did not appear in the VSSADMIN LIST WRITERS output. This might be review for the folks that know this component pretty ...read more
  • Understanding Pool Corruption Part 2 – Special Pool for Buffer Overruns

    In our previous article we discussed pool corruption that occurs when a driver writes too much data in a buffer.  In this article we will discuss how special pool can help identify the driver that writes too much data.   Pool is typically organized ...read more
  • Important Announcement: AD FS 2.0 and MS13-066

    Hi everyone, Adam and JR here with an important announcement. We’re tracking an important issue in support where some customers who have installed security update MS13-066 on their AD FS 2.0 servers are experiencing authentication outages. This ...read more
  • MD5 Signature Hash Deprecation and Your Infrastructure

    Hi everyone, David here with a quick announcement. Yesterday, MSRC announced a timeframe for deprecation of built-in support for certificates that use the MD5 signature hash. You can find more information here: http://blogs.technet.com/b/srd/archive ...read more
  • Emerging issue with Windows Server 2012 and the HpCISSs2.sys driver

    Hello AskPerf readers.  We wanted to alert you to an issue that is generating calls within support here recently.  The October 2012 release of the HP HpCISSs2.sys driver is causing Windows Server 2012 to hang or become unresponsive every 3 –7 days.

    Check out the blog below that details more information on this issue:

    Windows 2012 servers that use HpCISSs2.sys become unresponsive typically every 3-7 days

    -AskPerf Team

  • Task Manager in Windows Server 2012 and Windows 8

    Good morning AskPerf!  This is Digvijay from the Performance team in Bangalore. Following up on my previous blog about Task Manager for Windows Server 2008 R2, it was very apt to write another one about the Task Manager of Windows Server 2012 & Windows 8.

    My first impression of this new task manager was – Wow! This is really simplified but is still powerful.  Let’s have a look at what’s new:

    The first time you open update task manager on Windows Server 2012 / Windows 8, you will be presented with a minimal look:

    clip_image002

    However, clicking “More details” reveals much more…

    NOTE the Disk & Network columns do not appear on Windows Server 2012.  Network usage can be found on the Performance tab, and Disk activity can be found in Resource Monitor.

    clip_image004

    At one glance, we can see how any process is doing and what kind of load its adding on the computer.  This will come in handy for troubleshooting those slow response issues as just by looking at the task manager, we can now quickly know who’s thrashing the disk and who’s choking up the bandwidth.  Memory and CPU are pretty much the same as previous versions.

    clip_image005

    If you expand the Process, you will see more details about that process.  Example - expanding the Background processes shows the services that are running inside it, if any exist.

    clip_image006

    If you want to stop a service, we can do it right from here.  Just right click and select Stop.

    clip_image008

    We can also open Services snap-in directly from here.

    Apart from other enhancements, the other good thing I like about this tab is the Open File Location option when you right click on any of the processes. Please note that the context menus are different for a program and an open window inside it.

    Here is an example:

    Right click on Microsoft Word –

    clip_image010

    Right click on Document1

    clip_image012

    The next tab is Performance.  This tab has been redesigned and made easier to get all the required information about your computer in one place.

    clip_image014

    The main heads here are –

    • CPU – Shows you all the information you might want to know about your CPU at a glimpse. E.g.  The type of CPU on the machine ( the make and model), Clock speed, Total Sockets, Number of Cores, Number of Logical Processors being exposed to the OS as well as if the CPU supports virtualization or not. You also get to know about the L1, L2 and L3 cache present on the machine.

    This tab also shows the CPU usage history since the time Task Manager was opened.  We also expose information of the total number of Processes, Threads and Handles along with the Uptime of the machine.

    clip_image016

    • Memory- Again this tab has been modified to show information that is easy to understand and relate. We see the graph of the memory usage of the last 60 seconds since the time the Task Manager was open.

    The second graph segregates the amount of pages in the different lists of PFN Database:  

    In Use – Memory used by processes, drivers and the operating system.
    Modified - Memory whose contents must be written to disk before it can be used for another purpose.
    Standby - Memory that contains cached data and code that is not actively in use.
    Free - Memory that is not currently in use, and that will be repurposed first when processes, drivers, or the operating system need more memory.

    NOTE that this graph shows the same information that’s shown in the Reliability Monitor in Memory section.

    This page also shows hardware information about the physical memory like number of RAM slots present on the server and used, the Type of memory bus, the Speed of the FSB and any amount of memory reserved by the hardware. (PCI cards, Graphics, shadowing etc.)

    It also summarizes the memory in use and memory available along with committed and cached bytes.  

    In Use – The total memory currently used by the OS, the process and drivers running.
    Available - This is amount of physical memory that is currently available for use by the operating system. It is equal to the sum of the standby pages and the free pages from the above graph.
    Committed - Committed memory is the physical memory in use for which space has been reserved in the paging file should it need to be written to disk.
    Cached – This is the sum of the standby and the modified pages as shown in the task manager.
    Paged Pool – Memory allocated in the of kernel mode virtual address space to kernel mode components, device drivers etc. These can paged to the pagefile if there is any need to free up physical pages.
    Non-Paged pool - Memory allocated in kernel mode virtual address space to kernel mode components, device drivers etc which is guaranteed to be resident in physical memory all the time.

    clip_image018

    • Disk – Shows information about the disk drives connected to the machine. ( You will see one or more disks depending  on the number of connected physical drives including USB drives)

    The graph shows the percentage disk activity in the last 60 seconds since Task manager was open. The 2nd graph shows the speed at which data is being read/written to the disk in KB/MB per second.

    Here we have information about the make and model of the disk, total capacity and formatted disk space. We also show if a particular disk is a system disk or not (has the Boot files) and if it contains any page file.

    Active time- Percentage of time the disk is busy. The lower the better.
    Average response time – This is the time taken for the disk to complete individual read/write operation. This includes the time required for the spindles to rotate and the head to move and align to the specific sector for reading the content. 
    Read speed – Rate at which data is being read from the disk in KB/s.
    Write speed – Rate at which data is being written to the disk in KB/s.

    • Ethernet – Show all the info that you would like to know about your network card.

    clip_image019

    The graph shows the network throughout of the last 60 seconds since the time the task manager was open. It also shows the current upload and downloads speed along with the type of connection and the ipv4 and ipv6 address.

    The next tab is dedicated to the Modern UI apps and shows the history of the recently used apps and the amount of resources they have been using (CPU, Network bandwidth) for different operations like Tile updates, Uploads, Downloads, etc. Depending on the setting for downloading over a metered connection, we can also view those sections detailing the amount of network bandwidth used.

    clip_image021

    The Startup tab shows what we used to see in msconfig in the older OS’s (Win7/2008R2 and earlier) with better information like the impact of the startup application on the startup/login process.  If you are experiencing issues with slow logons, or delays in reaching the desktop, this is the first place you need to visit to check and disable the processes having high/medium impact.

    clip_image023

    We can also find information regarding the Disk I/O and CPU usage by the processes listed during the at the startup operation.

    Continuing on we come to the Users tab:

    clip_image025

    This tab shows you which users are logged on, and what processes are running under their context.  Clicking the Disconnect button will obviously disconnect that user, but not log them out.

    Details tab is next:

    clip_image027

    This Tab is pretty much similar to the Processes tab under XP/2003/Win7/Server 2008.  There are some new options available when you right-click a process name:

    clip_image029

    Finally, we come to the Services tab.

    clip_image031

    This one is also similar to the same one in Windows 7 / Server 2008.  It does add a few additional right-click options however:

    clip_image033

    We hope you are enjoying the newly designed Task Manager in Windows Server 2012 / Windows 8.  If you want to read more about Task Manager, check out the following links:

    Windows 8 / Windows Server 2012: The New Task Manager

    The Windows 8 Task Manager

    -Digvijay

  • How to Cleanup TPM information from AD for Windows 8 computers

    For Windows 7 machines, TPM Owner Password is stored in msTPM-OwnerInformation which is attribute of Computer object in AD. So if you delete the computer object, TPM Owner Password is also deleted.

    For Windows 8, TPM Owner Information is not stored directly under Computer Object. It is stored in a separate object which is linked to computer object. When we delete a computer object from AD, the msTPM-OwnerInformation attribute which holds the TPM Owner Password is not deleted automatically.

    As per Best Practices, TPM Owner Information is also backed in AD DS for all domain joined computers.

    In a Scenario, where an admin is doing a REFRESH of a computer and he will delete the existing computer object in AD, he should first delete the TPM information for the computer which is now stored under a different location in AD.

    If you will not delete the msTPM-InformationObject under TPM devices, they will remain in AD as stale entry.

    If administrator will not delete the original computer object from AD in a Refresh Scenario, then you do not have to delete the TPM Information under TPM devices container in AD.

    In Windows 8 TPM auto-provisioning feature, initializes the TPM and can escrow the TPM Owner Password in AD DS if GPO to backup TPM password is enabled.

    Windows 8 TPM GPO
    http://technet.microsoft.com/en-us/library/jj679889.aspx

    If your computer is not being joined to a domain the TPM owner authorization value will be stored in the local computer registry.

    TPM Owner Information for a Windows 8 machine is stored under msTPM-InformationObject in TPM devices container in Active Directory Users and Computer MMC snap-in.

    Note: If TPM devices container is not available then make sure you have done the schema extensions for Windows 8.

    Schema Extensions for Windows Server 2008 R2 to support AD DS backup of TPM information from Windows 8 clients
    http://technet.microsoft.com/en-us/library/jj635854.aspx

    clip_image002

     

    How to delete the msTPM-InformationObject in AD

    1. Connect to Active Directory Users and Computer MMC Snap-in and select the computer object which you want to delete from AD.

    2. Right Click on Computer Object and go to Properties and Select Attribute Editor tab.

    3. Choose msTPM-TpmInformationForComputer from the list of attributes and get the CN name.

    clip_image003

    4. Now in Active Directory Users and Computers MMC Snap-in select TPM Devices container.

    5. Search for the CN Name which you gather from Step 3. This is the msTPM-InformationObject for the computer.

    6. Right click on msTPM-InfomationObject & select Properties.

    7. In attribute list you will see the msTPM-OwnerInformation attribute under which holds the TPM owner password for the computer.

    clip_image004

    8. Delete the msTPM-InformationObject under TPM Devices Container which is collected from Step 5.

    9. Now you can delete the original computer object from AD.

     

    More Information:

    TPM Provisioning Feature
    http://technet.microsoft.com/en-us/library/jj131725.aspx

    Windows 8 TPM GPO
    http://technet.microsoft.com/en-us/library/jj679889.aspx

    Schema Extensions for Windows Server 2008 R2 to support AD DS backup of TPM information from Windows 8 clients
    http://technet.microsoft.com/en-us/library/jj635854.aspx

     

    Manoj Sehgal
    Senior Support Escalation Engineer
    Microsoft Enterprise Platforms Support

  • Troubleshooting a Stop 0x7B in Windows

    The purpose of this blog is to provide information about the common Stop 0x7B bugcheck/Reboot Loop issue, and the best methods available to resolve it.
    We will discuss the main components involved in this type of bugcheck/crash, what can affect them, and not only troubleshooting steps to perform, but also steps to resolve the issue.

    Basic Information

    A Stop 0x7B bugcheck has this error:

    INACCESSIBLE_BOOT_DEVICE

    So, what does that mean, exactly?

    I realize that not everybody is familiar with the boot process, and the files needed to boot to Windows, but that is another topic altogether; so I’ll just outline the basics:

    When a PC that uses a BIOS (Basic Input/Output System) boots from a hard disk that has a volume/partition formatted as a Master Boot Record (MBR), it searches the MBR for information about where the Windows directory is located.  Before we can boot Windows, we have to load some storage controller drivers.  Windows has the standard ones already “in the box”, but sometimes newer chipsets require updated drivers before Windows will be able to boot, and those are generally added during the Windows setup process.

    So, when the boot process searches for a bootable device, if the correct driver is not loaded, we will see this issue occur.  Additionally, there several other factors that could cause this type of issue to occur besides a problem storage controller driver, including (but not limited to):

    • We may actually have missing or corrupt storage controller drivers
    • We may have missing or corrupt Filter Drivers related to the storage stack, or in some cases, misbehaving Filter Drivers
    • We may have file system corruption
    • The storage controller mode or settings in the BIOS may have been changed
    • A different storage controller may be in use than the one used when Windows was installed, or the disk may have been moved to a different machine with a different controller
    • There may be a problem with the hardware—the motherboard or storage controller may be faulty

    There are essentially two different “Branches” or types of cases where a Stop 0x7B can occur—cases where the machine was running fine and then the issue appeared, or cases where a Windows is being deployed to a machine; we will discuss the former type of issue first.

    So, what do we do when we engage on a case where the machine was working fine, and suddenly is now in a “Reboot Loop” caused by a Stop 0x7B bug check? 
    Well, there is a sequence that we should follow:

    First—and this is important—find out everything you can about the circumstances of the issue—i.e.: any recent updates or new software installations, the last time the machine was booted, why the machine was booted (was it due to problems with the OS?), etc.  This will help guide the troubleshooting path. 

    The steps below outline the order in which we should proceed, assuming that there are no hardware problems:

    Troubleshooting step 1 if machine was previously was booting properly:

    In nearly every single case, we should always run chkdsk /f on the volume holding the Windows directory first.  In some, or perhaps many cases, this alone may resolve the issue. If we run chkdsk on the volume, and it finds errors, we should run it again, until there are no more errors present.  If we find errors and fix them, or if we do not find any errors, and the issue still exists, then we need to move to the next step in the sequence

    Troubleshooting step 2 if machine was previously was booting properly:

    If we know what the second parameter of the bugcheck is, and it is a 0x34 , and running chkdsk did not resolve the issue, then we may have a filter driver that is corrupt or missing, or misbehaving, so we would either need to boot to WinPE and load the system hive file. 

    To do this, we would boot to WinPE or WinRE (using a Windows Vista / Windows Server 2008 / Windows 7 / Windows Server 2008 R2 DVD).  Once there, use Regedit.exe to load the “SYSTEM” registry hive file.  Once booted, follow these steps to load the hive file:

    1. Type “regedit.exe” into the cmd window, and press enter
    2. Within Regedit, highlight the HKEY_USERS branch on the left pane
    3. At the top of Regedit, click “File” then select “Load Hive”
    4. On the next screen, navigate to the drive that holds the Windows folder, down to Windows\system32\config, and select the “SYSTEM” file (with no extension); when prompted for a name, use “TEST”
    5. Expand “TEST”, and click on the “Select” folder; the number listed next to “Current” and “Default” should be the same, and reflect the ControlSet00x that is used by Windows to boot; this is the ControlSet we will be editing.
    6. Highlight the ControlSet00x identified in the “Select” key from the previous step, and expand it out
    7. Highlight the “Control” folder on the left and expand it out

    We are looking for any UpperFilters or LowerFilters entries, but mainly as it relates to storage; we can compare to another, working machine’s registry entries so we know what would normally appear there, and if there are entries that don’t appear to be standard, we can remove them. 

    Here are some (but probably not all) of the different registry entries where we may find these filter drivers (located under the ControlSet designated as “Default”):

    \Control\Class\{4D36E96A-E325-11CE-BFC1-08002BE10318}
    \Control\Class\{4D36E967-E325-11CE-BFC1-08002BE10318}
    \Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}
    \Control\Class\{71A27CDD-812A-11D0-BEC7-08002BE2092F}

    If an UpperFilters or LowerFilters entry appears that is non-standard (i.e.: not a Windows default Filter Driver in these keys, such as PartMgr), we would remove the entry by double-clicking on it in the right pane, and deleting only that entry (there may be multiple entries).

    The reason that these entries may affect us is because in the “Services” branch, there may be an entry with a START type set to “0” or “1” (indicting it is loaded at the “Boot” or “System” portion of the boot process), and either the file referred to is missing, corrupt, or it may be named differently than what is listed in the entry.  It is important to note that if there indeed is a service set to “0” or “1” that corresponds to an UpperFilters or LowerFilters entry, Setting the service to disabled in the Services branch without removing the Filter Driver entry will cause the machine to crash with a 0x7b!

    If we find a Filter Driver entry that is not related to a storage component, such as the network stack, it is still possible that this may be related, if the machine is booting from SAN, and is using iSCSI to connect to that SAN.

    Troubleshooting step 3 if machine was previously was booting properly:

    Assuming that the second parameter of the bug check is not 0x34 (such as 0xC102), and running chkdsk did not find any problems, or found problems and fixed them, but the issue still exists, then, depending on the OS, we would perform different actions:

    • If the OS is Vista/Windows Server 2008 or higher, we could boot to WinPE/WinRE and run SFC /scannow in Offline mode; to do this, we would run this command: “sfc.exe /scannow /OffBootDir=C: /OffWinDir=C:\Windows”, where “C:\” is the volume that contains the Windows directory.  If SFC finds errors and fixes them, we should run it again, but if it is unable to fix the errors, we may not have many options left aside from rebuilding/restoring from backup.  If it finds no errors, then we may have other issues going on.
    • If the OS is Windows XP/Windows Server 2003, we can try to perform a Repair Install/In-Place Upgrade.

    Troubleshooting step 4 if machine was previously was booting properly:

    If running chkdsk and SFC /scannow on Windows Vista/Windows Server 2008 or higher does not resolve the issue, it is possible that the BCD has become corrupt; you can use the bootrec /rebuildbcd command to rebuild the BCD; Best Practice dictates creating a copy of the BCD before performing these steps, and Method 2 in KB927391 provides the commands to perform this action. 

    As it relates to Legacy OS’s (Windows XP and Windows Server 2003), aside from the steps outlined in Troubleshooting Step 3 above, the only options available if those do not resolve the issue would be to restore the machine from backup (if one is available) or rebuild the machine.
    Likewise—if SFC /scannow finds problems, but is unable to fix them, or if rebuilding the BCD outlined in Troubleshooting Step 4 above does not resolve the issue, we will probably need to restore from backup or rebuild the machine.

    Keep in mind that when we say, “rebuild the machine”, what we are essentially doing is re-installing the OS and any applications previously installed on the machine; any data on the machine on other drives should be unaffected (unless there is file system and file corruption on the drive, and the data is located on the same drive as the OS, where some of the data may also be corrupted), and we would just need to re-install any applications using that data after the OS is re-installed (Exchange, SQL, etc.).

    Troubleshooting when the issue occurs during deployment:

    If the issue is occurring when deploying an image, this generally means we do not have the drivers needed for the specific controller (and the mode it is operating in) loaded, or they are not loaded properly.

    When we are experiencing this type of issue, we need to check the BIOS settings and the driver being used to make sure that the correct mode and driver for that mode is being used (they must match, such as AHCI or RAID controller driver needed if the controller is configured in this mode). 

    If this is a standalone machine, where the machine is booting from CD/DVD and simply running Windows Setup, make sure that the (correct) drivers are entered using the “F6” prompt while booting (for XP/W2K3), or the add drivers button within Setup for Vista/W2K8 and above.

    If a custom image is being deployed  to a machine (a .WIM file) and we are seeing these issues, the first thing to check would be the controller installed on the machine, it’s settings, and make sure the controller and the settings for it matches the driver that is injected into the image.

    If the controller and the driver injected into an image don’t match, of course we will need to obtain the correct drivers and inject them into the image; I won’t go into those details, because there are many articles that describe how to do that, and performing those actions are beyond the scope of this blog.

    To eliminate issues that may be caused by hardware, we can try performing the installation/image deployment on another identical machine that has the same settings as the original target machine.

    Perry Johnson
    Senior Support Escalation Engineer
    Microsoft Support