• Troubleshooting Windows activation failures on Azure VMs

    If you are experiencing Windows activation failures on an Azure VM, please try the following steps to resolve the issue. An example of an error message you may see is: Error(s): Activating Windows(R), ServerDatacenter edition Error: 0xC004F074 The Software more
  • Disk Performance Internals

    Abstract: Storage is the slowest component of most computer systems. As such, storage is often a performance bottleneck. This article discusses the disk performance kernel provider, partition manager.  By understanding how the disk performance provider more
  • Driver Object Corruption Triggers Bugcheck 109

    My name is Victor Mei, I am an Escalation Engineer in Platforms Global Escalation Services in GCR.  Some customers I worked with have strong interests in debugging; but usually they got frustrated when I told them “To find the cause from this dump more
  • Recovering Azure VM by attaching OS disk to another Azure VM

    If you are unable to administer an Azure VM because of RDP or SSH failures, in many cases rebooting or resizing the VM may resolve the issue. You can troubleshoot the VM by attaching the OS disk as a data disk to a different Azure VM using the steps more
  • Surface Pro 3 Hibernation Doesn’t Occur on Enterprise Install

    Hi my name is Scott McArthur and I want to call out a recently published KB article:

    Surface Pro 3 doesn't hibernate after four hours in connected standby

    If you are deploying an image to Surface Pro 3, you are missing out on the feature where after 4 hours in Connected Standby the device will hibernate. This is a key feature related to battery life so I would recommend that all Enterprise customers install KB2955769 and incorporate these PowerCfg commands into your deployment.

    If you use Microsoft Deployment Toolkit 2013 for your deployments this is super easy. Here are the steps

    1. Under Packages, import KB2955769


    2. Create PowerCfg_Sp3.batthat contains the following commands:

    REM sets CS battery saver time-out to four hours:
    powercfg /setdcvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 7398e821-3937-4469-b07b-33eb785aaca1 14400
    powercfg /setacvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 7398e821-3937-4469-b07b-33eb785aaca1 14400

    REM sets CS battery saver trip point to 100:
    powercfg /setdcvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 1e133d45-a325-48da-8769-14ae6dc1170b 100
    powercfg /setacvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 1e133d45-a325-48da-8769-14ae6dc1170b 100

    REM sets the CS battery saver action to hibernate:
    powercfg /setdcvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f c10ce532-2eb1-4b3c-b3fe-374623cdcf07 001
    powercfg /setacvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f c10ce532-2eb1-4b3c-b3fe-374623cdcf07 001

    powercfg /setactive SCHEME_CURRENT

    3. Save PowerCfg_Sp3.bat to your Deploymentshare\Scriptsfolder

    4. Open up the task sequence you use to deploy Windows and add a custom task in the state restore phase called PowerCfg-SP3


    5. In the properties of this task sequence step, edit the following:


    6. Click the Options tab and add conditional for “Task Sequence variable model equals Surface Pro 3”


    Note:This ensures this only runs on Surface Pro 3 devices using the model variable

    Hope this helps with your Surface deployments and keep eye on this blog for more tips and tricks for Surface

    Scott McArthur
    Senior Support Escalation Engineer

  • Your technical answers and automated solutions via

    Hello folks,

    One of our Engineering PMs that supports our Diagnostics and Automated solutions published a blog regarding Bing and how you can use it to answer your technical questions and provide automated solutions.  Here is a brief overview:

    Bing Technical Instant Answers provide concise answers to technical questions directly within search results and hopefully answer your question (or help you solve an issue) without you having to actually visit the web pages linked within the answer. The answers are triggered by specific search phrases, and they try to provide a unique benefit either by precisely matching your intent or by providing additional content related to your intent. In some cases, the instant answer will link to an automated fix or troubleshooter that you can run directly from the Bing search results. Microsoft will constantly be adding new technical answers, so if you have a technical problem with a Microsoft product or service try asking Bing to see if we have an instant answer for you!

    Go check out his blog via the link below:

    Using Bing for technical instant answers and automated solutions


  • Cross Post: Using Bing for technical instant answers and automated solutions

    This is a cross post from William Keener’s Support Diagnostics and Automated Solutions blog that we wanted to add to our site.  It relates to Bing and instant answers about Microsoft Products/Technologies/Support issues and here on the AskCore site, we are all about getting this type of information out there.  Any comments made should be made on the originating post so it can be properly seen, heard, or answered.


    Using Bing for technical instant answers and automated solutions

    Bing has been providing factual instant answers (and translation instant answers) for some time now, but recently they added "technical" instant answers for questions about Microsoft products and technologies or technical support issues. My previous team built the content management system that our internal content delivery teams are now using to add technical instant answers to Bing. Here's an example technical instant answer for the "Cortana" search term: 

    Now that I'm working on support diagnostics and automated solutions again, I have been working with the Bing and content delivery teams to get some instant answers created with links to some of our automated solutions.

    And I'm happy to announce that the first one is live! So you can now search for "Windows Update Troubleshooter" (or a variety of related terms and error messages) and the first result will be a technical instant answer with a link to download and run our automated troubleshooter to fix problems with Windows Update.

    When you click the link in step 3, you will be prompted to open (or run) or save the troubleshooter.

    Just click Open (or Run) to launch the troubleshooter.

    The content delivery teams will be constantly adding more technical instant answers, and we hope to have more live with automated solutions soon!

    Note that technical instant answers are also available in the Bing app on Windows Phone. To see the phone experience, tap Search and then type or say "cortana" on your Windows Phone. Then click the "See More" link at the bottom of the second result (after the ad - "Meet Cortana on Windows Phone 8.1") and swipe left or right to view the content on each of the tabs.

  • Understanding ATQ performance counters, yet another twist in the world of TLAs

    Hello again, this is guest author Herbert from Germany. If you worked an Active Directory performance issue, you might have noticed a number of AD Performance counters for NTDS and “Directory Services” objects including some ATQ related counters. In this more
  • Unable to restart server due to registry bloat over 2GB

    Hello AskPerf!  Pushing up a blog today to discuss the registry bloat issue that has been recently addressed in the following KB:

    Computer cannot be restarted if the registry hives are larger than 2 GB


    • You have a computer that is running the x64-based version of Windows 8.1, Windows Server 2012 R2, Windows 8, or Windows Server 2012.
    • The registry hives for the computer are larger than 2 gigabyte (GB).

    This problem occurs because of the 2 GB size limit of the registry hives in x64-based version of Windows.

    Install this patch to resolve the issue.


    When you get into this state, you may experience one of the following issues:

    1. You can boot to a stop error.
    2. You can boot and not be able to log in due to the RQL (Registry Quota Limit).
    3. You can boot and be logged in with a temp profile and not be able to install any software due to the RQL.

    If this happens, KB2978366 should be installed.

    With that, the following questions may come to mind:

    • How does this issue occur?
    • How do I prevent this issue in the first place?
    • How do I fix this issue once the hotfix is installed?
    • What happens if I see this problem on another OS version?
    • Are there any tools I can use to troubleshoot this issue?

    Question: How does this issue occur?

    Answer: There are many reasons that cause registry hives/keys to bloat.  Some of the ones we have seen are related to KB2871131, which refers to the “..\Printers\DevModes2” key bloat.  This hotfix does not “fix” the issue, but prevents it from occurring in the first place.  You still have to clean the keys first.  Additionally, there is a known issue with SQL Server 2012 SP1 that can cause the registry to hit the 2GB limit and put the machine in a no-boot state.  Please see KB2793634 for more details on this.

    Question: How do I prevent this issue in the first place?

    Answer: There really is no good answer for this outside of installing the hotfixes noted above, and keeping a close eye out on your registry hives.  You can use Performance Monitor however to monitor the “System\ % Registry Quota In Use” counter.  If this counter gets over 50 %, then you should start investigating what registry keys/hives are growing.


    % Registry Quota In Use is the percentage of the Total Registry Quota Allowed that is currently being used by the system.  This counter displays the current percentage value only; it is not an average.

    NOTE The following Registry hives point to their corresponding files:

    • HKLM\BCD00000000 - \Boot\BCD
    • HKLM\COMPONENTS - %windir%\System32\config\Components
    • HKLM\SAM - %windir%\System32\config\SAM
    • HKLM\SECURITY - %windir%\System32\config\SECURITY
    • HKLM\SOFTWARE - %windir%\System32\config\SOFTWARE
    • HKLM\SYSTEM - %windir%\System32\config\SYSTEM
    • HKU\.DEFAULT - %windir%\System32\config\DEFAULT
    • HKCU - %userprofile%\NTUSER.DAT
    • HKLM\HARDWARE - This is dynamic and gets built with the OS boots (volatile hive)
    • HKLM\CLUSTER - %windir%\Cluster\CLUSDB
    • HKU\<SID of local service account> - %systemroot%\ServiceProfiles\LocalService\Ntuser.dat
    • HKU\<SID of network service account> - %systemroot%\ServiceProfiles\NetworkService\Ntuser.dat
    • HKU\<SID of username> - \Users\<username<\Ntuser.dat
    • HKU\<SID of username>\Classes - \Users\<username>\AppData\Local\Microsoft\Windows\Usrclass.dat

    Question: How do I fix this issue once the hotfix is installed?

    Answer:  After installing the hotfix, you may need to copy your Registry file to another machine that includes the hotfix.  After you have cleared out the bloated entries (whitespace will remain), then simply load the hive up, and then unload it.  This process will shrink your registry key back down pre-bloat. If a system is unbootable due to registry bloat install the hotfix on another system. Boot the problem system from DVD, copy the bloated registry hive to external storage, put on system with hotfix and use regedit to remove the bloated registry info and whitespace. The hive can then be copied back to problem system to allow it to boot normally.

    Question: What happens if I see this problem on another OS version?

    Answer:  Simply copy your hive over to a Win 8/ Server 2012 machine that has this hotfix installed, then follow the steps above.

    Question: Are there any tools I can use to troubleshoot this issue?

    Answer:  Coming Soon


    How to Compress "Bloated" Registry Hives


  • RDP Fails with Event ID 1058 & Event 36870 with Remote Desktop Session Host Certificate & SSL Communication

    Hello AskPerf!  Sanket here from the Windows Platforms team here to discuss an issue with Remote Desktop Services where RDP does not work when you try to connect from a remote machine.  With that, let’s get started!

    I’m sure most of you have come across the following message when connecting to a machine via RDP:

    Remote Desktop Connection

    This computer can't connect to the remote computer. Try connecting again. If the problem continues, contact the owner of the remote computer or your network administrator.

    This is a generic that can be caused by numerous varying reasons.  We have a fairly detailed troubleshooting KB article that talks about this error and what to do to fix it:

    Remote Desktop disconnected or can’t connect to remote computer or to Remote Desktop server (Terminal Server) that is running Windows Server 2008 R2

    Assumptions are that most of you have followed this KB and resolved your issue.  However, there could other reasons that could cause RDP to fail as well.

    I recently worked an issue with same error where RDP from a remote machine was not connecting to a Windows 2012 Server.  NOTE the same error can occur on previous OS versions as well.

    There was a mystery as to what was changed on the server that could have caused this start.  Possible assumptions were user intervention, or some application may have changed/removed certain permissions.

    During the course of troubleshooting, we double-checked the KB article noted above, and noted the following Error events in the System Log:

    Log Name:      System
    Source:        Microsoft-Windows-TerminalServices-RemoteConnectionManager
    Date:          7/27/2014 12:16:59 AM
    Event ID:      1058
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      XXXXXXX
    Description: The RD Session Host Server has failed to replace the expired self-signed certificate used for RD Session Host Server authentication on SSL connections.
    The relevant status code was Access is denied.

    This error indicates that there is already a Certificate in place, however there is no sufficient permissions, and/or the default permissions on “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys” may have been modified.
    Log Name:      System
    Source:        Schannel
    Date:          --
    Event ID:      36870
    Task Category: None
    Level:         Error
    User:          SYSTEM
    Computer:      XXXXX
    Description: A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x8009030D.
    The internal error state is 10001.

    There was a fatal error accessing the Private Key for secure communications.

    At this point, I decided to capture a Process Monitor (Procmon) log on the destination server where the connection was going to.  As you may already know, Procmon allows us to monitor/record real-time file system, Registry and process/thread activity on Windows Workstations/Servers.

    Per the Procmon log, we found an “Access Denied” error to the following path:


    The above cert key f686aace6942fb7f7ceb231212eef4a4_xxx is associated with RDS, and this GUID like number is the pair key for both the computer and user.

    If you use the certutil -key command, you would see this Cert key with TSSecKeySet1:

    f686aace6942fb7f7ceb231212eef4a4_xxxxxxxxxx: AT_KEYEXCHANGE

    From the Procmon Logs:
    12:39:53.5364585 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: N/A, Share Mode: Read, Allocation Size: N/A,
    12:40:24.3692803 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
    Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a, Impersonating: NT AUTHORITY\SYSTEM
    12:40:23.9265708 AM svchost.exe 1012 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
    Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a
    So, what are the default permissions?  Well, you can use icacls to find this:
    C:\>icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\
    Everyone :(R,W)
    BUILTIN\Administrators :(F)
    BUILTIN\Administrators ::(R)

    In case if you want to grant permission using icals you can provide the same using following command :
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :( R)
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\SYSTEM :(F)
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :(R )

    Fig 1.1 (Permission in Windows Explorer)

    As you can see above, the SYSTEM accounts needs the proper permissions.  If these permissions have been changed, then they need put back to defaults.  The certs under this key should be inheriting the above permissions from the parent folder MachineKeys.

    You can restore permissions, grant the permissions back using icacls, or use the Windows Explorer GUI.  Correcting the default permission on the cert should allow RDP to now work correctly.

    Considering if this would have been easily reproducible, there is always an option to enable the Auditing on the cert key f686aace6942fb7f7ceb231212eef4a4_xxxxx under “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys”.  This can be done using the Security Tab on Properties of the cert key as seen in the screenshot below:

    NOTE Adding Auditing on this object will log Events to the Security Event Logs.  You will want to keep this enabled until you are able to reproduce the connection issue.

    Hope you find this information helpful.

    Additional Resources

  • Updating Surface Pro 3 firmware (Cross Post)

    Hi this is Scott McArthur and I just wanted to call attention to a blog that I worked on with some of our PFE engineers that just posted related to Surface. 

    How to Update the Surface Pro 3 Firmware Offline using a USB Drive

    This blog shows you how you can update firmware from a Bootable WindowsPE USB flash drive.  This is useful for some scenarios where you need updated firmware BEFOREyou do a deployment to the device.  Hope it helps with your Surface deployments

    Scott McArthur
    Senior Support Escalation Engineer

  • Asset Tag Tool for Surface Pro 3

    Hi my name is Scott McArthur and I want to call out a tool that recently came out that allows Enterprise customers to set Asset Tags on Surface Pro 3.

    This tool is available for download at the following location:

    The tool requires the following:

    • Surface Pro 3(other Surface devices not supported)
    • UEFI firmware version or newer

    It can be run from within Windows or from WindowsPE.  The download comes with a README.TXT that contains the following reference:

    This tool gets or sets the proposed Asset Tag, which will be applied on next reboot.

    The current Asset Tag is an SMBIOS setting which can be queried via WMI:

    (Get-WmiObject -query "Select * from Win32_SystemEnclosure").SMBiosAssetTag

    To get the proposed asset tag:

    AssetTag -g

    To clear the proposed asset tag:

    AssetTag -s

    To set the proposed asset tag:

    AssetTag -s testassettag12

    Valid values for this can be:

    • up to 36 characters long
    • valid characters including A-Z, a-z, 0-9, period and hyphen

    You can view the Asset Tag in the UEFI settings under Device Information.


    Here is a PowerShell script demonstrating way to get proposed value and interpret errors.

    Note that stout contains the Asset Tag and stderr contains error messages.

    AssetTag -g > $asset_tag 2> $error_message
    $asset_tag_return_code = $LASTEXITCODE
    $asset_tag = $asset_tag.Trim("`r`n")

    if ($asset_tag_return_code -eq 0) {
         Write-Output ("Good Tag = " + $asset_tag)
    } else {
         Write-Output (
              "Failure: Code = " + $asset_tag_return_code +
              "Tag = " + $asset_tag +
              "Message = " + $error_message)


    Hope this helps with your Deployments.

    Scott McArthur
    Senior Support Escalation Engineer

  • Configuring Azure Virtual Machines for Optimal Storage Performance

    In support, one of the most common questions we get is: How do I achieve the best disk performance for Azure virtual machines? Platform Planning: In the standard tier of virtual machine in Azure, the maximum IOPS is 500 per disk . When planning more
  • Windows 10 Preview available for review

    Good morning AskPerf!  It’s been a while since our last post, and we apologize for that.  We’ve been quite busy here on the Support side knocking out customer issues…

    Any who, we have some upcoming blogs in the oven that need a little more time to bake.  One of which is a short series on Windows Event Forwarding which I am very excited about.  Look for that to come out in the coming months.

    Even though we are commonly known as the Performance team, internally we are known as the Reliability team.  Some of the technologies we support are as follows:

    Windows Client/Server OS

    • Printing
    • RDS / TS
    • Performance which includes System Hangs, High CPU, Memory issues, etc.
    • Base WMI functionality
    • COM/DCOM – base functionality
    • Explorer (Shell)
    • Desktop Search
    • MUI and IME
    • MSI – basic functionality
    • Themes/Fonts/Screen Savers/Wallpaper
    • Task Scheduler
    • WinRM – basic functionality
    • Windows PowerShell – install and basic functionality
    • ACT

    There are many other smaller technologies, but these are the main ones.

    Now back to our original topic:  The Windows 10 Preview is available for download/testing.  To get it, click the following link:

    Windows 10 Preview

    Finally, we always welcome feedback on topics you would like for us to blog about here on the AskPerf blog site.


  • Virtual Machine Checkpoint fails with Access Denied when running on a Clustered Shared Volume

    When you attempt to create a CheckPoint of a virtual machine that is running on a Cluster Shared Volume (CSV) , you may receive a General access denied error as shown below.


    You will receive this error if the virtual machine’s VHD is placed on the root of the drive.



    The reason for the access denied error is due to the VM worker process (VMMS) not having relevant permissions on the CSV volume.  Below are default permission that is present for a typical CSV volume.  It is strongly recommended that these permissions not be changed.


    To resolve the issue, migrate the storage from Failover Cluster Manager or reconfigure the VM and place the VHDX in a folder off the root.  By moving the VHDx to a subfolder or if the VM is reconfigured, the VMMS service updates the permissions on the subfolder as it should.

    For example, this is the current location of the file: 

    C:\ClusterStorage\Volume1\Test Lab.Vhdx

    You would want to move it (and any other VHDX files present) to a subfolder you can create, such as this: 

    C:\ClusterStorage\Volume1\Test Lab\Test Lab.Vhdx

    There are several options you can run through to accomplish this task.

    Option 1:

    Using the Virtual Machine Storageselection from Failover Cluster Manager, move it to the folder you created.  This is an option that can be done without affecting production as it can be done while the virtual machine is online and running.



    Option 2: 

    Shut the virtual machine down and, in Explorer, move the VHDx from the root of CSV to a folder you create.  In Failover Cluster Manager, bring up the settings of the virtual machine and manually change the path of the relocated VHDx.  This is an option that can be done but will affect production as it cannot be done while the virtual machine is online and running.  So you would need to schedule downtime to accomplish this task.

    General Rule:

    Microsoft has always not recommended to keep any type of data files in the root of a drive.  Even though things may appear to work fine, problems could arise from this configuration.

    Shasank Prasad
    Senior Support Escalation Engineer
    Microsoft Corporation

  • Remove Lingering Objects that cause AD Replication error 8606 and friends

    Introducing the Lingering Object Liquidator Hi all, Justin Turner here ---it's been a while since my last update . The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering more
  • How to identify a driver that calls a Windows API leading to a pool leak on behalf of NT Kernel?

    Hello my name is Gurpreet Singh Jutla and I would like to share information on how we can trace the caller which ends up allocating “Se  “ Pool tag. When we use the Windows debugger and investigate the pool allocation and the binary associated with more
  • Announcing public availability of MBAM Compliance Data Cleanup Tool 2.5

    We are happy to announce public availability of MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe), aka MBAMCDCT 2.5.
    MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe) is a command line tool which enables you to delete machine records from the ‘Compliance Status’ database of the MBAM 1.0 and MBAM 2.0, MBAM 2.0 SP1 and MBAM 2.5 standalone.


    There have been situation where you as a MBAM Admin had to delete the entries of older/reimaged machine records from the MBAM compliance database. The only solution in this case was to run complex SQL queries to delete machines from the database. This tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the MBAM Compliance Status database.


    This is a command line tool which enables you to schedule it stale data deletion as a task to automate deletion of obsolete machine records from the MBAM compliance database.

    This tool provides three different ways to delete machine records from the MBAM Compliance Status database:

    1.     Delete machines which have not reported in last X days.

    2.     Delete machines specified in a comma separated list via command line.

    3.     Delete machines specified in a text file.




    This tool doesn’t delete the recovery information or any other data from MBAM Recovery and Hardware Database. All delete operations are performed strictly on the MBAM Compliance Status Database.

    This tool is available for download from the TechNet website as a self-extractable compressed file, which includes the executable and documentation.

    Hope this tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the database.


    This tool and documentation are provided "as-is". You bear the risk of using it. No express warranties, guarantees or conditions are provided. The tool supplied in this document is not supported under any Microsoft standard support program or service. However, you can report issues and bugs in the comments section on this page. Microsoft will, at its sole discretion, address issues and bugs reported.



    Himanshu Singh

    Windows Core Team

  • We Are Hiring Windows Escalation Engineers in Charlotte, Dallas, and Redmond

    Would you like to join the world’s best and most elite debuggers to enable the success of Microsoft solutions?   As a trusted advisor to our top customers you will be working with to the most experienced IT professionals and developers in the industry more
  • Windows Troubleshooting – Stop 9E Explained

    What to do if a stop 9E occurs.  How you can solve the issue yourself. more
  • Windows Troubleshooting – Special Pool

    The Windows Support team has a new YouTube channel, “ Windows Troubleshooting ”.  The first set of videos cover debugging blue screens. In this video, Bob Golding, Senior Escalation Engineer, describes how the Special Pool Windows diagnostics tool more
  • Hate to see you go, but it’s time to move on to greener pastures. A farewell to Authorization Manger aka AzMan

    Hi all, Jason here. Long time reader, first time blogger. AzMan is Microsoft’s tool to manage authorization to applications based on a user’s role. AzMan has been around since 2003 and has had a good run. Now it’s time to send it out more
  • Go the modern (app) way

    Hello everyone,

    This is Ashfana from the Windows Performance team. I’m here to talk about the basic philosophy behind Windows Store Apps which were introduced in Windows 8 and Windows 8.1. Windows Store apps were originally called Metro or modern apps before the RTM of the product. These are apps developed using a variety of languages like C#, C++, VB, HTML and Javascript, - which provide the modern, full screen immersive experience.

    Windows has the largest software ecosystem thanks to millions of developers around the world, and Microsoft’s focus on ensuring backward compatibility for application software. This ecosystem has evolved over time and has reached a stage where – we aren’t able to fix some of the limitations and move ahead without causing existing applications to break. Some limitations are installation/uninstallation problems, Software state corruption, uncontrolled modification of important system registries and files, unreliable installer components, one app causing another one to crash, and security risks when running Apps under elevated token. This new App model was introduced to address these reliability and security issues and to develop a new ecosystem that allows Apps to run on multiple hardware platforms with minimum efforts for the developer.

    Here are some of the features of Store Apps that make it very attractive for consumers and enterprises:

    • Windows Store apps do not need an installer. They are either sideloaded to the image using DISM, or installed via Add-AppxPackage PowerShell Command, or installed directly from the Store or a Azure based portal. You no longer need MSI or other installer components, Windows already has the necessary components that can extract the Appx package and install it.
    • Apps are installed per-user (unless they are sideloaded into the image before deployment). They maintain their state within folders in the user’s profile. This ensures better security.
      Apps run within a sandbox environment called an App container. This restricts an App from accessing much of the system and the users profile by default. Additional resources can be requested by an App by declaring capabilities within its manifest file. During first launch of the App, the user is prompted to allow/confirm access to these additional resources.
    • Desktop Apps run under medium integrity and are able to access full privileges granted to a user. An untrusted desktop application can pose a greater risk. Windows Store Apps on the other hand run under base privilege and hence provides a greater level of protection.
    • The Appx manifest and blockmap tell Windows how to install the App, chances of an installation going corrupt are very less. Even if an installation goes corrupt – it would not affect the system in any way.
    • Store Apps provide an immersive experience, are generally very lightweight, and are best suited for scenarios where hard core processing can be done at the server side and results can be delivered to the App over the network.
    • With the introduction of Universal Apps in Visual Studio 2013 update 2, developers can now write an App once and port it to run on multiple devices such as Xbox, Windows phones, PC, tablets and laptops. This makes application development time much shorter.

    Structure of a Windows Store/Universal App:

    Windows Store app are packaged into one or more files with the extension .appx. This appx package is the unit of installation for a Windows Store app. Appx packages are ZIP-based container files that contains the app’s payload files plus info needed to validate, deploy, manage, and update the app. From a high-level view, each Windows Store app package contains these items:


    App payload
    App code files and assets
    Payload files are the code files and assets that you author when you create your Windows Store app.

    App manifest
    App manifest file (AppxManifest.xml)
    The app manifest declares the identity of the app, the app's capabilities, and info for deploying and updating.

    App block map
    App package’s block map file (AppxBlockMap.xml)
    The block map file lists all the app files contained in the package along with associated cryptographic hash values that the operating system uses to validate file integrity and to optimize an update for the app.

    App signature
    App package’s digital signature file (AppxSignature.p7x)
    The app package signature ensures that the package and contents haven't been modified after they were signed. If the signing certificate validates to a Trusted Root Certification Authorities Certificate, the signature also identifies who signed the package. The signer of the package is typically the publisher or author of the app.

    Packaging app:

    When you create an app using Visual Studio, it packages your app, it automatically adds the app block map and signature files to the package. But you can also use the standalone MakeAppx and SignTool utilities if you want to manually package your app.

    The steps that are involved in Package installation:


    The Windows Store app deployment process occurs in multiple phases.

    1) In the first phase, Windows acquires and validates the app manifest, app block map, and app signature. It checks the OS version and dependencies, sees if you have sufficient disk space, and also whether this app is already installed.

    2) In the second phase Windows checks the app package’s deployment criteria to ensure that the app deployment will be successful.

    3) Windows stages the package’s contents on the disk in the %ProgramFiles%\WindowsApps directory in a new directory named after the package identity:

    <Package Name>_<Version>_<Architecture>_<ResourceID>_<Publisher Hash>

    4) Windows registers the package into the user's account. During this phase, the extensions that are declared in the manifest are registered with the operating system.

    The Future:

    With this release of Visual Studio 2013, we have set out to accomplish three major goals:

    1) Reach customers across phones, tablets, and PCs;

    2) Deliver innovation that supports developer investments;

    3) Make cross-platform technology easier and more capable.


    Develop once for all Windows devices using a unified Windows runtime and VS tools that allow you to both support experiences unique to a device in XAML, HTML, and DirectX, and share the code that supports those experiences across all devices using C++, C#, or JavaScript. When your work is finished you can you can produce the app packages that you will submit to the Windows Store and Windows Phone Store with a single action to get your app out to customers on any Windows device. You may also just sign it and deploy this through SCCM.

    More links:

    Package manifest schema reference

    Capabilities of an App:

    Create an app package

    App packager (MakeAppx.exe)


    How to create an app package signing certificate

    Ashfana Begum
    Support Engineer
    Microsoft Performance Team

  • Deciphering Storport Traces 101

    Welcome back to the CORE Team Blog -- Paul Reynolds here. In previous blogs, I wrote about how to capture Storport traces in Windows 8 and Windows 2012. Please see:

    Tracing with Storport in Windows 2012 and Windows 8 with KB2819476 hotfix


    Tracing with Storport in Windows 2012 and Windows 8 without KB2819476 hotfix

    This time around, I would like to explore what information you can draw from the raw data contained in a Storport trace. What conclusions can you reach regarding your disk performance? Do you have a disk that does not perform well?

    To accomplish this, we will take advantage of free tools available to Windows and Office 2013 users:

    • Windows PowerShell
    • Windows Performance Toolkit
    • Excel PowerPivot

    First, we need to talk briefly about the Windows Storage Stack and where Storport traces are taken. It is important to note that Storport traces are at the very last “rung of the ladder” before Windows hands off I/O request packets to hardware. Hardware in this case encompasses firmware, drivers, HBAs, storage fabrics – anything after the Windows Operating System. It is important because it can help delineate where the problem is. Is the problem with the Operating System or with the hardware? Should I call Microsoft to open a support case or it is more appropriate to talk to my Storage Vendor? Storport traces are the perfect place to start to answer questions such as these.

    It is assumed you already have a Storport Trace in hand – an ETL file captured using the procedures documented in the two blogs above for Windows 8 and Windows 2012, or Windows 2008 (see Bob Golding’s blog):

    Storport ETW Logging to Measure Requests Made to a Disk Unit

    As the old adage goes: a picture is worth a thousand words. When it comes time to deciphering Storport Traces, viewing graphs that summarize data and show trends, and viewing charts that have average and maximums, are much more helpful than looking at raw data. The top-level steps we will undertake are:

    1. Capture the Storport Trace
    2. Convert to the Storport Trace into CSV format
    3. Scrub the data in the CSV file
    4. Import the data into an Excel Spreadsheet

    The easiest way to accomplish the last 3 steps above is to automate it using a Windows PowerShell script. At the end of the blog, there is a zip to download that has this file. Save the PowerShell script as StorPortPACMAN.PS1 in a directory on your C: drive called StorPortPACMAN :

    PLEASE NOTE: you must have the following perquisites installed to successfully run the script above:

    1. XPERF needs to be installed and in your system path.

    Windows 7 or 8 – use the Windows Assessment and Deployment Kit (ADK):

    2. PowerPivot is part of Office 2013 but it needs to be added as a COM Addin. Click here for instructions.

    At the end of this blog, there is also 2 spreadsheets in the zip you must download into the C:\StorPortPACMAN directory.

    Run the script, and depending on which version of Windows was being run on the server, one of the Excel Spreadsheets will open and be refreshed with data in your Storport trace.

    Here are screenshots of sample graphs and charts you will see in the first page of the spreadsheets:


    This first graph shows request duration values over the time of the Storport trace. It is very useful to get the “big picture”. For the most part, the disk in this graph is performing fine except for a period near the beginning. Depending on the application you are investigating, this may be fine and the average value is what is important to you. Averages and maximums are shown in the chart below, which is next to this graph in the spreadsheet:


    Finally, the chart data is presented in the graph below:


    The Excel spreadsheets and their corresponding graphs and charts are only samples of what you can do for your disk analysis using Storport traces and Excel PowerPivot tools. Why use PowerPivot? For Storport traces it means being able to view and summarize more data. Excel by itself is an excellent tool, but problems may start to develop when your data approaches or exceeds a million rows of data.

    It is not uncommon for a Storport trace to contain tens of millions of rows of data, especially if you decide to not use a filter while capturing the data. I generally suggest to not use a filter as your averages will be closer to results you obtain from other tools such as Perfmon or XPERF. Using filters will cause your disk to look worse than it really is.

    Request Duration Times, as a rule of thumb, can be summed up as follows for SCSI Read(10) and Non-cached Write(10) I/O:

    <  9ms = excellent
    < 15ms = good
    < 25ms = fair
    > 25ms = poor

    We purposely use the caveat, rule of thumb, because these values are using an assumed 64KB data I/O size. If a read or write is larger or smaller than that, these values should be adjusted.

    Using an Excel Pivot table and chart lends well to focusing on a specific LUN through the use of filters built into the table or chart.

    Special Cases:

    We occasionally see a Storport trace that does not make sense. One example is a VM that has more than one SCSI disk and the Excel spreadsheet shows only one LUN. What happened to my other disk? It is there, but because of the way storage may be presented, the same LUN number is used for more than one disk. So what does one do in a situation such at this? The easiest way is to drop and drag a second property of a disk into the chart or graph you are viewing. There are 4 properties we gather in a Storport trace:

    1. LUN
    2. Target
    3. Bus
    4. Port

    Since using LUN is not enough, we can add a second property to the chart, Target. To do this, at the top of the spreadsheet click on the Analyze tab under Pivottable Tools, then click on Field List to expose the data available to you. Drag and Drop the Target field onto Rows and underneath LUN. You will see something similar to this:


    This will let you see all your disks if they have the same LUN number.
    If you are at all like me, viewing charts and graphs might be fine for seeing the big picture and giving presentations, but your inner engineer is screaming to look at the data in detail, to make sure you are comfortable with it and understand it. To do this, click on the POWERPIVOT tab and then the Manage button in the Data Model area. This will open a new window that exposes the raw data to you so that you may filter it, sort it, or anything else you wish to make sure it all makes sense to you before using it.

    I hope you find this blog helpful and gives you a new tool to use when investigating disk performance.

    Paul Reynolds

  • WMI: How to Troubleshoot WMI High Handle Count


    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is experiencing high handle count

    Your first thing to do is check the Application Event log for following event:

    Source: Microsoft-Windows-WMI

    Event 5612 Wmiprvse.exe exceeding handle quota limit Event

    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    If you find the above Event, you can try and bump up the handle quota limit to see if it resolves your issue. If it is a leak, then bumping limit will only mean it will take longer to reach the new limit. If it’s just load related, then bumping the limit could resolve the issue.

    The event will tell you what the handle count was, and if it is higher than the 8192 value I suggest below.  You can then skip this section and move on to data collection to figure out the cause of high handle count.

    How to increase the handle quota limit for the WMI Provider Service

    1. Go to Start--> run and type wbemtest.exe
    2. Click Connect on the Windows Management Instrumentation Tester
    3. In the namespace text box just enter "root" (without quotes)
    4. Click Connect


    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    1. Click "Enum Instances…"


    1. In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK.

    Note: a double underscore __ precedes ProviderHostQuotaConfiguration


    1. A query Result window will come up. In this windows now double click "__ProviderHostQuotaConfiguration=@"


    1. An Object Editor windows will come up now
    2. Under properties find the property "HandlesPerHost"


    1. Change the value from default of 4096 to 8192
    2. Click Save Property
    3. Click Save Object in the Object Editor window
    4. Close the other windows now and exit WMI Tester
    5. Restart Windows Management Instrumentation Service.

    If after bumping up quota limit and wmiprvse still exceeding quota limit, accomplish following actions below. You will want to read through the rest in its entirety to ensure you get all of the necessary tools downloaded before taking any actions.

    Configure System for Complete Memory Dump by referring to:

    Windows 8 and Windows Server 2012 Automatic Memory Dump:

    Windows does not create a memory dump file when a Stop error occurs in Windows 8 or Windows Server 2012:

    Windows 2008, Windows Vista, Windows 7, Windows 2008 R2:

    Windows Server 2003 and Windows XP:

    Collect perfmon logging using logman method

    Directions below will create 2 perfmon logs, one at a 5 minute interval (PerfLog-Long) and a short 5 second interval log (PerfLog-Short) and they will be placed in C:\Perflogs folder.

    • Long log (5 min intervals) – no thread counter, 250 MB:

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Long -o "c:\perflogs\PerfLog-Long.blg" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues\*" "\System\*"  -si 00:05:00

    3. Start the log with:

    Logman.exe start PerfLog-Long

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Long

    • Short, high resolution log – 5 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:05

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Configure Handle Tracing

    You probably just need the standalone version since we only need the debugging tool and not the whole WDK package.

      1. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator
      2. Click on the ‘Kernel Flags’ tab
      3. Check the box next to ‘Enable bad handles detection’
      4. Click on the ‘Image File’ tab
      5. Next to ‘Image: (TAB to refresh)’, enter wmiprvse.exe or if tracing WMI Service enter the unique name given to the svchost process for WMI Service per directions in next section below
      6. And then click on the ‘Tab’ key
      7. Next to ‘Stack Backtrace: (Megs):’ enter ‘10’
      8. Click on Apply
      9. Click on Ok
      10. Restart WMI service

    If it is a svchost process showing high handle count, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From there in a command prompt you can run tasklist /svc and look for the PID then identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From experience it will be the WMI service more times than not but not always, as such I would try to break it out first on its own and monitor to see if it is the one driving up high handle count in the shared svchost process.

    WMI (Windows Management Instrumentation) service, you can break it out by accomplishing the following.

    Break WMI Service out into its own unique svchost process

    a.      Open command prompt with elevated privileges

    b.      Run following command: sc config winmgmt type= own

    c.      Restart Wmi service

    d.      Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    sc config <service name> type= share

    e. Restart the service or machine and verify result is Win32_SHARE_PROCESS when you run sc query winmgmt command again

    f. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe

    g. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt

    h. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    I. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    j. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    k. You would now substitute wmisvchost.exe in lieu of wmiprvse.exe in step 6. under Configure Handle Tracing above

    Using debugger to attach to the process in windbg.exe and running !htrace –enable command

    1. Launch WinDbg program from under Debugging Tools for Windows that you installed earlier.

    2. Created folder c:\websymbols

    3. Click on File-Symbol path and add the following symbol path to the debugger: SRV*c:\websymbols*

    4. Click on File-Save Workspace

    Attach to process to accomplish handle tracking using htrace

    To do this:

    1. From Windbg - File - Attach a Process - Select the instance of wmiprvse.exe with high handle count

    Note: If it is WMI Service (run tasklist /svc or Task Manager with PID column added first to find the PID of svchost.exe containing winmgmt which you should have broken out and uniquely named wmisvchost.exe per earlier directions)

    2. Run following command from the debugger:

    .logopen "C:\debug.log" then hit <ENTER> key

    !htrace -enable 0x20000 then hit <ENTER> key

    Note:  By default, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 keep a history of 4000 handles open and close operations. 

    With !htrace you can enable to keep a much higher history by doing the following:

    !htrace -enable 0x20000

    In this example, we are increasing the handle history to 131072 (decimal, 0x20000 hexadecimal)

    !htrace –snapshot then hit <ENTER> key

    g then hit <ENTER> key

    3. Now, let the process run until the number of handle has increased a lot and gotten high.

    Final htrace log

    1. Break into debugger with Keyboard keys (Ctrl+Break)

    2. Run following commands:

    !htrace –diff then hit <ENTER> key

    .logclose then hit <ENTER> key

    .detach then hit <ENTER> key

    3. Close WinDbg

    Now complete the following actions once you have gotten your final htrace log

    1. If high handle count is with wmiprvse, download the latest version of the Windows Sysinternals tool Process Explorer.

    2. Find the instance of wmiprvse.exe with high handle count and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    3. If the WMI Service was the process with the high handle count, then dump out the WMI service process which should be wmisvchost.exe per previous directions and all instances of wmiprvse.exe using procdump. If it is wmiprvse.exe that is exhibiting the high handle count, then only need to dump out that instance and nothing else.

    a. Download Windows Sysinternals tool called Procdump from URL:

    b. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    c. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID

    d. Run the following command: procdump –ma <PID>

    e. Note: Replace <PID> with actual PID you documented for instances of wmiprvse.exe and/or wmisvchost.exe as it applies based on directions above

    4. Stop Perfmon logging

    5. Do a complete memory dump of the machine

    At this point with data in hand you will want to open a Support Incident with Microsoft to get the data analyzed to help determine cause of high handle count.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.