SupportingWindows

  • Driver Object Corruption Triggers Bugcheck 109

    My name is Victor Mei, I am an Escalation Engineer in Platforms Global Escalation Services in GCR.  Some customers I worked with have strong interests in debugging; but usually they got frustrated when I told them “To find the cause from this dump ...read more
  • Recovering Azure VM by attaching OS disk to another Azure VM

    If you are unable to administer an Azure VM because of RDP or SSH failures, in many cases rebooting or resizing the VM may resolve the issue. You can troubleshoot the VM by attaching the OS disk as a data disk to a different Azure VM using the steps ...read more
  • Surface Pro 3 Hibernation Doesn’t Occur on Enterprise Install

    Hi my name is Scott McArthur and I want to call out a recently published KB article:

    Surface Pro 3 doesn't hibernate after four hours in connected standby
    http://support2.microsoft.com/kb/2998588

    If you are deploying an image to Surface Pro 3, you are missing out on the feature where after 4 hours in Connected Standby the device will hibernate. This is a key feature related to battery life so I would recommend that all Enterprise customers install KB2955769 and incorporate these PowerCfg commands into your deployment.

    If you use Microsoft Deployment Toolkit 2013 for your deployments this is super easy. Here are the steps

    1. Under Packages, import KB2955769

    image

    2. Create PowerCfg_Sp3.batthat contains the following commands:

    REM sets CS battery saver time-out to four hours:
    powercfg /setdcvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 7398e821-3937-4469-b07b-33eb785aaca1 14400
    powercfg /setacvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 7398e821-3937-4469-b07b-33eb785aaca1 14400

    REM sets CS battery saver trip point to 100:
    powercfg /setdcvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 1e133d45-a325-48da-8769-14ae6dc1170b 100
    powercfg /setacvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f 1e133d45-a325-48da-8769-14ae6dc1170b 100

    REM sets the CS battery saver action to hibernate:
    powercfg /setdcvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f c10ce532-2eb1-4b3c-b3fe-374623cdcf07 001
    powercfg /setacvalueindex SCHEME_CURRENT e73a048d-bf27-4f12-9731-8b2076e8891f c10ce532-2eb1-4b3c-b3fe-374623cdcf07 001

    powercfg /setactive SCHEME_CURRENT

    3. Save PowerCfg_Sp3.bat to your Deploymentshare\Scriptsfolder

    4. Open up the task sequence you use to deploy Windows and add a custom task in the state restore phase called PowerCfg-SP3

    image

    5. In the properties of this task sequence step, edit the following:

    image

    6. Click the Options tab and add conditional for “Task Sequence variable model equals Surface Pro 3”

    image

    Note:This ensures this only runs on Surface Pro 3 devices using the model variable

    Hope this helps with your Surface deployments and keep eye on this blog for more tips and tricks for Surface

    Scott McArthur
    Senior Support Escalation Engineer

  • Your technical answers and automated solutions via Bing.com

    Hello folks,

    One of our Engineering PMs that supports our Diagnostics and Automated solutions published a blog regarding Bing and how you can use it to answer your technical questions and provide automated solutions.  Here is a brief overview:

    Bing Technical Instant Answers provide concise answers to technical questions directly within search results and hopefully answer your question (or help you solve an issue) without you having to actually visit the web pages linked within the answer. The answers are triggered by specific search phrases, and they try to provide a unique benefit either by precisely matching your intent or by providing additional content related to your intent. In some cases, the instant answer will link to an automated fix or troubleshooter that you can run directly from the Bing search results. Microsoft will constantly be adding new technical answers, so if you have a technical problem with a Microsoft product or service try asking Bing to see if we have an instant answer for you!

    Go check out his blog via the link below:

    Using Bing for technical instant answers and automated solutions

    -Blake

  • Cross Post: Using Bing for technical instant answers and automated solutions

    This is a cross post from William Keener’s Support Diagnostics and Automated Solutions blog that we wanted to add to our site.  It relates to Bing and instant answers about Microsoft Products/Technologies/Support issues and here on the AskCore site, we are all about getting this type of information out there.  Any comments made should be made on the originating post so it can be properly seen, heard, or answered.

    -------------------------------------------------------------

    Using Bing for technical instant answers and automated solutions
    http://blogs.msdn.com/b/williamk/archive/2014/10/31/using-bing-for-technical-instant-answers-and-automated-solutions.aspx

    Bing has been providing factual instant answers (and translation instant answers) for some time now, but recently they added "technical" instant answers for questions about Microsoft products and technologies or technical support issues. My previous team built the content management system that our internal content delivery teams are now using to add technical instant answers to Bing. Here's an example technical instant answer for the "Cortana" search term: 

    Now that I'm working on support diagnostics and automated solutions again, I have been working with the Bing and content delivery teams to get some instant answers created with links to some of our automated solutions.

    And I'm happy to announce that the first one is live! So you can now search for "Windows Update Troubleshooter" (or a variety of related terms and error messages) and the first result will be a technical instant answer with a link to download and run our automated troubleshooter to fix problems with Windows Update.

    When you click the link in step 3, you will be prompted to open (or run) or save the troubleshooter.

    Just click Open (or Run) to launch the troubleshooter.

    The content delivery teams will be constantly adding more technical instant answers, and we hope to have more live with automated solutions soon!

    Note that technical instant answers are also available in the Bing app on Windows Phone. To see the phone experience, tap Search and then type or say "cortana" on your Windows Phone. Then click the "See More" link at the bottom of the second result (after the ad - "Meet Cortana on Windows Phone 8.1") and swipe left or right to view the content on each of the tabs.

  • Understanding ATQ performance counters, yet another twist in the world of TLAs

    Hello again, this is guest author Herbert from Germany. If you worked an Active Directory performance issue, you might have noticed a number of AD Performance counters for NTDS and “Directory Services” objects including some ATQ related counters. In this ...read more
  • Unable to restart server due to registry bloat over 2GB

    Hello AskPerf!  Pushing up a blog today to discuss the registry bloat issue that has been recently addressed in the following KB:

    Computer cannot be restarted if the registry hives are larger than 2 GB

    Symptom

    • You have a computer that is running the x64-based version of Windows 8.1, Windows Server 2012 R2, Windows 8, or Windows Server 2012.
    • The registry hives for the computer are larger than 2 gigabyte (GB).

    Cause
    This problem occurs because of the 2 GB size limit of the registry hives in x64-based version of Windows.

    Resolution
    Install this patch to resolve the issue.

     

    When you get into this state, you may experience one of the following issues:

    1. You can boot to a stop error.
    2. You can boot and not be able to log in due to the RQL (Registry Quota Limit).
    3. You can boot and be logged in with a temp profile and not be able to install any software due to the RQL.

    If this happens, KB2978366 should be installed.

    With that, the following questions may come to mind:

    • How does this issue occur?
    • How do I prevent this issue in the first place?
    • How do I fix this issue once the hotfix is installed?
    • What happens if I see this problem on another OS version?
    • Are there any tools I can use to troubleshoot this issue?

    Question: How does this issue occur?

    Answer: There are many reasons that cause registry hives/keys to bloat.  Some of the ones we have seen are related to KB2871131, which refers to the “..\Printers\DevModes2” key bloat.  This hotfix does not “fix” the issue, but prevents it from occurring in the first place.  You still have to clean the keys first.  Additionally, there is a known issue with SQL Server 2012 SP1 that can cause the registry to hit the 2GB limit and put the machine in a no-boot state.  Please see KB2793634 for more details on this.

    Question: How do I prevent this issue in the first place?

    Answer: There really is no good answer for this outside of installing the hotfixes noted above, and keeping a close eye out on your registry hives.  You can use Performance Monitor however to monitor the “System\ % Registry Quota In Use” counter.  If this counter gets over 50 %, then you should start investigating what registry keys/hives are growing.

    clip_image001

    % Registry Quota In Use is the percentage of the Total Registry Quota Allowed that is currently being used by the system.  This counter displays the current percentage value only; it is not an average.

    NOTE The following Registry hives point to their corresponding files:

    • HKLM\BCD00000000 - \Boot\BCD
    • HKLM\COMPONENTS - %windir%\System32\config\Components
    • HKLM\SAM - %windir%\System32\config\SAM
    • HKLM\SECURITY - %windir%\System32\config\SECURITY
    • HKLM\SOFTWARE - %windir%\System32\config\SOFTWARE
    • HKLM\SYSTEM - %windir%\System32\config\SYSTEM
    • HKU\.DEFAULT - %windir%\System32\config\DEFAULT
    • HKCU - %userprofile%\NTUSER.DAT
    • HKLM\HARDWARE - This is dynamic and gets built with the OS boots (volatile hive)
    • HKLM\CLUSTER - %windir%\Cluster\CLUSDB
    • HKU\<SID of local service account> - %systemroot%\ServiceProfiles\LocalService\Ntuser.dat
    • HKU\<SID of network service account> - %systemroot%\ServiceProfiles\NetworkService\Ntuser.dat
    • HKU\<SID of username> - \Users\<username<\Ntuser.dat
    • HKU\<SID of username>\Classes - \Users\<username>\AppData\Local\Microsoft\Windows\Usrclass.dat

    Question: How do I fix this issue once the hotfix is installed?

    Answer:  After installing the hotfix, you may need to copy your Registry file to another machine that includes the hotfix.  After you have cleared out the bloated entries (whitespace will remain), then simply load the hive up, and then unload it.  This process will shrink your registry key back down pre-bloat. If a system is unbootable due to registry bloat install the hotfix on another system. Boot the problem system from DVD, copy the bloated registry hive to external storage, put on system with hotfix and use regedit to remove the bloated registry info and whitespace. The hive can then be copied back to problem system to allow it to boot normally.

    Question: What happens if I see this problem on another OS version?

    Answer:  Simply copy your hive over to a Win 8/ Server 2012 machine that has this hotfix installed, then follow the steps above.

    Question: Are there any tools I can use to troubleshoot this issue?

    Answer:  Coming Soon

    Reference

    How to Compress "Bloated" Registry Hives

    -Blake

  • RDP Fails with Event ID 1058 & Event 36870 with Remote Desktop Session Host Certificate & SSL Communication

    Hello AskPerf!  Sanket here from the Windows Platforms team here to discuss an issue with Remote Desktop Services where RDP does not work when you try to connect from a remote machine.  With that, let’s get started!

    I’m sure most of you have come across the following message when connecting to a machine via RDP:

    Remote Desktop Connection

    This computer can't connect to the remote computer. Try connecting again. If the problem continues, contact the owner of the remote computer or your network administrator.

    This is a generic that can be caused by numerous varying reasons.  We have a fairly detailed troubleshooting KB article that talks about this error and what to do to fix it:

    Remote Desktop disconnected or can’t connect to remote computer or to Remote Desktop server (Terminal Server) that is running Windows Server 2008 R2

    Assumptions are that most of you have followed this KB and resolved your issue.  However, there could other reasons that could cause RDP to fail as well.

    I recently worked an issue with same error where RDP from a remote machine was not connecting to a Windows 2012 Server.  NOTE the same error can occur on previous OS versions as well.

    There was a mystery as to what was changed on the server that could have caused this start.  Possible assumptions were user intervention, or some application may have changed/removed certain permissions.

    During the course of troubleshooting, we double-checked the KB article noted above, and noted the following Error events in the System Log:

    Log Name:      System
    Source:        Microsoft-Windows-TerminalServices-RemoteConnectionManager
    Date:          7/27/2014 12:16:59 AM
    Event ID:      1058
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      XXXXXXX
    Description: The RD Session Host Server has failed to replace the expired self-signed certificate used for RD Session Host Server authentication on SSL connections.
    The relevant status code was Access is denied.

     
    This error indicates that there is already a Certificate in place, however there is no sufficient permissions, and/or the default permissions on “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys” may have been modified.
     
    &
     
    Log Name:      System
    Source:        Schannel
    Date:          --
    Event ID:      36870
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      XXXXX
    Description: A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x8009030D.
    The internal error state is 10001.

     
    There was a fatal error accessing the Private Key for secure communications.

    At this point, I decided to capture a Process Monitor (Procmon) log on the destination server where the connection was going to.  As you may already know, Procmon allows us to monitor/record real-time file system, Registry and process/thread activity on Windows Workstations/Servers.

    Per the Procmon log, we found an “Access Denied” error to the following path:

    “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_”

    The above cert key f686aace6942fb7f7ceb231212eef4a4_xxx is associated with RDS, and this GUID like number is the pair key for both the computer and user.

    If you use the certutil -key command, you would see this Cert key with TSSecKeySet1:

    f686aace6942fb7f7ceb231212eef4a4_xxxxxxxxxx: AT_KEYEXCHANGE

    From the Procmon Logs:
     
    12:39:53.5364585 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: N/A, Share Mode: Read, Allocation Size: N/A,
    Impersonating: NT AUTHORITY\NETWORK SERVICE
     
    12:40:24.3692803 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
    Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a, Impersonating: NT AUTHORITY\SYSTEM
     
    12:40:23.9265708 AM svchost.exe 1012 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
    Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a
     
    So, what are the default permissions?  Well, you can use icacls to find this:
     
    C:\>icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\
    Everyone :(R,W)
    BUILTIN\Administrators :(F)
     
    c:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx
    NT AUTHORITY\NETWORK SERVICE :(R)
    NT AUTHORITY\SYSTEM :(F)
    BUILTIN\Administrators ::(R)

    In case if you want to grant permission using icals you can provide the same using following command :
     
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :( R)
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\SYSTEM :(F)
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :(R )

    Fig 1.1 (Permission in Windows Explorer)

    As you can see above, the SYSTEM accounts needs the proper permissions.  If these permissions have been changed, then they need put back to defaults.  The certs under this key should be inheriting the above permissions from the parent folder MachineKeys.

    You can restore permissions, grant the permissions back using icacls, or use the Windows Explorer GUI.  Correcting the default permission on the cert should allow RDP to now work correctly.

    Considering if this would have been easily reproducible, there is always an option to enable the Auditing on the cert key f686aace6942fb7f7ceb231212eef4a4_xxxxx under “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys”.  This can be done using the Security Tab on Properties of the cert key as seen in the screenshot below:

    NOTE Adding Auditing on this object will log Events to the Security Event Logs.  You will want to keep this enabled until you are able to reproduce the connection issue.

    Hope you find this information helpful.

    Additional Resources

  • Updating Surface Pro 3 firmware (Cross Post)

    Hi this is Scott McArthur and I just wanted to call attention to a blog that I worked on with some of our PFE engineers that just posted related to Surface. 

    How to Update the Surface Pro 3 Firmware Offline using a USB Drive
    http://blogs.technet.com/b/askpfeplat/archive/2014/10/20/how-to-update-the-surface-pro-3-firmware-offline-using-a-usb-drive.aspx

    This blog shows you how you can update firmware from a Bootable WindowsPE USB flash drive.  This is useful for some scenarios where you need updated firmware BEFOREyou do a deployment to the device.  Hope it helps with your Surface deployments

    Scott McArthur
    Senior Support Escalation Engineer

  • Asset Tag Tool for Surface Pro 3

    Hi my name is Scott McArthur and I want to call out a tool that recently came out that allows Enterprise customers to set Asset Tags on Surface Pro 3.

    This tool is available for download at the following location:

    http://www.microsoft.com/en-us/download/details.aspx?id=44076

    The tool requires the following:

    • Surface Pro 3(other Surface devices not supported)
    • UEFI firmware version 3.9.150.0 or newer

    It can be run from within Windows or from WindowsPE.  The download comes with a README.TXT that contains the following reference:

    This tool gets or sets the proposed Asset Tag, which will be applied on next reboot.

    The current Asset Tag is an SMBIOS setting which can be queried via WMI:

    (Get-WmiObject -query "Select * from Win32_SystemEnclosure").SMBiosAssetTag

    To get the proposed asset tag:

    AssetTag -g

    To clear the proposed asset tag:

    AssetTag -s

    To set the proposed asset tag:

    AssetTag -s testassettag12

    Valid values for this can be:

    • up to 36 characters long
    • valid characters including A-Z, a-z, 0-9, period and hyphen

    You can view the Asset Tag in the UEFI settings under Device Information.

    clip_image001

    Here is a PowerShell script demonstrating way to get proposed value and interpret errors.

    Note that stout contains the Asset Tag and stderr contains error messages.

    AssetTag -g > $asset_tag 2> $error_message
    $asset_tag_return_code = $LASTEXITCODE
    $asset_tag = $asset_tag.Trim("`r`n")

    if ($asset_tag_return_code -eq 0) {
         Write-Output ("Good Tag = " + $asset_tag)
    } else {
         Write-Output (
              "Failure: Code = " + $asset_tag_return_code +
              "Tag = " + $asset_tag +
              "Message = " + $error_message)

    }

    Hope this helps with your Deployments.

    Scott McArthur
    Senior Support Escalation Engineer

  • Configuring Azure Virtual Machines for Optimal Storage Performance

    In support, one of the most common questions we get is: How do I achieve the best disk performance for Azure virtual machines? Platform Planning: In the standard tier of virtual machine in Azure, the maximum IOPS is 500 per disk . When planning ...read more
  • Windows 10 Preview available for review

    Good morning AskPerf!  It’s been a while since our last post, and we apologize for that.  We’ve been quite busy here on the Support side knocking out customer issues…

    Any who, we have some upcoming blogs in the oven that need a little more time to bake.  One of which is a short series on Windows Event Forwarding which I am very excited about.  Look for that to come out in the coming months.

    Even though we are commonly known as the Performance team, internally we are known as the Reliability team.  Some of the technologies we support are as follows:

    Windows Client/Server OS

    • Printing
    • RDS / TS
    • Performance which includes System Hangs, High CPU, Memory issues, etc.
    • Base WMI functionality
    • COM/DCOM – base functionality
    • Explorer (Shell)
    • Desktop Search
    • MUI and IME
    • MSI – basic functionality
    • Themes/Fonts/Screen Savers/Wallpaper
    • Task Scheduler
    • WinRM – basic functionality
    • Windows PowerShell – install and basic functionality
    • ACT

    There are many other smaller technologies, but these are the main ones.

    Now back to our original topic:  The Windows 10 Preview is available for download/testing.  To get it, click the following link:

    Windows 10 Preview

    Finally, we always welcome feedback on topics you would like for us to blog about here on the AskPerf blog site.

    -Blake

  • Virtual Machine Checkpoint fails with Access Denied when running on a Clustered Shared Volume

    When you attempt to create a CheckPoint of a virtual machine that is running on a Cluster Shared Volume (CSV) , you may receive a General access denied error as shown below.

    clip_image001

    You will receive this error if the virtual machine’s VHD is placed on the root of the drive.

    clip_image002

    clip_image003

    The reason for the access denied error is due to the VM worker process (VMMS) not having relevant permissions on the CSV volume.  Below are default permission that is present for a typical CSV volume.  It is strongly recommended that these permissions not be changed.

    clip_image005

    To resolve the issue, migrate the storage from Failover Cluster Manager or reconfigure the VM and place the VHDX in a folder off the root.  By moving the VHDx to a subfolder or if the VM is reconfigured, the VMMS service updates the permissions on the subfolder as it should.

    For example, this is the current location of the file: 

    C:\ClusterStorage\Volume1\Test Lab.Vhdx

    You would want to move it (and any other VHDX files present) to a subfolder you can create, such as this: 

    C:\ClusterStorage\Volume1\Test Lab\Test Lab.Vhdx

    There are several options you can run through to accomplish this task.

    Option 1:

    Using the Virtual Machine Storageselection from Failover Cluster Manager, move it to the folder you created.  This is an option that can be done without affecting production as it can be done while the virtual machine is online and running.

    clip_image007

    clip_image008

    Option 2: 

    Shut the virtual machine down and, in Explorer, move the VHDx from the root of CSV to a folder you create.  In Failover Cluster Manager, bring up the settings of the virtual machine and manually change the path of the relocated VHDx.  This is an option that can be done but will affect production as it cannot be done while the virtual machine is online and running.  So you would need to schedule downtime to accomplish this task.

    General Rule:

    Microsoft has always not recommended to keep any type of data files in the root of a drive.  Even though things may appear to work fine, problems could arise from this configuration.

    Shasank Prasad
    Senior Support Escalation Engineer
    Microsoft Corporation

  • Remove Lingering Objects that cause AD Replication error 8606 and friends

    Introducing the Lingering Object Liquidator Hi all, Justin Turner here ---it's been a while since my last update . The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering ...read more
  • How to identify a driver that calls a Windows API leading to a pool leak on behalf of NT Kernel?

    Hello my name is Gurpreet Singh Jutla and I would like to share information on how we can trace the caller which ends up allocating “Se  “ Pool tag. When we use the Windows debugger and investigate the pool allocation and the binary associated with ...read more
  • Announcing public availability of MBAM Compliance Data Cleanup Tool 2.5

    We are happy to announce public availability of MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe), aka MBAMCDCT 2.5.
     
    MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe) is a command line tool which enables you to delete machine records from the ‘Compliance Status’ database of the MBAM 1.0 and MBAM 2.0, MBAM 2.0 SP1 and MBAM 2.5 standalone.

     

    There have been situation where you as a MBAM Admin had to delete the entries of older/reimaged machine records from the MBAM compliance database. The only solution in this case was to run complex SQL queries to delete machines from the database. This tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the MBAM Compliance Status database.

     

    This is a command line tool which enables you to schedule it stale data deletion as a task to automate deletion of obsolete machine records from the MBAM compliance database.


    This tool provides three different ways to delete machine records from the MBAM Compliance Status database:

    1.     Delete machines which have not reported in last X days.

    2.     Delete machines specified in a comma separated list via command line.

    3.     Delete machines specified in a text file.

     

    Note:

     

    This tool doesn’t delete the recovery information or any other data from MBAM Recovery and Hardware Database. All delete operations are performed strictly on the MBAM Compliance Status Database.

    This tool is available for download from the TechNet website http://gallery.technet.microsoft.com/MBAM-Compliance-Data-9b4c950d as a self-extractable compressed file, which includes the executable and documentation.

    Hope this tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the database.

    Disclaimer:

    This tool and documentation are provided "as-is". You bear the risk of using it. No express warranties, guarantees or conditions are provided. The tool supplied in this document is not supported under any Microsoft standard support program or service. However, you can report issues and bugs in the comments section on this page. Microsoft will, at its sole discretion, address issues and bugs reported.

     

     

    Himanshu Singh

    Windows Core Team

  • We Are Hiring Windows Escalation Engineers in Charlotte, Dallas, and Redmond

    Would you like to join the world’s best and most elite debuggers to enable the success of Microsoft solutions?   As a trusted advisor to our top customers you will be working with to the most experienced IT professionals and developers in the industry ...read more
  • Windows Troubleshooting – Stop 9E Explained

    What to do if a stop 9E occurs.  How you can solve the issue yourself. ...read more
  • Windows Troubleshooting – Special Pool

    The Windows Support team has a new YouTube channel, “ Windows Troubleshooting ”.  The first set of videos cover debugging blue screens. In this video, Bob Golding, Senior Escalation Engineer, describes how the Special Pool Windows diagnostics tool ...read more
  • Hate to see you go, but it’s time to move on to greener pastures. A farewell to Authorization Manger aka AzMan

    Hi all, Jason here. Long time reader, first time blogger. AzMan is Microsoft’s tool to manage authorization to applications based on a user’s role. AzMan has been around since 2003 and has had a good run. Now it’s time to send it out ...read more
  • Go the modern (app) way

    Hello everyone,

    This is Ashfana from the Windows Performance team. I’m here to talk about the basic philosophy behind Windows Store Apps which were introduced in Windows 8 and Windows 8.1. Windows Store apps were originally called Metro or modern apps before the RTM of the product. These are apps developed using a variety of languages like C#, C++, VB, HTML and Javascript, - which provide the modern, full screen immersive experience.

    Windows has the largest software ecosystem thanks to millions of developers around the world, and Microsoft’s focus on ensuring backward compatibility for application software. This ecosystem has evolved over time and has reached a stage where – we aren’t able to fix some of the limitations and move ahead without causing existing applications to break. Some limitations are installation/uninstallation problems, Software state corruption, uncontrolled modification of important system registries and files, unreliable installer components, one app causing another one to crash, and security risks when running Apps under elevated token. This new App model was introduced to address these reliability and security issues and to develop a new ecosystem that allows Apps to run on multiple hardware platforms with minimum efforts for the developer.

    Here are some of the features of Store Apps that make it very attractive for consumers and enterprises:

    • Windows Store apps do not need an installer. They are either sideloaded to the image using DISM, or installed via Add-AppxPackage PowerShell Command, or installed directly from the Store or a Azure based portal. You no longer need MSI or other installer components, Windows already has the necessary components that can extract the Appx package and install it.
    • Apps are installed per-user (unless they are sideloaded into the image before deployment). They maintain their state within folders in the user’s profile. This ensures better security.
      Apps run within a sandbox environment called an App container. This restricts an App from accessing much of the system and the users profile by default. Additional resources can be requested by an App by declaring capabilities within its manifest file. During first launch of the App, the user is prompted to allow/confirm access to these additional resources.
    • Desktop Apps run under medium integrity and are able to access full privileges granted to a user. An untrusted desktop application can pose a greater risk. Windows Store Apps on the other hand run under base privilege and hence provides a greater level of protection.
    • The Appx manifest and blockmap tell Windows how to install the App, chances of an installation going corrupt are very less. Even if an installation goes corrupt – it would not affect the system in any way.
    • Store Apps provide an immersive experience, are generally very lightweight, and are best suited for scenarios where hard core processing can be done at the server side and results can be delivered to the App over the network.
    • With the introduction of Universal Apps in Visual Studio 2013 update 2, developers can now write an App once and port it to run on multiple devices such as Xbox, Windows phones, PC, tablets and laptops. This makes application development time much shorter.

    Structure of a Windows Store/Universal App:

    Windows Store app are packaged into one or more files with the extension .appx. This appx package is the unit of installation for a Windows Store app. Appx packages are ZIP-based container files that contains the app’s payload files plus info needed to validate, deploy, manage, and update the app. From a high-level view, each Windows Store app package contains these items:

    image

    App payload
    App code files and assets
    Payload files are the code files and assets that you author when you create your Windows Store app.

    App manifest
    App manifest file (AppxManifest.xml)
    The app manifest declares the identity of the app, the app's capabilities, and info for deploying and updating.

    App block map
    App package’s block map file (AppxBlockMap.xml)
    The block map file lists all the app files contained in the package along with associated cryptographic hash values that the operating system uses to validate file integrity and to optimize an update for the app.

    App signature
    App package’s digital signature file (AppxSignature.p7x)
    The app package signature ensures that the package and contents haven't been modified after they were signed. If the signing certificate validates to a Trusted Root Certification Authorities Certificate, the signature also identifies who signed the package. The signer of the package is typically the publisher or author of the app.

    Packaging app:

    When you create an app using Visual Studio, it packages your app, it automatically adds the app block map and signature files to the package. But you can also use the standalone MakeAppx and SignTool utilities if you want to manually package your app.

    The steps that are involved in Package installation:

    image

    The Windows Store app deployment process occurs in multiple phases.

    1) In the first phase, Windows acquires and validates the app manifest, app block map, and app signature. It checks the OS version and dependencies, sees if you have sufficient disk space, and also whether this app is already installed.

    2) In the second phase Windows checks the app package’s deployment criteria to ensure that the app deployment will be successful.

    3) Windows stages the package’s contents on the disk in the %ProgramFiles%\WindowsApps directory in a new directory named after the package identity:

    <Package Name>_<Version>_<Architecture>_<ResourceID>_<Publisher Hash>

    4) Windows registers the package into the user's account. During this phase, the extensions that are declared in the manifest are registered with the operating system.

    The Future:

    With this release of Visual Studio 2013, we have set out to accomplish three major goals:

    1) Reach customers across phones, tablets, and PCs;

    2) Deliver innovation that supports developer investments;

    3) Make cross-platform technology easier and more capable.

    image

    Develop once for all Windows devices using a unified Windows runtime and VS tools that allow you to both support experiences unique to a device in XAML, HTML, and DirectX, and share the code that supports those experiences across all devices using C++, C#, or JavaScript. When your work is finished you can you can produce the app packages that you will submit to the Windows Store and Windows Phone Store with a single action to get your app out to customers on any Windows device. You may also just sign it and deploy this through SCCM.

    More links:

    Package manifest schema reference
    http://msdn.microsoft.com/en-us/library/windows/apps/br211473.aspx

    Capabilities of an App:
    http://msdn.microsoft.com/en-us/library/windows/apps/br211422.aspx

    Create an app package
    http://msdn.microsoft.com/en-us/library/windows/apps/hh975357.aspx

    App packager (MakeAppx.exe)
    http://msdn.microsoft.com/en-us/library/windows/apps/hh446767.aspx

    SignTool
    http://msdn.microsoft.com/en-us/library/windows/apps/ff551778.aspx

    How to create an app package signing certificate
    http://msdn.microsoft.com/en-us/library/windows/apps/jj835832.aspx

    Ashfana Begum
    Support Engineer
    Microsoft Performance Team

  • Deciphering Storport Traces 101

    Welcome back to the CORE Team Blog -- Paul Reynolds here. In previous blogs, I wrote about how to capture Storport traces in Windows 8 and Windows 2012. Please see:

    Tracing with Storport in Windows 2012 and Windows 8 with KB2819476 hotfix

    And

    Tracing with Storport in Windows 2012 and Windows 8 without KB2819476 hotfix

    This time around, I would like to explore what information you can draw from the raw data contained in a Storport trace. What conclusions can you reach regarding your disk performance? Do you have a disk that does not perform well?

    To accomplish this, we will take advantage of free tools available to Windows and Office 2013 users:

    • Windows PowerShell
    • Windows Performance Toolkit
    • Excel PowerPivot

    First, we need to talk briefly about the Windows Storage Stack and where Storport traces are taken. It is important to note that Storport traces are at the very last “rung of the ladder” before Windows hands off I/O request packets to hardware. Hardware in this case encompasses firmware, drivers, HBAs, storage fabrics – anything after the Windows Operating System. It is important because it can help delineate where the problem is. Is the problem with the Operating System or with the hardware? Should I call Microsoft to open a support case or it is more appropriate to talk to my Storage Vendor? Storport traces are the perfect place to start to answer questions such as these.

    It is assumed you already have a Storport Trace in hand – an ETL file captured using the procedures documented in the two blogs above for Windows 8 and Windows 2012, or Windows 2008 (see Bob Golding’s blog):

    Storport ETW Logging to Measure Requests Made to a Disk Unit

    As the old adage goes: a picture is worth a thousand words. When it comes time to deciphering Storport Traces, viewing graphs that summarize data and show trends, and viewing charts that have average and maximums, are much more helpful than looking at raw data. The top-level steps we will undertake are:

    1. Capture the Storport Trace
    2. Convert to the Storport Trace into CSV format
    3. Scrub the data in the CSV file
    4. Import the data into an Excel Spreadsheet

    The easiest way to accomplish the last 3 steps above is to automate it using a Windows PowerShell script. At the end of the blog, there is a zip to download that has this file. Save the PowerShell script as StorPortPACMAN.PS1 in a directory on your C: drive called StorPortPACMAN :

    PLEASE NOTE: you must have the following perquisites installed to successfully run the script above:

    1. XPERF needs to be installed and in your system path.

    Windows 7 or 8 – use the Windows Assessment and Deployment Kit (ADK): http://www.microsoft.com/en-US/download/details.aspx?id=39982

    2. PowerPivot is part of Office 2013 but it needs to be added as a COM Addin. Click here for instructions.

    At the end of this blog, there is also 2 spreadsheets in the zip you must download into the C:\StorPortPACMAN directory.

    Run the script, and depending on which version of Windows was being run on the server, one of the Excel Spreadsheets will open and be refreshed with data in your Storport trace.

    Here are screenshots of sample graphs and charts you will see in the first page of the spreadsheets:

    image

    This first graph shows request duration values over the time of the Storport trace. It is very useful to get the “big picture”. For the most part, the disk in this graph is performing fine except for a period near the beginning. Depending on the application you are investigating, this may be fine and the average value is what is important to you. Averages and maximums are shown in the chart below, which is next to this graph in the spreadsheet:

    image

    Finally, the chart data is presented in the graph below:

    image

    The Excel spreadsheets and their corresponding graphs and charts are only samples of what you can do for your disk analysis using Storport traces and Excel PowerPivot tools. Why use PowerPivot? For Storport traces it means being able to view and summarize more data. Excel by itself is an excellent tool, but problems may start to develop when your data approaches or exceeds a million rows of data.

    It is not uncommon for a Storport trace to contain tens of millions of rows of data, especially if you decide to not use a filter while capturing the data. I generally suggest to not use a filter as your averages will be closer to results you obtain from other tools such as Perfmon or XPERF. Using filters will cause your disk to look worse than it really is.

    Request Duration Times, as a rule of thumb, can be summed up as follows for SCSI Read(10) and Non-cached Write(10) I/O:

    <  9ms = excellent
    < 15ms = good
    < 25ms = fair
    > 25ms = poor

    We purposely use the caveat, rule of thumb, because these values are using an assumed 64KB data I/O size. If a read or write is larger or smaller than that, these values should be adjusted.

    Using an Excel Pivot table and chart lends well to focusing on a specific LUN through the use of filters built into the table or chart.

    Special Cases:

    We occasionally see a Storport trace that does not make sense. One example is a VM that has more than one SCSI disk and the Excel spreadsheet shows only one LUN. What happened to my other disk? It is there, but because of the way storage may be presented, the same LUN number is used for more than one disk. So what does one do in a situation such at this? The easiest way is to drop and drag a second property of a disk into the chart or graph you are viewing. There are 4 properties we gather in a Storport trace:

    1. LUN
    2. Target
    3. Bus
    4. Port

    Since using LUN is not enough, we can add a second property to the chart, Target. To do this, at the top of the spreadsheet click on the Analyze tab under Pivottable Tools, then click on Field List to expose the data available to you. Drag and Drop the Target field onto Rows and underneath LUN. You will see something similar to this:

    image

    This will let you see all your disks if they have the same LUN number.
    If you are at all like me, viewing charts and graphs might be fine for seeing the big picture and giving presentations, but your inner engineer is screaming to look at the data in detail, to make sure you are comfortable with it and understand it. To do this, click on the POWERPIVOT tab and then the Manage button in the Data Model area. This will open a new window that exposes the raw data to you so that you may filter it, sort it, or anything else you wish to make sure it all makes sense to you before using it.

    I hope you find this blog helpful and gives you a new tool to use when investigating disk performance.

    Paul Reynolds

  • WMI: How to Troubleshoot WMI High Handle Count

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is experiencing high handle count

    Your first thing to do is check the Application Event log for following event:

    Source: Microsoft-Windows-WMI

    Event 5612 Wmiprvse.exe exceeding handle quota limit Event

    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    If you find the above Event, you can try and bump up the handle quota limit to see if it resolves your issue. If it is a leak, then bumping limit will only mean it will take longer to reach the new limit. If it’s just load related, then bumping the limit could resolve the issue.

    The event will tell you what the handle count was, and if it is higher than the 8192 value I suggest below.  You can then skip this section and move on to data collection to figure out the cause of high handle count.

    How to increase the handle quota limit for the WMI Provider Service

    1. Go to Start--> run and type wbemtest.exe
    2. Click Connect on the Windows Management Instrumentation Tester
    3. In the namespace text box just enter "root" (without quotes)
    4. Click Connect

    clip_image001

    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    1. Click "Enum Instances…"

    clip_image002

    1. In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK.

    Note: a double underscore __ precedes ProviderHostQuotaConfiguration

    clip_image003

    1. A query Result window will come up. In this windows now double click "__ProviderHostQuotaConfiguration=@"

    clip_image004

    1. An Object Editor windows will come up now
    2. Under properties find the property "HandlesPerHost"

    clip_image005

    1. Change the value from default of 4096 to 8192
    2. Click Save Property
    3. Click Save Object in the Object Editor window
    4. Close the other windows now and exit WMI Tester
    5. Restart Windows Management Instrumentation Service.

    If after bumping up quota limit and wmiprvse still exceeding quota limit, accomplish following actions below. You will want to read through the rest in its entirety to ensure you get all of the necessary tools downloaded before taking any actions.

    Configure System for Complete Memory Dump by referring to:

    Windows 8 and Windows Server 2012 Automatic Memory Dump: http://blogs.technet.com/b/askcore/archive/2012/09/12/windows-8-and-windows-server-2012-automatic-memory-dump.aspx

    Windows does not create a memory dump file when a Stop error occurs in Windows 8 or Windows Server 2012: http://support.microsoft.com/kb/2853466

    Windows 2008, Windows Vista, Windows 7, Windows 2008 R2: http://support.microsoft.com/kb/969028

    Windows Server 2003 and Windows XP: http://support.microsoft.com/kb/972110

    Collect perfmon logging using logman method

    Directions below will create 2 perfmon logs, one at a 5 minute interval (PerfLog-Long) and a short 5 second interval log (PerfLog-Short) and they will be placed in C:\Perflogs folder.

    • Long log (5 min intervals) – no thread counter, 250 MB:

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Long -o "c:\perflogs\PerfLog-Long.blg" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues\*" "\System\*"  -si 00:05:00

    3. Start the log with:

    Logman.exe start PerfLog-Long

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Long

    • Short, high resolution log – 5 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:05

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Configure Handle Tracing

    You probably just need the standalone version since we only need the debugging tool and not the whole WDK package.

      1. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator
      2. Click on the ‘Kernel Flags’ tab
      3. Check the box next to ‘Enable bad handles detection’
      4. Click on the ‘Image File’ tab
      5. Next to ‘Image: (TAB to refresh)’, enter wmiprvse.exe or if tracing WMI Service enter the unique name given to the svchost process for WMI Service per directions in next section below
      6. And then click on the ‘Tab’ key
      7. Next to ‘Stack Backtrace: (Megs):’ enter ‘10’
      8. Click on Apply
      9. Click on Ok
      10. Restart WMI service

    If it is a svchost process showing high handle count, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From there in a command prompt you can run tasklist /svc and look for the PID then identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From experience it will be the WMI service more times than not but not always, as such I would try to break it out first on its own and monitor to see if it is the one driving up high handle count in the shared svchost process.

    WMI (Windows Management Instrumentation) service, you can break it out by accomplishing the following.

    Break WMI Service out into its own unique svchost process

    a.      Open command prompt with elevated privileges

    b.      Run following command: sc config winmgmt type= own

    c.      Restart Wmi service

    d.      Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    sc config <service name> type= share

    e. Restart the service or machine and verify result is Win32_SHARE_PROCESS when you run sc query winmgmt command again

    f. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe

    g. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt

    h. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    I. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    j. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    k. You would now substitute wmisvchost.exe in lieu of wmiprvse.exe in step 6. under Configure Handle Tracing above

    Using debugger to attach to the process in windbg.exe and running !htrace –enable command

    1. Launch WinDbg program from under Debugging Tools for Windows that you installed earlier.

    2. Created folder c:\websymbols

    3. Click on File-Symbol path and add the following symbol path to the debugger: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    4. Click on File-Save Workspace

    Attach to process to accomplish handle tracking using htrace

    To do this:

    1. From Windbg - File - Attach a Process - Select the instance of wmiprvse.exe with high handle count

    Note: If it is WMI Service (run tasklist /svc or Task Manager with PID column added first to find the PID of svchost.exe containing winmgmt which you should have broken out and uniquely named wmisvchost.exe per earlier directions)

    2. Run following command from the debugger:

    .logopen "C:\debug.log" then hit <ENTER> key

    !htrace -enable 0x20000 then hit <ENTER> key

    Note:  By default, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 keep a history of 4000 handles open and close operations. 

    With !htrace you can enable to keep a much higher history by doing the following:

    !htrace -enable 0x20000

    In this example, we are increasing the handle history to 131072 (decimal, 0x20000 hexadecimal)

    !htrace –snapshot then hit <ENTER> key

    g then hit <ENTER> key

    3. Now, let the process run until the number of handle has increased a lot and gotten high.

    Final htrace log

    1. Break into debugger with Keyboard keys (Ctrl+Break)

    2. Run following commands:

    !htrace –diff then hit <ENTER> key

    .logclose then hit <ENTER> key

    .detach then hit <ENTER> key

    3. Close WinDbg

    Now complete the following actions once you have gotten your final htrace log

    1. If high handle count is with wmiprvse, download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    2. Find the instance of wmiprvse.exe with high handle count and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    3. If the WMI Service was the process with the high handle count, then dump out the WMI service process which should be wmisvchost.exe per previous directions and all instances of wmiprvse.exe using procdump. If it is wmiprvse.exe that is exhibiting the high handle count, then only need to dump out that instance and nothing else.

    a. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    b. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    c. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID

    d. Run the following command: procdump –ma <PID>

    e. Note: Replace <PID> with actual PID you documented for instances of wmiprvse.exe and/or wmisvchost.exe as it applies based on directions above

    4. Stop Perfmon logging

    5. Do a complete memory dump of the machine

    At this point with data in hand you will want to open a Support Incident with Microsoft to get the data analyzed to help determine cause of high handle count.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

  • WMI: How to troubleshoot High CPU Usage by WMI Components

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of CPU.

    In the directions below, you may have already broken out WMI Service to troubleshoot your issue.  By default, WMI runs in the main shared networking svchost process with several other services.

    If it is a svchost process showing high cpu usage, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From inside a command prompt you can type in  tasklist /svc and look for the PID #, and identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From my experience, it will be the WMI service more times than not but not always.  As such, I would suggest breaking it out first into its own, and monitor to see if it is the one driving up high memory usage in the shared svchost process.

    If you suspect the WMI (Windows Management Instrumentation) service, you can break it out following directions below.

    Break WMI Service out into its own svchost process

    1. Open command prompt with elevated privileges
    2. Run following command: sc config winmgmt type= own
    3. Restart Wmi service
    4. Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    • sc config <service name> type= share
    • Restart the service or machine and verify result is Win32_SHARE_PROCESS when you
    • run sc query winmgmt command again

    Configure Perfmon Collection using logman.exe method. Capture 15 minutes while issue is occurring.

    Short, high resolution log – 1 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window (if this does not work, you may need to manually type it in):

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:01

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Collect and Xperf trace for High CPU by using the Windows Performance Recorder form the Windows Performance Toolkit which you can install from the ADK

    Note: If the Operating System is a 64 bit box, you must first accomplish the following registry setting before collecting Xperf trace.

    Registry Path
    HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
    Setting
    DisablePagingExecutive
    Data Type:
    REG_DWORD
    Value:
    1

    NOTE setting this key is not needed on Windows Server 2012 & 2012 R2

    Reboot machine to place registry setting into effect.

    1. Download the Windows 8 ADK (Windows Assessment Deployment Kit) from here.
    2. Open the adksetup.exe and hit next until you get you the option to select feature options
    3. Select "Windows Performance Toolkit" and hit "Install"

    clip_image002

    After installation has finished, start creating a trace by starting the "Windows Performance Recorder"

    clip_image004

    Select CPU usage under Resource Analysis

    Logging mode can be left set to “Memory”, or you can change to “File”. Just be conscious of your disk space if you chose “File” as the etl file can become large fast

    Capture high cpu occurrence, but do not let the recording run for no more than 10 minutes.

    Immediately after capturing the event using Windows Performance Recorder (WPR), now use process explorer to dump out the process exhibiting high cpu usage.

    1. Download Windows Sysinternals tool called Procdump: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    2. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    3. Open Task Manager and add the PID column view, then go locate the instance of wmiprvse.exe with high cpu usage and note the PID. If it was the WMI service that had the high cpu, then you should already have it broken out to run in its own svchost process and note the PID of that svchost process. To confirm you have the right svchost process, you can run tasklist /svc from administrative command prompt and verify the PID noted in task manager and ensure it is the svchost process running winmgmt in it.

    4. Run the following command: procdump –ma -s 60 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe or for the svchost process running winmgmt exhibiting high memory usage

    The above command will produce 3 dumps spaced 1 minute apart each in same directory you ran the procdump command from

    5. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    6. If it was wmiprvse.exe that had the high CPU usage, then find the instance and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high CPU usage.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up:  WMI: How to Troubleshoot WMI High Handle Count

  • WMI: High Memory Usage by WMI Service or Wmiprvse.exe

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of memory.

    There are many reasons why WMI might experience high memory consumptions. This can occur in the WMI (Windows Management Instrumentation) service (winmgmt) or in the WMI provider hosting vehicle wmiprvse.exe. Both scenarios will be addressed below.

    High memory usage may simply be due to load, as opposed to some type of leak. By default, the memory quota limit for instances of wmiprvse.exe on Windows XP and Windows Server 2003 is 128 MB, and 512 MB on newer Operating Systems (Vista and higher). Hit those limits and wmi functionality will become problematic if not come to a grinding halt.

    There is another case where the quota limit problem could happen. There is limit for all wmiprvse’s cumulatively. If the total memory of all instances of wmiprvse’s (with one exception) together reaches 1GB, then all new memory allocations fail in all wmiprvse processes.

    As a first effort you can try to bump up that quota limit to see if it gives enough room for wmiprvse to operate, but if you still reach the new quota limit, then you will want to proceed with the Scenarios below

    How To Bump up WMI memory quota limit:

    • Go to Start--> Run and type wbemtest.exe
    • Click Connect

    clip_image001 

     

    • In the namespace text box type "root" (without quotes).

    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    • Click Connect
    • Click Enum Instances…

    clip_image002

     

    • In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK. Note: the Superclass name includes a double underscore at the front.

    clip_image003

     

    • In the Query Result window, double-click "__ProviderHostQuotaConfiguration=@"

    clip_image004

     

    • In the Object Editor window, double-click whichever Property name you wish to modify the quota for. In this case that will be MemoryPerHost

    clip_image005

     

    • In the Value dialog, type in 536870912 (512 MB) for XP and Windows 2003.  For Vista and higher, start with new value of 805306368 (768), then move to 1073741824 (1024) if still needing room. At the same time, if you are going to bump up the MemoryPerHostLimit, you should also then bump up the MemoryAllHost limit to 2147483648 (2048) provided you have the available ram to do so. If this does not resolve quota violation issue, then need to troubleshoot cause of high memory usage following below Scenario section
    • Click Save Property.
    • Click Save Object.
    • Close Wbemtest.
    • Restart the machine

    Note: if you cannot restart the machine, then as a workaround, you can break Windows Management Instrumentation service into its own svchost process outline in step 2. a-c below.

    If bumping the quota limits does not resolve the issue, then as workaround you can try to move suspected leaking providers into their own group (wmiprvse) to avoid the memory quota caused by other providers running in the group or kill the instance of wmiprvse exhibiting high memory until issue is resolved. This is outlined below in Scenarios 1a and 2a.

    First thing we need to determine is if the memory consumption being caused by private data or heap data. We have to address the 2 types differently.

    1. Download a Windows Sysinternals tool call VMMap from following url: http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx

    VMMap is a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types.

    This tool is used to attach to an individual process allowing a snapshot to be taken to see the memory map for that process.

    2. Simply launch VMMap and from the process list it displays, pick the instance of wmiprvse showing the high working set memory usage.

    3. If it is a svchost process that is exhibiting high memory, from a command run scqueryex winmgmt to identify the PID of the svchost process that is hosting WMI Service (winmgmt). From experience it will be the WMI service more times than not but not always as the service using majority of the memory; as such I would try to break it out first on its own and monitor to see if it is the one driving up high memory usage in the shared svchost process.

    From this point, I will assume it is the WMI service as this article is only addressing WMI and no other services.

    You will need to break the WMI service out into its own svchost process first if it hasn’t already been accomplished so you can run VMMap specifically against the WMI service to truly know if it is being consumed by Heap or Private Data. By default the WMI service runs in a shared svchost process with a community of other services. You can do so by the following directions. If it is caused by Heap, you will find later in the Scenario directions below that you will have to go back and uniquely name the svchost process for the service to run in to be able to enable User Stack Tracing against it.

    a. Open command prompted with elevated privileges

    b. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own

    c. Restart wmi service with net stop winmgmt and net start winmgmt commands

    d. Verify winmgmt service running in its own svchost process by runnning tasklist /svc

    4. Once it displays the result, look under the size column for Private Data and Heap. This should tell you if the majority of the memory being consumed is Private Data or Heap.

    5. Follow appropriate Scenario below based on if issue is with Windows Management Instrumentation service or wmiprvse and if caused by Heap or Private Data memory.

    Scenario 1a: High Memory Consumption by Wmiprvse.exe caused by Heap memory

    1. Download Windows Debugging Tools for Windows from following link: http://msdn.microsoft.com/en-us/windows/hh852365.aspx

    You can probably just get the standalone version since we only need the debugging tool and not the whole WDK package.

    2. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator

    3. Click on “Image File” tab

    4. Type in wmiprvse.exe

    5. Hit the keyboard TAB key

    6. Place check mark in “Create user mode stack trace database

    7. Click Ok

    8. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    9. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high memory usage:

    wmic path msft_providers get Provider,HostProcessIdentifier /format:list

    10. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

    11. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

    $prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
    <ENTER>
    $prv.HostingModel = $Prv.HostingModel + ":OWN"
    <ENTER>
    set-ciminstance -inputobject $prv
    <ENTER>

    Repeat the above for each Provider found in step 9

    12. Restarting of Wmi service will put the settings into effect without having to reboot machine

    13. Now you will want to take a process dump of the instance of wmiprvse when memory usage is high. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    14. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

    15. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

    16. Run the following command: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

    The above command will produce 3 dumps spaced at 5 minutes apart each in same directory you ran the procdump command from

    17. Note also as workaround until issue gets resolved you could simply kill the instance of wmiprvse with high memory from task manager or from PowerShell can run following:

    kill -f <pid of suspect wmiprvse>

    At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high memory usage

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 1b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Heap memory

    The Windows Management Instrumentation service runs under the display name of winmgmt and is located in networking svchost.exe process shared along with several other services. You need to break the WMI service out into its own unique svchost process for data collection purposes.

    NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 6 to uniquely name the svchost process it is running in. We do this because we only want to enable User Stack Tracing against just the WMI service and not every svchost process.

    1. Open command prompted with elevated privileges
    2. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own
    3. Restart wmi service with net stop winmgmt and net start winmgmt commands
    4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
    5. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe
    6. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt
    7. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    Note: It is important that you go back and reverse what you did in step 7 and modify path back to original after you are no longer needing the service to be broken out and uniquely named as failure to do this can prevent future WMI hotfixes from being installed.

    Simply run following command then restart wmi service: sc config winmgmt type= share

    8. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    9. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    10. For sake of brevity here, follow steps 4-14 from Scenario 1a with the exception of typing in wmisvchost.exe in place or wmiprvse.exe for process we are enabling “Create user mode stack trace database” against.

    11. Here we want to use procdump to dump the wmisvchost.exe process along with every instance of wmiprvse.exe that is running. Reference steps 9-11 listed in Scenario 1a above.

    First command will dump wmisvchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running.

    Command for dumping wmisvchost.exe would be: procdump –ma -s 300 -n 3 wmisvchost.exe

    Command for dumping each instance of wmiprvse just once: procdump –ma <PID>

    Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

    At this point you will now need to open a Support Incident Case with Microsoft to get the process dumps analyzed to determine cause of high memory usage

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2a: High Memory Consumption by Wmiprvse.exe caused by Private Data memory

    This scenario is a little bit different than being caused by Heap, as just dumps in of themselves normally do not tell the full story. In this type of case, we need to know what type of wmi activity is occurring and it becomes significantly harder and more involved to try and determine cause.

    We will need to collect procdumps still, but also need to add in WMI activity logging.

    1. From start run type in eventvwr.msc

    2. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it

    3. Expand Applications and Services\Microsoft\Windows\WMI-Activity

              Right click on items Debug and Trace and select “Enable” for each one

    4. Download the latest version of the Windows Sysinternals tool Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx.

    5. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high

    memory usage:

    wmic path msft_providers get Provider,HostProcessIdentifier /format:list

    6. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

    7. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

    $prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
    <ENTER>
    $prv.HostingModel = $Prv.HostingModel + ":OWN"
    <ENTER>
    set-ciminstance -inputobject $prv
    <ENTER>

    Repeat the above for each Provider found in step 5

    8. Restarting of Wmi service will put the settings into effect without having to reboot machine

    9. When memory usage is high, use Procdump tool to dump out the wmiprvse.exe exhibiting high memory usage. http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    10. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

    11. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

    12. Run the following command: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

    The above command will produce 3 dumps space out 5 minutes apart each in same directory you ran the procdump command from

    13. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865

    14. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

    At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Private Data memory

    NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 5

    1. Open command prompted with elevated privileges
    2. Break WMI service out into its own svchost process by running following command: sc config winmgmt type= own
    3. Restart WMI service with net stop winmgmt and net start winmgmt commands
    4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
    5. From start run type in eventvwr.msc
    6. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it
    7. Expand Applications and Services\Microsoft\Windows\WMI-Activity

              Right click on items Debug and Trace and select “Enable” for each one

    8. Here we want to use procdump to dump the svchost process housing the WMI Service along with every instance of wmiprvse.exe that is running. Reference steps 9-10 listed in Scenario 1a above

    a. From command prompt run tasklist /svc and locate the instance of svchost that winmgmt is running in and note the PID

    b. First command will dump svchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running just once

    Command for dumping svchost.exe housing winmgmt would be: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID for svchost process housing winmgmt

    Command for dumping each instance of wmiprvse running would be: procdump –ma <PID>

    Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

     

    1. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865
    2. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

    At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up: WMI: How to troubleshoot High CPU Usage by WMI Components