SupportingWindows

  • Understanding ATQ performance counters, yet another twist in the world of TLAs

    Hello again, this is guest author Herbert from Germany. If you worked an Active Directory performance issue, you might have noticed a number of AD Performance counters for NTDS and “Directory Services” objects including some ATQ related counters. In this ...read more
  • Unable to restart server due to registry bloat over 2GB

    Hello AskPerf!  Pushing up a blog today to discuss the registry bloat issue that has been recently addressed in the following KB:

    Computer cannot be restarted if the registry hives are larger than 2 GB

    Symptom

    • You have a computer that is running the x64-based version of Windows 8.1, Windows Server 2012 R2, Windows 8, or Windows Server 2012.
    • The registry hives for the computer are larger than 2 gigabyte (GB).

    Cause
    This problem occurs because of the 2 GB size limit of the registry hives in x64-based version of Windows.

    Resolution
    Install this patch to resolve the issue.

     

    When you get into this state, you may experience one of the following issues:

    1. You can boot to a stop error.
    2. You can boot and not be able to log in due to the RQL (Registry Quota Limit).
    3. You can boot and be logged in with a temp profile and not be able to install any software due to the RQL.

    If this happens, KB2978366 should be installed.

    With that, the following questions may come to mind:

    • How does this issue occur?
    • How do I prevent this issue in the first place?
    • How do I fix this issue once the hotfix is installed?
    • What happens if I see this problem on another OS version?
    • Are there any tools I can use to troubleshoot this issue?

    Question: How does this issue occur?

    Answer: There are many reasons that cause registry hives/keys to bloat.  Some of the ones we have seen are related to KB2871131, which refers to the “..\Printers\DevModes2” key bloat.  This hotfix does not “fix” the issue, but prevents it from occurring in the first place.  You still have to clean the keys first.  Additionally, there is a known issue with SQL Server 2012 SP1 that can cause the registry to hit the 2GB limit and put the machine in a no-boot state.  Please see KB2793634 for more details on this.

    Question: How do I prevent this issue in the first place?

    Answer: There really is no good answer for this outside of installing the hotfixes noted above, and keeping a close eye out on your registry hives.  You can use Performance Monitor however to monitor the “System\ % Registry Quota In Use” counter.  If this counter gets over 50 %, then you should start investigating what registry keys/hives are growing.

    clip_image001

    % Registry Quota In Use is the percentage of the Total Registry Quota Allowed that is currently being used by the system.  This counter displays the current percentage value only; it is not an average.

    NOTE The following Registry hives point to their corresponding files:

    • HKLM\BCD00000000 - \Boot\BCD
    • HKLM\COMPONENTS - %windir%\System32\config\Components
    • HKLM\SAM - %windir%\System32\config\SAM
    • HKLM\SECURITY - %windir%\System32\config\SECURITY
    • HKLM\SOFTWARE - %windir%\System32\config\SOFTWARE
    • HKLM\SYSTEM - %windir%\System32\config\SYSTEM
    • HKU\.DEFAULT - %windir%\System32\config\DEFAULT
    • HKCU - %userprofile%\NTUSER.DAT
    • HKLM\HARDWARE - This is dynamic and gets built with the OS boots (volatile hive)
    • HKLM\CLUSTER - %windir%\Cluster\CLUSDB
    • HKU\<SID of local service account> - %systemroot%\ServiceProfiles\LocalService\Ntuser.dat
    • HKU\<SID of network service account> - %systemroot%\ServiceProfiles\NetworkService\Ntuser.dat
    • HKU\<SID of username> - \Users\<username<\Ntuser.dat
    • HKU\<SID of username>\Classes - \Users\<username>\AppData\Local\Microsoft\Windows\Usrclass.dat

    Question: How do I fix this issue once the hotfix is installed?

    Answer:  After installing the hotfix, you may need to copy your Registry file to another machine that includes the hotfix.  After you have cleared out the bloated entries (whitespace will remain), then simply load the hive up, and then unload it.  This process will shrink your registry key back down pre-bloat. If a system is unbootable due to registry bloat install the hotfix on another system. Boot the problem system from DVD, copy the bloated registry hive to external storage, put on system with hotfix and use regedit to remove the bloated registry info and whitespace. The hive can then be copied back to problem system to allow it to boot normally.

    Question: What happens if I see this problem on another OS version?

    Answer:  Simply copy your hive over to a Win 8/ Server 2012 machine that has this hotfix installed, then follow the steps above.

    Question: Are there any tools I can use to troubleshoot this issue?

    Answer:  Coming Soon

    Reference

    How to Compress "Bloated" Registry Hives

    -Blake

  • RDP Fails with Event ID 1058 & Event 36870 with Remote Desktop Session Host Certificate & SSL Communication

    Hello AskPerf!  Sanket here from the Windows Platforms team here to discuss an issue with Remote Desktop Services where RDP does not work when you try to connect from a remote machine.  With that, let’s get started!

    I’m sure most of you have come across the following message when connecting to a machine via RDP:

    Remote Desktop Connection

    This computer can't connect to the remote computer. Try connecting again. If the problem continues, contact the owner of the remote computer or your network administrator.

    This is a generic that can be caused by numerous varying reasons.  We have a fairly detailed troubleshooting KB article that talks about this error and what to do to fix it:

    Remote Desktop disconnected or can’t connect to remote computer or to Remote Desktop server (Terminal Server) that is running Windows Server 2008 R2

    Assumptions are that most of you have followed this KB and resolved your issue.  However, there could other reasons that could cause RDP to fail as well.

    I recently worked an issue with same error where RDP from a remote machine was not connecting to a Windows 2012 Server.  NOTE the same error can occur on previous OS versions as well.

    There was a mystery as to what was changed on the server that could have caused this start.  Possible assumptions were user intervention, or some application may have changed/removed certain permissions.

    During the course of troubleshooting, we double-checked the KB article noted above, and noted the following Error events in the System Log:

    Log Name:      System
    Source:        Microsoft-Windows-TerminalServices-RemoteConnectionManager
    Date:          7/27/2014 12:16:59 AM
    Event ID:      1058
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      XXXXXXX
    Description: The RD Session Host Server has failed to replace the expired self-signed certificate used for RD Session Host Server authentication on SSL connections.
    The relevant status code was Access is denied.

     
    This error indicates that there is already a Certificate in place, however there is no sufficient permissions, and/or the default permissions on “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys” may have been modified.
     
    &
     
    Log Name:      System
    Source:        Schannel
    Date:          --
    Event ID:      36870
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      XXXXX
    Description: A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x8009030D.
    The internal error state is 10001.

     
    There was a fatal error accessing the Private Key for secure communications.

    At this point, I decided to capture a Process Monitor (Procmon) log on the destination server where the connection was going to.  As you may already know, Procmon allows us to monitor/record real-time file system, Registry and process/thread activity on Windows Workstations/Servers.

    Per the Procmon log, we found an “Access Denied” error to the following path:

    “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_”

    The above cert key f686aace6942fb7f7ceb231212eef4a4_xxx is associated with RDS, and this GUID like number is the pair key for both the computer and user.

    If you use the certutil -key command, you would see this Cert key with TSSecKeySet1:

    f686aace6942fb7f7ceb231212eef4a4_xxxxxxxxxx: AT_KEYEXCHANGE

    From the Procmon Logs:
     
    12:39:53.5364585 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: N/A, Share Mode: Read, Allocation Size: N/A,
    Impersonating: NT AUTHORITY\NETWORK SERVICE
     
    12:40:24.3692803 AM lsass.exe 588 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
    Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a, Impersonating: NT AUTHORITY\SYSTEM
     
    12:40:23.9265708 AM svchost.exe 1012 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxx ACCESS DENIED
    Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, Share Mode: Read, Allocation Size: n/a
     
    So, what are the default permissions?  Well, you can use icacls to find this:
     
    C:\>icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\
    Everyone :(R,W)
    BUILTIN\Administrators :(F)
     
    c:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx
    NT AUTHORITY\NETWORK SERVICE :(R)
    NT AUTHORITY\SYSTEM :(F)
    BUILTIN\Administrators ::(R)

    In case if you want to grant permission using icals you can provide the same using following command :
     
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :( R)
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\SYSTEM :(F)
    icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_xxxxx /grant " NT AUTHORITY\NETWORK SERVICE :(R )

    Fig 1.1 (Permission in Windows Explorer)

    As you can see above, the SYSTEM accounts needs the proper permissions.  If these permissions have been changed, then they need put back to defaults.  The certs under this key should be inheriting the above permissions from the parent folder MachineKeys.

    You can restore permissions, grant the permissions back using icacls, or use the Windows Explorer GUI.  Correcting the default permission on the cert should allow RDP to now work correctly.

    Considering if this would have been easily reproducible, there is always an option to enable the Auditing on the cert key f686aace6942fb7f7ceb231212eef4a4_xxxxx under “C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys”.  This can be done using the Security Tab on Properties of the cert key as seen in the screenshot below:

    NOTE Adding Auditing on this object will log Events to the Security Event Logs.  You will want to keep this enabled until you are able to reproduce the connection issue.

    Hope you find this information helpful.

    Additional Resources

  • Updating Surface Pro 3 firmware (Cross Post)

    Hi this is Scott McArthur and I just wanted to call attention to a blog that I worked on with some of our PFE engineers that just posted related to Surface. 

    How to Update the Surface Pro 3 Firmware Offline using a USB Drive
    http://blogs.technet.com/b/askpfeplat/archive/2014/10/20/how-to-update-the-surface-pro-3-firmware-offline-using-a-usb-drive.aspx

    This blog shows you how you can update firmware from a Bootable WindowsPE USB flash drive.  This is useful for some scenarios where you need updated firmware BEFOREyou do a deployment to the device.  Hope it helps with your Surface deployments

    Scott McArthur
    Senior Support Escalation Engineer

  • Asset Tag Tool for Surface Pro 3

    Hi my name is Scott McArthur and I want to call out a tool that recently came out that allows Enterprise customers to set Asset Tags on Surface Pro 3.

    This tool is available for download at the following location:

    http://www.microsoft.com/en-us/download/details.aspx?id=44076

    The tool requires the following:

    • Surface Pro 3(other Surface devices not supported)
    • UEFI firmware version 3.9.150.0 or newer

    It can be run from within Windows or from WindowsPE.  The download comes with a README.TXT that contains the following reference:

    This tool gets or sets the proposed Asset Tag, which will be applied on next reboot.

    The current Asset Tag is an SMBIOS setting which can be queried via WMI:

    (Get-WmiObject -query "Select * from Win32_SystemEnclosure").SMBiosAssetTag

    To get the proposed asset tag:

    AssetTag -g

    To clear the proposed asset tag:

    AssetTag -s

    To set the proposed asset tag:

    AssetTag -s testassettag12

    Valid values for this can be:

    • up to 36 characters long
    • valid characters including A-Z, a-z, 0-9, period and hyphen

    You can view the Asset Tag in the UEFI settings under Device Information.

    clip_image001

    Here is a PowerShell script demonstrating way to get proposed value and interpret errors.

    Note that stout contains the Asset Tag and stderr contains error messages.

    AssetTag -g > $asset_tag 2> $error_message
    $asset_tag_return_code = $LASTEXITCODE
    $asset_tag = $asset_tag.Trim("`r`n")

    if ($asset_tag_return_code -eq 0) {
         Write-Output ("Good Tag = " + $asset_tag)
    } else {
         Write-Output (
              "Failure: Code = " + $asset_tag_return_code +
              "Tag = " + $asset_tag +
              "Message = " + $error_message)

    }

    Hope this helps with your Deployments.

    Scott McArthur
    Senior Support Escalation Engineer

  • Configuring Azure Virtual Machines for Optimal Storage Performance

    In support, one of the most common questions we get is: How do I achieve the best disk performance for Azure virtual machines? Platform Planning: In the standard tier of virtual machine in Azure, the maximum IOPS is 500 per disk . When planning ...read more
  • Windows 10 Preview available for review

    Good morning AskPerf!  It’s been a while since our last post, and we apologize for that.  We’ve been quite busy here on the Support side knocking out customer issues…

    Any who, we have some upcoming blogs in the oven that need a little more time to bake.  One of which is a short series on Windows Event Forwarding which I am very excited about.  Look for that to come out in the coming months.

    Even though we are commonly known as the Performance team, internally we are known as the Reliability team.  Some of the technologies we support are as follows:

    Windows Client/Server OS

    • Printing
    • RDS / TS
    • Performance which includes System Hangs, High CPU, Memory issues, etc.
    • Base WMI functionality
    • COM/DCOM – base functionality
    • Explorer (Shell)
    • Desktop Search
    • MUI and IME
    • MSI – basic functionality
    • Themes/Fonts/Screen Savers/Wallpaper
    • Task Scheduler
    • WinRM – basic functionality
    • Windows PowerShell – install and basic functionality
    • ACT

    There are many other smaller technologies, but these are the main ones.

    Now back to our original topic:  The Windows 10 Preview is available for download/testing.  To get it, click the following link:

    Windows 10 Preview

    Finally, we always welcome feedback on topics you would like for us to blog about here on the AskPerf blog site.

    -Blake

  • Virtual Machine Checkpoint fails with Access Denied when running on a Clustered Shared Volume

    When you attempt to create a CheckPoint of a virtual machine that is running on a Cluster Shared Volume (CSV) , you may receive a General access denied error as shown below.

    clip_image001

    You will receive this error if the virtual machine’s VHD is placed on the root of the drive.

    clip_image002

    clip_image003

    The reason for the access denied error is due to the VM worker process (VMMS) not having relevant permissions on the CSV volume.  Below are default permission that is present for a typical CSV volume.  It is strongly recommended that these permissions not be changed.

    clip_image005

    To resolve the issue, migrate the storage from Failover Cluster Manager or reconfigure the VM and place the VHDX in a folder off the root.  By moving the VHDx to a subfolder or if the VM is reconfigured, the VMMS service updates the permissions on the subfolder as it should.

    For example, this is the current location of the file: 

    C:\ClusterStorage\Volume1\Test Lab.Vhdx

    You would want to move it (and any other VHDX files present) to a subfolder you can create, such as this: 

    C:\ClusterStorage\Volume1\Test Lab\Test Lab.Vhdx

    There are several options you can run through to accomplish this task.

    Option 1:

    Using the Virtual Machine Storageselection from Failover Cluster Manager, move it to the folder you created.  This is an option that can be done without affecting production as it can be done while the virtual machine is online and running.

    clip_image007

    clip_image008

    Option 2: 

    Shut the virtual machine down and, in Explorer, move the VHDx from the root of CSV to a folder you create.  In Failover Cluster Manager, bring up the settings of the virtual machine and manually change the path of the relocated VHDx.  This is an option that can be done but will affect production as it cannot be done while the virtual machine is online and running.  So you would need to schedule downtime to accomplish this task.

    General Rule:

    Microsoft has always not recommended to keep any type of data files in the root of a drive.  Even though things may appear to work fine, problems could arise from this configuration.

    Shasank Prasad
    Senior Support Escalation Engineer
    Microsoft Corporation

  • Remove Lingering Objects that cause AD Replication error 8606 and friends

    Introducing the Lingering Object Liquidator Hi all, Justin Turner here ---it's been a while since my last update . The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering ...read more
  • How to identify a driver that calls a Windows API leading to a pool leak on behalf of NT Kernel?

    Hello my name is Gurpreet Singh Jutla and I would like to share information on how we can trace the caller which ends up allocating “Se  “ Pool tag. When we use the Windows debugger and investigate the pool allocation and the binary associated with ...read more
  • Announcing public availability of MBAM Compliance Data Cleanup Tool 2.5

    We are happy to announce public availability of MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe), aka MBAMCDCT 2.5.
     
    MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe) is a command line tool which enables you to delete machine records from the ‘Compliance Status’ database of the MBAM 1.0 and MBAM 2.0, MBAM 2.0 SP1 and MBAM 2.5 standalone.

     

    There have been situation where you as a MBAM Admin had to delete the entries of older/reimaged machine records from the MBAM compliance database. The only solution in this case was to run complex SQL queries to delete machines from the database. This tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the MBAM Compliance Status database.

     

    This is a command line tool which enables you to schedule it stale data deletion as a task to automate deletion of obsolete machine records from the MBAM compliance database.


    This tool provides three different ways to delete machine records from the MBAM Compliance Status database:

    1.     Delete machines which have not reported in last X days.

    2.     Delete machines specified in a comma separated list via command line.

    3.     Delete machines specified in a text file.

     

    Note:

     

    This tool doesn’t delete the recovery information or any other data from MBAM Recovery and Hardware Database. All delete operations are performed strictly on the MBAM Compliance Status Database.

    This tool is available for download from the TechNet website http://gallery.technet.microsoft.com/MBAM-Compliance-Data-9b4c950d as a self-extractable compressed file, which includes the executable and documentation.

    Hope this tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the database.

    Disclaimer:

    This tool and documentation are provided "as-is". You bear the risk of using it. No express warranties, guarantees or conditions are provided. The tool supplied in this document is not supported under any Microsoft standard support program or service. However, you can report issues and bugs in the comments section on this page. Microsoft will, at its sole discretion, address issues and bugs reported.

     

     

    Himanshu Singh

    Windows Core Team

  • We Are Hiring Windows Escalation Engineers in Charlotte, Dallas, and Redmond

    Would you like to join the world’s best and most elite debuggers to enable the success of Microsoft solutions?   As a trusted advisor to our top customers you will be working with to the most experienced IT professionals and developers in the industry ...read more
  • Windows Troubleshooting – Stop 9E Explained

    What to do if a stop 9E occurs.  How you can solve the issue yourself. ...read more
  • Windows Troubleshooting – Special Pool

    The Windows Support team has a new YouTube channel, “ Windows Troubleshooting ”.  The first set of videos cover debugging blue screens. In this video, Bob Golding, Senior Escalation Engineer, describes how the Special Pool Windows diagnostics tool ...read more
  • Hate to see you go, but it’s time to move on to greener pastures. A farewell to Authorization Manger aka AzMan

    Hi all, Jason here. Long time reader, first time blogger. AzMan is Microsoft’s tool to manage authorization to applications based on a user’s role. AzMan has been around since 2003 and has had a good run. Now it’s time to send it out ...read more
  • Go the modern (app) way

    Hello everyone,

    This is Ashfana from the Windows Performance team. I’m here to talk about the basic philosophy behind Windows Store Apps which were introduced in Windows 8 and Windows 8.1. Windows Store apps were originally called Metro or modern apps before the RTM of the product. These are apps developed using a variety of languages like C#, C++, VB, HTML and Javascript, - which provide the modern, full screen immersive experience.

    Windows has the largest software ecosystem thanks to millions of developers around the world, and Microsoft’s focus on ensuring backward compatibility for application software. This ecosystem has evolved over time and has reached a stage where – we aren’t able to fix some of the limitations and move ahead without causing existing applications to break. Some limitations are installation/uninstallation problems, Software state corruption, uncontrolled modification of important system registries and files, unreliable installer components, one app causing another one to crash, and security risks when running Apps under elevated token. This new App model was introduced to address these reliability and security issues and to develop a new ecosystem that allows Apps to run on multiple hardware platforms with minimum efforts for the developer.

    Here are some of the features of Store Apps that make it very attractive for consumers and enterprises:

    • Windows Store apps do not need an installer. They are either sideloaded to the image using DISM, or installed via Add-AppxPackage PowerShell Command, or installed directly from the Store or a Azure based portal. You no longer need MSI or other installer components, Windows already has the necessary components that can extract the Appx package and install it.
    • Apps are installed per-user (unless they are sideloaded into the image before deployment). They maintain their state within folders in the user’s profile. This ensures better security.
      Apps run within a sandbox environment called an App container. This restricts an App from accessing much of the system and the users profile by default. Additional resources can be requested by an App by declaring capabilities within its manifest file. During first launch of the App, the user is prompted to allow/confirm access to these additional resources.
    • Desktop Apps run under medium integrity and are able to access full privileges granted to a user. An untrusted desktop application can pose a greater risk. Windows Store Apps on the other hand run under base privilege and hence provides a greater level of protection.
    • The Appx manifest and blockmap tell Windows how to install the App, chances of an installation going corrupt are very less. Even if an installation goes corrupt – it would not affect the system in any way.
    • Store Apps provide an immersive experience, are generally very lightweight, and are best suited for scenarios where hard core processing can be done at the server side and results can be delivered to the App over the network.
    • With the introduction of Universal Apps in Visual Studio 2013 update 2, developers can now write an App once and port it to run on multiple devices such as Xbox, Windows phones, PC, tablets and laptops. This makes application development time much shorter.

    Structure of a Windows Store/Universal App:

    Windows Store app are packaged into one or more files with the extension .appx. This appx package is the unit of installation for a Windows Store app. Appx packages are ZIP-based container files that contains the app’s payload files plus info needed to validate, deploy, manage, and update the app. From a high-level view, each Windows Store app package contains these items:

    image

    App payload
    App code files and assets
    Payload files are the code files and assets that you author when you create your Windows Store app.

    App manifest
    App manifest file (AppxManifest.xml)
    The app manifest declares the identity of the app, the app's capabilities, and info for deploying and updating.

    App block map
    App package’s block map file (AppxBlockMap.xml)
    The block map file lists all the app files contained in the package along with associated cryptographic hash values that the operating system uses to validate file integrity and to optimize an update for the app.

    App signature
    App package’s digital signature file (AppxSignature.p7x)
    The app package signature ensures that the package and contents haven't been modified after they were signed. If the signing certificate validates to a Trusted Root Certification Authorities Certificate, the signature also identifies who signed the package. The signer of the package is typically the publisher or author of the app.

    Packaging app:

    When you create an app using Visual Studio, it packages your app, it automatically adds the app block map and signature files to the package. But you can also use the standalone MakeAppx and SignTool utilities if you want to manually package your app.

    The steps that are involved in Package installation:

    image

    The Windows Store app deployment process occurs in multiple phases.

    1) In the first phase, Windows acquires and validates the app manifest, app block map, and app signature. It checks the OS version and dependencies, sees if you have sufficient disk space, and also whether this app is already installed.

    2) In the second phase Windows checks the app package’s deployment criteria to ensure that the app deployment will be successful.

    3) Windows stages the package’s contents on the disk in the %ProgramFiles%\WindowsApps directory in a new directory named after the package identity:

    <Package Name>_<Version>_<Architecture>_<ResourceID>_<Publisher Hash>

    4) Windows registers the package into the user's account. During this phase, the extensions that are declared in the manifest are registered with the operating system.

    The Future:

    With this release of Visual Studio 2013, we have set out to accomplish three major goals:

    1) Reach customers across phones, tablets, and PCs;

    2) Deliver innovation that supports developer investments;

    3) Make cross-platform technology easier and more capable.

    image

    Develop once for all Windows devices using a unified Windows runtime and VS tools that allow you to both support experiences unique to a device in XAML, HTML, and DirectX, and share the code that supports those experiences across all devices using C++, C#, or JavaScript. When your work is finished you can you can produce the app packages that you will submit to the Windows Store and Windows Phone Store with a single action to get your app out to customers on any Windows device. You may also just sign it and deploy this through SCCM.

    More links:

    Package manifest schema reference
    http://msdn.microsoft.com/en-us/library/windows/apps/br211473.aspx

    Capabilities of an App:
    http://msdn.microsoft.com/en-us/library/windows/apps/br211422.aspx

    Create an app package
    http://msdn.microsoft.com/en-us/library/windows/apps/hh975357.aspx

    App packager (MakeAppx.exe)
    http://msdn.microsoft.com/en-us/library/windows/apps/hh446767.aspx

    SignTool
    http://msdn.microsoft.com/en-us/library/windows/apps/ff551778.aspx

    How to create an app package signing certificate
    http://msdn.microsoft.com/en-us/library/windows/apps/jj835832.aspx

    Ashfana Begum
    Support Engineer
    Microsoft Performance Team

  • Deciphering Storport Traces 101

    Welcome back to the CORE Team Blog -- Paul Reynolds here. In previous blogs, I wrote about how to capture Storport traces in Windows 8 and Windows 2012. Please see:

    Tracing with Storport in Windows 2012 and Windows 8 with KB2819476 hotfix

    And

    Tracing with Storport in Windows 2012 and Windows 8 without KB2819476 hotfix

    This time around, I would like to explore what information you can draw from the raw data contained in a Storport trace. What conclusions can you reach regarding your disk performance? Do you have a disk that does not perform well?

    To accomplish this, we will take advantage of free tools available to Windows and Office 2013 users:

    • Windows PowerShell
    • Windows Performance Toolkit
    • Excel PowerPivot

    First, we need to talk briefly about the Windows Storage Stack and where Storport traces are taken. It is important to note that Storport traces are at the very last “rung of the ladder” before Windows hands off I/O request packets to hardware. Hardware in this case encompasses firmware, drivers, HBAs, storage fabrics – anything after the Windows Operating System. It is important because it can help delineate where the problem is. Is the problem with the Operating System or with the hardware? Should I call Microsoft to open a support case or it is more appropriate to talk to my Storage Vendor? Storport traces are the perfect place to start to answer questions such as these.

    It is assumed you already have a Storport Trace in hand – an ETL file captured using the procedures documented in the two blogs above for Windows 8 and Windows 2012, or Windows 2008 (see Bob Golding’s blog):

    Storport ETW Logging to Measure Requests Made to a Disk Unit

    As the old adage goes: a picture is worth a thousand words. When it comes time to deciphering Storport Traces, viewing graphs that summarize data and show trends, and viewing charts that have average and maximums, are much more helpful than looking at raw data. The top-level steps we will undertake are:

    1. Capture the Storport Trace
    2. Convert to the Storport Trace into CSV format
    3. Scrub the data in the CSV file
    4. Import the data into an Excel Spreadsheet

    The easiest way to accomplish the last 3 steps above is to automate it using a Windows PowerShell script. At the end of the blog, there is a zip to download that has this file. Save the PowerShell script as StorPortPACMAN.PS1 in a directory on your C: drive called StorPortPACMAN :

    PLEASE NOTE: you must have the following perquisites installed to successfully run the script above:

    1. XPERF needs to be installed and in your system path.

    Windows 7 or 8 – use the Windows Assessment and Deployment Kit (ADK): http://www.microsoft.com/en-US/download/details.aspx?id=39982

    2. PowerPivot is part of Office 2013 but it needs to be added as a COM Addin. Click here for instructions.

    At the end of this blog, there is also 2 spreadsheets in the zip you must download into the C:\StorPortPACMAN directory.

    Run the script, and depending on which version of Windows was being run on the server, one of the Excel Spreadsheets will open and be refreshed with data in your Storport trace.

    Here are screenshots of sample graphs and charts you will see in the first page of the spreadsheets:

    image

    This first graph shows request duration values over the time of the Storport trace. It is very useful to get the “big picture”. For the most part, the disk in this graph is performing fine except for a period near the beginning. Depending on the application you are investigating, this may be fine and the average value is what is important to you. Averages and maximums are shown in the chart below, which is next to this graph in the spreadsheet:

    image

    Finally, the chart data is presented in the graph below:

    image

    The Excel spreadsheets and their corresponding graphs and charts are only samples of what you can do for your disk analysis using Storport traces and Excel PowerPivot tools. Why use PowerPivot? For Storport traces it means being able to view and summarize more data. Excel by itself is an excellent tool, but problems may start to develop when your data approaches or exceeds a million rows of data.

    It is not uncommon for a Storport trace to contain tens of millions of rows of data, especially if you decide to not use a filter while capturing the data. I generally suggest to not use a filter as your averages will be closer to results you obtain from other tools such as Perfmon or XPERF. Using filters will cause your disk to look worse than it really is.

    Request Duration Times, as a rule of thumb, can be summed up as follows for SCSI Read(10) and Non-cached Write(10) I/O:

    <  9ms = excellent
    < 15ms = good
    < 25ms = fair
    > 25ms = poor

    We purposely use the caveat, rule of thumb, because these values are using an assumed 64KB data I/O size. If a read or write is larger or smaller than that, these values should be adjusted.

    Using an Excel Pivot table and chart lends well to focusing on a specific LUN through the use of filters built into the table or chart.

    Special Cases:

    We occasionally see a Storport trace that does not make sense. One example is a VM that has more than one SCSI disk and the Excel spreadsheet shows only one LUN. What happened to my other disk? It is there, but because of the way storage may be presented, the same LUN number is used for more than one disk. So what does one do in a situation such at this? The easiest way is to drop and drag a second property of a disk into the chart or graph you are viewing. There are 4 properties we gather in a Storport trace:

    1. LUN
    2. Target
    3. Bus
    4. Port

    Since using LUN is not enough, we can add a second property to the chart, Target. To do this, at the top of the spreadsheet click on the Analyze tab under Pivottable Tools, then click on Field List to expose the data available to you. Drag and Drop the Target field onto Rows and underneath LUN. You will see something similar to this:

    image

    This will let you see all your disks if they have the same LUN number.
    If you are at all like me, viewing charts and graphs might be fine for seeing the big picture and giving presentations, but your inner engineer is screaming to look at the data in detail, to make sure you are comfortable with it and understand it. To do this, click on the POWERPIVOT tab and then the Manage button in the Data Model area. This will open a new window that exposes the raw data to you so that you may filter it, sort it, or anything else you wish to make sure it all makes sense to you before using it.

    I hope you find this blog helpful and gives you a new tool to use when investigating disk performance.

    Paul Reynolds

  • WMI: How to Troubleshoot WMI High Handle Count

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is experiencing high handle count

    Your first thing to do is check the Application Event log for following event:

    Source: Microsoft-Windows-WMI

    Event 5612 Wmiprvse.exe exceeding handle quota limit Event

    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    If you find the above Event, you can try and bump up the handle quota limit to see if it resolves your issue. If it is a leak, then bumping limit will only mean it will take longer to reach the new limit. If it’s just load related, then bumping the limit could resolve the issue.

    The event will tell you what the handle count was, and if it is higher than the 8192 value I suggest below.  You can then skip this section and move on to data collection to figure out the cause of high handle count.

    How to increase the handle quota limit for the WMI Provider Service

    1. Go to Start--> run and type wbemtest.exe
    2. Click Connect on the Windows Management Instrumentation Tester
    3. In the namespace text box just enter "root" (without quotes)
    4. Click Connect

    clip_image001

    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    1. Click "Enum Instances…"

    clip_image002

    1. In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK.

    Note: a double underscore __ precedes ProviderHostQuotaConfiguration

    clip_image003

    1. A query Result window will come up. In this windows now double click "__ProviderHostQuotaConfiguration=@"

    clip_image004

    1. An Object Editor windows will come up now
    2. Under properties find the property "HandlesPerHost"

    clip_image005

    1. Change the value from default of 4096 to 8192
    2. Click Save Property
    3. Click Save Object in the Object Editor window
    4. Close the other windows now and exit WMI Tester
    5. Restart Windows Management Instrumentation Service.

    If after bumping up quota limit and wmiprvse still exceeding quota limit, accomplish following actions below. You will want to read through the rest in its entirety to ensure you get all of the necessary tools downloaded before taking any actions.

    Configure System for Complete Memory Dump by referring to:

    Windows 8 and Windows Server 2012 Automatic Memory Dump: http://blogs.technet.com/b/askcore/archive/2012/09/12/windows-8-and-windows-server-2012-automatic-memory-dump.aspx

    Windows does not create a memory dump file when a Stop error occurs in Windows 8 or Windows Server 2012: http://support.microsoft.com/kb/2853466

    Windows 2008, Windows Vista, Windows 7, Windows 2008 R2: http://support.microsoft.com/kb/969028

    Windows Server 2003 and Windows XP: http://support.microsoft.com/kb/972110

    Collect perfmon logging using logman method

    Directions below will create 2 perfmon logs, one at a 5 minute interval (PerfLog-Long) and a short 5 second interval log (PerfLog-Short) and they will be placed in C:\Perflogs folder.

    • Long log (5 min intervals) – no thread counter, 250 MB:

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Long -o "c:\perflogs\PerfLog-Long.blg" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues\*" "\System\*"  -si 00:05:00

    3. Start the log with:

    Logman.exe start PerfLog-Long

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Long

    • Short, high resolution log – 5 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:05

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Configure Handle Tracing

    You probably just need the standalone version since we only need the debugging tool and not the whole WDK package.

      1. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator
      2. Click on the ‘Kernel Flags’ tab
      3. Check the box next to ‘Enable bad handles detection’
      4. Click on the ‘Image File’ tab
      5. Next to ‘Image: (TAB to refresh)’, enter wmiprvse.exe or if tracing WMI Service enter the unique name given to the svchost process for WMI Service per directions in next section below
      6. And then click on the ‘Tab’ key
      7. Next to ‘Stack Backtrace: (Megs):’ enter ‘10’
      8. Click on Apply
      9. Click on Ok
      10. Restart WMI service

    If it is a svchost process showing high handle count, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From there in a command prompt you can run tasklist /svc and look for the PID then identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From experience it will be the WMI service more times than not but not always, as such I would try to break it out first on its own and monitor to see if it is the one driving up high handle count in the shared svchost process.

    WMI (Windows Management Instrumentation) service, you can break it out by accomplishing the following.

    Break WMI Service out into its own unique svchost process

    a.      Open command prompt with elevated privileges

    b.      Run following command: sc config winmgmt type= own

    c.      Restart Wmi service

    d.      Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    sc config <service name> type= share

    e. Restart the service or machine and verify result is Win32_SHARE_PROCESS when you run sc query winmgmt command again

    f. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe

    g. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt

    h. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    I. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    j. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    k. You would now substitute wmisvchost.exe in lieu of wmiprvse.exe in step 6. under Configure Handle Tracing above

    Using debugger to attach to the process in windbg.exe and running !htrace –enable command

    1. Launch WinDbg program from under Debugging Tools for Windows that you installed earlier.

    2. Created folder c:\websymbols

    3. Click on File-Symbol path and add the following symbol path to the debugger: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    4. Click on File-Save Workspace

    Attach to process to accomplish handle tracking using htrace

    To do this:

    1. From Windbg - File - Attach a Process - Select the instance of wmiprvse.exe with high handle count

    Note: If it is WMI Service (run tasklist /svc or Task Manager with PID column added first to find the PID of svchost.exe containing winmgmt which you should have broken out and uniquely named wmisvchost.exe per earlier directions)

    2. Run following command from the debugger:

    .logopen "C:\debug.log" then hit <ENTER> key

    !htrace -enable 0x20000 then hit <ENTER> key

    Note:  By default, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 keep a history of 4000 handles open and close operations. 

    With !htrace you can enable to keep a much higher history by doing the following:

    !htrace -enable 0x20000

    In this example, we are increasing the handle history to 131072 (decimal, 0x20000 hexadecimal)

    !htrace –snapshot then hit <ENTER> key

    g then hit <ENTER> key

    3. Now, let the process run until the number of handle has increased a lot and gotten high.

    Final htrace log

    1. Break into debugger with Keyboard keys (Ctrl+Break)

    2. Run following commands:

    !htrace –diff then hit <ENTER> key

    .logclose then hit <ENTER> key

    .detach then hit <ENTER> key

    3. Close WinDbg

    Now complete the following actions once you have gotten your final htrace log

    1. If high handle count is with wmiprvse, download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    2. Find the instance of wmiprvse.exe with high handle count and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    3. If the WMI Service was the process with the high handle count, then dump out the WMI service process which should be wmisvchost.exe per previous directions and all instances of wmiprvse.exe using procdump. If it is wmiprvse.exe that is exhibiting the high handle count, then only need to dump out that instance and nothing else.

    a. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    b. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    c. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID

    d. Run the following command: procdump –ma <PID>

    e. Note: Replace <PID> with actual PID you documented for instances of wmiprvse.exe and/or wmisvchost.exe as it applies based on directions above

    4. Stop Perfmon logging

    5. Do a complete memory dump of the machine

    At this point with data in hand you will want to open a Support Incident with Microsoft to get the data analyzed to help determine cause of high handle count.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

  • WMI: How to troubleshoot High CPU Usage by WMI Components

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of CPU.

    In the directions below, you may have already broken out WMI Service to troubleshoot your issue.  By default, WMI runs in the main shared networking svchost process with several other services.

    If it is a svchost process showing high cpu usage, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From inside a command prompt you can type in  tasklist /svc and look for the PID #, and identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From my experience, it will be the WMI service more times than not but not always.  As such, I would suggest breaking it out first into its own, and monitor to see if it is the one driving up high memory usage in the shared svchost process.

    If you suspect the WMI (Windows Management Instrumentation) service, you can break it out following directions below.

    Break WMI Service out into its own svchost process

    1. Open command prompt with elevated privileges
    2. Run following command: sc config winmgmt type= own
    3. Restart Wmi service
    4. Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    • sc config <service name> type= share
    • Restart the service or machine and verify result is Win32_SHARE_PROCESS when you
    • run sc query winmgmt command again

    Configure Perfmon Collection using logman.exe method. Capture 15 minutes while issue is occurring.

    Short, high resolution log – 1 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window (if this does not work, you may need to manually type it in):

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:01

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Collect and Xperf trace for High CPU by using the Windows Performance Recorder form the Windows Performance Toolkit which you can install from the ADK

    Note: If the Operating System is a 64 bit box, you must first accomplish the following registry setting before collecting Xperf trace.

    Registry Path
    HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
    Setting
    DisablePagingExecutive
    Data Type:
    REG_DWORD
    Value:
    1

    NOTE setting this key is not needed on Windows Server 2012 & 2012 R2

    Reboot machine to place registry setting into effect.

    1. Download the Windows 8 ADK (Windows Assessment Deployment Kit) from here.
    2. Open the adksetup.exe and hit next until you get you the option to select feature options
    3. Select "Windows Performance Toolkit" and hit "Install"

    clip_image002

    After installation has finished, start creating a trace by starting the "Windows Performance Recorder"

    clip_image004

    Select CPU usage under Resource Analysis

    Logging mode can be left set to “Memory”, or you can change to “File”. Just be conscious of your disk space if you chose “File” as the etl file can become large fast

    Capture high cpu occurrence, but do not let the recording run for no more than 10 minutes.

    Immediately after capturing the event using Windows Performance Recorder (WPR), now use process explorer to dump out the process exhibiting high cpu usage.

    1. Download Windows Sysinternals tool called Procdump: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    2. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    3. Open Task Manager and add the PID column view, then go locate the instance of wmiprvse.exe with high cpu usage and note the PID. If it was the WMI service that had the high cpu, then you should already have it broken out to run in its own svchost process and note the PID of that svchost process. To confirm you have the right svchost process, you can run tasklist /svc from administrative command prompt and verify the PID noted in task manager and ensure it is the svchost process running winmgmt in it.

    4. Run the following command: procdump –ma -s 60 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe or for the svchost process running winmgmt exhibiting high memory usage

    The above command will produce 3 dumps spaced 1 minute apart each in same directory you ran the procdump command from

    5. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    6. If it was wmiprvse.exe that had the high CPU usage, then find the instance and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high CPU usage.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up:  WMI: How to Troubleshoot WMI High Handle Count

  • WMI: High Memory Usage by WMI Service or Wmiprvse.exe

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of memory.

    There are many reasons why WMI might experience high memory consumptions. This can occur in the WMI (Windows Management Instrumentation) service (winmgmt) or in the WMI provider hosting vehicle wmiprvse.exe. Both scenarios will be addressed below.

    High memory usage may simply be due to load, as opposed to some type of leak. By default, the memory quota limit for instances of wmiprvse.exe on Windows XP and Windows Server 2003 is 128 MB, and 512 MB on newer Operating Systems (Vista and higher). Hit those limits and wmi functionality will become problematic if not come to a grinding halt.

    There is another case where the quota limit problem could happen. There is limit for all wmiprvse’s cumulatively. If the total memory of all instances of wmiprvse’s (with one exception) together reaches 1GB, then all new memory allocations fail in all wmiprvse processes.

    As a first effort you can try to bump up that quota limit to see if it gives enough room for wmiprvse to operate, but if you still reach the new quota limit, then you will want to proceed with the Scenarios below

    How To Bump up WMI memory quota limit:

    • Go to Start--> Run and type wbemtest.exe
    • Click Connect

    clip_image001 

     

    • In the namespace text box type "root" (without quotes).

    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    • Click Connect
    • Click Enum Instances…

    clip_image002

     

    • In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK. Note: the Superclass name includes a double underscore at the front.

    clip_image003

     

    • In the Query Result window, double-click "__ProviderHostQuotaConfiguration=@"

    clip_image004

     

    • In the Object Editor window, double-click whichever Property name you wish to modify the quota for. In this case that will be MemoryPerHost

    clip_image005

     

    • In the Value dialog, type in 536870912 (512 MB) for XP and Windows 2003.  For Vista and higher, start with new value of 805306368 (768), then move to 1073741824 (1024) if still needing room. At the same time, if you are going to bump up the MemoryPerHostLimit, you should also then bump up the MemoryAllHost limit to 2147483648 (2048) provided you have the available ram to do so. If this does not resolve quota violation issue, then need to troubleshoot cause of high memory usage following below Scenario section
    • Click Save Property.
    • Click Save Object.
    • Close Wbemtest.
    • Restart the machine

    Note: if you cannot restart the machine, then as a workaround, you can break Windows Management Instrumentation service into its own svchost process outline in step 2. a-c below.

    If bumping the quota limits does not resolve the issue, then as workaround you can try to move suspected leaking providers into their own group (wmiprvse) to avoid the memory quota caused by other providers running in the group or kill the instance of wmiprvse exhibiting high memory until issue is resolved. This is outlined below in Scenarios 1a and 2a.

    First thing we need to determine is if the memory consumption being caused by private data or heap data. We have to address the 2 types differently.

    1. Download a Windows Sysinternals tool call VMMap from following url: http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx

    VMMap is a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types.

    This tool is used to attach to an individual process allowing a snapshot to be taken to see the memory map for that process.

    2. Simply launch VMMap and from the process list it displays, pick the instance of wmiprvse showing the high working set memory usage.

    3. If it is a svchost process that is exhibiting high memory, from a command run scqueryex winmgmt to identify the PID of the svchost process that is hosting WMI Service (winmgmt). From experience it will be the WMI service more times than not but not always as the service using majority of the memory; as such I would try to break it out first on its own and monitor to see if it is the one driving up high memory usage in the shared svchost process.

    From this point, I will assume it is the WMI service as this article is only addressing WMI and no other services.

    You will need to break the WMI service out into its own svchost process first if it hasn’t already been accomplished so you can run VMMap specifically against the WMI service to truly know if it is being consumed by Heap or Private Data. By default the WMI service runs in a shared svchost process with a community of other services. You can do so by the following directions. If it is caused by Heap, you will find later in the Scenario directions below that you will have to go back and uniquely name the svchost process for the service to run in to be able to enable User Stack Tracing against it.

    a. Open command prompted with elevated privileges

    b. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own

    c. Restart wmi service with net stop winmgmt and net start winmgmt commands

    d. Verify winmgmt service running in its own svchost process by runnning tasklist /svc

    4. Once it displays the result, look under the size column for Private Data and Heap. This should tell you if the majority of the memory being consumed is Private Data or Heap.

    5. Follow appropriate Scenario below based on if issue is with Windows Management Instrumentation service or wmiprvse and if caused by Heap or Private Data memory.

    Scenario 1a: High Memory Consumption by Wmiprvse.exe caused by Heap memory

    1. Download Windows Debugging Tools for Windows from following link: http://msdn.microsoft.com/en-us/windows/hh852365.aspx

    You can probably just get the standalone version since we only need the debugging tool and not the whole WDK package.

    2. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator

    3. Click on “Image File” tab

    4. Type in wmiprvse.exe

    5. Hit the keyboard TAB key

    6. Place check mark in “Create user mode stack trace database

    7. Click Ok

    8. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    9. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high memory usage:

    wmic path msft_providers get Provider,HostProcessIdentifier /format:list

    10. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

    11. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

    $prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
    <ENTER>
    $prv.HostingModel = $Prv.HostingModel + ":OWN"
    <ENTER>
    set-ciminstance -inputobject $prv
    <ENTER>

    Repeat the above for each Provider found in step 9

    12. Restarting of Wmi service will put the settings into effect without having to reboot machine

    13. Now you will want to take a process dump of the instance of wmiprvse when memory usage is high. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    14. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

    15. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

    16. Run the following command: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

    The above command will produce 3 dumps spaced at 5 minutes apart each in same directory you ran the procdump command from

    17. Note also as workaround until issue gets resolved you could simply kill the instance of wmiprvse with high memory from task manager or from PowerShell can run following:

    kill -f <pid of suspect wmiprvse>

    At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high memory usage

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 1b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Heap memory

    The Windows Management Instrumentation service runs under the display name of winmgmt and is located in networking svchost.exe process shared along with several other services. You need to break the WMI service out into its own unique svchost process for data collection purposes.

    NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 6 to uniquely name the svchost process it is running in. We do this because we only want to enable User Stack Tracing against just the WMI service and not every svchost process.

    1. Open command prompted with elevated privileges
    2. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own
    3. Restart wmi service with net stop winmgmt and net start winmgmt commands
    4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
    5. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe
    6. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt
    7. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    Note: It is important that you go back and reverse what you did in step 7 and modify path back to original after you are no longer needing the service to be broken out and uniquely named as failure to do this can prevent future WMI hotfixes from being installed.

    Simply run following command then restart wmi service: sc config winmgmt type= share

    8. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    9. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    10. For sake of brevity here, follow steps 4-14 from Scenario 1a with the exception of typing in wmisvchost.exe in place or wmiprvse.exe for process we are enabling “Create user mode stack trace database” against.

    11. Here we want to use procdump to dump the wmisvchost.exe process along with every instance of wmiprvse.exe that is running. Reference steps 9-11 listed in Scenario 1a above.

    First command will dump wmisvchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running.

    Command for dumping wmisvchost.exe would be: procdump –ma -s 300 -n 3 wmisvchost.exe

    Command for dumping each instance of wmiprvse just once: procdump –ma <PID>

    Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

    At this point you will now need to open a Support Incident Case with Microsoft to get the process dumps analyzed to determine cause of high memory usage

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2a: High Memory Consumption by Wmiprvse.exe caused by Private Data memory

    This scenario is a little bit different than being caused by Heap, as just dumps in of themselves normally do not tell the full story. In this type of case, we need to know what type of wmi activity is occurring and it becomes significantly harder and more involved to try and determine cause.

    We will need to collect procdumps still, but also need to add in WMI activity logging.

    1. From start run type in eventvwr.msc

    2. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it

    3. Expand Applications and Services\Microsoft\Windows\WMI-Activity

              Right click on items Debug and Trace and select “Enable” for each one

    4. Download the latest version of the Windows Sysinternals tool Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx.

    5. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high

    memory usage:

    wmic path msft_providers get Provider,HostProcessIdentifier /format:list

    6. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

    7. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

    $prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
    <ENTER>
    $prv.HostingModel = $Prv.HostingModel + ":OWN"
    <ENTER>
    set-ciminstance -inputobject $prv
    <ENTER>

    Repeat the above for each Provider found in step 5

    8. Restarting of Wmi service will put the settings into effect without having to reboot machine

    9. When memory usage is high, use Procdump tool to dump out the wmiprvse.exe exhibiting high memory usage. http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    10. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

    11. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

    12. Run the following command: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

    The above command will produce 3 dumps space out 5 minutes apart each in same directory you ran the procdump command from

    13. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865

    14. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

    At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Private Data memory

    NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 5

    1. Open command prompted with elevated privileges
    2. Break WMI service out into its own svchost process by running following command: sc config winmgmt type= own
    3. Restart WMI service with net stop winmgmt and net start winmgmt commands
    4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
    5. From start run type in eventvwr.msc
    6. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it
    7. Expand Applications and Services\Microsoft\Windows\WMI-Activity

              Right click on items Debug and Trace and select “Enable” for each one

    8. Here we want to use procdump to dump the svchost process housing the WMI Service along with every instance of wmiprvse.exe that is running. Reference steps 9-10 listed in Scenario 1a above

    a. From command prompt run tasklist /svc and locate the instance of svchost that winmgmt is running in and note the PID

    b. First command will dump svchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running just once

    Command for dumping svchost.exe housing winmgmt would be: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID for svchost process housing winmgmt

    Command for dumping each instance of wmiprvse running would be: procdump –ma <PID>

    Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

     

    1. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865
    2. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

    At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up: WMI: How to troubleshoot High CPU Usage by WMI Components

  • WMI: Missing or Failing WMI Providers or Invalid WMI Class

    Scenario

    Windows Management Instrumentation fails due to receiving an event or error concerning missing or failure to load WMI Provider, or Invalid WMI class, or WMI Invalid Namespace.

    Below are some common errors indicating issues with a WMI Provider or Class:

    • Failed to initialize all required WMI classes
    • Win32_processor: WMI: Invalid namespace
    • Win32_WMISetting: WMI: Invalid namespace
    • Win32_OperatingSystem: WMI: Invalid namespace
    • WBEM_E_NOT_FOUND 0x80041002
    • WBEM_E_PROVIDER_FAILURE 0x80041004
    • WBEM_E_INVALID_NAMESPACE 0x8004100E
    • WBEM_E_INVALID_CLASS 0x80041010
    • WBEM_E_PROVIDER_NOT_FOUND 0x80041011
    • WBEM_E_INVALID_PROVIDER_REGISTRATION 0x80041012
    • WBEM_E_PROVIDER_LOAD_FAILURE    0x80041013

    The common urge by many is to simply take a hammer to WMI and either rebuild the repository or recompile all of the .mof files in the C:\Windows\System32\Wbem folder. While the hammer may seem a good approach to solve your problem in some cases or what seems like a quick fix, it can actually cause more problems and issues that are not immediately visible or felt. These newly created problems as a result of the hammer approach are not always immediately obvious and tend to surface later, and usually when the last thing you need is another problem to have to deal with on your plate, especially if it created a critical business situation.

    Rebuilding the repository or recompiling all of the .mof files as a first action when other steps should be taken first can cause damage to the system and/or to installed applications.

    First we want to just check right up front if our repository is corrupted or not to save time, then continue from there with appropriate actions.

    Check the Windows Application log, look for events in the past week where Source = Microsoft-Windows-WMI, check if any of the following WMI event IDs exist: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository or core infrastructure problem.

    Check if the repository is corrupted or not by running following command from command prompt with admin rights: winmgmt /verifyrepository. If it is corrupted the returned result will be “Repository is inconsistent.” If it comes back as consistent, then repository is not corrupted so do not rebuild repository at this point.

    Note: if results came back as “inconsistent”, then skip to bottom of the page and reference the section titled “Wmi Repository Corrupted”, otherwise continue with the methods below.

    If issue is with a 3rd party provider or namespace, the vendor should be engaged as they are responsible for their own namespaces and providers.

    For the 3 scenarios listed below for WMI Invalid Namespace, WMI Invalid Class, and WMI Provider Load failure, if a class is present and operation still errors out with invalid class, then the most likely reason is that service/wmiprvse is hitting memory quota limit or issues.

    When a quota issue is hit in wmiprvse, new memory allocations fail and based on where the failure happen client gets different errors “invalid query”/Invalid class/Provider load failure/quota check”
    When a quota issue is hit when a new operation is for a provider which is not loaded already in wmiprvse and thus requiring it to be loaded in a new instance of wmiprvse, then the provider loading would fail with “provider load failure”.

    Scenario 1: WMI Invalid Namespace

    First we want to take any scripts or programs out of the equation by using local built in tools. The two most common tools used to check wmi functionality is the WMI console (winmgmt.msc) and Wbemtest (Windows Management Instrumentation Tester).

    Ensure the Namespace in question actually exist and functional.

    1. Go to start-run and type in wmimgmt.msc
    2. Right click on Local Wmi Control (Local)and select properties
    3. On the general tab, if there is any failures noted on that box, that indicates a core WMI issue and most likely with the Cimv2 namespace.
    4. Click on the Security tab and expand Root folder. This is where you will see all of the namespace listed for WMI
    5. Find the namespace referenced in the error message you are getting
    6. If you find the namespace is missing, do the following, otherwise skip to step 6 if the namespace is listed

    a. Open up a command prompt with administrative rights or elevated privileges

    b. Go to www.msdn.com and search for the provider of the missing namespace. We are looking for the associated .mof file. In the below example, we will use MSCluster as our missing namespace.

    e.g. search words: wmi mscluster provider or wmi mscluster namespace. In this case you should find ClusWmi.mof as the WMI provider

    In case the CimV2 namespace is missing, the provider is CimWin32.mof

    If it also lists the associated .dll file, you will want to re-register it also as this would be the registration for the DCOM side as WMI and DCOM work hand in hand.

    c. From the command prompt with administrative rights or elevated privileges change directory to C:\Windows\System32\Wbem and run the following command: mofcomp.exe <provider.mof>

    Note: Replace <provider.mof> with actual .mof name you found searching MSDN. In our example above that command would be: mofcomp.exe ClusWmi.mof

    For re-registering associated .dll if one exists use following command: regsvr32 <provider.dll> and again replacing <provider.dl> with actual dll name

    d. Restart the Windows Management Instrumentation Service

    e. Use wmimgmt.msc console again to now check if the missing namespace is now listed. First close the console and reopen if it was still open from previous action

    7. Go to start-run and type in wbemtest

    8. Click on the “Connect Button

    9. In the Namespace Box type in the path to the namespace for which getting invalid namespace error for. This path would have the same look and feel of a Windows Directory, so just as you see the structure in wmimgmt.msc console on the Security tab, so is how you will type in path

    Examples:

    Root\Cimv2
    Root\Mscluster
    Root\RSOP\Computer

    10. Click on the “Connect” button

    11. Now all of the buttons should no longer be greyed out on the main wbemtest console page. Click on the “Enum Classes” button

    12. Leave “Enter Superclass Name” blank and select “Recursive” then click OK. If you don’t get any error messages then you can access the name successfully without issue using built in Windows Management Instrumentation Tester

    13. To test further, let’s see if we can access some classes.

    a. Pick any class and double click on it to bring up the Object Editor for that class

    b. Next click on the “Instances” button on the right

    c. If it doesn’t sit there hung at “Operation in progress” or doesn’t return any error, then access to that class would appear to be okay.

    NOTE: Not every class you click in the “Instances” button will actually return results. Some classes will and some won’t depending on the class. Don’t sweat it if it doesn’t as long as the box above says “Done”

    Repeat this on several classes in the namespace for sanity check just to see if any produce an error message or Operation in progress hangs.

    If you did not get any errors connecting to the namespace or accessing some of the classes in that namespace, then your issue may be with the application or whatever method being used outside of Wbemtest that is the problem and would advise opening case with Microsoft if it is a Microsoft application or process, otherwise first contact the vendor of the application for assistance in troubleshooting.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2: WMI Invalid Class

    To troubleshoot an Invalid WMI Class, you basically would follow the same procedure as above that you would for WMI Invalid Namespace.

    Here is a good link that you could probably also click your way thru various links to find the provider .mof for Microsoft classes.

    WMI Classes

    Scenario 3: WMI Provider Load Failure

    Again, you will follow the same procedures as above for testing, and also locating the provider .mof file and associated .dll if one is listed and recompiling and re-registering of that particular provider.

    Wmi Repository Corrupted

    There are some exceptions where you have accomplished all of the above and from Wbemtest (note I said Wbemtest, not some method outside of Wbemtest) and you are still getting failures with a namespace, class, or provider. In these scenarios you may have to revert to using the proverbial hammer as a last ditch effort even though the repository check came back as consistent when your ran winmgmt /verifyrepository.

    Before doing such, it would be a good idea to open a Support Incident Case with Microsoft for assistance before reverting to rebuilding of the repository if the winmgmt /verifyrepository had come back as Consistent.

    IMPORTANT

    If this becomes the case, please refer to my Ask the Performance Team Blog article, WMI: Rebuilding the WMI Repository before rebuilding the repository as there are some important gotchas that you need to be aware of.

    As a final note, if you are running into a reoccurring WMI corruption issue in your environment, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption in WMI.

    At this point you may also want to open a Support Incident with Microsoft for further assistance if needed.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up: WMI: High Memory Usage by WMI Service or Wmiprvse.exe

  • WMI: Repository Corruption, or Not?

    Scenario

    Windows Management Instrumentation failing due to repository being corrupted

    The WMI Repository (%windir%System32\Wbem\Repository) is the database that stores meta-information and definitions for WMI classes; in some cases the repository also stores static class data as well. If Repository becomes corrupted, then the WMI service will not be able to function correctly.

    Before grabbing that preverbal hammer approach and just rebuilding your repository, ask yourself, “Is the WMI repository OK?”

    Common symptoms that lead to this question are: provider load failure, access denied, class not found, invalid namespace, and namespace not found to mention a few.

    If you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows or to installed applications. Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption. Noting here that having a repository too large also creates problems; an issue that can sometimes be interpreted as a corrupt repository, which is not always the case. If issues are due to a large repository, rebuilding the repository is currently the only method available to reduce it back to a working size.

    Since I mentioned “large repository”, let me set some guidelines up front. There is no hard fast number per say as to when you will start feeling performance problems with a large repository. As a guideline, if the objects.data file, located in (%windir%System32\Wbem\Repository, is 1 GB or larger, then I would recommend rebuilding your repository to reduce it back down to a working and manageable size. If the size is between 600-900 MB, and you are not feeling any noticeable performance issues, then I would recommend against rebuilding the repository.

    If WMI is corrupted, you can receive various errors and symptoms, depending on what activity was being done at the time. Below is a few errors and symptoms that could indicate that the repository is corrupted:

    1. Unable to connect to root\default or root\cimv2 namespaces. Fails returning error code 0x80041002 pointing to WBEM_E_NOT_FOUND.
    2. When we open Computer Management and Right Click on Computer Management (Local) and select Properties, you get the following error: "WMI: Not Found" or it hangs trying connect
    3. 0x80041010 WBEM_E_INVALID_CLASS
    4. Trying to use wbemtest, it hangs
    5. Schemas/Objects missing
    6. Strange connection/operation errors (0x8007054e):

    get-cimclass : Unable to complete the requested operation because of either a catastrophic media failure or a data structure corruption on the disk.

    At line:1 char:1

    + get-cimclass -Namespace root\cimv2\TerminalServices

    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo : NotSpecified: (:) [Get-CimClass], CimException

    + FullyQualifiedErrorId : HRESULT 0x8007054e,Microsoft.Management.Infrastructure.CimCmdlets.GetCimClassCommand

    Check the Windows Application log for events in the past week where Source = Microsoft-Windows-WMI.  Look for any of the following WMI event IDs: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository issue or core infrastructure problem.

    If you do not find any of these events logged, your next action is to use the built in repository checker. From an elevated command prompt run "winmgmt /verifyrepository". If the repository has an issue, it will respond "repository is not consistent".

    If repository check comes back as “consistent”, then look at my other Ask Perf blogs for applicability:

    WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

    WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

    How to troubleshoot High CPU Usage by WMI Components (COMING SOON)

    WMI Self-Recover

    When the WMI service restarts or detects Repository corruption, the self-recovery procedure will trigger automatically in two approaches (one or the other):

    1. AutoRestore: if the VSS backup mechanism enable snapshot the timestamp backup images In the system (ex. Win7 feature: previous fileversion), WMI will apply the AutoRestore approach to restore backup valid images in version queue. (if possible)
      • EVT: 65 (start restore) / 66 (succeed recovered with VSS Path)
    1. AutoRecovery: the rebuild process to generate fresh images of Repository based on registered mofs( listed @ HKLM\Software\Microsoft\WBEM\CIMOM: AutoRecover Mofs).
      • EVT: 5616 (complete recovery), eventually, there are lots of EVT:63 for mof warning about Localsystem registration of providers.

    Note: Under almost no circumstance should you use the script that rebuilds the WMI repository from the MOF files

    The script is inherently flawed, for 2 reasons:

      1. If you navigate to the %systemroot%\system32\wbem folder, and list the MOF files, you see find MOFs named (some provider name)_uninstall.mof. When you mofcomp those, they remove the classes in the MOF. The script mofcomps everything, so it can very easily install then immediately uninstall the classes, resulting in the class not being accessible.
      2. Replaying mofs is often sequence dependent. Example: classes in mof1 can depend on or have associations with classes in mof2. If they aren't present, MOFCOMP will not insert the classes. It's extremely difficult to know what is / is not the right sequence, so any script that simply MOFCOMPs everything is not going to be fully successful.

    In addition to causing damage to your system that's almost impossible to fix correctly, if you take that approach you will blow away all information that could be used to determine the root-cause.

    If the repository check (winmgmt /verifyrepository) comes back as inconsistent, your first action is to run “winmmgmt /salvagerepository” followed by running “winmgmt /verifyrepository” again to see if it now comes back as consistent.

    If it is still comes back inconsistent, then you need to run “winmgmt /resetrepository.” Before running this, please read the important note below for Server 2012.

    Force-recovery process -- rebuild based on the registry list of AutoRecover Mofs

     

    1. Check regkey value is empty or not @ HKLM\Software\Microsoft\WBEM\CIMOM: 'Autorecover Mofs' (** first line on some OSs is empty, review it in opening the regkey value)
    2. If above regkey is empty, copy/paste the regkey value from another machine of equal System/OS to the suspect machine
    3. Run the following command from command prompt with admin rights: “Winmgmt /resetrepositoy”
    4. If you get the error noted below, stop all dependency services on the WMI service by running following command: 

    net stop winmgmt /y

    then run

    Winmgmt /resetrepositoy

    WMI repository reset failed
    Error code:     0x8007041B
    Facility:       Win32
    Description:    A stop control has been sent to a service that other running services are dependent on

    NOTE: Applies to Server 2012

    We have encountered some issues when running the mofcomp command on Windows Server 2012 which has caused the Cluster namespace to be removed due to the cluswmiuninstall.mof contained in the c:\windows\system32\wbem folder. It has also caused unregistering class names for Cluster Aware Updating (CAU) because of its cauwmiv2.mof also in the wbem folder. Those could also affect other namespace that have an uninstall type .mof in the wbem folder beyond the two mentioned above.

    Furthermore, the uninstall .mof files for servers running Microsoft Cluster are also part of the autorecover folder that is used when you run the winmgmt /resetrepository command which will end up having the same effect of first installing the Cluster namespace, then uninstalling it just as if you had run a script to rebuild the repository that contained the “for” command to recompile all of the MOFs in the WBEM folder.

    Take the following actions to confirm if the uninstall problem for this scenario exists on your server. If it doesn’t, then you can run the winmgmt /reset repository, otherwise follow my directions below for manually accomplishing rebuild.

    1. Open regedit, and navigate to hklm\software\microsoft\wbem\cimom, and open “Autorecover MOFS
    2. Copy the data from that string value, and paste it into notepad
    3. Do a search for ClusWmiUninstall.mof. If the cluster provider uninstall has autorecover, it will be listed here
    4. If found, then continue to manual rebuild below, if not found then go ahead and use the winmgmt /resetrepository command

    How to manually rebuild repository on Server 2012 Cluster machine when cluster provider uninstall has an autorecover

    First ensure you have run winmgmt /verifyrepository to ensure that it is “inconsistent” and that you have tried winmgmt /salvagerepository to see if it resolves your issue.

    Change the startup type for the Window Management Instrumentation (WMI) Service to disabled.

     

    1. Stop the WMI Service, you may need to stop services dependent on the WMI service first before it allow you to successfully stop WMI Service
    2. Rename the repository folder:  C:\WINDOWS\system32\wbem\Repository to Repository.old
    3. Open a CMD Prompt with elevated privileges
    4. Cd windows\system32\wbem
    5. Run following command to re-register all the dlls: for /f %s in ('dir /b /s *.dll') do regsvr32 /s %s
    6. Set the WMI Service type back to Automatic and restart WMI Service
    7. cd /d c:\  ((go to the root of the c drive, this is important))
    8. Run following command specifically adopted for 2012 Clustered servers to recompile the MOFs: “dir /b *.mof *.mfl | findstr /v /i uninstall > moflist.txt & for /F %s in (moflist.txt) do mofcomp %s”
    9. Restart WMI service

    As a final note, if you run into a reoccurring corruption issue in your environment with WMI, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption and other issues in WMI.

    Other repository recovery solutions:

    Note: in following solutions (1 & 2), if the backup images (repository) are in large size (>100MB), restoring the repository will take some time.

    1. Apply WMI AutoRestore story in the system to recover the repository image quickly and keep it in sync with previous state.
    2. Enable VSS backup-related features for storing image snapshots
      • ex.  Volume Shadow Copy (VSS), or check any valid copies listed Local Disk(C:) Properties >> Shadow copies
    3. Make sure registry key has following setting: HKLM\Software\Microsoft\WBEM\CIMOM: AutoRestoreEnabled=1
    4. Frequently snapshot the restore-points of the system (if needed, refer to the following PowerShell scripts)

    $filterStmt = [String]::Format("DriveLetter='{0}'",$env:SystemDrive)

    # Get systemdrive Volume info

    $vlm = gcim win32_Volume -Filter $filterStmt

    # create a shadowcopy

    $res = icim win32_ShadowCopy -MethodName Create -Arguments @{Volume=$Vlm.DeviceID}

    if ($res.ReturnValue -eq 0)

    { gcim win32_ShadowCopy -Filter ("ID='"+$res.ShadowID+"'") } # **

    else

    { $res | fc }

      • AutoRestore only searches the top 3 queued snapshots for the latest valid backup, if no valid one is found, AutoRecovery will apply.
      • To restore others in the queue of snapshots ( manually )
        1. In Server sku, looking at 'previous version' tab of the repository folder to find the expected backup path 
        2. stop WMI service: Net stop winmgmt /y
        3. Replace all of the files in %windir%\system32\wbem\Repository folder with the files from the backup path found in step 1

    Note The WMI Service has auto start setting, and if it comes back alive, you will not be able to replace the files.  The service needs to be in a stopped state (if WMI service is alive at the time, repeat step: 2~3)

    ex. Directory of \\localhost\C$\@GMT-2014.03.13-01.02.49Windows\system32\wbem\repository

    03/11/2014  11:53 AM    <DIR>          .
    03/11/2014  11:53 AM    <DIR>          ..
    03/12/2014  05:30 PM         4,759,552 INDEX.BTR
    03/12/2014  05:30 PM            90,640 MAPPING1.MAP
    03/12/2014  03:26 PM            90,640 MAPPING2.MAP
    03/12/2014  05:24 PM            90,640 MAPPING3.MAP
    03/12/2014  05:30 PM        27,541,504 OBJECTS.DATA

    4. Run following wmic command to bring WMI service back to life: wmic os

    You should set up a regular Scheduled Task to backup the latest repository:

        • winmgmt /backup <backup image path>
        • tracing EVT: 67,68

    You could also schedule restores as necessary

    • winmgmt /restore <above backup image path> 1
    • tracing EVT: 65,66

    If the issue is not a repository issue, and the objects are not retrievable:

    • Re-install the product. This is the first place to start.
    • If there is a specific provider that is not showing up, you can re-run mofcomp of a single provider. See Ask the Performance Team Blog article WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

    If the issue persists or keeps returning, then at this point you will now need to open a Support Incident Case with Microsoft for further assistance.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and well help us track the effectiveness of the blog.

    Next up: WMI: Missing or Failing WMI Providers or Invalid WMI Class

  • WMI: Common Symptoms and Errors

    Quota Violation Issues: Memory and/or Handle

    Symptoms

    • General WMI-based scripts or applications fail or fail intermittently
    • Applications such as SMS/SCCM produce errors on server and/or inventories fail or fail intermittently
    • Applications such as Exchange or SQL fail on server or fail intermittently
    • Unable to connect to specific namespace via WBEMTEST or unable to query specific classes in a namespace. May be intermittent.
    • WMI appears to be hung or non-responsive
    • Unable to run msinfo32 or tasklist
    • Events for unexpected termination/crash of wmiprvse.exe
    • Lower than normal available memory on the system
    • Out of memory errors when running certain WMI tasks
    • 0x80041033 -- SHUTDOWN of the target provider
    • 0x80041006 – OUTOFMEMORY
    • 0x80041013 -- fail to load the provider
    • 0x800705af -- The paging file is too small for this operation to complete
    • 0x8004106C-- Quota violation
    • Event logs contain WMI-related events
    • WMI service crashing

    Event Source: Service Control Manager
    Event ID: 7031
    Description: The Windows Management Instrumentation service terminated unexpectedly

    • Handle Quota Violation

    Source: Microsoft-Windows-WMI
    Event 5612 Wmiprvse.exe exceeding handle quota limit Event
    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    • Memory Quota Violation does not log and event such as Handle Quota Violation does. You can check to see if any instance of wmiprvse is using 500 mb of memory or greater as the default limit is 512 mb on Vista an newer. Once you hit around 500 mb and any new allocation that would cross the 512 mb limit would fail. All instances of wmiprvse combined have a 1 gb limit. For Windows XP and 2003 the default is 128 mb. Exceed the default and you have reached a Quota Violation. Also check for Windows Management Instrumentation service crashing

    Memory Quota Violations refer to Ask Perf blog: WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

    Handle Quota Violations refer to Ask Perf blog: How to Troubleshoot WMI High Handle Count (COMING SOON)

    Missing or Failing WMI Providers or Invalid WMI Class

    Symptoms

    • General WMI-based scripts or applications fail
    • Applications such as SMS/SCCM produce errors on server and/or inventories fail
    • Applications such as Exchange or SQL fail on server
    • Unable to connect to specific namespace via WBEMTEST or unable to query specific classes in a namespace
    • WMI functionality appears normal on local node but unable to connect to/from machine via WMI scripts, tools or applications
    • WMI (Local) properties in wmimgmt.msc console and see the following

    Failed to initialize all required WMI classes
    Win32_processor: WMI: Invalid namespace
    Win32_WMISetting: WMI: Invalid namespace
    Security information: Successful
    Win32_OperatingSystem: WMI: Invalid namespace
    Win32_processor: WMI: Invalid namespace
    Win32_WMISetting: WMI: Invalid namespace
    Win32_OperatingSystem: WMI: Invalid namespace

    • WBEM_E_NOT_FOUND 0x80041002
    • WBEM_E_PROVIDER_FAILURE 0x80041004
    • WBEM_E_INVALID_NAMESPACE 0x8004100E
    • WBEM_E_INVALID_CLASS 0x80041010
    • WBEM_E_PROVIDER_NOT_FOUND 0x80041011
    • WBEM_E_INVALID_PROVIDER_REGISTRATION 0x80041012
    • WBEM_E_PROVIDER_LOAD_FAILURE 0x80041013

    Refer to Ask Perf blog: WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

    High Memory Usage by WMI Service or Wmiprvse.exe

    Symptoms

    • Wmiprvse.exe memory quota violations – refer to top of blog
    • Lower than normal available memory on the system
    • Delayed or slow logons to the box
    • Excessive or slow return times to queries to WMI or scripts that are running that call to WMI
    • Spinning donut when trying to bring up WMI (Local) properties in wmimgmt.msc console or using Wbemtest (Windows Management Instrumentation Tester) built in tool
    • Sluggish or slow responding system
    • Server hang
    • Unable to run msinfo32 or tasklist
    • Svchost process housing WMI service (winmgmt) exhibiting high memory usage or leak
    • Instance of wmiprvse reaching or exceeding 512 mb on Vista and newer, or 128 mb on XP or Windows 2003: Quota Violation issue
    • Large repository C:\Windows\System32\Wbem\Repository folder and objects.data file is 1gb or larger
    • Cluster management tools not working

    Refer to Ask Perf blog: WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

    High CPU Usage by WMI Components

    Symptoms

    • High cpu usage by svchost hosting WMI service (winmgmt)
    • High cpu usage by wmiprvse
    • System sluggish or slow performance

    Refer to Ask Perf blog: How to troubleshoot High CPU Usage by WMI Components (COMING SOON)

    High Handle Count on WMI Components

    Symptoms

    • Refer to top of blog: Quota Violation Issues: Memory and/or Handle 
    • Following Event being logged

    Source: Microsoft-Windows-WMI
    Event 5612 Wmiprvse.exe exceeding handle quota limit Event
    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    • Resource depletion type messages
    • Error message: No more treads can be created in the system
    • Message when trying to RDP

    The User Profile Service service failed the logon

     

    • High handle count on svchost process hosting WMI service (winmgmt)
    • Cluster management tools not working

    Refer to Ask Perf blog: How to Troubleshoot WMI High Handle Count (COMING SOON)

    Repository Corruption

    Symptoms

    • General WMI-based scripts or applications fail
    • Applications such as SMS/SCCM produce errors on server and/or inventories fail
    • Applications such as Exchange and SQL fail on server
    • WINMGMT.MSC Security tab shows missing, repeating and/or gibberish namespace entries
    • Unable to connect to specific or possibly any namespace via WBEMTEST
    • Winmgmt.msc console WMI (local) properties Security tab shows missing, repeating and/or gibberish namespace entries
    • Unable to connect to root\default or root\cimv2 namespaces. Fails returning error code 0x80041002 pointing to WBEM_E_NOT_FOUND
    • Opening Computer Management and Right Click on Computer Management (Local) and select Properties, you get the following error: "WMI: Not Found" or it hangs trying connect
    • Wbemtest (Windows Management Instrumentation Tester) built in tool hangs
    • Error 0x80041010 WBEM_E_INVALID_CLASS
    • Any events with Source = Microsoft-Windows-WMI, look for any of the following WMI event IDs: 28, 65, 5600, 5601, 5614 as any of these can be indicators or repository corruption
    • WMI namespaces end up missing

    Refer to Ask Perf blog: WMI: Repository Corruption, or Not? (COMING SOON)

    Next up: WMI: Repository Corruption, or Not?

  • Troubleshooting WMI Series coming

    Hello AskPerf! My name is Jeffrey Worline and I wanted to let you know that I will be writing several troubleshooting WMI blog posts over the next few days. These will not be your typical Blog post, but more around the KB/Technet format.

    There has been a lot of internal discussions here at Microsoft with Product Group, Developers, and myself concerning the high amount of WMI issues that have come into Support. Whenever there is an issue with WMI itself, or suspected issue with WMI, the first words that always seem to come out is that “WMI is corrupted”. Oh no, say it isn’t so. What do we do? Grab the “Big Hammer” and rebuild the repository! Excuse my humor as we have a little fun, as the upcoming blogs are going to be more straight forward and dry.

    While rebuilding the repository may seem like a good idea and a quick fix at times, what most do not realize is that if you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows and/or to your installed applications. That last sentence is worth repeating; “Deleting and rebuilding the repository can cause damage to Windows and/or to your installed applications.” When it becomes necessary to rebuild the repository and this has been verified, there are right and wrong ways to go about it.

    Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption before taking such actions as rebuilding the repository. There are also other proactive steps that can be taken as alternatives to rebuilding the repository if such an occasion arises.

    So, the following blog posts will address various scenarios surrounding WMI, how to troubleshoot them, and the data we should collect to help resolve WMI related issues. These posts will equip you with the information you need to address WMI issues in the correct manner, and to help prevent you from causing more damage by just using the proverbial “Big Hammer” approach.

    With that, here is a list of the Posts/Topics you will see in the coming days: 

    • WMI: Common Symptoms and Errors
    • WMI: Repository Corruption, or Not?
    • WMI: Missing or Failing WMI Providers or Invalid WMI Class
    • WMI: High Memory Usage by WMI Service or Wmiprvse.exe
    • WMI: How to troubleshoot High CPU Usage by WMI Components
    • WMI: How to Troubleshoot WMI High Handle Count

    NOTE the above bullet points will become active URL links once each post becomes available.

    -Jeff

  • How to Monitor for Storage Account Throttling

    For Azure Virtual Machines, we document storage limits of 500 IOPS per disk, and the guidance to not place more than 40 highly used VHDs in any single storage account since there is a 20,000 IOPS limit per storage account. By default, each subscription ...read more