SupportingWindows

  • Remove Lingering Objects that cause AD Replication error 8606 and friends

    Introducing the Lingering Object Liquidator Hi all, Justin Turner here ---it's been a while since my last update . The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering ...read more
  • How to identify a driver that calls a Windows API leading to a pool leak on behalf of NT Kernel?

    Hello my name is Gurpreet Singh Jutla and I would like to share information on how we can trace the caller which ends up allocating “Se  “ Pool tag. When we use the Windows debugger and investigate the pool allocation and the binary associated with ...read more
  • Announcing public availability of MBAM Compliance Data Cleanup Tool 2.5

    We are happy to announce public availability of MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe), aka MBAMCDCT 2.5.
     
    MBAM Compliance Data Cleanup Tool 2.5 (clean-mbam.exe) is a command line tool which enables you to delete machine records from the ‘Compliance Status’ database of the MBAM 1.0 and MBAM 2.0, MBAM 2.0 SP1 and MBAM 2.5 standalone.

     

    There have been situation where you as a MBAM Admin had to delete the entries of older/reimaged machine records from the MBAM compliance database. The only solution in this case was to run complex SQL queries to delete machines from the database. This tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the MBAM Compliance Status database.

     

    This is a command line tool which enables you to schedule it stale data deletion as a task to automate deletion of obsolete machine records from the MBAM compliance database.


    This tool provides three different ways to delete machine records from the MBAM Compliance Status database:

    1.     Delete machines which have not reported in last X days.

    2.     Delete machines specified in a comma separated list via command line.

    3.     Delete machines specified in a text file.

     

    Note:

     

    This tool doesn’t delete the recovery information or any other data from MBAM Recovery and Hardware Database. All delete operations are performed strictly on the MBAM Compliance Status Database.

    This tool is available for download from the TechNet website http://gallery.technet.microsoft.com/MBAM-Compliance-Data-9b4c950d as a self-extractable compressed file, which includes the executable and documentation.

    Hope this tool helps you report the true state of encryption compliance in your environment by deleting the obsolete information from the database.

    Disclaimer:

    This tool and documentation are provided "as-is". You bear the risk of using it. No express warranties, guarantees or conditions are provided. The tool supplied in this document is not supported under any Microsoft standard support program or service. However, you can report issues and bugs in the comments section on this page. Microsoft will, at its sole discretion, address issues and bugs reported.

     

     

    Himanshu Singh

    Windows Core Team

  • We Are Hiring Windows Escalation Engineers in Charlotte, Dallas, and Redmond

    Would you like to join the world’s best and most elite debuggers to enable the success of Microsoft solutions?   As a trusted advisor to our top customers you will be working with to the most experienced IT professionals and developers in the industry ...read more
  • Windows Troubleshooting – Stop 9E Explained

    What to do if a stop 9E occurs.  How you can solve the issue yourself. ...read more
  • Windows Troubleshooting – Special Pool

    The Windows Support team has a new YouTube channel, “ Windows Troubleshooting ”.  The first set of videos cover debugging blue screens. In this video, Bob Golding, Senior Escalation Engineer, describes how the Special Pool Windows diagnostics tool ...read more
  • Hate to see you go, but it’s time to move on to greener pastures. A farewell to Authorization Manger aka AzMan

    Hi all, Jason here. Long time reader, first time blogger. AzMan is Microsoft’s tool to manage authorization to applications based on a user’s role. AzMan has been around since 2003 and has had a good run. Now it’s time to send it out ...read more
  • Go the modern (app) way

    Hello everyone,

    This is Ashfana from the Windows Performance team. I’m here to talk about the basic philosophy behind Windows Store Apps which were introduced in Windows 8 and Windows 8.1. Windows Store apps were originally called Metro or modern apps before the RTM of the product. These are apps developed using a variety of languages like C#, C++, VB, HTML and Javascript, - which provide the modern, full screen immersive experience.

    Windows has the largest software ecosystem thanks to millions of developers around the world, and Microsoft’s focus on ensuring backward compatibility for application software. This ecosystem has evolved over time and has reached a stage where – we aren’t able to fix some of the limitations and move ahead without causing existing applications to break. Some limitations are installation/uninstallation problems, Software state corruption, uncontrolled modification of important system registries and files, unreliable installer components, one app causing another one to crash, and security risks when running Apps under elevated token. This new App model was introduced to address these reliability and security issues and to develop a new ecosystem that allows Apps to run on multiple hardware platforms with minimum efforts for the developer.

    Here are some of the features of Store Apps that make it very attractive for consumers and enterprises:

    • Windows Store apps do not need an installer. They are either sideloaded to the image using DISM, or installed via Add-AppxPackage PowerShell Command, or installed directly from the Store or a Azure based portal. You no longer need MSI or other installer components, Windows already has the necessary components that can extract the Appx package and install it.
    • Apps are installed per-user (unless they are sideloaded into the image before deployment). They maintain their state within folders in the user’s profile. This ensures better security.
      Apps run within a sandbox environment called an App container. This restricts an App from accessing much of the system and the users profile by default. Additional resources can be requested by an App by declaring capabilities within its manifest file. During first launch of the App, the user is prompted to allow/confirm access to these additional resources.
    • Desktop Apps run under medium integrity and are able to access full privileges granted to a user. An untrusted desktop application can pose a greater risk. Windows Store Apps on the other hand run under base privilege and hence provides a greater level of protection.
    • The Appx manifest and blockmap tell Windows how to install the App, chances of an installation going corrupt are very less. Even if an installation goes corrupt – it would not affect the system in any way.
    • Store Apps provide an immersive experience, are generally very lightweight, and are best suited for scenarios where hard core processing can be done at the server side and results can be delivered to the App over the network.
    • With the introduction of Universal Apps in Visual Studio 2013 update 2, developers can now write an App once and port it to run on multiple devices such as Xbox, Windows phones, PC, tablets and laptops. This makes application development time much shorter.

    Structure of a Windows Store/Universal App:

    Windows Store app are packaged into one or more files with the extension .appx. This appx package is the unit of installation for a Windows Store app. Appx packages are ZIP-based container files that contains the app’s payload files plus info needed to validate, deploy, manage, and update the app. From a high-level view, each Windows Store app package contains these items:

    image

    App payload
    App code files and assets
    Payload files are the code files and assets that you author when you create your Windows Store app.

    App manifest
    App manifest file (AppxManifest.xml)
    The app manifest declares the identity of the app, the app's capabilities, and info for deploying and updating.

    App block map
    App package’s block map file (AppxBlockMap.xml)
    The block map file lists all the app files contained in the package along with associated cryptographic hash values that the operating system uses to validate file integrity and to optimize an update for the app.

    App signature
    App package’s digital signature file (AppxSignature.p7x)
    The app package signature ensures that the package and contents haven't been modified after they were signed. If the signing certificate validates to a Trusted Root Certification Authorities Certificate, the signature also identifies who signed the package. The signer of the package is typically the publisher or author of the app.

    Packaging app:

    When you create an app using Visual Studio, it packages your app, it automatically adds the app block map and signature files to the package. But you can also use the standalone MakeAppx and SignTool utilities if you want to manually package your app.

    The steps that are involved in Package installation:

    image

    The Windows Store app deployment process occurs in multiple phases.

    1) In the first phase, Windows acquires and validates the app manifest, app block map, and app signature. It checks the OS version and dependencies, sees if you have sufficient disk space, and also whether this app is already installed.

    2) In the second phase Windows checks the app package’s deployment criteria to ensure that the app deployment will be successful.

    3) Windows stages the package’s contents on the disk in the %ProgramFiles%\WindowsApps directory in a new directory named after the package identity:

    <Package Name>_<Version>_<Architecture>_<ResourceID>_<Publisher Hash>

    4) Windows registers the package into the user's account. During this phase, the extensions that are declared in the manifest are registered with the operating system.

    The Future:

    With this release of Visual Studio 2013, we have set out to accomplish three major goals:

    1) Reach customers across phones, tablets, and PCs;

    2) Deliver innovation that supports developer investments;

    3) Make cross-platform technology easier and more capable.

    image

    Develop once for all Windows devices using a unified Windows runtime and VS tools that allow you to both support experiences unique to a device in XAML, HTML, and DirectX, and share the code that supports those experiences across all devices using C++, C#, or JavaScript. When your work is finished you can you can produce the app packages that you will submit to the Windows Store and Windows Phone Store with a single action to get your app out to customers on any Windows device. You may also just sign it and deploy this through SCCM.

    More links:

    Package manifest schema reference
    http://msdn.microsoft.com/en-us/library/windows/apps/br211473.aspx

    Capabilities of an App:
    http://msdn.microsoft.com/en-us/library/windows/apps/br211422.aspx

    Create an app package
    http://msdn.microsoft.com/en-us/library/windows/apps/hh975357.aspx

    App packager (MakeAppx.exe)
    http://msdn.microsoft.com/en-us/library/windows/apps/hh446767.aspx

    SignTool
    http://msdn.microsoft.com/en-us/library/windows/apps/ff551778.aspx

    How to create an app package signing certificate
    http://msdn.microsoft.com/en-us/library/windows/apps/jj835832.aspx

    Ashfana Begum
    Support Engineer
    Microsoft Performance Team

  • Deciphering Storport Traces 101

    Welcome back to the CORE Team Blog -- Paul Reynolds here. In previous blogs, I wrote about how to capture Storport traces in Windows 8 and Windows 2012. Please see:

    Tracing with Storport in Windows 2012 and Windows 8 with KB2819476 hotfix

    And

    Tracing with Storport in Windows 2012 and Windows 8 without KB2819476 hotfix

    This time around, I would like to explore what information you can draw from the raw data contained in a Storport trace. What conclusions can you reach regarding your disk performance? Do you have a disk that does not perform well?

    To accomplish this, we will take advantage of free tools available to Windows and Office 2013 users:

    • Windows PowerShell
    • Windows Performance Toolkit
    • Excel PowerPivot

    First, we need to talk briefly about the Windows Storage Stack and where Storport traces are taken. It is important to note that Storport traces are at the very last “rung of the ladder” before Windows hands off I/O request packets to hardware. Hardware in this case encompasses firmware, drivers, HBAs, storage fabrics – anything after the Windows Operating System. It is important because it can help delineate where the problem is. Is the problem with the Operating System or with the hardware? Should I call Microsoft to open a support case or it is more appropriate to talk to my Storage Vendor? Storport traces are the perfect place to start to answer questions such as these.

    It is assumed you already have a Storport Trace in hand – an ETL file captured using the procedures documented in the two blogs above for Windows 8 and Windows 2012, or Windows 2008 (see Bob Golding’s blog):

    Storport ETW Logging to Measure Requests Made to a Disk Unit

    As the old adage goes: a picture is worth a thousand words. When it comes time to deciphering Storport Traces, viewing graphs that summarize data and show trends, and viewing charts that have average and maximums, are much more helpful than looking at raw data. The top-level steps we will undertake are:

    1. Capture the Storport Trace
    2. Convert to the Storport Trace into CSV format
    3. Scrub the data in the CSV file
    4. Import the data into an Excel Spreadsheet

    The easiest way to accomplish the last 3 steps above is to automate it using a Windows PowerShell script. At the end of the blog, there is a zip to download that has this file. Save the PowerShell script as StorPortPACMAN.PS1 in a directory on your C: drive called StorPortPACMAN :

    PLEASE NOTE: you must have the following perquisites installed to successfully run the script above:

    1. XPERF needs to be installed and in your system path.

    Windows 7 or 8 – use the Windows Assessment and Deployment Kit (ADK): http://www.microsoft.com/en-US/download/details.aspx?id=39982

    2. PowerPivot is part of Office 2013 but it needs to be added as a COM Addin. Click here for instructions.

    At the end of this blog, there is also 2 spreadsheets in the zip you must download into the C:\StorPortPACMAN directory.

    Run the script, and depending on which version of Windows was being run on the server, one of the Excel Spreadsheets will open and be refreshed with data in your Storport trace.

    Here are screenshots of sample graphs and charts you will see in the first page of the spreadsheets:

    image

    This first graph shows request duration values over the time of the Storport trace. It is very useful to get the “big picture”. For the most part, the disk in this graph is performing fine except for a period near the beginning. Depending on the application you are investigating, this may be fine and the average value is what is important to you. Averages and maximums are shown in the chart below, which is next to this graph in the spreadsheet:

    image

    Finally, the chart data is presented in the graph below:

    image

    The Excel spreadsheets and their corresponding graphs and charts are only samples of what you can do for your disk analysis using Storport traces and Excel PowerPivot tools. Why use PowerPivot? For Storport traces it means being able to view and summarize more data. Excel by itself is an excellent tool, but problems may start to develop when your data approaches or exceeds a million rows of data.

    It is not uncommon for a Storport trace to contain tens of millions of rows of data, especially if you decide to not use a filter while capturing the data. I generally suggest to not use a filter as your averages will be closer to results you obtain from other tools such as Perfmon or XPERF. Using filters will cause your disk to look worse than it really is.

    Request Duration Times, as a rule of thumb, can be summed up as follows for SCSI Read(10) and Non-cached Write(10) I/O:

    <  9ms = excellent
    < 15ms = good
    < 25ms = fair
    > 25ms = poor

    We purposely use the caveat, rule of thumb, because these values are using an assumed 64KB data I/O size. If a read or write is larger or smaller than that, these values should be adjusted.

    Using an Excel Pivot table and chart lends well to focusing on a specific LUN through the use of filters built into the table or chart.

    Special Cases:

    We occasionally see a Storport trace that does not make sense. One example is a VM that has more than one SCSI disk and the Excel spreadsheet shows only one LUN. What happened to my other disk? It is there, but because of the way storage may be presented, the same LUN number is used for more than one disk. So what does one do in a situation such at this? The easiest way is to drop and drag a second property of a disk into the chart or graph you are viewing. There are 4 properties we gather in a Storport trace:

    1. LUN
    2. Target
    3. Bus
    4. Port

    Since using LUN is not enough, we can add a second property to the chart, Target. To do this, at the top of the spreadsheet click on the Analyze tab under Pivottable Tools, then click on Field List to expose the data available to you. Drag and Drop the Target field onto Rows and underneath LUN. You will see something similar to this:

    image

    This will let you see all your disks if they have the same LUN number.
    If you are at all like me, viewing charts and graphs might be fine for seeing the big picture and giving presentations, but your inner engineer is screaming to look at the data in detail, to make sure you are comfortable with it and understand it. To do this, click on the POWERPIVOT tab and then the Manage button in the Data Model area. This will open a new window that exposes the raw data to you so that you may filter it, sort it, or anything else you wish to make sure it all makes sense to you before using it.

    I hope you find this blog helpful and gives you a new tool to use when investigating disk performance.

    Paul Reynolds

  • WMI: How to Troubleshoot WMI High Handle Count

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is experiencing high handle count

    Your first thing to do is check the Application Event log for following event:

    Source: Microsoft-Windows-WMI

    Event 5612 Wmiprvse.exe exceeding handle quota limit Event

    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    If you find the above Event, you can try and bump up the handle quota limit to see if it resolves your issue. If it is a leak, then bumping limit will only mean it will take longer to reach the new limit. If it’s just load related, then bumping the limit could resolve the issue.

    The event will tell you what the handle count was, and if it is higher than the 8192 value I suggest below.  You can then skip this section and move on to data collection to figure out the cause of high handle count.

    How to increase the handle quota limit for the WMI Provider Service

    1. Go to Start--> run and type wbemtest.exe
    2. Click Connect on the Windows Management Instrumentation Tester
    3. In the namespace text box just enter "root" (without quotes)
    4. Click Connect

    clip_image001

    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    1. Click "Enum Instances…"

    clip_image002

    1. In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK.

    Note: a double underscore __ precedes ProviderHostQuotaConfiguration

    clip_image003

    1. A query Result window will come up. In this windows now double click "__ProviderHostQuotaConfiguration=@"

    clip_image004

    1. An Object Editor windows will come up now
    2. Under properties find the property "HandlesPerHost"

    clip_image005

    1. Change the value from default of 4096 to 8192
    2. Click Save Property
    3. Click Save Object in the Object Editor window
    4. Close the other windows now and exit WMI Tester
    5. Restart Windows Management Instrumentation Service.

    If after bumping up quota limit and wmiprvse still exceeding quota limit, accomplish following actions below. You will want to read through the rest in its entirety to ensure you get all of the necessary tools downloaded before taking any actions.

    Configure System for Complete Memory Dump by referring to:

    Windows 8 and Windows Server 2012 Automatic Memory Dump: http://blogs.technet.com/b/askcore/archive/2012/09/12/windows-8-and-windows-server-2012-automatic-memory-dump.aspx

    Windows does not create a memory dump file when a Stop error occurs in Windows 8 or Windows Server 2012: http://support.microsoft.com/kb/2853466

    Windows 2008, Windows Vista, Windows 7, Windows 2008 R2: http://support.microsoft.com/kb/969028

    Windows Server 2003 and Windows XP: http://support.microsoft.com/kb/972110

    Collect perfmon logging using logman method

    Directions below will create 2 perfmon logs, one at a 5 minute interval (PerfLog-Long) and a short 5 second interval log (PerfLog-Short) and they will be placed in C:\Perflogs folder.

    • Long log (5 min intervals) – no thread counter, 250 MB:

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Long -o "c:\perflogs\PerfLog-Long.blg" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues\*" "\System\*"  -si 00:05:00

    3. Start the log with:

    Logman.exe start PerfLog-Long

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Long

    • Short, high resolution log – 5 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window:

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:05

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Configure Handle Tracing

    You probably just need the standalone version since we only need the debugging tool and not the whole WDK package.

      1. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator
      2. Click on the ‘Kernel Flags’ tab
      3. Check the box next to ‘Enable bad handles detection’
      4. Click on the ‘Image File’ tab
      5. Next to ‘Image: (TAB to refresh)’, enter wmiprvse.exe or if tracing WMI Service enter the unique name given to the svchost process for WMI Service per directions in next section below
      6. And then click on the ‘Tab’ key
      7. Next to ‘Stack Backtrace: (Megs):’ enter ‘10’
      8. Click on Apply
      9. Click on Ok
      10. Restart WMI service

    If it is a svchost process showing high handle count, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From there in a command prompt you can run tasklist /svc and look for the PID then identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From experience it will be the WMI service more times than not but not always, as such I would try to break it out first on its own and monitor to see if it is the one driving up high handle count in the shared svchost process.

    WMI (Windows Management Instrumentation) service, you can break it out by accomplishing the following.

    Break WMI Service out into its own unique svchost process

    a.      Open command prompt with elevated privileges

    b.      Run following command: sc config winmgmt type= own

    c.      Restart Wmi service

    d.      Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    sc config <service name> type= share

    e. Restart the service or machine and verify result is Win32_SHARE_PROCESS when you run sc query winmgmt command again

    f. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe

    g. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt

    h. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    I. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    j. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    k. You would now substitute wmisvchost.exe in lieu of wmiprvse.exe in step 6. under Configure Handle Tracing above

    Using debugger to attach to the process in windbg.exe and running !htrace –enable command

    1. Launch WinDbg program from under Debugging Tools for Windows that you installed earlier.

    2. Created folder c:\websymbols

    3. Click on File-Symbol path and add the following symbol path to the debugger: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    4. Click on File-Save Workspace

    Attach to process to accomplish handle tracking using htrace

    To do this:

    1. From Windbg - File - Attach a Process - Select the instance of wmiprvse.exe with high handle count

    Note: If it is WMI Service (run tasklist /svc or Task Manager with PID column added first to find the PID of svchost.exe containing winmgmt which you should have broken out and uniquely named wmisvchost.exe per earlier directions)

    2. Run following command from the debugger:

    .logopen "C:\debug.log" then hit <ENTER> key

    !htrace -enable 0x20000 then hit <ENTER> key

    Note:  By default, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 keep a history of 4000 handles open and close operations. 

    With !htrace you can enable to keep a much higher history by doing the following:

    !htrace -enable 0x20000

    In this example, we are increasing the handle history to 131072 (decimal, 0x20000 hexadecimal)

    !htrace –snapshot then hit <ENTER> key

    g then hit <ENTER> key

    3. Now, let the process run until the number of handle has increased a lot and gotten high.

    Final htrace log

    1. Break into debugger with Keyboard keys (Ctrl+Break)

    2. Run following commands:

    !htrace –diff then hit <ENTER> key

    .logclose then hit <ENTER> key

    .detach then hit <ENTER> key

    3. Close WinDbg

    Now complete the following actions once you have gotten your final htrace log

    1. If high handle count is with wmiprvse, download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    2. Find the instance of wmiprvse.exe with high handle count and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    3. If the WMI Service was the process with the high handle count, then dump out the WMI service process which should be wmisvchost.exe per previous directions and all instances of wmiprvse.exe using procdump. If it is wmiprvse.exe that is exhibiting the high handle count, then only need to dump out that instance and nothing else.

    a. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    b. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    c. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID

    d. Run the following command: procdump –ma <PID>

    e. Note: Replace <PID> with actual PID you documented for instances of wmiprvse.exe and/or wmisvchost.exe as it applies based on directions above

    4. Stop Perfmon logging

    5. Do a complete memory dump of the machine

    At this point with data in hand you will want to open a Support Incident with Microsoft to get the data analyzed to help determine cause of high handle count.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

  • WMI: How to troubleshoot High CPU Usage by WMI Components

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of CPU.

    In the directions below, you may have already broken out WMI Service to troubleshoot your issue.  By default, WMI runs in the main shared networking svchost process with several other services.

    If it is a svchost process showing high cpu usage, you can use Task Manager and add PID column, then identify which svchost process has the high memory usage. From inside a command prompt you can type in  tasklist /svc and look for the PID #, and identify if a single service is running in that svchost process or multiple services. If multiple services, it may become necessary to break each service out to run in its own svchost process to determine if it is the WMI service (winmgmt) that is causing the issue. From my experience, it will be the WMI service more times than not but not always.  As such, I would suggest breaking it out first into its own, and monitor to see if it is the one driving up high memory usage in the shared svchost process.

    If you suspect the WMI (Windows Management Instrumentation) service, you can break it out following directions below.

    Break WMI Service out into its own svchost process

    1. Open command prompt with elevated privileges
    2. Run following command: sc config winmgmt type= own
    3. Restart Wmi service
    4. Run sc query winmgmt to ensure status of service now reflects “own” indicating running in its own svchost process 

    When issue had been resolved or no longer needing the service broken out into its own svchost process, place it back into the shared svchost process by running following command from command prompt:

    • sc config <service name> type= share
    • Restart the service or machine and verify result is Win32_SHARE_PROCESS when you
    • run sc query winmgmt command again

    Configure Perfmon Collection using logman.exe method. Capture 15 minutes while issue is occurring.

    Short, high resolution log – 1 sec interval with thread counter, 250MB

    1. Click on Start

    <<Start Search>>, enter "CMD.exe" w/o the quotation marks and then press Enter.

    2. Copy and paste the following command into the command prompt window (if this does not work, you may need to manually type it in):

    Logman.exe create counter PerfLog-Short -o "c:\perflogs\PerfLog-Short" -f bincirc -v mmddhhmm -max 250 -c "\Cache\*" "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Server Work Queues\*" "\Thread(*)\*" -si 00:00:01

    3. Start the log with:

    Logman.exe start PerfLog-Short

    4. Please stop the performance log as soon as the issue returns with the following command:

    Logman.exe stop PerfLog-Short

    Please note that if you reboot the server, you will need to start the logs again as they will not automatically restart on boot.

    Collect and Xperf trace for High CPU by using the Windows Performance Recorder form the Windows Performance Toolkit which you can install from the ADK

    Note: If the Operating System is a 64 bit box, you must first accomplish the following registry setting before collecting Xperf trace.

    Registry Path
    HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
    Setting
    DisablePagingExecutive
    Data Type:
    REG_DWORD
    Value:
    1

    NOTE setting this key is not needed on Windows Server 2012 & 2012 R2

    Reboot machine to place registry setting into effect.

    1. Download the Windows 8 ADK (Windows Assessment Deployment Kit) from here.
    2. Open the adksetup.exe and hit next until you get you the option to select feature options
    3. Select "Windows Performance Toolkit" and hit "Install"

    clip_image002

    After installation has finished, start creating a trace by starting the "Windows Performance Recorder"

    clip_image004

    Select CPU usage under Resource Analysis

    Logging mode can be left set to “Memory”, or you can change to “File”. Just be conscious of your disk space if you chose “File” as the etl file can become large fast

    Capture high cpu occurrence, but do not let the recording run for no more than 10 minutes.

    Immediately after capturing the event using Windows Performance Recorder (WPR), now use process explorer to dump out the process exhibiting high cpu usage.

    1. Download Windows Sysinternals tool called Procdump: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    2. Open a command prompt with elevated or administrative rights and change to the directory were you saved Procdump

    3. Open Task Manager and add the PID column view, then go locate the instance of wmiprvse.exe with high cpu usage and note the PID. If it was the WMI service that had the high cpu, then you should already have it broken out to run in its own svchost process and note the PID of that svchost process. To confirm you have the right svchost process, you can run tasklist /svc from administrative command prompt and verify the PID noted in task manager and ensure it is the svchost process running winmgmt in it.

    4. Run the following command: procdump –ma -s 60 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe or for the svchost process running winmgmt exhibiting high memory usage

    The above command will produce 3 dumps spaced 1 minute apart each in same directory you ran the procdump command from

    5. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    6. If it was wmiprvse.exe that had the high CPU usage, then find the instance and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high CPU usage.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up:  WMI: How to Troubleshoot WMI High Handle Count

  • WMI: High Memory Usage by WMI Service or Wmiprvse.exe

    Scenario

    Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of memory.

    There are many reasons why WMI might experience high memory consumptions. This can occur in the WMI (Windows Management Instrumentation) service (winmgmt) or in the WMI provider hosting vehicle wmiprvse.exe. Both scenarios will be addressed below.

    High memory usage may simply be due to load, as opposed to some type of leak. By default, the memory quota limit for instances of wmiprvse.exe on Windows XP and Windows Server 2003 is 128 MB, and 512 MB on newer Operating Systems (Vista and higher). Hit those limits and wmi functionality will become problematic if not come to a grinding halt.

    There is another case where the quota limit problem could happen. There is limit for all wmiprvse’s cumulatively. If the total memory of all instances of wmiprvse’s (with one exception) together reaches 1GB, then all new memory allocations fail in all wmiprvse processes.

    As a first effort you can try to bump up that quota limit to see if it gives enough room for wmiprvse to operate, but if you still reach the new quota limit, then you will want to proceed with the Scenarios below

    How To Bump up WMI memory quota limit:

    • Go to Start--> Run and type wbemtest.exe
    • Click Connect

    clip_image001 

     

    • In the namespace text box type "root" (without quotes).

    Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

    • Click Connect
    • Click Enum Instances…

    clip_image002

     

    • In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK. Note: the Superclass name includes a double underscore at the front.

    clip_image003

     

    • In the Query Result window, double-click "__ProviderHostQuotaConfiguration=@"

    clip_image004

     

    • In the Object Editor window, double-click whichever Property name you wish to modify the quota for. In this case that will be MemoryPerHost

    clip_image005

     

    • In the Value dialog, type in 536870912 (512 MB) for XP and Windows 2003.  For Vista and higher, start with new value of 805306368 (768), then move to 1073741824 (1024) if still needing room. At the same time, if you are going to bump up the MemoryPerHostLimit, you should also then bump up the MemoryAllHost limit to 2147483648 (2048) provided you have the available ram to do so. If this does not resolve quota violation issue, then need to troubleshoot cause of high memory usage following below Scenario section
    • Click Save Property.
    • Click Save Object.
    • Close Wbemtest.
    • Restart the machine

    Note: if you cannot restart the machine, then as a workaround, you can break Windows Management Instrumentation service into its own svchost process outline in step 2. a-c below.

    If bumping the quota limits does not resolve the issue, then as workaround you can try to move suspected leaking providers into their own group (wmiprvse) to avoid the memory quota caused by other providers running in the group or kill the instance of wmiprvse exhibiting high memory until issue is resolved. This is outlined below in Scenarios 1a and 2a.

    First thing we need to determine is if the memory consumption being caused by private data or heap data. We have to address the 2 types differently.

    1. Download a Windows Sysinternals tool call VMMap from following url: http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx

    VMMap is a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types.

    This tool is used to attach to an individual process allowing a snapshot to be taken to see the memory map for that process.

    2. Simply launch VMMap and from the process list it displays, pick the instance of wmiprvse showing the high working set memory usage.

    3. If it is a svchost process that is exhibiting high memory, from a command run scqueryex winmgmt to identify the PID of the svchost process that is hosting WMI Service (winmgmt). From experience it will be the WMI service more times than not but not always as the service using majority of the memory; as such I would try to break it out first on its own and monitor to see if it is the one driving up high memory usage in the shared svchost process.

    From this point, I will assume it is the WMI service as this article is only addressing WMI and no other services.

    You will need to break the WMI service out into its own svchost process first if it hasn’t already been accomplished so you can run VMMap specifically against the WMI service to truly know if it is being consumed by Heap or Private Data. By default the WMI service runs in a shared svchost process with a community of other services. You can do so by the following directions. If it is caused by Heap, you will find later in the Scenario directions below that you will have to go back and uniquely name the svchost process for the service to run in to be able to enable User Stack Tracing against it.

    a. Open command prompted with elevated privileges

    b. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own

    c. Restart wmi service with net stop winmgmt and net start winmgmt commands

    d. Verify winmgmt service running in its own svchost process by runnning tasklist /svc

    4. Once it displays the result, look under the size column for Private Data and Heap. This should tell you if the majority of the memory being consumed is Private Data or Heap.

    5. Follow appropriate Scenario below based on if issue is with Windows Management Instrumentation service or wmiprvse and if caused by Heap or Private Data memory.

    Scenario 1a: High Memory Consumption by Wmiprvse.exe caused by Heap memory

    1. Download Windows Debugging Tools for Windows from following link: http://msdn.microsoft.com/en-us/windows/hh852365.aspx

    You can probably just get the standalone version since we only need the debugging tool and not the whole WDK package.

    2. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator

    3. Click on “Image File” tab

    4. Type in wmiprvse.exe

    5. Hit the keyboard TAB key

    6. Place check mark in “Create user mode stack trace database

    7. Click Ok

    8. Download the latest version of the Windows Sysinternals tool Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    9. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high memory usage:

    wmic path msft_providers get Provider,HostProcessIdentifier /format:list

    10. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

    11. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

    $prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
    <ENTER>
    $prv.HostingModel = $Prv.HostingModel + ":OWN"
    <ENTER>
    set-ciminstance -inputobject $prv
    <ENTER>

    Repeat the above for each Provider found in step 9

    12. Restarting of Wmi service will put the settings into effect without having to reboot machine

    13. Now you will want to take a process dump of the instance of wmiprvse when memory usage is high. Download Windows Sysinternals tool called Procdump from URL: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    14. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

    15. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

    16. Run the following command: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

    The above command will produce 3 dumps spaced at 5 minutes apart each in same directory you ran the procdump command from

    17. Note also as workaround until issue gets resolved you could simply kill the instance of wmiprvse with high memory from task manager or from PowerShell can run following:

    kill -f <pid of suspect wmiprvse>

    At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high memory usage

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 1b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Heap memory

    The Windows Management Instrumentation service runs under the display name of winmgmt and is located in networking svchost.exe process shared along with several other services. You need to break the WMI service out into its own unique svchost process for data collection purposes.

    NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 6 to uniquely name the svchost process it is running in. We do this because we only want to enable User Stack Tracing against just the WMI service and not every svchost process.

    1. Open command prompted with elevated privileges
    2. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own
    3. Restart wmi service with net stop winmgmt and net start winmgmt commands
    4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
    5. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe
    6. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt
    7. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

    Note: It is important that you go back and reverse what you did in step 7 and modify path back to original after you are no longer needing the service to be broken out and uniquely named as failure to do this can prevent future WMI hotfixes from being installed.

    Simply run following command then restart wmi service: sc config winmgmt type= share

    8. Restart wmi service with net stop winmgmt and net start winmgmt commands again

    9. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

    10. For sake of brevity here, follow steps 4-14 from Scenario 1a with the exception of typing in wmisvchost.exe in place or wmiprvse.exe for process we are enabling “Create user mode stack trace database” against.

    11. Here we want to use procdump to dump the wmisvchost.exe process along with every instance of wmiprvse.exe that is running. Reference steps 9-11 listed in Scenario 1a above.

    First command will dump wmisvchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running.

    Command for dumping wmisvchost.exe would be: procdump –ma -s 300 -n 3 wmisvchost.exe

    Command for dumping each instance of wmiprvse just once: procdump –ma <PID>

    Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

    At this point you will now need to open a Support Incident Case with Microsoft to get the process dumps analyzed to determine cause of high memory usage

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2a: High Memory Consumption by Wmiprvse.exe caused by Private Data memory

    This scenario is a little bit different than being caused by Heap, as just dumps in of themselves normally do not tell the full story. In this type of case, we need to know what type of wmi activity is occurring and it becomes significantly harder and more involved to try and determine cause.

    We will need to collect procdumps still, but also need to add in WMI activity logging.

    1. From start run type in eventvwr.msc

    2. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it

    3. Expand Applications and Services\Microsoft\Windows\WMI-Activity

              Right click on items Debug and Trace and select “Enable” for each one

    4. Download the latest version of the Windows Sysinternals tool Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx.

    5. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

    Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high

    memory usage:

    wmic path msft_providers get Provider,HostProcessIdentifier /format:list

    6. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

    7. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

    $prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
    <ENTER>
    $prv.HostingModel = $Prv.HostingModel + ":OWN"
    <ENTER>
    set-ciminstance -inputobject $prv
    <ENTER>

    Repeat the above for each Provider found in step 5

    8. Restarting of Wmi service will put the settings into effect without having to reboot machine

    9. When memory usage is high, use Procdump tool to dump out the wmiprvse.exe exhibiting high memory usage. http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    10. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

    11. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

    12. Run the following command: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

    The above command will produce 3 dumps space out 5 minutes apart each in same directory you ran the procdump command from

    13. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865

    14. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

    At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Private Data memory

    NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 5

    1. Open command prompted with elevated privileges
    2. Break WMI service out into its own svchost process by running following command: sc config winmgmt type= own
    3. Restart WMI service with net stop winmgmt and net start winmgmt commands
    4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
    5. From start run type in eventvwr.msc
    6. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it
    7. Expand Applications and Services\Microsoft\Windows\WMI-Activity

              Right click on items Debug and Trace and select “Enable” for each one

    8. Here we want to use procdump to dump the svchost process housing the WMI Service along with every instance of wmiprvse.exe that is running. Reference steps 9-10 listed in Scenario 1a above

    a. From command prompt run tasklist /svc and locate the instance of svchost that winmgmt is running in and note the PID

    b. First command will dump svchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running just once

    Command for dumping svchost.exe housing winmgmt would be: procdump –ma -s 300 -n 3 <PID>

    Note: Replace <PID> with actual PID for svchost process housing winmgmt

    Command for dumping each instance of wmiprvse running would be: procdump –ma <PID>

    Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

     

    1. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: http://www.microsoft.com/en-us/download/details.aspx?id=4865
    2. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

    At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up: WMI: How to troubleshoot High CPU Usage by WMI Components

  • WMI: Missing or Failing WMI Providers or Invalid WMI Class

    Scenario

    Windows Management Instrumentation fails due to receiving an event or error concerning missing or failure to load WMI Provider, or Invalid WMI class, or WMI Invalid Namespace.

    Below are some common errors indicating issues with a WMI Provider or Class:

    • Failed to initialize all required WMI classes
    • Win32_processor: WMI: Invalid namespace
    • Win32_WMISetting: WMI: Invalid namespace
    • Win32_OperatingSystem: WMI: Invalid namespace
    • WBEM_E_NOT_FOUND 0x80041002
    • WBEM_E_PROVIDER_FAILURE 0x80041004
    • WBEM_E_INVALID_NAMESPACE 0x8004100E
    • WBEM_E_INVALID_CLASS 0x80041010
    • WBEM_E_PROVIDER_NOT_FOUND 0x80041011
    • WBEM_E_INVALID_PROVIDER_REGISTRATION 0x80041012
    • WBEM_E_PROVIDER_LOAD_FAILURE    0x80041013

    The common urge by many is to simply take a hammer to WMI and either rebuild the repository or recompile all of the .mof files in the C:\Windows\System32\Wbem folder. While the hammer may seem a good approach to solve your problem in some cases or what seems like a quick fix, it can actually cause more problems and issues that are not immediately visible or felt. These newly created problems as a result of the hammer approach are not always immediately obvious and tend to surface later, and usually when the last thing you need is another problem to have to deal with on your plate, especially if it created a critical business situation.

    Rebuilding the repository or recompiling all of the .mof files as a first action when other steps should be taken first can cause damage to the system and/or to installed applications.

    First we want to just check right up front if our repository is corrupted or not to save time, then continue from there with appropriate actions.

    Check the Windows Application log, look for events in the past week where Source = Microsoft-Windows-WMI, check if any of the following WMI event IDs exist: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository or core infrastructure problem.

    Check if the repository is corrupted or not by running following command from command prompt with admin rights: winmgmt /verifyrepository. If it is corrupted the returned result will be “Repository is inconsistent.” If it comes back as consistent, then repository is not corrupted so do not rebuild repository at this point.

    Note: if results came back as “inconsistent”, then skip to bottom of the page and reference the section titled “Wmi Repository Corrupted”, otherwise continue with the methods below.

    If issue is with a 3rd party provider or namespace, the vendor should be engaged as they are responsible for their own namespaces and providers.

    For the 3 scenarios listed below for WMI Invalid Namespace, WMI Invalid Class, and WMI Provider Load failure, if a class is present and operation still errors out with invalid class, then the most likely reason is that service/wmiprvse is hitting memory quota limit or issues.

    When a quota issue is hit in wmiprvse, new memory allocations fail and based on where the failure happen client gets different errors “invalid query”/Invalid class/Provider load failure/quota check”
    When a quota issue is hit when a new operation is for a provider which is not loaded already in wmiprvse and thus requiring it to be loaded in a new instance of wmiprvse, then the provider loading would fail with “provider load failure”.

    Scenario 1: WMI Invalid Namespace

    First we want to take any scripts or programs out of the equation by using local built in tools. The two most common tools used to check wmi functionality is the WMI console (winmgmt.msc) and Wbemtest (Windows Management Instrumentation Tester).

    Ensure the Namespace in question actually exist and functional.

    1. Go to start-run and type in wmimgmt.msc
    2. Right click on Local Wmi Control (Local)and select properties
    3. On the general tab, if there is any failures noted on that box, that indicates a core WMI issue and most likely with the Cimv2 namespace.
    4. Click on the Security tab and expand Root folder. This is where you will see all of the namespace listed for WMI
    5. Find the namespace referenced in the error message you are getting
    6. If you find the namespace is missing, do the following, otherwise skip to step 6 if the namespace is listed

    a. Open up a command prompt with administrative rights or elevated privileges

    b. Go to www.msdn.com and search for the provider of the missing namespace. We are looking for the associated .mof file. In the below example, we will use MSCluster as our missing namespace.

    e.g. search words: wmi mscluster provider or wmi mscluster namespace. In this case you should find ClusWmi.mof as the WMI provider

    In case the CimV2 namespace is missing, the provider is CimWin32.mof

    If it also lists the associated .dll file, you will want to re-register it also as this would be the registration for the DCOM side as WMI and DCOM work hand in hand.

    c. From the command prompt with administrative rights or elevated privileges change directory to C:\Windows\System32\Wbem and run the following command: mofcomp.exe <provider.mof>

    Note: Replace <provider.mof> with actual .mof name you found searching MSDN. In our example above that command would be: mofcomp.exe ClusWmi.mof

    For re-registering associated .dll if one exists use following command: regsvr32 <provider.dll> and again replacing <provider.dl> with actual dll name

    d. Restart the Windows Management Instrumentation Service

    e. Use wmimgmt.msc console again to now check if the missing namespace is now listed. First close the console and reopen if it was still open from previous action

    7. Go to start-run and type in wbemtest

    8. Click on the “Connect Button

    9. In the Namespace Box type in the path to the namespace for which getting invalid namespace error for. This path would have the same look and feel of a Windows Directory, so just as you see the structure in wmimgmt.msc console on the Security tab, so is how you will type in path

    Examples:

    Root\Cimv2
    Root\Mscluster
    Root\RSOP\Computer

    10. Click on the “Connect” button

    11. Now all of the buttons should no longer be greyed out on the main wbemtest console page. Click on the “Enum Classes” button

    12. Leave “Enter Superclass Name” blank and select “Recursive” then click OK. If you don’t get any error messages then you can access the name successfully without issue using built in Windows Management Instrumentation Tester

    13. To test further, let’s see if we can access some classes.

    a. Pick any class and double click on it to bring up the Object Editor for that class

    b. Next click on the “Instances” button on the right

    c. If it doesn’t sit there hung at “Operation in progress” or doesn’t return any error, then access to that class would appear to be okay.

    NOTE: Not every class you click in the “Instances” button will actually return results. Some classes will and some won’t depending on the class. Don’t sweat it if it doesn’t as long as the box above says “Done”

    Repeat this on several classes in the namespace for sanity check just to see if any produce an error message or Operation in progress hangs.

    If you did not get any errors connecting to the namespace or accessing some of the classes in that namespace, then your issue may be with the application or whatever method being used outside of Wbemtest that is the problem and would advise opening case with Microsoft if it is a Microsoft application or process, otherwise first contact the vendor of the application for assistance in troubleshooting.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Scenario 2: WMI Invalid Class

    To troubleshoot an Invalid WMI Class, you basically would follow the same procedure as above that you would for WMI Invalid Namespace.

    Here is a good link that you could probably also click your way thru various links to find the provider .mof for Microsoft classes.

    WMI Classes

    Scenario 3: WMI Provider Load Failure

    Again, you will follow the same procedures as above for testing, and also locating the provider .mof file and associated .dll if one is listed and recompiling and re-registering of that particular provider.

    Wmi Repository Corrupted

    There are some exceptions where you have accomplished all of the above and from Wbemtest (note I said Wbemtest, not some method outside of Wbemtest) and you are still getting failures with a namespace, class, or provider. In these scenarios you may have to revert to using the proverbial hammer as a last ditch effort even though the repository check came back as consistent when your ran winmgmt /verifyrepository.

    Before doing such, it would be a good idea to open a Support Incident Case with Microsoft for assistance before reverting to rebuilding of the repository if the winmgmt /verifyrepository had come back as Consistent.

    IMPORTANT

    If this becomes the case, please refer to my Ask the Performance Team Blog article, WMI: Rebuilding the WMI Repository before rebuilding the repository as there are some important gotchas that you need to be aware of.

    As a final note, if you are running into a reoccurring WMI corruption issue in your environment, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption in WMI.

    At this point you may also want to open a Support Incident with Microsoft for further assistance if needed.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

    Next up: WMI: High Memory Usage by WMI Service or Wmiprvse.exe

  • WMI: Repository Corruption, or Not?

    Scenario

    Windows Management Instrumentation failing due to repository being corrupted

    The WMI Repository (%windir%System32\Wbem\Repository) is the database that stores meta-information and definitions for WMI classes; in some cases the repository also stores static class data as well. If Repository becomes corrupted, then the WMI service will not be able to function correctly.

    Before grabbing that preverbal hammer approach and just rebuilding your repository, ask yourself, “Is the WMI repository OK?”

    Common symptoms that lead to this question are: provider load failure, access denied, class not found, invalid namespace, and namespace not found to mention a few.

    If you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows or to installed applications. Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption. Noting here that having a repository too large also creates problems; an issue that can sometimes be interpreted as a corrupt repository, which is not always the case. If issues are due to a large repository, rebuilding the repository is currently the only method available to reduce it back to a working size.

    Since I mentioned “large repository”, let me set some guidelines up front. There is no hard fast number per say as to when you will start feeling performance problems with a large repository. As a guideline, if the objects.data file, located in (%windir%System32\Wbem\Repository, is 1 GB or larger, then I would recommend rebuilding your repository to reduce it back down to a working and manageable size. If the size is between 600-900 MB, and you are not feeling any noticeable performance issues, then I would recommend against rebuilding the repository.

    If WMI is corrupted, you can receive various errors and symptoms, depending on what activity was being done at the time. Below is a few errors and symptoms that could indicate that the repository is corrupted:

    1. Unable to connect to root\default or root\cimv2 namespaces. Fails returning error code 0x80041002 pointing to WBEM_E_NOT_FOUND.
    2. When we open Computer Management and Right Click on Computer Management (Local) and select Properties, you get the following error: "WMI: Not Found" or it hangs trying connect
    3. 0x80041010 WBEM_E_INVALID_CLASS
    4. Trying to use wbemtest, it hangs
    5. Schemas/Objects missing
    6. Strange connection/operation errors (0x8007054e):

    get-cimclass : Unable to complete the requested operation because of either a catastrophic media failure or a data structure corruption on the disk.

    At line:1 char:1

    + get-cimclass -Namespace root\cimv2\TerminalServices

    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo : NotSpecified: (:) [Get-CimClass], CimException

    + FullyQualifiedErrorId : HRESULT 0x8007054e,Microsoft.Management.Infrastructure.CimCmdlets.GetCimClassCommand

    Check the Windows Application log for events in the past week where Source = Microsoft-Windows-WMI.  Look for any of the following WMI event IDs: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository issue or core infrastructure problem.

    If you do not find any of these events logged, your next action is to use the built in repository checker. From an elevated command prompt run "winmgmt /verifyrepository". If the repository has an issue, it will respond "repository is not consistent".

    If repository check comes back as “consistent”, then look at my other Ask Perf blogs for applicability:

    WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

    WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

    How to troubleshoot High CPU Usage by WMI Components (COMING SOON)

    WMI Self-Recover

    When the WMI service restarts or detects Repository corruption, the self-recovery procedure will trigger automatically in two approaches (one or the other):

    1. AutoRestore: if the VSS backup mechanism enable snapshot the timestamp backup images In the system (ex. Win7 feature: previous fileversion), WMI will apply the AutoRestore approach to restore backup valid images in version queue. (if possible)
      • EVT: 65 (start restore) / 66 (succeed recovered with VSS Path)
    1. AutoRecovery: the rebuild process to generate fresh images of Repository based on registered mofs( listed @ HKLM\Software\Microsoft\WBEM\CIMOM: AutoRecover Mofs).
      • EVT: 5616 (complete recovery), eventually, there are lots of EVT:63 for mof warning about Localsystem registration of providers.

    Note: Under almost no circumstance should you use the script that rebuilds the WMI repository from the MOF files

    The script is inherently flawed, for 2 reasons:

      1. If you navigate to the %systemroot%\system32\wbem folder, and list the MOF files, you see find MOFs named (some provider name)_uninstall.mof. When you mofcomp those, they remove the classes in the MOF. The script mofcomps everything, so it can very easily install then immediately uninstall the classes, resulting in the class not being accessible.
      2. Replaying mofs is often sequence dependent. Example: classes in mof1 can depend on or have associations with classes in mof2. If they aren't present, MOFCOMP will not insert the classes. It's extremely difficult to know what is / is not the right sequence, so any script that simply MOFCOMPs everything is not going to be fully successful.

    In addition to causing damage to your system that's almost impossible to fix correctly, if you take that approach you will blow away all information that could be used to determine the root-cause.

    If the repository check (winmgmt /verifyrepository) comes back as inconsistent, your first action is to run “winmmgmt /salvagerepository” followed by running “winmgmt /verifyrepository” again to see if it now comes back as consistent.

    If it is still comes back inconsistent, then you need to run “winmgmt /resetrepository.” Before running this, please read the important note below for Server 2012.

    Force-recovery process -- rebuild based on the registry list of AutoRecover Mofs

     

    1. Check regkey value is empty or not @ HKLM\Software\Microsoft\WBEM\CIMOM: 'Autorecover Mofs' (** first line on some OSs is empty, review it in opening the regkey value)
    2. If above regkey is empty, copy/paste the regkey value from another machine of equal System/OS to the suspect machine
    3. Run the following command from command prompt with admin rights: “Winmgmt /resetrepositoy”
    4. If you get the error noted below, stop all dependency services on the WMI service by running following command: 

    net stop winmgmt /y

    then run

    Winmgmt /resetrepositoy

    WMI repository reset failed
    Error code:     0x8007041B
    Facility:       Win32
    Description:    A stop control has been sent to a service that other running services are dependent on

    NOTE: Applies to Server 2012

    We have encountered some issues when running the mofcomp command on Windows Server 2012 which has caused the Cluster namespace to be removed due to the cluswmiuninstall.mof contained in the c:\windows\system32\wbem folder. It has also caused unregistering class names for Cluster Aware Updating (CAU) because of its cauwmiv2.mof also in the wbem folder. Those could also affect other namespace that have an uninstall type .mof in the wbem folder beyond the two mentioned above.

    Furthermore, the uninstall .mof files for servers running Microsoft Cluster are also part of the autorecover folder that is used when you run the winmgmt /resetrepository command which will end up having the same effect of first installing the Cluster namespace, then uninstalling it just as if you had run a script to rebuild the repository that contained the “for” command to recompile all of the MOFs in the WBEM folder.

    Take the following actions to confirm if the uninstall problem for this scenario exists on your server. If it doesn’t, then you can run the winmgmt /reset repository, otherwise follow my directions below for manually accomplishing rebuild.

    1. Open regedit, and navigate to hklm\software\microsoft\wbem\cimom, and open “Autorecover MOFS
    2. Copy the data from that string value, and paste it into notepad
    3. Do a search for ClusWmiUninstall.mof. If the cluster provider uninstall has autorecover, it will be listed here
    4. If found, then continue to manual rebuild below, if not found then go ahead and use the winmgmt /resetrepository command

    How to manually rebuild repository on Server 2012 Cluster machine when cluster provider uninstall has an autorecover

    First ensure you have run winmgmt /verifyrepository to ensure that it is “inconsistent” and that you have tried winmgmt /salvagerepository to see if it resolves your issue.

    Change the startup type for the Window Management Instrumentation (WMI) Service to disabled.

     

    1. Stop the WMI Service, you may need to stop services dependent on the WMI service first before it allow you to successfully stop WMI Service
    2. Rename the repository folder:  C:\WINDOWS\system32\wbem\Repository to Repository.old
    3. Open a CMD Prompt with elevated privileges
    4. Cd windows\system32\wbem
    5. Run following command to re-register all the dlls: for /f %s in ('dir /b /s *.dll') do regsvr32 /s %s
    6. Set the WMI Service type back to Automatic and restart WMI Service
    7. cd /d c:\  ((go to the root of the c drive, this is important))
    8. Run following command specifically adopted for 2012 Clustered servers to recompile the MOFs: “dir /b *.mof *.mfl | findstr /v /i uninstall > moflist.txt & for /F %s in (moflist.txt) do mofcomp %s”
    9. Restart WMI service

    As a final note, if you run into a reoccurring corruption issue in your environment with WMI, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption and other issues in WMI.

    Other repository recovery solutions:

    Note: in following solutions (1 & 2), if the backup images (repository) are in large size (>100MB), restoring the repository will take some time.

    1. Apply WMI AutoRestore story in the system to recover the repository image quickly and keep it in sync with previous state.
    2. Enable VSS backup-related features for storing image snapshots
      • ex.  Volume Shadow Copy (VSS), or check any valid copies listed Local Disk(C:) Properties >> Shadow copies
    3. Make sure registry key has following setting: HKLM\Software\Microsoft\WBEM\CIMOM: AutoRestoreEnabled=1
    4. Frequently snapshot the restore-points of the system (if needed, refer to the following PowerShell scripts)

    $filterStmt = [String]::Format("DriveLetter='{0}'",$env:SystemDrive)

    # Get systemdrive Volume info

    $vlm = gcim win32_Volume -Filter $filterStmt

    # create a shadowcopy

    $res = icim win32_ShadowCopy -MethodName Create -Arguments @{Volume=$Vlm.DeviceID}

    if ($res.ReturnValue -eq 0)

    { gcim win32_ShadowCopy -Filter ("ID='"+$res.ShadowID+"'") } # **

    else

    { $res | fc }

      • AutoRestore only searches the top 3 queued snapshots for the latest valid backup, if no valid one is found, AutoRecovery will apply.
      • To restore others in the queue of snapshots ( manually )
        1. In Server sku, looking at 'previous version' tab of the repository folder to find the expected backup path 
        2. stop WMI service: Net stop winmgmt /y
        3. Replace all of the files in %windir%\system32\wbem\Repository folder with the files from the backup path found in step 1

    Note The WMI Service has auto start setting, and if it comes back alive, you will not be able to replace the files.  The service needs to be in a stopped state (if WMI service is alive at the time, repeat step: 2~3)

    ex. Directory of \\localhost\C$\@GMT-2014.03.13-01.02.49Windows\system32\wbem\repository

    03/11/2014  11:53 AM    <DIR>          .
    03/11/2014  11:53 AM    <DIR>          ..
    03/12/2014  05:30 PM         4,759,552 INDEX.BTR
    03/12/2014  05:30 PM            90,640 MAPPING1.MAP
    03/12/2014  03:26 PM            90,640 MAPPING2.MAP
    03/12/2014  05:24 PM            90,640 MAPPING3.MAP
    03/12/2014  05:30 PM        27,541,504 OBJECTS.DATA

    4. Run following wmic command to bring WMI service back to life: wmic os

    You should set up a regular Scheduled Task to backup the latest repository:

        • winmgmt /backup <backup image path>
        • tracing EVT: 67,68

    You could also schedule restores as necessary

    • winmgmt /restore <above backup image path> 1
    • tracing EVT: 65,66

    If the issue is not a repository issue, and the objects are not retrievable:

    • Re-install the product. This is the first place to start.
    • If there is a specific provider that is not showing up, you can re-run mofcomp of a single provider. See Ask the Performance Team Blog article WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

    If the issue persists or keeps returning, then at this point you will now need to open a Support Incident Case with Microsoft for further assistance.

    Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and well help us track the effectiveness of the blog.

    Next up: WMI: Missing or Failing WMI Providers or Invalid WMI Class

  • WMI: Common Symptoms and Errors

    Quota Violation Issues: Memory and/or Handle

    Symptoms

    • General WMI-based scripts or applications fail or fail intermittently
    • Applications such as SMS/SCCM produce errors on server and/or inventories fail or fail intermittently
    • Applications such as Exchange or SQL fail on server or fail intermittently
    • Unable to connect to specific namespace via WBEMTEST or unable to query specific classes in a namespace. May be intermittent.
    • WMI appears to be hung or non-responsive
    • Unable to run msinfo32 or tasklist
    • Events for unexpected termination/crash of wmiprvse.exe
    • Lower than normal available memory on the system
    • Out of memory errors when running certain WMI tasks
    • 0x80041033 -- SHUTDOWN of the target provider
    • 0x80041006 – OUTOFMEMORY
    • 0x80041013 -- fail to load the provider
    • 0x800705af -- The paging file is too small for this operation to complete
    • 0x8004106C-- Quota violation
    • Event logs contain WMI-related events
    • WMI service crashing

    Event Source: Service Control Manager
    Event ID: 7031
    Description: The Windows Management Instrumentation service terminated unexpectedly

    • Handle Quota Violation

    Source: Microsoft-Windows-WMI
    Event 5612 Wmiprvse.exe exceeding handle quota limit Event
    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    • Memory Quota Violation does not log and event such as Handle Quota Violation does. You can check to see if any instance of wmiprvse is using 500 mb of memory or greater as the default limit is 512 mb on Vista an newer. Once you hit around 500 mb and any new allocation that would cross the 512 mb limit would fail. All instances of wmiprvse combined have a 1 gb limit. For Windows XP and 2003 the default is 128 mb. Exceed the default and you have reached a Quota Violation. Also check for Windows Management Instrumentation service crashing

    Memory Quota Violations refer to Ask Perf blog: WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

    Handle Quota Violations refer to Ask Perf blog: How to Troubleshoot WMI High Handle Count (COMING SOON)

    Missing or Failing WMI Providers or Invalid WMI Class

    Symptoms

    • General WMI-based scripts or applications fail
    • Applications such as SMS/SCCM produce errors on server and/or inventories fail
    • Applications such as Exchange or SQL fail on server
    • Unable to connect to specific namespace via WBEMTEST or unable to query specific classes in a namespace
    • WMI functionality appears normal on local node but unable to connect to/from machine via WMI scripts, tools or applications
    • WMI (Local) properties in wmimgmt.msc console and see the following

    Failed to initialize all required WMI classes
    Win32_processor: WMI: Invalid namespace
    Win32_WMISetting: WMI: Invalid namespace
    Security information: Successful
    Win32_OperatingSystem: WMI: Invalid namespace
    Win32_processor: WMI: Invalid namespace
    Win32_WMISetting: WMI: Invalid namespace
    Win32_OperatingSystem: WMI: Invalid namespace

    • WBEM_E_NOT_FOUND 0x80041002
    • WBEM_E_PROVIDER_FAILURE 0x80041004
    • WBEM_E_INVALID_NAMESPACE 0x8004100E
    • WBEM_E_INVALID_CLASS 0x80041010
    • WBEM_E_PROVIDER_NOT_FOUND 0x80041011
    • WBEM_E_INVALID_PROVIDER_REGISTRATION 0x80041012
    • WBEM_E_PROVIDER_LOAD_FAILURE 0x80041013

    Refer to Ask Perf blog: WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

    High Memory Usage by WMI Service or Wmiprvse.exe

    Symptoms

    • Wmiprvse.exe memory quota violations – refer to top of blog
    • Lower than normal available memory on the system
    • Delayed or slow logons to the box
    • Excessive or slow return times to queries to WMI or scripts that are running that call to WMI
    • Spinning donut when trying to bring up WMI (Local) properties in wmimgmt.msc console or using Wbemtest (Windows Management Instrumentation Tester) built in tool
    • Sluggish or slow responding system
    • Server hang
    • Unable to run msinfo32 or tasklist
    • Svchost process housing WMI service (winmgmt) exhibiting high memory usage or leak
    • Instance of wmiprvse reaching or exceeding 512 mb on Vista and newer, or 128 mb on XP or Windows 2003: Quota Violation issue
    • Large repository C:\Windows\System32\Wbem\Repository folder and objects.data file is 1gb or larger
    • Cluster management tools not working

    Refer to Ask Perf blog: WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

    High CPU Usage by WMI Components

    Symptoms

    • High cpu usage by svchost hosting WMI service (winmgmt)
    • High cpu usage by wmiprvse
    • System sluggish or slow performance

    Refer to Ask Perf blog: How to troubleshoot High CPU Usage by WMI Components (COMING SOON)

    High Handle Count on WMI Components

    Symptoms

    • Refer to top of blog: Quota Violation Issues: Memory and/or Handle 
    • Following Event being logged

    Source: Microsoft-Windows-WMI
    Event 5612 Wmiprvse.exe exceeding handle quota limit Event
    WMI has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: %1 Value: %2 Maximum value: %3 WMIPRVSE PID: %4

    • Resource depletion type messages
    • Error message: No more treads can be created in the system
    • Message when trying to RDP

    The User Profile Service service failed the logon

     

    • High handle count on svchost process hosting WMI service (winmgmt)
    • Cluster management tools not working

    Refer to Ask Perf blog: How to Troubleshoot WMI High Handle Count (COMING SOON)

    Repository Corruption

    Symptoms

    • General WMI-based scripts or applications fail
    • Applications such as SMS/SCCM produce errors on server and/or inventories fail
    • Applications such as Exchange and SQL fail on server
    • WINMGMT.MSC Security tab shows missing, repeating and/or gibberish namespace entries
    • Unable to connect to specific or possibly any namespace via WBEMTEST
    • Winmgmt.msc console WMI (local) properties Security tab shows missing, repeating and/or gibberish namespace entries
    • Unable to connect to root\default or root\cimv2 namespaces. Fails returning error code 0x80041002 pointing to WBEM_E_NOT_FOUND
    • Opening Computer Management and Right Click on Computer Management (Local) and select Properties, you get the following error: "WMI: Not Found" or it hangs trying connect
    • Wbemtest (Windows Management Instrumentation Tester) built in tool hangs
    • Error 0x80041010 WBEM_E_INVALID_CLASS
    • Any events with Source = Microsoft-Windows-WMI, look for any of the following WMI event IDs: 28, 65, 5600, 5601, 5614 as any of these can be indicators or repository corruption
    • WMI namespaces end up missing

    Refer to Ask Perf blog: WMI: Repository Corruption, or Not? (COMING SOON)

    Next up: WMI: Repository Corruption, or Not?

  • Troubleshooting WMI Series coming

    Hello AskPerf! My name is Jeffrey Worline and I wanted to let you know that I will be writing several troubleshooting WMI blog posts over the next few days. These will not be your typical Blog post, but more around the KB/Technet format.

    There has been a lot of internal discussions here at Microsoft with Product Group, Developers, and myself concerning the high amount of WMI issues that have come into Support. Whenever there is an issue with WMI itself, or suspected issue with WMI, the first words that always seem to come out is that “WMI is corrupted”. Oh no, say it isn’t so. What do we do? Grab the “Big Hammer” and rebuild the repository! Excuse my humor as we have a little fun, as the upcoming blogs are going to be more straight forward and dry.

    While rebuilding the repository may seem like a good idea and a quick fix at times, what most do not realize is that if you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows and/or to your installed applications. That last sentence is worth repeating; “Deleting and rebuilding the repository can cause damage to Windows and/or to your installed applications.” When it becomes necessary to rebuild the repository and this has been verified, there are right and wrong ways to go about it.

    Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption before taking such actions as rebuilding the repository. There are also other proactive steps that can be taken as alternatives to rebuilding the repository if such an occasion arises.

    So, the following blog posts will address various scenarios surrounding WMI, how to troubleshoot them, and the data we should collect to help resolve WMI related issues. These posts will equip you with the information you need to address WMI issues in the correct manner, and to help prevent you from causing more damage by just using the proverbial “Big Hammer” approach.

    With that, here is a list of the Posts/Topics you will see in the coming days: 

    • WMI: Common Symptoms and Errors
    • WMI: Repository Corruption, or Not?
    • WMI: Missing or Failing WMI Providers or Invalid WMI Class
    • WMI: High Memory Usage by WMI Service or Wmiprvse.exe
    • WMI: How to troubleshoot High CPU Usage by WMI Components
    • WMI: How to Troubleshoot WMI High Handle Count

    NOTE the above bullet points will become active URL links once each post becomes available.

    -Jeff

  • It turns out that weird things can happen when you mix Windows Server 2003 and Windows Server 2012 R2 domain controllers

    We have been getting quite a few calls lately where Kerberos authentication fails intermittently and users are unable to log on. By itself, that’s a type of call that we’re used to and we help our customers with all the time. Most experienced AD admins ...read more
  • Windows Performance Monitor Overview

    Hello AskPerf!  My name is Matt Graham and I will be writing a high level overview of the capabilities of Windows Performance Monitor.  The intention of this blog post is to introduce new users to this powerful, and often underutilized, tool.  So rather than going through each part of Performance Monitor and explaining it in depth, my aim here is to offer a quick guide to the tool.

    When you first open Performance Monitor (perfmon), you see the following:

    clip_image001

    Let's briefly go through each one and talk about what they do. 

    Performance

    At the very top level "Performance" gives you an overview of your systems memory usage, network usage, disk usage, etc.  You can right click on "Performance" and connect to another computer to view a remote computers performance statistics. (NOTE: Should add brief comments about what is required in order to remotely connect to another machine…)

    Monitoring Tools

    From the Monitoring Tools icon you can right click and launch the Resource Monitor.  Resource Monitor is another powerful tool that can help you see how your system resources are being used.  You also have the ability to launch the System Reliability Monitor.  This utility allows you to see information about software updates and installations.  You can also see critical events that occurred and on what day those events occurred.  Finally, you can see all of the problem reports that have been sent from your computer by clicking on the "View all problem reports" link at the bottom of the window.

    clip_image002

    Performance Monitor

    The Performance Monitor is primarily for viewing real time statistics.  By default only one counter is selected; the %Processor Time counter.  However you can add additional counters by clicking on the green plus sign.  This will allow you to monitor any counters you wish in real time.

    clip_image003

    While you can see all of the performance counters you like here, the real power of Performance Monitor is found in its ability to capture performance metrics over an elapsed period of time.  Capturing data over a period of time allows you to see trends and these trends are what are most useful for determining the overall performance of your system.  To capture this data, you can create what are called "Data Collector Sets".

    Data Collector Sets

    Data Collector Sets are aptly named.  They collect data from your system so that you can view changes in configuration information and performance information over a specified period of time.

    There are basically three types of data collector sets:

    Performance Counter

    Capture data based on the polling of an event at a specified time interval

    Event Traces

    Capture data based on events that occur rather than based on a specified time interval

    System Configuration Information

    Capture configuration information

    Under Data Collector Sets you can see the following:

    clip_image004

    User Defined

    Under User Defined, you can create your own custom Data Collector Set.  These Data Collections Sets can contain counters, traces, and configuration collectors.  You can right click on User Defined and select New -> New Data Collector Set to create one.  You can create a data collector set from a template or create your own custom set.  Let's create a custom one:

    1. Name your Data Collector Set and select "Create manually (Advanced)".

    2. On this screen you can choose to create data logs (counter / trace / config) or you can create a Performance Counter Alert.  The "Performance Counter Alert" option allows you to create alerts based off of certain performance values and thresholds.  For now, we will select "Create Data Logs" and place a check box in all three boxes:

    clip_image005

    3. On the screen below you can set the counter interval rate (how often do you want it to capture the selected data) and the specific counters that you want to capture.

    clip_image006

    4. Once you click Add, you can select counters and then add them to the "Added Counters" box.  Note that you have options in terms of whether you want Perfmon to collect the data as a total or if you want to break the data up, in this case, per processor.  You should pay attention to which one you select as the meaning of these counters depends on what is being counted.

    clip_image007

    5. You will then be prompted to add trace providers.  Trace providers simply provide information to perfmon about a specific set of events.  For example, if you wanted to collect event information about the Windows Firewall, you would select the "Microsoft-Windows-Firewall" provider.  You can then edit the properties (if you know what you are doing) and even record registry keys (after you hit next you can specify which keys to record).

    clip_image008

    6. You can also specify a location where you would like to save the data.  By default, the data is saved to: %systemdrive%\PerfLogs\Admin\New Data Collector Set.

    7. Finally, you can select a user that you would like to run the data collector set as.  This may be helpful in environments where desktops and servers are locked down for security purposes.

    If you look at the "New Data Collector Set" that we just created, you can see that it contains a performance counter, a trace, and a configuration collector.  You can right click on any of these to modify them as you see fit.

    clip_image009

    Finally, if you look at the remaining items under Data Collector Sets, you can see a bunch of preconfigured collector sets.

    clip_image010

    Reports

    The final part of Performance Monitor is the Reports section.  Here you can view the information that was collected by your data collector sets.  If you have never run your data collector set, then you will not see any information when you click on it.

    clip_image011

    However, once you have run your data collector set, you can click on it and see the reports and information collected:

    clip_image012

     

    clip_image013

    So this is a basic overview of Windows Performance Monitor.  Once you are familiar with the parts, you can then dive into learning about which counters to use when, and what your counter interval rate should be when you are trying to capture this kind of data vs that kind of data.

    Additional Resources

  • Deploy Windows to Surface Pro 3 using Microsoft Deployment Toolkit

    Hi, my name is Scott McArthur and I am Senior Support Escalation Engineer on the Deployment/Devices team. In today’s blog I am going to go over the steps to deploy Windows 8.1 Enterprise X64 Update to a Surface Pro 3. In this example I will be using the following deployment technologies

    • Microsoft Deployment Toolkit 2013 Server
    • Windows Server 2012 R2 WDS server

    I will be using the Microsoft USB to Ethernet adapter to PXE boot the MDT 2013 Lite Touch Images from the WDS server. If you don’t have the adapter you could utilize a USB hard drive and Media Deployment from MDT (not covered in this blog). There are various ways to deploy Windows to a device so this is just one example.

    Before starting you need to gather up the following:

    • Note: We are going to make the download of this update easier but in the meantime you can grab this update from this link.
    • Optional: Existing Surface Pro 3 with OEM image installed. Used to gather files for Pen Pairing during OOBE

    Note: In this blog I am using the Surface Pro 3 as the hardware to build the reference image on. In environment where you are building an image that will only go on a Surface Pro 3 this is generally not a problem but if you create reference image that is going to many different types of systems we recommend for you to build your reference image in a Generation 1 Hyper-V virtual machine so that the reference image is “clean” of any drivers and then you use the features of MDT or SCCM to layer the device specific drivers down during deployment. Since there are so many factors involved I opted to show the simpler of scenarios and then you can decide what fits best for your environment and goals.

    Step #1: Extract the contents of the Surface Pro 3 Firmware and Driver pack

    After downloading the Surface Pro 3 firmware and driver pack you will see the following files:

    • Surface Ethernet Adapter.zip
    • Surface Gigabit Ethernet Adapter.zip
    • Surface Pro – July 2104.zip
    • Surface Pro 2 – July 2014.zip
    • Surface Pro 3 – July 2014.zip

    Note: This package is updated on regular basis so the filenames be slightly different but overall package organization should be similar.

    Extract the contents of the following files:

    • Surface Pro 3 – July 2014.zip
    • Surface Ethernet Adapter.zip
    • Surface Gigabit Ethernet Adapter.zip

    For the next steps we will assume they were extracted to the following locations

    • C:\Surface_Pro3_July_2014
    • C:\Surface_Ethernet_Adapter
    • C:\Surface_Gigabit_Ethernet_Adapter
    • C:\KB2968599

    Step #2: Import OS

    In this step we will import the OS. Surface Pro 3 only supports Windows 8.1 X64 Update. This can be Enterprise or Professional.

    • Right click Operating Systems and choose import
    • Browse to your location of your VL Windows 8.1 Enterprise Update X64 ISO
    • Provide directory name such as “Windows 8.1 Enterprise Update X64”
    • Click next and Finish

    Step #3: Add the Surface Pro 3 Firmware and Driver pack drivers to MDT

    In the Microsoft Deployment Toolkit Workbench create the following folder structure under Out-Of-Box Drivers

    image

    Note: The last folder must be called “Surface Pro 3”

    • Right click Out-Of-Box Drivers\WindowsPEX64 folder and choose import drivers. Browse to C:\Surface_Ethernet_Adapter and import the driver
    • Right click Out-Of-Box Drivers\WindowsPEX64 folder and choose import drivers. Browse to C:\Surface_Gigabit_Ethernet_Adapter and import the driver
    • Right click Out-Of-Box Drivers\X64\Surface Pro 3 and choose import drivers. Browse to C:\Surface_Pro3_July_2014

    Step #4: Create Selection Profile for Windows PE drivers

    This still will create a selection profile for Windows PE drivers. This helps to ensure only the necessary drivers are imported into Lite Touch boot image.

    • In the Microsoft Deployment Toolkit workbench navigate to Advanced Configuration\Selection Profiles.
    • Right click and choose new selection profile
    • Name the selection Profile WindowsPEx64
    • Browse to Out-Of-Box Drivers\WindowsPEX64
    • Select WindowsPEX64 folder

    image

    • Next
    • Finish

    Step #5: Assign Selection Profile for Windows PE

    This step will assign the previously created selection profile to Windows PE Lite touch so that only the drivers under WindowsPEx64 are added to the boot image

    • Right click the Deployment share and choose properties
    • Choose Windows PE tab
    • Choose Platform X64
    • Choose Drivers and Patches tab
    • For selection profile choose WindowsPEx64

    image

    Step #6: Import Updates

    In this step we will import the update that enables the Pen button functionality with modern OneNote. In most cases you would probably add other security updates and other updates to your deployment at this point also.

    In the Microsoft Deployment Workbench right click packages and choose import and then browse to C:\KB2968599\Windows8.1-KB2968599-x64.msu

    image

    Step #7: Create Task Sequence

    In this step we will create a task sequence to deploy Windows 8.1 Enterprise Update X64

    • In the Microsoft Deployment Workbench right click Task Sequences and choose new
    • Task Sequence ID=BLDWin81ENTUPX64
    • Task Sequence Name=Build Windows 8.1 Enterprise Update X64 reference image
    • Choose Standard Client Task Sequence
    • Choose the Windows 8.1 Enterprise Update X64 reference image OS
    • Choose Do not Specify a product key at this time
    • Fill out Organization and other information
    • Fill out local administrator password
    • Finish

    Step #8: Edit Task Sequence for Drivers

    In this step we will edit the task sequence to modify the driver injection step. There are a number of ways to address drivers in MDT. The key to preventing driver installation issues to make sure that the only drivers used during the deployment are the ones designed for the Surface. If your Out-Of-Box drivers contain drivers for other systems and you do not use one of the options below then you cannot control what drivers get used during the deployment. This can lead to problems so we would recommend you use Selection Profiles or other methods to ensure only the drivers designed for the Surface are used during the deployment. For additional reading on this topic I encourage you to take a look at this blog

    Option #1:Create a selection profile for Out-Of-Box Drivers\Windows81Update\X64\Surface Pro 3 and then set the Inject Drivers TS step to this selection profile. It is recommended to choose the “Install all drivers from this selection profile option also. Disadvantage to this option is that this TS would be specific to Surface Pro 3. If you configure this option it will look like this in the task sequence

    image

    Option #2 (Recommended approach):Use the DriverGroup001 variable to set this based on the Model of the system. This is more flexible since it will take the Model (WMI variable from the BIOS) information and use this to decide which folder to use. This allows this task sequence to work for a variety of devices. The folder names have to match EXACTLY with the Model exposed by the system (MSINFO32 will show you the model)

    We will use Option #2 for these steps

    In Microsoft Deployment Toolkit workbench right click the task sequence you created earlier and choose properties

    • Choose the Task Sequence tab
    • Browse to the Preinstall phase and look for step called “Inject Drivers”
    • Click the Enable Bitlocker step which is right before the “Inject Drivers” step
    • Click Add, General, Set Task Sequence Variable
    • Set the following:

    Name: Set DriverGroup001 variable to Model
    Task Sequence Variable: DriverGroup001
    Value: Windows81Update\x64\%model%

     

    image

    • Choose the Inject Drivers step that occurs after this step and set the Selection profile to Nothing and choose “install all drivers from the selection Profile”. This is important so all the firmware updates and drivers for devices that are not present(for example keyboard) are added to the deployment

    image

    • Click apply and save the task sequence

    Step #9: Modify the Unattend.xml

    In this step we will modify the Unattend.xml to make sure OOBE is completely automated. There is additional prompt during OOBE to join wireless network if the wireless driver is available. The TS Unattend.xml does not contain the entry to automate this since this is a new setting with Windows and the template in MDT 2013 doesn’t contain it

    • In Microsoft Deployment Toolkit workbench right click the TS and choose properties
    • Choose OS info tab
    • Choose Edit Unattend.xml

    Note: This will take a while the first time a catalog is created. If you encounter error take a look at KB2524737.

    • Navigate to 7 OOBESyetm\Microsoft-Windows-Shell-Setup\OOBE
    • For HideWirelessSetupInOOBE choose True

    image

    Another option to consider modifying at this point is configuring whether or not the Power button shows on the start screen. The OEM image that ships with Surface Pro 3 is configured to show the Power button on the start screen. If you do a new install the default behavior is not to show the power button (by design). For additional information on this behavior and Unattend option to configure this see the following:

    KB2959188: Power/shutdown button may be missing from the Windows 8.1 start screen

    image

    Step #10: Configure Image for Pen Pairing during OOBE (Optional)

    During the 1st bootup of the OEM image that ships with the Surface Pro 3 you are prompted during OOBE to pair the pen. In most cases you will probably want to pairthe pen after the deployment is complete but if you would like to add this step to the deployment you can use the following instructions.

    Note: The pairing prompt will occur during OOBE so it will interrupt MDT’s automated deployment. Once paired you must click next for it to continue. Ideally this is something IT person would handle for the user before handing over the device to the user.

    1. Take one of your existing Surface Pro 3 devices that has the OEM image on it and copy the following files to USB flash drive or other location:

    %windir%\system32\oobe\info\default\1033\oobe.xml
    %windir%\system32\oobe\info\default\1033\PenPairing_en-US.png
    %windir%\system32\oobe\info\default\1033\PenError_en-US.png
    %windir%\system32\oobe\info\default\1033\PenSuccess_en-US.png

    2. On the MDT server open Deployment and Imaging Tools Environment cmd prompt

    3. Use the DISM command to mount the image you are deploying

    Dism /mount-wim /wimfile:d:\deploymentshare\operating systems\<name of image>\sources\install.wim /index:1 /mountdir:c:\mount

    4. Create the following pathing in the image

    C:\mount\windows\system32\oobe\info\default\1033

    5. Copy all the files from Step #1 above into this folder

    6. Close any explorer Windows and switch to C:\ to make sure no open file handles to the c:\mount folder

    7. Unmount the image and save changes

    Dism /unmount-wim /mountdir:c:\mount /commit

    Step #11: Configure Default Display Resolution

    The default display resolution for the Surface Pro 3 is 2160x1440. To set this automatically you can add the following entry to your customsettings.ini (Right click the Deployment share, properties, rules):

    [Settings]
    Priority=Model, Default

    [Surface Pro 3]
    XResolution=2160
    YResolution=1440

    This uses the MDT functionality of where it knows the Model (Surface Pro 3) and based on these entries adds the resolution settings to the Unattend.xml for you

    Step #12: Update MDT server and WDS server

    At this point you would want to do a full generation of the deployment share to create the Lite Touch boot images to ensure the Surface Ethernet Adapter driver is incorporated into the MDT Lite Touch boot images and then import these images to your Windows Deployment Service (WDS) server. I would recommend you utilize a 2012R2 WDS server. For additional information on support for UEFI in WDS take a look at KB2938884.

    Step #13: PXE boot

    The final step is to PXE boot the Surface Pro 3. To PXE boot do the following:

    • Shut the device down
    • Press and hold volume down button
    • Press the Power button
    • When you see the Surface Logo you can let go
    • You should see prompt to PXE boot. The Surface Pro 3 supports a On Screen Keyboard(OSK)
    • Press the Keyboard icon in upper right of screen
    • Press Enter button on OSK
    • Using arrow keys on OSK choose your MDT 2013 Lite Touch image from the WDS server
    • Then follow the prompts during Lite Touch to initiate the deployment

    If you can’t get the Surface Pro 3 to PXE boot check the following:

    • Make sure you are using Microsoft USB to Ethernet Adapter. 3rd party adapters are not supported for PXE booting
    • Check and make sure this issue does not apply to your environment
    • 2602043: Invalid Boot File Received Error Message When PXE booting from WDS

    Additional Notes

    Some additional tips:

    • Check out my other blog for some additional tips for the PEN at “Deploying Surface Pro 3 Pen and OneNote Tips
    • If you do not want to see the Deployment Summary at the end of the deployment you can add the following entries customsettings.ini:

    [Default]
    ;Skip Final Summary Screen
    SkipFinalSummary=Yes
    ;Control behavior after system is complete
    FinishAction=Shutdown|Reboot|Restart|Logoff

    Thanks for reading this blog and good luck with your Surface Pro 3 deployments.

    Scott McArthur
    Senior Support Escalation Engineer

  • Deploying Surface Pro 3 Pen and OneNote Tips

    Hi, my name is Scott McArthur and I am Senior Support Escalation Engineer on the Deployment/Devices team. In today’s blog I am going to go over some tips for deploying Surface Pro 3 related to the Pen and OneNote integration.

    Tip #1: Deploying custom image to Surface Pro 3

    You may have noticed that if you deploy a custom image to Surface Pro 3 the Pen button does not bring up modern OneNote. The image that ships with Surface Pro 3 contains additional update that adds this functionality. If you are deploying a custom image you will need to incorporate that update into your deployment or reference image.

    KB2968599: Quick Note-Taking Experience Feature for Windows 8.1

    At this time we are working on making this update easier to download but in the meantime you can download it from the following direct link:

    http://download.windowsupdate.com/d/msdownload/update/software/crup/2014/06/windows8.1-kb2968599-x64_de4ca043bf6ba84330fd96cb374e801071c4b8aa.msu

    Tip #2: Setting default OneNote for the Pen

    If you use the Desktop version of OneNote 2013 you may want it to be the default application when the Pen button is pressed. To change the default OneNote application you can set this in OneNote 2013 using following steps

    1. Click File, Options
    2. Choose Advanced
    3. Under Default OneNote Application

    image

    NOTE: If you do not have this option in OneNote 2013 make sure you have the following update for OneNote installed:

    KB2881082: July 8, 2014 update for OneNote 2013

    Tip #3: Double click functionality for screenshots

    One of the other nice features is the ability to double click the pen button to send a screenshot to OneNote.

    Magic Tricks with OneNote and Surface Pro 3
    http://blogs.office.com/2014/06/18/magic-tricks-with-onenote-and-surface-pro-3

    In order to support this functionality, the Modern OneNote app must be the latest version available from the store.  So, if this functionality does not work, make sure Modern OneNote app has been updated.

    If you configure the Desktop OneNote as the default OneNote application, it should work by default with the double-click feature.

    Tip #4: Adding Pen pairing to a deployment

    During the 1st bootup of the OEM image that ships with the Surface Pro 3, you are prompted during OOBE to pair the pen. If you are deploying a custom image and want to add this setup screen to your deployment (to be completed by technician or user), you can use the following steps:

    Requirements:

    • System with the Windows ADK installed
    • Install.wim are you are deploying
    • Another Surface Pro 3 system that has the OEM system image that ships from Microsoft

    1. Take one of your existing Surface Pro 3 devices that has the OEM image on it and copy the following files to USB flash drive or other location:

    %windir%\system32\oobe\info\default\1033\oobe.xml
    %windir%\system32\oobe\info\default\1033\PenPairing_en-US.png
    %windir%\system32\oobe\info\default\1033\PenError_en-US.png
    %windir%\system32\oobe\info\default\1033\PenSuccess_en-US.png

    2. Open Deployment and Imaging Tools Environment cmd prompt

    3. Use the DISM command to mount the image you are deploying

    Dism /mount-wim /wimfile:c:\install.wim /index:1 /mountdir:c:\mount

    4. Create the following folder structure in the image

    C:\mount\windows\system32\oobe\info\default\1033

    5. Copy all the files from Step #1 above into this folder

    6. Close any explorer Windows and switch to C:\ to make sure no open file handles to the c:\mount folder

    7. Unmount the image and save changes

    Dism /unmount-wim /mountdir:c:\mount /commit

    Tip #5: Troubleshooting the pen

    For additional information on troubleshooting the pen take a look at:

    http://www.microsoft.com/surface/en-us/support/touch-mouse-and-search/troubleshoot-surface-pen#penshows

    Hope this helps with your Surface Pro 3 deployments

    Scott McArthur
    Senior Support Escalation Engineer

  • Windows Azure Pack: Infrastructure as a Service Jump Start

    Free online event with live Q&A with the WAP team: http://aka.ms/WAPIaaS

    Two half-days – Wednesday July 16th & Thursday July 17th– 9am-1pm PST

    IT Pros, you know that enterprises desire the flexibility and affordability of the cloud, and service providers want the ability to support more enterprise customers. Join us for an exploration of Windows Azure Pack's (WAP's) infrastructure services (IaaS), which bring Microsoft Azure technologies to your data center (on your hardware) and build on the power of Windows Server and System Center to deliver an enterprise-class, cost-effective solution for self-service, multitenant cloud infrastructure and application services.

    Join Microsoft’s leading experts as they focus on the infrastructure services from WAP, including self-service and automation of virtual machine roles, virtual networking, clouds, plans, and more. See helpful demos, and hear examples that will help speed up your journey to the cloud. Bring your questions for the live Q&A!

    Register here: http://aka.ms/WAPIaaS

    Course Outline:

    Day One

    • Introduction to the Windows Azure Pack
    • Install and Configure WAP
    • Integrate the Fabric
    • Deliver Self-Service

    Day Two

    • Automate Services
    • Extend Services with Third Parties
    • Create Tenant Experiences
    • Real-World WAP Deployments

    Instructor Team

    Andrew Zeller | Microsoft Senior Technical Program Manager
    Andrew Zeller is a Technical Program Manager at Microsoft, focusing on service delivery and automation with Windows Server, System Center, and the Windows Azure Pack.

    Symon Perriman | Microsoft Senior Technical Evangelist
    As Microsoft Senior Technical Evangelist and worldwide technical lead covering virtualization (Hyper-V), infrastructure (Windows Server), management (System Center), and cloud (Microsoft Azure), Symon Perriman is an internationally recognized industry expert, author, keynote presenter, executive briefing specialist, and technology personality. He started in the technology industry in 2002 and has been at Microsoft for seven years, working with multiple teams, including engineering, evangelism, and technical marketing. Symon holds several patents and more than two dozen industry certifications, including Microsoft Certified Trainer (MCT), MCSE Private Cloud, and VMware Certified Professional (VCP). In 2013, he co-authored Introduction to System Center 2012 R2 for IT Professionals (Microsoft Press) and he has contributed to five other technical books. Symon co-hosts the weekly Edge Show for IT Professionals, and his technologies have been featured in PC Magazine, Reuters News, and The Wall Street Journal. He graduated from Duke University with degrees in Computer Science, Economics, and Film & Digital Studies, and he also serves as the technical lead for several startups and entertainment production companies.​

  • Software-Defined Networking with Windows Server and System Center Jump Start

    Free online event with live Q&A with the WAP team: http://aka.ms/WAPIaaS

    Two half-days – Wednesday July 16th & Thursday July 17th– 9am-1pm PST

    IT Pros, you know that enterprises desire the flexibility and affordability of the cloud, and service providers want the ability to support more enterprise customers. Join us for an exploration of Windows Azure Pack's (WAP's) infrastructure services (IaaS), which bring Microsoft Azure technologies to your data center (on your hardware) and build on the power of Windows Server and System Center to deliver an enterprise-class, cost-effective solution for self-service, multitenant cloud infrastructure and application services.

    Join Microsoft’s leading experts as they focus on the infrastructure services from WAP, including self-service and automation of virtual machine roles, virtual networking, clouds, plans, and more. See helpful demos, and hear examples that will help speed up your journey to the cloud. Bring your questions for the live Q&A!

    Register here: http://aka.ms/WAPIaaS

    Course Outline:

    Day One

    • Introduction to the Windows Azure Pack
    • Install and Configure WAP
    • Integrate the Fabric
    • Deliver Self-Service

    Day Two

    • Automate Services
    • Extend Services with Third Parties
    • Create Tenant Experiences
    • Real-World WAP Deployments

    Instructor Team

    Andrew Zeller | Microsoft Senior Technical Program Manager
    Andrew Zeller is a Technical Program Manager at Microsoft, focusing on service delivery and automation with Windows Server, System Center, and the Windows Azure Pack.

    Symon Perriman | Microsoft Senior Technical Evangelist
    As Microsoft Senior Technical Evangelist and worldwide technical lead covering virtualization (Hyper-V), infrastructure (Windows Server), management (System Center), and cloud (Microsoft Azure), Symon Perriman is an internationally recognized industry expert, author, keynote presenter, executive briefing specialist, and technology personality. He started in the technology industry in 2002 and has been at Microsoft for seven years, working with multiple teams, including engineering, evangelism, and technical marketing. Symon holds several patents and more than two dozen industry certifications, including Microsoft Certified Trainer (MCT), MCSE Private Cloud, and VMware Certified Professional (VCP). In 2013, he co-authored Introduction to System Center 2012 R2 for IT Professionals (Microsoft Press) and he has contributed to five other technical books. Symon co-hosts the weekly Edge Show for IT Professionals, and his technologies have been featured in PC Magazine, Reuters News, and The Wall Street Journal. He graduated from Duke University with degrees in Computer Science, Economics, and Film & Digital Studies, and he also serves as the technical lead for several startups and entertainment production companies.​ ​

  • Is Offloaded Data Transfers (ODX) working?

    Offloaded Data Transfers (ODX) was a new data transfer strategy that makes advancements in moving files.  Only storage devices that comply with the SPC4 and SBC3 specifications will work with this feature.  With this feature, copying files from one server to another is much quicker.  This feature is only available in Windows 8/2012 and above as both the source and the destination.

    This is how it works at very high level:

    • A user copies or moves a file by using Windows Explorer, a command line interface, or as part of a virtual machine migration.
    • Windows Server 2012/2012R2 automatically translates this transfer request into an ODX (if supported by the storage array) and it receives a token that represents the data.
    • The token is copied between the source server and destination server.
    • The token is delivered to the storage array.
    • The storage array internally performs the copy or move and provides status information to the user.

    Below is a picture representation of what it looks like.  The top box is the way we are used to seeing it.  If you copy a file from one machine to another, the entire file is copied over the network.  In the bottom box, you see that the token is passed between the machines and the data is transferred on the storage.  This makes copying files tremendously faster, especially if you these files are in the gigabytes.

    image

    For more information on Offloaded Data Transfers, please refer to

    Windows Offloaded Data Transfers Overview

    Many Windows installations have additional filter drivers loaded on the Storage stack.  This could be antivirus, backup agents, encryption agents, etc.  So you will need to determine if the installed filter drivers support ODX.  As a quick note, if the filter driver supports ODX, but the storage does not (or vice versa), then ODX will not be used.

    We have filter manager supported features (SprtFtrs) which will tell if filter drivers support ODX.  We can use the FLTMC command, as shown below, to list filter drivers and their supported features.  For example:

    X:\> fltmc instances
    Filter      Volume Name    Altitude   Instance Name  Frame  SprtFtrs
    ----------  -------------  ---------  -------------  -----  --------
    FileInfo    C:             45000      FileInfo       0      00000003
    FileInfo    I:             45000      FileInfo       0      00000003
    FileInfo    D:             45000      FileInfo       0      00000003 <-It supports both Offload Read and Write
    FileInfo    K:             45000      FileInfo       0      00000003
    FileInfo    \Device\Mup    45000      FileInfo       0      00000003

    You can also see the Supported Features available for a filter driver in the registry:

    HKLM\system\CurrentControlset\services\<FilterName>

    The SupportedFeatures registry value contains an entry. If it is 3 as in the above FLTMC output, is supports ODX.

    Now that we have determined that ODX is supported by the required components, is it actually working?  You can see the ODX commands FSCTL_OFFLOAD_WRITE and FSCTL_OFFLOAD_READ captured in a Process Monitor trace when it is working.  When ODX is working, we see the following in Process Monitor.

    clip_image001

    In case a target fails the Offload, it does not support ODX, or does not recognize the token, it can give a STATUS_INVALID_TOKEN and/or STATUS_INVALID_DEVICE_REQUEST as the result.

    Other reasons why it might not work :

    1)    Something above Storage stack such as encryption or File system driver (Encryption filter, etc.) can cause it to fail.
    2)    Even though two disks/volumes might support offload they might be incompatible. This has to be established by involving the Storage Vendors.

    It is not a recommendation, but for informational purposes, you can disable ODX functionality in the registry if so desired.  You can do this  with a PowerShell command:

    Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1

    Or, you can edit the registry directly.  The value of 1 (false) means that it is disabled while the value of 0 (true) means it is enabled.  When this change is made, you will need to reboot the system for it to take effect.

    One last thing to mention is that you should always keep current on hotfixes.  This is especially true if you are running Failover Clustering.  Below are the recommended hotfixes you should be running on Clusters, which include fixes for ODX.

    Recommended hotfixes and updates for Windows Server 2012-based failover clusters
    Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters

    Other references:
    http://technet.microsoft.com/en-in/library/jj200627.aspx
    http://msdn.microsoft.com/en-us/library/windows/hardware/dn265439(v=vs.85).aspx
    http://msdn.microsoft.com/en-us/library/windows/desktop/hh848056(v=vs.85).aspx

    Shasank Prasad
    Senior Support Escalation Engineer
    Microsoft Corporation

  • Bugchecking a Computer on A Usermode Application Crash

    Hello my name is Gurpreet Singh Jutla and I would like to share information on how we can bugcheck a box on any usermode application crash. Set the application as a critical process when the application crash is reproducible. We may sometimes need a complete ...read more
  • Understanding ARM Assembly Part 3

    My name is Marion Cole, and I am a Sr. Escalation Engineer in Microsoft Platforms Serviceability group.  This is Part 3 of my series of articles about ARM assembly.  In part 1 we talked about the processor that is supported.  In part 2 ...read more