Home is where I lay my head

Optimization Guidance for Windows 7

pwigle — Wed, 27 Mar 2013 21:15:08 GMT

The Microsoft SBSL (Slow Boot Slow Logon) Community continuously works on improving performance during system startup and logon. This can be anything from hardware to software issues, this blog is a summary of a fair amount of those recommendations. Do note, this list is far from complete and it doesn’t have to reflect Microsoft’s best practices.

The idea

Boot and logon duration is primarily a function of workload. Everything you add to base OS adds workload. Bigger workloads mean bigger delays. The more policies, scripts, and data you move, the more services you start, the longer end-users wait.
Operations that take a long time after logon will affect boot and logon performance in an exaggerated way when resource consumption and concurrency conflicts exists. Restated, boot and logon is somewhat scenario specific. Inefficiencies that impact your post logon performance will crush your boot and logon performance.
Cost justify everything you do. Does the benefit of added workloads during boot and logon cost justify themselves against the added productivity cost. When calculating the total costs of added workload, think about:
- The number of boot and logons per day
- The amount of business days per year
- Average employee tenure in years
- Employee cost per minute
- The minutes the additional workload contributes to boot and logon delay

Hardware

During system startup Windows needs to read a lot of data from the local storage device. Depending on whether this is a clean Windows installation or an image containing provisioned applications, this can be anywhere between 150mb and 500mb (or possibly more), most of it random being random I/O.

Rotational hard drives have high seek times due to the need to travel the arm to the track of the disk where the data will be read or written. This makes them relatively bad in random I/O.
Solid State Drives (SSD) are similar to the memory you put in your camera, no physical movement is needed to read or write data. Have a look at the random I/O speed difference on a my local system containing two drives, one SSD for the operating system and one Hard Disk Drive (HDD) for bulk data storage.

Figure 1 – Winsat disk results on a SSD drive

Figure 2 – Winsat disk results on a HDD drive

In the figures above you can see that the SSD is about 150 times faster compared with the HDD in doing small random reads.
The main takeaway is that fast storage is crucial for quick startup and logon performance, in most scenarios much more important than the CPU. When considering the purchase of new hardware, I’d rather have a fast SSD with a slower CPU than visa versa.

Having enough physical memory for the current operating system, applications and future service packs is also crucial. Due to the low prices of physical memory this bottleneck which is getting rare on todays Windows Clients.

Some other settings that can impact performance on hardware level are:

Drive Acoustic modes, this limits the noise a HDD makes but impacts the performance.
Power Saving modes, can reduce the speed of disk I/O and potentially CPU I/O.

OS Servicing

On existing Windows 7 (and Windows 2008 R2) computers, consider and test the installation of all known fixes for known SBLS delays to avoid troubleshooting known issues and potentially improve boot and logon performance. This boils down to the installation of KB2775511 as mentioned in my other blog post, this rollup contains 90 fixes. Make sure this goes through a proper test cycle before deploying in production.

Also consider the installation of KB2792026 - Windows 7 SP1-based or Windows Server 2008 R2 SP1-based SMBv2 client computer freezes when the computer is under a heavy load.

Also integrate the same fixes into your OS build process for Windows 7 SP1 and Windows 2008 R2 Server images.

Base OS

A much seen issue is big WMI repositories, caused by applications that incorrectly treat repository as long term store. Have a look at %SystemRoot%\System32\WBEM\Repository and the size of the objects.data file.
Registry hives (system, security, software) must be of reasonable size, some applications are known for sources of registry bloat.

Application

Make sure you run the latest version, service packs and fixes for all security software.
Consider defining antivirus exclusions like described in KB822158.
Monitor the way applications use the registry+WMI.
Create a baseline using the Windows Performance Toolkit and note performance impact after comparing this with a trace after adding application or changing settings to a new image.
For each application that you install, examine known boot, logon and post logon performance issues and memory/handle or other resource leaks and their mitigation with the application vendor.

Logon Scripts

Use fully qualified DNS names to remove any ambiguity remote target computers.
When using the “Net use /D” command, it’s recommended to add the /Y parameter.
Scripts must be fault tolerant/resilient around password mismatches and inaccessible target computers.
Logon delays caused by script logic failures are frequently hidden by tools or when configured to run minimized or hidden mode.
Decrease the default value for “Specify maximum wait time for Computer Configuration” in Administrative Templates -> System-> Scripts Group Policy scripts (which changes the registry setting: HKLM\software\Microsoft\Windows\CurrentVersion\Policies\System\MaxGPOScriptWait) from the default 10 minutes to reasonable execution time for the set of scripts being executed over the links at your company)
Use Group Policy or Group Policy Preferences where applicable instead of KIX or other scripts.
Reduce the amount of WMI queries inside of a script.

Locator

Sites and subnets must be defined subnets containing by member workstations, servers and domain controllers.
Locator on clients and servers must pick the in-site or next closest DC for authentication, sourcing policy, profiles, logon scripts and LDAP/NSPI queries during boot, logon and post logon operations. The use of remote site servers adds WAN latency, reduces packet size, increases potential for port blockage and packet fragmentation.
Windows 2003 Domain Controllers need the Read Only Domain Controller Compatibility Pack installed if RODCs are deployed to prevent W2K3 DCs from auto-site covering for RODC-covered sites

Network

To get optimal performance in redirection and protocol stacks, servers consulted during boot and logon should be of the same or newer OS version as the clients you are deploying. For example, if deploying Windows 7 on the clients, servers should be Windows 2008 R2 or later. If deploying Windows 8, servers should be Windows Server 2012 or later.
Hard code link speeds in NCPA on clients and servers should match speed of underlying network.
Hard code link speeds on routers must match speed of underlying network.
Network connectivity must be allowed over all well-known and ephemeral ports used by the Operating System versions and applications considered during boot and logon. Low and high range ephemeral ports must be open for Windows Server operating systems deployed on the network. Windows 2003 and earlier servers use the low port range. Windows 2008 and later use the “high” port range.
Any delay in fundamental operations like SMB dialect negotiation, SMB session setup and Tree Connect during boot, logon and post logon operations is very likely to reduce performance.
Test network throughput in a post logon command prompt or Explorer instance for common operations performed during boot and logon. Note size and time it takes to copy policy, profiles, logon scripts.
Spanning tree enabled switches should have PORTFAST enabled.
EnablePMTUDiscovery = 0 reduces packet size to >512 bytes. Override with a policy and resolve the real problem that required you to set this key in the 1^st place.
Assume that Network infrastructure devices, especially WAN accelerators block required ports, reduce packet size, decelerate network throughput and connect you to sub-optimal target until proven otherwise at recurring intervals.

Policy

Reduce the amount of policies.
Reduce the amount of WMI filters on policies.
Make a concerted effort to reduce the amount of synchronous policies
Already mentioned at scripts: Decrease the default value for “Specify maximum wait time for Computer Configuration” in Administrative Templates -> System-> Scripts Group Policy scripts (which changes the registry setting:
HKLM\software\Microsoft\Windows\CurrentVersion\Policies\System\MaxGPOScriptWait) from the default 10 minutes to reasonable execution time for the set of scripts being executed over the links at your company)
Item level targeting fixes have to be in place on clients subject to item level targeting to improve logon performance and decrease CPU load caused by excessive LDAP traffic on DC role computers and network I/O.
Track the execution time, network I/O and DC load as you make changes to your computer and user policy. E.g. Inefficiently configured item level targeting can cause long delays on the client, massive increases in network I/O and high CPU load on the domain controllers servicing the resulting security group or LDAP requests.

Profile

Default and roaming user profiles must be of reasonable size.
Default and roaming profile must be modified in a supported way.
Investigate workarounds for Active Setup if using mandatory roaming profiles.
The link to the server hosting the profiles should be a fast and low latency connection.

Services

Auto-Start only the services that need to be started during the boot and logon phase.
Use the Windows Performance Toolkit to track any service requiring a high startup time, generally speaking 0.5 seconds or higher.

Hope this information helped you, if you have any comments or questions please consider leaving a reply.

Important Windows 7/2008 R2 Performance Rollup Package Is Available

pwigle — Wed, 27 Mar 2013 13:34:23 GMT

You want to improve performance of your Windows OS but don’t know where to begin? This rollup is a good starter – it might improve the performance of your Windows 7 SP1 PC or Windows 2008 R2 Server!

Every now and than an update/hotfix is released to address potential performance issue with Windows 7 SP1 and/or Windows Server 2008 R2. The good news: 90 of these updates now have been combined in one big rollup packages resolving many SBLS* issues.

The package can be downloaded from http://support.microsoft.com/kb/2775511 and a quick summary of the fixes are as follows:

Improves the Windows Client Remote File System components. These components include the following:
- Web-based Distributed Authoring and Versioning (WebDAV)
- DFSN client
- Folder Redirection
- Offline Files and Folders (CSC)
- SMB client
- Redirected Drive Buffering Subsystem (RDB)
- Multiple UNC Provider (MUP)
Improves the SMB Service and TCP protocol components. These improvements work together with other improvements to help improve the overall networking performance on Windows 7 SP1-based and Windows Server 2008 R2 SP1-based computers.
Improves the processing of Group Policies and Group Policy preferences. The performance of computers is improved after you install this rollup update on Windows 7-based computers that have several Group Policy preferences configured. Additionally, the network load and the domain controller usage may be reduced. We recommend that you install this hotfix rollup on every Windows 7 computer that has Group Policy preferences configured.
Improves the Windows Management Instrumentation (WMI) components to reduce the CPU usage and to improve the repository verification performance.

If you still experience Slow Boot Slow Logon issues, it’s time to take a boot trace using the Windows Performance Recorder.

Good luck!

*Slow Boot Slow Logon

How to migrate your Microsoft Account (a.k.a. Live ID) to a different country

pwigle — Mon, 21 Jan 2013 11:03:24 GMT

Yes! It’s possible to migrate your Microsoft Account, previously known as Live ID, to another country in a few clicks! In the past this wasn’t possible but since 16th of January 2013 this can be done in a few clicks.

In order to change country of your Microsoft Account, follow these steps:

Log in to http://xbox.com and choose “My Account”
On the bottom right, choose “Change Xbox account region”
Choose “Next” and select your country.
Choose “I agree” to finalize your migration.

FYI: it’s only possible to change region once every three months.

That’s it!

Outlook: Mail stuck after editing in Outbox

pwigle — Tue, 18 Dec 2012 09:42:09 GMT

Do you have a situation where you can’t send e-mail after editing a “waiting to be sent” e-mail in your outbox? I helped someone with the same situation, read how I troubleshooted and found the solution.

The scenario

This problem occurred in Outlook 2013 but could very well happen in earlier versions. This user disabled the option to “Send immediately when connected”

After composing and sending e-mail, it would be moved to the outbox. The actual sending of the e-mail would only happen when the user initiated a manual “Send and Receive”.

When editing an e-mail which was waiting to be sent (in the outbox) the icon would change from “waiting to sent” to a “draft”:

E-mail waiting to be sent in Outbox:

After editing the e-mail which was located in the outbox, the icon changed to draft:

As soon as you will press “Send” in the edited e-mail, the icon will change again to the little envelope and the e-mail would be sent after manually pressing “Send and Receive”. So far, this is all by design.

In this scenario, after pressing “Send” the e-mail would retain the draft icon and therefore would never leave the building. We found no way of sending an e-mail which was edited in the outbox.

How to troubleshoot

I tested it on a different machine and the behavior was different, an e-mail which was edited in the outbox would be waiting to be sent after hitting “send”.
To test if any plugins are responsible for this behavior we started outlook in safe mode:

When Outlook was running in safe mode, no plugins are loaded and the problem did not occur!

The quickest way to find the responsible plugin is by simply eliminating them, enable a few and restart outlook (not in safe mode) and see if the problem occurs.

After doing a few test I found the responsible plugin:

After disabling the iCloud plugin the problem did not occur.

This user did not use the Outlook functionality of iCloud so disabling was the solution. If the functionality is used, an update or follow-up with the vendor would be necessary.

Upgrading Windows Server 2008 R2 to 2012 – via RDP

pwigle — Wed, 12 Dec 2012 11:55:59 GMT

In order to save some fuel/time/energy I’m upgrading one or our test servers from Windows Server 2008 R2 to Windows Server 2012 via RDP. Not recommended or supported in any scenario <insert disclaimer here>.

Before you start, make sure all Virtual Machines are in a OFF state, setup wont be possible if they are set to SAVED or RUNNING.

Here we go!

Start a RDP session, I decided to take the Admin session:

Check the version currently installed:

Download and extract the ISO of the new Operating System:

Start the installer

Following the wizard:

- Download the latest updates for setup.

- Enter a valid product key (you can even ALT-TAB to copy it from somewhere)

- Choose a Windows Edition (with or without a GUI).

- Decide whether you accept the License Terms or not.

- Choose to upgrade.

And now… we wait…

After a while (and some reboots) the server responds to ping request again:

It took some more coffee until I could reach the server using RDP again.

Install the latest updates and: upgrade finished!

Error: "We cant connect to xbox service right now”

pwigle — Tue, 11 Dec 2012 19:52:15 GMT

Are you gettings this error, maybe together with a 0xc1010190 or 0xc00d1198? I didn’t spend much time troubleshooting but this is what fixed it for me:

1. Uninstall the Modern App by right clicking the tile

2. Reboot the PC

3. Re-install the application from the store.

Without the reboot in between I couldn’t get the app to work.
Hope it helps, please leave a reply!

Windows 8 - Nothing happens when you initiate a shutdown or hangs at the “shutting down” donut.

pwigle — Thu, 06 Dec 2012 15:33:57 GMT

On my corporate machine I had the issue where Windows would either:

Endlessly show the “Shutting Down” screen.
OR
When initiating a shutdown, nothing would happen except that Winlogon.exe would take a 100% CPU utilization on a single core.

I realized that when I didn’t use a hibernate file (powercfg /h off) scenario 2 would happen.

Troubleshooting the first scenario:

When troubleshooting the first scenario I configured the registry keys as described in “Windows feature lets you generate a memory dump file by using the keyboard” as can be found here. In order to get a full memory.dmp I configured the registry like described in KB254649.
After that I initiated a shutdown and waited for a few minutes before creating a memory dump by holding down the right CTRL key and pressing the SCROLL LOCK key two times.

After rebooting the machine there was a file C:\Windows\Memory.dmp ready to be analyzed in the Windows Debugger. The interesting stack (thanks Rob Scheepens) turned out to be:

I wasn’t sure why it was checking whether CHKDSK was scheduled for a next boot.

Troubleshooting the second scenario:

Since I also noticed the “IsHiberboot” part in the stack of the previous scenario, I enabled my hibernation file to check if this was to blame (using “Powercfg /h on").

After initiating a shutdown, nothing would happen except that winlogon.exe would take a 100% usage on a single core. After giving my PC a cold-reset I troubleshooting this using the following steps:

Note: If you don’t have the Windows 8 Performance Recorder installed, consult my previous blog for instructions:

http://blogs.technet.com/b/home_is_where_i_lay_my_head/archive/2012/07/30/how-to-install-the-windows-8-performance-recorder.aspx

I started Windows Performance Recorder, pressed Start and initiated a shutdown, waited a few minutes and finally pressed Save.

The result was a .ETL file which I loaded in Windows Performance Analyzer.
We already know the problem is related to the process winlogon.exe as shown in Task Manager, so we need symbols to get more information. To configure this use “Trace/Configure Symbol Path” and entered the Microsoft Public Symbol path:

Now looking at the trace, we can confirm it’s winlogon.exe taking a 100% CPU usage on a single core:

Zooming in on the responsible thread and doing some stackwalking, the same stack was to blame:

Because the stack is identical to problem troubleshooted in scenario 1, we can confirm it’s the same issue.

So what do we do next?

The thing I didn’t understand was the QueryIsDiskCheckScheduledForNextBoot part.
I looked in my registry to see if any CheckDisks (chkdsk) was scheduled and found this entry:

HKLM\System\CurrentControlSet\Control\Session Manager

I searched the registry for HarddiskVolume7 but couldn’t find any other entry and realized this was a VHD that was mounted a while ago.

Turned out to be that in the past:

I mounted a VHD
The volume got the dirty bit set
Before rebooting I dismounted the VHD again.

I followed up with the developers responsible for this code, turns out there might be a bug in autochk where it is not properly handling an explicit request to check a volume that no longer exists. This is currently being investigated.

The workaround is simple, remove this line from BootExecute:

autocheck autochk /m /f \Device\HarddiskVolum7

If this info has helped you, please consider leaving a comment.

Windows 8 Interactive Services Detection “Error 1: Incorrect Function”

pwigle — Tue, 09 Oct 2012 19:00:40 GMT

When an application or service displays an interactive message in session 0, Windows 7 displays a message to inform you about this application. You can use this feature to switch to session 0 and have a look which message is waiting:

The process responsible for this detection is UI0Detection.exe and controlled by a service called “Interactive Services Detection”

In Windows 8 this service now has been set to Manual and stopped by default (as opposed to Running under Windows 7). When you try to start this service in Windows 8 you will get the following message: “Error 1: Incorrect Function”

In order to get this service to run you will have to adjust a registry key:

Hive: HKLM
Path: SYSTEM\CurrentControlSet\Control\Windows
DWORD: NoInteractiveServices

Change value “1” to “0”

Start the Interactive Detection Service by services.msc or commandline using “net start ui0detect”

If you want to force a (test) message in session 0, get psexec.exe from Sysinternals and start it using the following parameters:

psexec -si 0 c:\windows\system32\calc.exe

This will start the Windows Calculator with System permissions in Session 0. This mechanism can also be used to troubleshoot logon/logoff issues by starting procmon or other troubleshooting tool in Session 0 which will survive a logoff.

How to install Windows 7 or 8 from USB using UEFI

pwigle — Tue, 02 Oct 2012 11:48:59 GMT

In order to benefit from the new UEFI features (including faster system startup), you will have to prepare your USB device slightly different.

Main take-away here: UEFI accepts boot loaders in files placed a the FAT formatted boot volume

To prepare your USB stick (4GB minimum) manually follow these steps:

Open an elevated command
Start diskpart
Enter list disk

Locate your USB stick, no room for mistakes here or you might destroy the wrong thing.
Enter select disk # (where # is the number of your USB drive, in the example above it’s number 3)
Enter the following commands, keep in mind that all data on the USB device will be deleted:

clean
create partition primary
select partition 1
active
format quick fs=fat32
assign
exit

Mount or extract the Windows 7 or 8 ISO
Copy the contents from the ISO to the USB stick

xcopy S:\* D:\ /s /e (S = the ISO drive or contents, D = the USB device)

Reboot your computer and enter the “BIOS” by pressing one of the F keys, this key varies depending on hardware manufacturer and model.
Enable the option “UEFI only” or similar option.
Save and exit, boot from your USB stick to start the installation.

Do you want to check if Windows is installed in such a way that its actually using UEFI, please consult my other blog post.

How to check in Windows if you are using UEFI

pwigle — Tue, 02 Oct 2012 11:04:31 GMT

You might be wondering if Windows is using UEFI or the legacy BIOS, it’s easy to check.

Just fire up MSINFO32 and look for this entry:

There are two possibilities here, either UEFI or Legacy.