Microsoft.com Operations

The Mystery of the Missing Process Performance Counter in Perfmon

One typical day I receive a request from a performance tester who is trying to collect some baseline production data about www.microsoft.com , the request seemed simple collect a couple of performance counter values during peak hours and post the data back to him .  I started to collecting data from my machines. One of the counters that he was interested in was Process à Handle Count  à w3wp. I could not find the process counter the counters what I had was as follows.

I quickly remembered about resource kit utility in win2k that was used to resolve this type of issue exctrlst.exe . Found a copy of it and copied it over to my 2k8 server not sure it would really work on 2k8 I decided to test my luck. I had a feeling that it was just a registry change so at the same time I started regmon on the server to capture registry changes.  Launched the tool and selected the perfproc counter and check marked performance counters enabled.  

Stopped regmon and filleted operation by regsetvalue Viola got it!!

 Closed Perfmon and opened it again. Verified the perfmon to make sure the process counter showed up.

Now I know what it sets in registry so I did want to do this remotely.   Quickly put through a for loop for the following command

Reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance /v "Disable Performance Counters" /t REG_DWORD /d 0

 Eg) for /f %i in (server.txt) do Reg add \\%i\HKLM\SYSTEM\CurrentControlSet\Services\PerfProc\Performance /v "Disable Performance Counters" /t REG_DWORD /d 0

One more issue resolved

 

Posted by MSCOM | 1 Comments

Filed under: Performance Monitor

SE Tips and Tricks -- Save aTS Session Run Remote... Find Hyper V information

Editors Note: One the System Engineers that run the servers that host www.Microsoft.com told me that he wanted to start posting some of the methods and tips and tricks that they use when managing one of the most heavily visitied web site in the world.  This post kicks off this series. 

1) How to remotely find out if a machine has Hyper V is installed?

 

When you install the hyperv Role on windows 2008 it updates the BCD store (replacement of boot.ini in 2008 & Vista) of the machine with an entry "hypervisorlaunchtype" and sets it to auto. So you can remotely query the BCD store of a machine to find out if Hyper V is installed or not. 

 

Here is a quick way from command prompt you can use the following command if you have psexec in the system path . You can use WINRS too, but psexec seems to be easy.

psexec \\machinename bcdedit | find "hyper"

e.g.) C:\>psexec \\hypervhost2k8 bcdedit | find "hyper"
 
PsExec v1.94 - Execute processes remotely
Copyright (C) 2001-2008 Mark Russinovich
Sysinternals - www.sysinternals.com
  
hypervisorlaunchtype    Auto
bcdedit exited on hypervhost2k8 with error code 0.
 

There are other quick ways too by checking existence of vmbus.sys, hvboot.sys and using WMI to query BCD.

 
2) How to remotely find out which is the Physical host of a VM?

 

In HyperV language physical host is a machine that hosts the root partition and guest will be called a

child partition installed on top of HyperV. You can remotely query the registry value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters\hostname of the child partition to find out the machine name that hosts the root partition.
 
Here is command that can be run remotely 
 
reg query "\\machinename\HKLM\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters" /V hostname 
 
 e.g.) C:\>reg query "\\hypervguest01\HKLM\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters" /v   physicalHostName

 

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters

    PhysicalHostName    REG_SZ    hypervhost2k8

Microsoft.com Operations Performance Analysis of IIS 7.0/Windows Server 2008

Now that Windows Server 2008 has been released earlier this year many enterprises are assessing the value proposition of moving to the Windows Server 2008 and IIS 7.0.  As an IT Pro or decision maker in this process you want real world working proof to make your decision clearer.  Sure, the new file based configuration system and modular architecture of IIS 7.0 make administrating your web servers a whole lot more flexible.  But how does it perform against its predecessor IIS 6.0 / Windows Server 2003?

 

Microsoft.com Operations has been an early and active adopters of IIS 7.0 / Windows Server 2008.  We actually started running a single server on the www.microsoft.com site back in June 2005.  We set out to answer this question.  Is IIS 7.0 / Windows Server 2008 more performant then IIS 6.0 / Windows Server 2003?  We performed a side by side performance analysis on the www.microsoft.com site which was taking live internet traffic during peak traffic hours.

 

During this performance assessment we found that the CPU usage was higher on Microsoft Windows Server 2008 RTM servers, which is driven by the higher number of RPS the servers were handling.  As mentioned below the hardware load balancers are sending more traffic to the Windows Server 2008 RTM servers (31% more RPS) as they are completing the outstanding web requests faster than the W2K3 SP2 server.  A good metric listed below is the “efficiency” or “cost” of the number of requests per CPU cycle (RPS / CPU Utilization %).  Using this metric, IIS 7.0/W2K8 RTM is over 10% more efficient then IIS 6.0/WS03 SP2 in handling live web platform traffic for the www.microsoft.com site.

 

Listed below the graphical data results is the methodology used in this performance analysis.

Performance Metrics	W2K3 SP2	WS2K8 RTM	Change
RPS / CPU Utilization (%) =  Requests per CPU Cycle	4.36	4.84	10.9%

 

	W2K3 SP2	W2K8 RTM	Change
Performance Indicators	A	B	(A-B)/A
CPU Utilization (%)	44.8%	52.8%	-17.9%
Web Service -Total Methods Requests/Sec “RPS”	194	255	31.4%
Web Service – Current Connections	280	294	5.0%
Load Balancing –  Current Client Connections	116	116	0
.NET CLR Memory – % Time in GC	1.1%	2.5%

 

Server Efficiency (RPS/ CPU %) – Efficiency of serving live web platform traffic

W2K3 SP2     4.36   “requests per CPU cycle”

W2K8 RTM   4.84   ~ 10.9% increased efficiency

 

 

CPU Utilization (%)

W2K3 SP2     44.8%

W2K8 RTM   52.8%   ~ 17.9% degradation (This is impacted by the increased RPS the W2K8 servers are handling)

 

 

Web Service – Total Methods Requests/Sec (RPS)

W2K3 SP2     194

W2K8 RTM    255   ~ 31.4% more traffic is being sent to the W2K8 RTM servers

 

 

Web Service – Current Connections

W2K3 SP2     280

W2K8 RTM   294   ~ 5% increase

 

 Load Balancing – Current Client Connections

W2K3 SP2     116

W2K8 RTM   116   Equal – as the hardware load balancer maintains the same amount of outstanding open client connections.

  

.NET CLR Memory – % Time in GC

W2K3 SP2    1.1%

W2K8 RTM   2.5%  No significant degradation in “Time in GC”

 

 

Performance Overview and methodology:

Site: www.microsoft.com 

During this assessment the live www.microsoft.com site was comprised on 80 servers.  There were four load balanced clusters of twenty servers each, located in multiple datacenters.  This performance analysis was developed by using a single twenty server cluster on the www.microsoft.com site.  The cluster is comprised of 19 Windows Server 2008 RTM servers (W2K8 RTM) and a single Windows Server 2003 SP2 (W2K3 SP2) server all taking live internet facing traffic.  This Lone Windows Server 2003 has become quite famous.  In fact, he actually has his own blog located at http://blogs.technet.com/windowsserver/pages/about-lone-server.aspx

 

Hardware:

  Model: HP DL585 G1 (4 dual-core CPUs)

  RAM: 32GB 

 

OS:

 Windows Server 2008 RTM (Build: 6.0.6001.18000) Enterprise version x64

 

Load Balancing:

We are using a hardware load balancing solution.  The load balancing algorithm we are using is based on “Least Current Client Connections” to each load balanced member server of the cluster (not round robin, or other any other load balancing algorithms).  The hardware load balancer will maintain the same number of current client connections to each member of the cluster.  So if a W2K8 server is completing web requests faster than a W2K3 server, the load balancer will send more traffic to the server W2K8 RTM server.

 

Methodology:

We gathered performance monitor logs for 72 hours during peak traffic (Monday - Wednesday) for a cluster of www.microsoft.com web servers.  In all the charts above the W2K8 RTM data is based on an average of all the W2K8 RTM servers in a single cluster.  While the W2K3 SP2 data is based on the single (Lone) server that was in rotation.

New IIS 7.0 White Paper…Internet Information Services 7.0 in Windows Server 2008 Provides an Improved Web and Application Platform

This paper discusses the management, security, performance and extensibility improvements when Windows Server 2008 is deployed to host and manage applications and services that run on the server and/or over the Web. Check it out on this TechNet link!

Modern .NET Development and The Joy of Simple LINQ to SQL

Introduction – Modern .NET Development

It took me a while to realize it, but Microsoft .NET application development has significantly evolved in the past year and a half. This is not your dad’s .NET. The following releases have added many considerable advances in .NET application development technologies:

                                               

Microsoft Product	Release Date	Major Features
.NET Framework version 3.0	November 11th, 2006	WPF, WCF
Silverlight version 1.0	September 5th, 2007	XAML rendering for web applications, JavaScript API
.NET Framework version 3.5	November 19th, 2007	Language integrated query (LINQ)
Visual Studio 2008	November 19th, 2007	IDE for all things .NET
Silverlight version 2.0	Beta 1 on March 5, 2008, RTM late summer 2008	Managed CLR, C#, .NET Framework, greater parity with WPF XAML
Expression Blend	v1 May 2007, v2 beta March 2008	XAML user experience (UX) IDE

 

If you fancy yourself an adept .NET developer, you had better be ramped up on these new technologies.

Today, if you are considering user interface/experience development, you should consider leveraging the rich UX elements and platform independence offered by Silverlight, and the power of authoring with the Expression Studio. If you are designing some middleware, you should consider developing software services using Windows Communication Foundation (WCF) technology. For coding backend data access it is now wise to evaluate LINQ to Entities or LINQ to SQL techniques. And before you write any C# foreach loop on an array or other collection, you might be able to write it better using LINQ to Objects techniques instead.

The bottom line here is that .NET developers have many new tools available to leverage for developing just about any application. Using these new techniques should eventually accelerate the delivery of applications that look better and are better-built. I say “eventually” because these new tricks will take some ramp up time to learn and master. And first, the developer needs to be aware of them and know how to judiciously leverage them.

However, we are talking about a pretty big load of new stuff to digest here. So let us discuss just one example of one these new tricks: LINQ to SQL. The primary purpose of rest of this article is to illustrate one simple example application of LINQ to SQL.

 

Simple LINQ to SQL

A couple of months ago, I was tasked with developing an internal application that would periodically transfer some data from a SQL database into a monitoring system. The SQL database was a Hewlett Packard Systems Insight Manager installation. The monitoring system is a Microsoft Systems Center Operations Manager (SCOM) 2007 installation, accessed via the System Center Operations Manager 2007 SDK, and the Microsoft.EnterpriseManagement Namespace.

LINQ to SQL is well-suited for this situation as it helps avoid changing either the source or destination installations, and with a minimal “footprint” of moving parts. With a single stand-alone console application, I was able to get this job done, and with a not-too-kludgey implementation. I developed a single, modest-length class file to do the whole thing, with no hard-coded transact SQL code!

I won’t drag you through my entire solution. I pared down the code to illustrate just the LINQ to SQL mechanics, as run against that Hewlett Packard Systems Insight Manager installation:

using System;

using System.Data.Linq;

using System.Data.Linq.Mapping;

using System.Data.Linq;

 

namespace LinqLite

{

    class Program

    {

        static void Main(string[] args)

        {

            DataContext db = new DataContext("YOUR HPSIM SQL SERVER CONNECTION STRING GOES HERE");

            Table<CIM_Chassis> TableCIM_Chassis = db.GetTable<CIM_Chassis>();

            Table<Devices> TableDevices = db.GetTable<Devices>();

            Table<Notices> TableNotices = db.GetTable<Notices>();

            var LinqQuery = from c in TableCIM_Chassis

                            join d in TableDevices on c.NodeID equals d.DeviceKey

                            join n in TableNotices on d.DeviceKey equals n.DeviceKey

                            where n.NoticeId > 35000    //TODO: make this a LastIdProcessed variable

                            orderby n.NoticeId

                            select new { n.NoticeId, n.NoticeSeverity, n.Generated, n.Comments, d.DeviceKey, d.ProductName, d.Name };

 

            foreach (var qdata in LinqQuery)

                Console.WriteLine(string.Format("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}"

                    , qdata.NoticeId

                    , qdata.NoticeSeverity

                    , qdata.Generated

                    , qdata.Comments

                    , qdata.DeviceKey

                    , qdata.ProductName

                    , qdata.Name

                ));

            Console.ReadLine();

        }

    }

 

    [Table]

    class CIM_Chassis

    {

        [Column] public long NodeID = 0;

    }

    [Table]

    class Devices

    {

        [Column] public int DeviceKey = 0;

        [Column] public string ProductName = string.Empty;

        [Column] public string Name = string.Empty;

    }

    [Table]

    class Notices

    {

        [Column] public int NoticeId = 0;

        [Column] public int NoticeSeverity = 0;

        [Column] public long Generated = 0;

        [Column] public string Comments = string.Empty;

        [Column] public int DeviceKey = 0;

    }

}

 

I call this solution simple LINQ to SQL primarily because it is all done in one code file, and does not involve using the fancy Visual Studio 2008 Object Relational Designer (O/R Designer). When you use the O/R Designer, you get a handy GUI for assembling some classes that map to the SQL data. You can easily end up with several auto-generated class files with many code stubs, just in case you might need them.   You use these classes in your LINQ queries to access the data. But if you are doing simple read-only selections, you can hand-code the classes, make them lean, and place them in-line in the code, like I did:

    [Table]

    class CIM_Chassis

    {

        [Column] public long NodeID = 0;

    }

    [Table]

    class Devices

    {

        [Column] public int DeviceKey = 0;

        [Column] public string ProductName = string.Empty;

        [Column] public string Name = string.Empty;

    }

    [Table]

    class Notices

    {

        [Column] public int NoticeId = 0;

        [Column] public int NoticeSeverity = 0;

        [Column] public long Generated = 0;

        [Column] public string Comments = string.Empty;

        [Column] public int DeviceKey = 0;

    }

 

These class definitions only list the fields that I am interested in, and not the entire tables. Another simple part of my example is the brief “using” list:

using System;

using System.Data.Linq;

using System.Data.Linq.Mapping;

using System.Data.Linq;

 

Hopefully my core LINQ selection code is not too hard to decipher – seven fields selected from two joined tables, with a third table joined in to filter the data to only notices for “chassis” devices. First, a DataContext is created to set up a connection the SQL Server database. Next, references are made to the table classes using DataContext.GetTable(TEntity) Generic Method:

            Table<CIM_Chassis> TableCIM_Chassis = db.GetTable<CIM_Chassis>();

            Table<Devices> TableDevices = db.GetTable<Devices>();

            Table<Notices> TableNotices = db.GetTable<Notices>();

 

Then the LINQ query is defined, complete with joins, a where clause, and orderby:

            var LinqQuery = from c in TableCIM_Chassis

                            join d in TableDevices on c.NodeID equals d.DeviceKey

                            join n in TableNotices on d.DeviceKey equals n.DeviceKey

                            where n.NoticeId > 35000    //TODO: make this a LastIdProcessed variable

                            orderby n.NoticeId

                            select new { n.NoticeId, n.NoticeSeverity, n.Generated, n.Comments, d.DeviceKey, d.ProductName, d.Name };

 

I put the “TODO” comment on the where clause to indicate that a LastIdProcessed variable could be persisted and recalled to use for filtering the notices data. Thus, you might fetch only new notices that have occurred after the last time this process was executed.

Finally, a foreach construct is employed to “harvest” and display the selected data:

            foreach (var qdata in LinqQuery)

                Console.WriteLine(string.Format("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}"

                    , qdata.NoticeId

                    , qdata.NoticeSeverity

                    , qdata.Generated

                    , qdata.Comments

                    , qdata.DeviceKey

                    , qdata.ProductName

                    , qdata.Name

                ));

 

The Console.ReadLine() at the very end merely serves as a way to pause program execution, so the displayed data can be viewed before the console window disappears.

MSDN’s “LINQ to SQL: .NET Language-Integrated Query for Relational Data” article does well to explain more about how this code works.

 Conclusion

I hope this article has helped demystify LINQ for you, and has maybe inspired you to consider using this powerful .NET feature. I have a lot yet to learn about LINQ and the rest of those modern .NET technologies previously listed in that first table. But, I feel it is ripe time to dig into these things to advance the craft.

Posted by MSCOM | 3 Comments

Filed under: .Net Framework, SQL, LINQ

Microsoft.com Engineering Operations TechCenter NEW on TechNet.Microsoft.com

The Microsoft.com Operations Team is pleased to announce the launch of our new Microsoft.com Engineering Operations TechCenter. This site ids designed with the ITPro/Systems Engineer in mind. We get to work on some cutting edge technologies and our goal is to share our experiences, best practices, and findings with the ITPro/Systems Engineering community. Take a look at the key articles that are in this inaugural launch:

 

Introduction to the Microsoft.com Engineering Operations Team

 

Introduces the Microsoft.com Engineering Operations team, which is made up of systems engineers who support some of the largest and most heavily visited sites on the Internet, and describes the architectural infrastructure of some of the properties we support, including www.microsoft.com,  Microsoft® Update, MSDN® and TechNet.

 

Migrating a Web Server from IIS 6.0 to IIS 7.0 by Using the Microsoft Web Deploy Tool

 

Get an inside look at how to install the new Microsoft Web Deploy Utility (MS Deploy) Tool as well as migrating your Web server or Web site from a computer that is running Information Services (IIS) version 6.0 on Microsoft® Windows Server™ 2003 to a computer that is running Internet Information Services (IIS) version 7.0 on Windows Server 2008.

 

How Microsoft.com Engineering Operations Delegates Configuration in IIS 7.0

 

Reduce administration overhead with IIS7.0 delegated configuration. Find out how the Microsoft.com Engineering Operations team uses delegated configuration in Internet Information Services (IIS) 7.0. Topics include performance and availability considerations, security, and delegating administrative control to Web sites.

 

Customizing IIS 7.0 by Installing and Configuring Roles, Role Services, Features, and Modules

 

Modular architecture in IIS7.0 enables administrators to reduce your web server footprint, by only installing features you need. Get a look under the hood and see how Microsoft.com Engineering Operations team installs role services, features and the associated Internet Information Services (IIS) version 7.0 modules that are required to run www.microsoft.com Web servers.

 

The Microsoft.com Engineering Operations TechCenter. will have also provide links to all of the existing content that we had produced over  the last couple of years.

 

In the works are content about:

1.       How MSCOM Ops Configure Application Pools in IIS7.0

2.       How MSCOM Ops Adopts New Technology

 

Also in the planning stages are information about how we are using Hyper-V, as well as findings on how the new TCP/IP stack effects performance on SQL replication in high latency situations (i.e., replication from West Coast to East Coast).

 

We are very excited to get this project launched, it is the first TechCenter that is focused on the ITPro/System Engineer.

Posted by MSCOM | 2 Comments

Filed under: IIS7, Infrastructure, Architecture, Webcasts, .Net Framework, ASP.Net, App Pools, Operations Management, Technology Adoption, Operations, TCP/IP, x64

Microsoft.com Engineering Operations Forum Is Now LIVE!!

 

We are pleased to announce that the Microsoft.com Engineering Operations Forum is now live on TechNet.  Come join us in discussions that are focused on Engineering Operations. Ask questions, give us your insights, help us build the community of folks that are engaged in real-world systems engineering. Log in with your Windows Live ID and be a part of the discussions.

Systems Engineering Architecture Consultation…”Help Us to Help You!”

MSCOM Operations get lots of requests from both internal and external customers on how we operate www.microsoft.com, Microsoft Update, and the Microsoft Download Center (just to name a few). Those customers are asking about a wide variety of topics that we may be able to help them with. Topics like our best practices we use in rolling out new technologies like Windows 2008 and IIS7.0 to our production web environment ,or how we use Peer-to Peer replication in our SQL topologies. Sometimes they just want to chew the fat with fellow System Engineers about the common problems that we all face as SEs.

 

These customer interactions are one of the best parts of working in MSCOM Operations. We truly learn as much in these customer engagements as (hopefully) our customers learn from us. To help us target these discussions we have created the following Infrastructure Architecture Straw Man. We provide this to customers that have pending engagements with us to try and get a sense of what their environments look like. This really helps us put the right Subject Matter Expert (SME) from our team in front of the customer with a good idea of what direction the discussion is likely to go.

 

Hopefully this will be of some use to you as well.

 

Infrastructure Architecture Questions/Topics Straw Man

This is intended to provide a list of topics to address with customers to assist in providing infrastructure architectural guidance.

First question(s) to ask is what are the business problems that need to be solved, the “must haves”. These then need to be prioritized. The infrastructure architecture can be very different depending on the result of this prioritization.

Next try to get a data flow diagram. Where are the calls coming from, what are the expected results from those calls and what components will need to be touched for each call. This will also help to flesh out the infrastructure.

Then try and get a high level diagram of the number of hosting locations/data centers; server clusters etc. which may or may not be known. Also try to determine upfront the target audience (public internet, corporate user’s intranet, partners extranet) and the approximate number of end users.

Finally get a list of requirements the customer thinks they need, then ensure that they understand the ramifications of those requirements. Example: Customer: “We need 5 nines availability.” Architect: “Great, please understand that equates to 2.59 sec of downtime per month.”

Below is a non prioritized list of requirement topics.

1.     Availability

a.     Defined as providing the required functional benefit to the users, not simply that “a server is up” metric.

Availability %	Downtime per year	Downtime per month*		Downtime per week
98%	7.30 days		14.4 hours	3.36 hours
99%	3.65 days		7.20 hours	1.68 hours
99.5%	1.83 days		3.60 hours	50.4 min
99.9%	8.76 hours		43.2 min	10.1 min
99.99%	52.6 min		4.32 min	1.01 min
99.999%	5.26 min		25.9 sec	6.05 sec
99.9999%	31.5 sec		2.59 sec	0.605 sec

b.    Each one comes with an associated cost.

2.     Performance

a.     Web Site

                                          i.    Size of the page

                                         ii.    Number of calls to render the page

                                        iii.    Type of calls (http, https etc.)

b.    Web Services

                                          i.    Will this be used to call SQL

c.     Back End (SQL)

d.    Scalability

                                          i.    100’s or 1000’s or 1,000,000 Req/sec

e.     Performance testing – what is the acceptable

3.     Monitoring

a.     What need to be monitored

                                          i.    Servers

                                         ii.    Applications

                                        iii.    Web services

                                        iv.    Connectivity

                                         v.    Events collected by monitoring should be “actionable”

4.     Manageability

a.     Directly related to Ops costs (people are the most expensive component)

b.    Simple cookie-cutter vs. complex one-offs

5.     Scale

a.     Firewall/DMZ

b.    Bandwidth

c.     Server specific

                                          i.    Global Load Balancing

                                         ii.    Local Load Balancing

6.     Content Distribution

a.     Content delivery networks (CDN)

b.    Caching strategies

c.     Nature of content

                                          i.    Static

                                         ii.    Dynamic

                                        iii.    Make up

1.     Jpegs

2.     gifs

3.     Flash

4.     Silverlight

7.     Performance

a.     Geo Location (high latency links)

b.    Server resource utilization under various loads (always good to load stress a server until it starts to fail to develop maximum capacity levels)

a.     Running “cool” . i.e. CPU 40%, memory (depends on server role)

                                                          i.    Determine load thresholds

b.    Running “normal” i.e. CPU sustained 60-75%, memory (depends on server role)

                                                          i.    Determine load thresholds

c.     Running “Hot” i.e. CPU sustained 90%, memory (depends on server role)

                                                          i.    Determine load thresholds

8.     Security

a.     Application security

b.    Infrastructure security (DDoS, Intrusion detection, etc.)

                                          i.    Firewalls

                                         ii.    DMZ

                                        iii.    Router ACLs

9.     Traffic Analysis

a.     Volume of traffic

                                          i.    Requests/sec

b.    Geo location of traffic

c.     Nature of traffic

                                          i.    ASP.net, etc.

10.  Geo-location of traffic

b.    Type of client requests

                                          i.    For static content

                                         ii.    For Dymanic contest

11.  Network (Frontend)

a.     Latency concerns

b.    Geo-redundancy needs

12.  Network Backend

a.     Latency concerns

b.    Geo-redundancy needs

13.  Virtualization

c.     Clients/Application Servers

d.    Storage

e.     Applications

14.  Identity Management/Authentication

a.     Windows auth

b.    Certificate based

c.     Protecting Personal Identifiable Information (PII)

d.    Cookies

 

IIS7.0 Lessons Learned In MSCOM Production Environment - On Demand Webcast

MSCOM has been running IIS7.0 in production since June 12, 2007. On November 27 we did a webcast entitled:

How Microsoft IT Is Providing an Improved Web and Applications Platform with Internet Information Services 7.0 in Windows Server 2008

In case you missed this live event we would like to invite you to use the link above to view this On Demand. Learn about some of the new benefits and features in Microsoft Internet Information Services (IIS) version 7.0 in the Windows Server 2008 operating system as experienced during the deployment of IIS7 as the Web and application platform for Microsoft.com. IIS7 in Windows Server 2008 provides management, security, performance, and extensibility improvements when Windows Server 2008 is deployed to host and manage applications and services that run on the server and/or over the Web.

Analyzing HTTP Traffic On Your IIS 7.0 Cluster

If you have ever run an enterprise web site you have probably received a panicked call from a customer or from your Tier 1 monitoring team that your site was responding slowly or throwing 500 errors and that you needed to resolve this issue ASAP.   Maybe you are rolling out a new application or software across your site and want to perform side by side analysis of servers to ensure their health.

 

Over the past year we have rolled out multiple pre-release versions of Windows Server 2008 / IIS 7.0 onto the www.microsoft.com site.  During this process we have had to monitor the overall health of the site.  Which included being able to quickly identify content/application that were not behaving well on a server, cluster of servers or across the entire site.   When running an enterprise of IIS servers it is invaluable to understand the traffic patterns of your site and to have the tools to help identify problem areas.  There are many tools in the wild to monitor the health of your enterprise, but I will show you how to use Logparser to harvest your IIS logs and pinpoint a site problem.   The goal of this blog is to help you identify issues or anomalous traffic patterns on your IIS web site.

 

For the purposes of this blog I will use an eight server cluster for our analysis.  Sometimes tracking down issues on your site can be like trying to find a needle in a haystack.  I like to start by looking at the overall health of the site or for the purposes of my examples, a cluster of servers.   Next, I like to drill down into more detail on the server that looks unhealthy.  In my examples I will drill down by URI and followed next by the associated window of time that the server/URI was unhealthy.

 

Here is where the fun begins!   The first step is to perform a query by HTTP status codes for the cluster.  Using Logparser I created a file called statuscodes_by_servers_in_cluster.sql and fed in the input IIS log files from all the IIS servers in the cluster which are called u_ex07111409.log.  This result in an easy to read table showing you the summarized results by HTTP status code for each of the servers in your cluster. 

 

Summarized HTTP status codes by server for a complete a complete cluster

logparser -rtp:-1 file:statuscodes_by_servers_in_cluster.sql?logfilename=u_ex07111409.log

sc-status Srv1   Srv2   Srv3   Srv4   Srv5   Srv6   Srv7   Srv8

--------- ------ ------ ------ ------ ------ ------ ------ ------

200       576313 585078 573956 574218 566388 599807 572982 589815

206       242    238    247    262    236    216    192    246

301       6148   6388   6378   6154   6140   6600   6293   6398

302       166817 168700 166471 165909 163386 172658 166764 170363

304       125493 126640 125908 126204 124139 130669 125855 129026

400       4      3      3      2      2      7      7      3

403       130    106    146    170    148    154    148    95

404       9023   9149   9835   8833   8811   9489   9250   9369

405       0      1      0      0      0      0      0      0

406       2      10     5      11     4      7      8      4

500       1361   342    363    337    332    323    340    351

501       18     22     21     25     31     27     34     30

 

I created the following Logparser file (statuscodes_by_servers_in_cluster.sql) to create the site by side server table of HTTP status code values.

 

statuscodes_by_servers_in_cluster.sql

SELECT

sc-status,

       SUM(_Srv1) AS Srv1,

       SUM(_Srv2) AS Srv2,

       SUM(_Srv3) AS Srv3,

       SUM(_Srv4) AS Srv4,

       SUM(_Srv5) AS Srv5,

       SUM(_Srv6) AS Srv6,

       SUM(_Srv7) AS Srv7,

       SUM(_Srv8) AS Srv8

      

USING

       CASE s-computername WHEN 'ServerName1' THEN 1 ELSE 0 END AS _Srv1,

       CASE s-computername WHEN 'ServerName2' THEN 1 ELSE 0 END AS _Srv2,

       CASE s-computername WHEN 'ServerName3' THEN 1 ELSE 0 END AS _Srv3,

       CASE s-computername WHEN 'ServerName4' THEN 1 ELSE 0 END AS _Srv4,

       CASE s-computername WHEN 'ServerName5' THEN 1 ELSE 0 END AS _Srv5,

       CASE s-computername WHEN 'ServerName6' THEN 1 ELSE 0 END AS _Srv6,

       CASE s-computername WHEN 'ServerName7' THEN 1 ELSE 0 END AS _Srv7,

       CASE s-computername WHEN 'ServerName8' THEN 1 ELSE 0 END AS _Srv8

 

FROM

       \\ServerName1\logshare$\%logfilename%,

       \\ServerName2\logshare$\%logfilename%,

       \\ServerName3\logshare$\%logfilename%,

       \\ServerName4\logshare$\%logfilename%,

       \\ServerName5\logshare$\%logfilename%,

       \\ServerName6\logshare$\%logfilename%,

       \\ServerName7\logshare$\%logfilename%,

       \\ServerName8\logshare$\%logfilename%

 

GROUP BY

        sc-status

 

ORDER BY

        sc-status

 

You will notice that the HTTP 500 status codes are rather high on one of our servers (SRV1).   Now that we have identified which server looks unhealthy we can drill down and identify the problem application or URI which is causing the high level of 500’s server errors.

 

The next Logparser query I like to use is a spill by URI for the specific HTTP 500 status code on the cluster.   Using Logparser I created a file called cs-uri-stem_by_servers_in_cluster.sql and fed in the same input IIS log files from all the IIS servers in the cluster called u_ex07111409.log.

 

URI spill for ‘500’ HTTP status code for a complete cluster:

logparser -rtp:-1 file:cs-uri-stem_by_servers_in_cluster.sql?logfilename=u_ex07111409.log

sc-status Srv1 Srv2 Srv3 Srv4 Srv5 Srv6 Srv7 Srv8 cs-uri-stem

--------- ---- ---- ---- ---- ---- ---- ---- ---- ----------------------------------------------

500       0    0    0    0    0    0    0    1    /brasil/pr/2002/ms_edu_minas.stm

500       0    0    0    1    0    0    0    0    /brasil/pr/2002/ms_games_br.stm

500       0    0    0    0    0    1    0    0    /brasil/technet/topicos/codered.stm

500       1    0    0    0    0    0    0    0    /brasil/windows2000/requisitos.stm

500       0    0    0    0    2    0    0    0    /egypt/

500       0    0    0    0    1    0    0    1    /esp

500       0    0    0    0    0    0    0    35   /fwquery/

500       28   9    10   35   21   20   23   31   /gbadapp/errorpages/error.aspx

500       267  160  184  260  270  270  274  275  /ibadapp/errorpages/error.aspx

500       1    0    0    0    0    0    0    0    /middleeast/press/2001/may/gold_cert.stm

500       0    0    0    1    0    0    0    0    /middleeast/press/2001/may/officexpa.stm

500       1060 7    17   38   35   32   38   0    /obadapp/errorpages/error.aspx

500       0    0    0    0    0    0    1    0    /projectserver

500       4    2    1    2    1    0    4    5    /shared/1/navigation.asmx/DisplayDlNavHtml

500       0    0    0    0    2    0    0    1    /shared/2/navigation.asmx/DisplayDlNavHtml

500       0    0    0    0    0    0    0    2    /xml/overview.asp

 

I created the following Logparser file (cs-uri-stem_by_servers_in_cluster.sql) to create the URI spill by server for the HTTP 500 status code.

 

cs-uri-stem_by_servers_in_cluster.sql

SELECT

sc-status,

       SUM(_Srv1) AS Srv1,

       SUM(_Srv2) AS Srv2,

       SUM(_Srv3) AS Srv3,

       SUM(_Srv4) AS Srv4,

       SUM(_Srv5) AS Srv5,

       SUM(_Srv6) AS Srv6,

       SUM(_Srv7) AS Srv7,

       SUM(_Srv8) AS Srv8,

       cs-uri-stem

 

      

USING

       CASE s-computername WHEN 'ServerName1' THEN 1 ELSE 0 END AS _Srv1,

       CASE s-computername WHEN 'ServerName2' THEN 1 ELSE 0 END AS _Srv2,

       CASE s-computername WHEN 'ServerName3' THEN 1 ELSE 0 END AS _Srv3,

       CASE s-computername WHEN 'ServerName4' THEN 1 ELSE 0 END AS _Srv4,

       CASE s-computername WHEN 'ServerName5' THEN 1 ELSE 0 END AS _Srv5,

       CASE s-computername WHEN 'ServerName6' THEN 1 ELSE 0 END AS _Srv6,

       CASE s-computername WHEN 'ServerName7' THEN 1 ELSE 0 END AS _Srv7,

       CASE s-computername WHEN 'ServerName8' THEN 1 ELSE 0 END AS _Srv8

 

FROM

       \\ServerName1\logshare$\%logfilename%,

       \\ServerName2\logshare$\%logfilename%,

       \\ServerName3\logshare$\%logfilename%,

       \\ServerName4\logshare$\%logfilename%,

       \\ServerName5\logshare$\%logfilename%,

       \\ServerName6\logshare$\%logfilename%,

       \\ServerName7\logshare$\%logfilename%,

       \\ServerName8\logshare$\%logfilename%

 

WHERE  sc-status = 500

 

GROUP BY

       cs-uri-stem,

       sc-status

 

ORDER BY

        cs-uri-stem

 

 

There are a couple URIs that are throwing the 500 errors on SRV1.  The majority of the errors coming from the URI “/obadapp/errorpages/error.aspx”.    The next step is to determine if there was a window where this URI/application was performing poorly.

 

The next Logparser query I like to use is to spill by HTTP status code and by time (in minutes) for the problem server in question.  This will determine if the URI / application on ServerName1 (SRV1) was throwing the 500 errors consistently throughout the hour or during a specific window of time.   Using Logparser I created a file called httpstatuspivot.sql and fed in the same input IIS log file from ServerName1 called u_ex07111409.log.

 

Spill of HTTP status codes by time

logparser -rtp:-1 file:httpstatuspivot.sql?logfilename=\\ServerName1\logshare$\u_ex07111409.log

Minute              200s  206s 301s 302s 304s 403s 404s 405s 406s 500s 501s

------------------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

2007-11-14 09:00:00 11143 3    128  3302 2477 6    158  6    0    3    0

2007-11-14 09:01:00 10659 4    103  3158 2399 2    142  2    0    3    0

2007-11-14 09:02:00 11139 6    129  3215 2410 1    154  1    0    4    0

2007-11-14 09:03:00 10749 1    116  3141 2192 3    161  3    0    4    0

2007-11-14 09:04:00 10814 2    132  3130 2179 2    180  2    0    3    0

2007-11-14 09:05:00 11139 3    134  3240 2527 5    172  5    0    6    0

2007-11-14 09:06:00 11005 8    136  3285 2384 2    189  2    0    8    0

2007-11-14 09:07:00 10913 6    116  3125 2409 6    165  6    0    7    0

2007-11-14 09:08:00 11291 4    136  3267 2315 2    204  2    1    5    0

2007-11-14 09:09:00 10854 5    140  3239 2358 2    162  2    0    5    1

2007-11-14 09:10:00 10219 4    107  2975 2215 1    166  1    1    8    0

2007-11-14 09:11:00 10006 6    124  2854 2050 1    180  1    0    6    0

2007-11-14 09:12:00 10555 5    112  2895 2163 1    160  1    0    7    0

2007-11-14 09:13:00 10404 2    90   3007 2228 0    190  0    0    5    0

2007-11-14 09:14:00 9246  9    103  2839 1852 0    143  0    0    3    0

2007-11-14 09:15:00 9751  4    114  2964 2180 3    135  3    0    4    0

2007-11-14 09:16:00 11884 4    137  3411 2645 2    169  2    0    7    0

2007-11-14 09:17:00 10314 5    94   2838 2203 2    146  2    0    3    0

2007-11-14 09:18:00 9844  2    93   2911 2123 3    132  3    0    6    2

2007-11-14 09:19:00 10080 2    115  2883 2158 1    132  1    0    6    0

2007-11-14 09:20:00 10410 3    114  2993 2309 3    155  3    0    4    0

2007-11-14 09:21:00 10817 3    117  3173 2385 0    188  0    0    4    0

2007-11-14 09:22:00 10313 4    117  3016 2324 3    185  3    0    9    1

2007-11-14 09:23:00 10210 2    107  2792 2058 2    143  2    0    5    0

2007-11-14 09:24:00 10122 6    85   2785 2204 2    156  2    0    6    0

2007-11-14 09:25:00 10852 4    100  3095 2332 3    161  3    0    7    0

2007-11-14 09:26:00 9857  6    100  2911 2100 3    135  3    0    6    0

2007-11-14 09:27:00 9707  4    103  2820 2081 3    164  3    0    8    0

2007-11-14 09:28:00 9858  7    90   2775 2142 6    139  6    0    4    0

2007-11-14 09:29:00 9508  3    77   2767 2064 3    137  3    0    2    0

2007-11-14 09:30:00 8810  3    90   2593 2094 1    110  1    0    11   3

2007-11-14 09:31:00 8913  4    96   2522 1955 1    153  1    0    6    1

2007-11-14 09:32:00 9214  6    112  2789 2176 3    150  3    0    9    0

2007-11-14 09:33:00 9930  3    106  2871 2118 5    140  5    0    4    3

2007-11-14 09:34:00 9136  6    95   2548 1794 3    123  3    0    5    1

2007-11-14 09:35:00 9431  3    106  2734 2136 1    173  1    0    3    0

2007-11-14 09:36:00 9174  13   99   2667 2034 0    116  0    0    5    0

2007-11-14 09:37:00 8292  4    91   2507 1807 1    115  1    0    7    0

2007-11-14 09:38:00 9275  2    82   2496 1911 3    139  3    0    5    0

2007-11-14 09:39:00 8290  3    84   2396 1741 2    116  2    0    108  0

2007-11-14 09:40:00 8802  7    117  2556 2034 5    151  5    0    617  0

2007-11-14 09:41:00 8549  6    90   2332 2002 1    140  1    0    311  1

2007-11-14 09:42:00 8857  5    146  2521 1901 0    167  0    0    1    2

2007-11-14 09:43:00 8200  3    91   2315 1819 1    136  1    0    4    0

2007-11-14 09:44:00 8397  5    92   2388 1953 3    128  3    0    6    0

2007-11-14 09:45:00 8204  1    79   2453 1911 1    124  1    0    4    0

2007-11-14 09:46:00 8979  3    93   2657 2035 1    142  1    0    6    0

2007-11-14 09:47:00 9369  5    79   2757 2066 2    166  2    0    8    0

2007-11-14 09:48:00 8086  2    80   2375 1751 0    148  0    0    5    0

2007-11-14 09:49:00 8119  2    76   2193 1597 2    96   2    0    10   0

2007-11-14 09:50:00 8642  4    104  2494 2066 2    147  2    0    10   0

2007-11-14 09:51:00 8032  4    83   2292 1889 2    147  2    0    7    0

2007-11-14 09:52:00 8349  3    88   2585 1956 1    159  1    0    10   1

2007-11-14 09:53:00 8507  4    88   2589 1792 1    157  1    0    4    0

2007-11-14 09:54:00 9180  2    84   2538 1918 3    154  3    0    3    2

2007-11-14 09:55:00 8475  1    91   2523 1930 3    134  3    0    8    0

2007-11-14 09:56:00 8357  1    83   2428 1855 1    143  1    0    8    0

2007-11-14 09:57:00 8144  5    80   2354 1785 2    130  2    0    7    0

2007-11-14 09:58:00 8930  1    76   2567 1890 2    147  2    0    5    0

2007-11-14 09:59:00 10007 4    98   2971 2111 4    169  4    0    6    0

 

 

I created the following Logparser file (httpstatuspivot.sql) to create the spill by time (in minutes) for the server which was throwing the 500 errors.  Also, an excellent TechNet article entitled “Inside Microsoft.com - Analyze Web Stats with Log Parser” also covers pivoting output of your Logparser query.

 

httpstatuspivot.sql

SELECT quantize(to_localtime(to_timestamp(date,time)),60) AS Minute,

    SUM([_200]) AS [200s],

    SUM([_206]) AS [206s],

    SUM([_301]) AS [301s],

    SUM([_302]) AS [302s],

    SUM([_304]) AS [304s],

    SUM([_403]) AS [403s],

    SUM([_404]) AS [404s],

    SUM([_403]) AS [405s],

    SUM([_406]) AS [406s],

    SUM([_500]) AS [500s],

    SUM([_501]) AS [501s]

 

 

USING

    CASE sc-status WHEN 200 THEN 1 ELSE 0 END AS [_200],

    CASE sc-status WHEN 206 THEN 1 ELSE 0 END AS [_206],

    CASE sc-status WHEN 301 THEN 1 ELSE 0 END AS [_301],

    CASE sc-status WHEN 302 THEN 1 ELSE 0 END AS [_302],

    CASE sc-status WHEN 304 THEN 1 ELSE 0 END AS [_304],

    CASE sc-status WHEN 403 THEN 1 ELSE 0 END AS [_403],

    CASE sc-status WHEN 404 THEN 1 ELSE 0 END AS [_404],

    CASE sc-status WHEN 401 THEN 1 ELSE 0 END AS [_405],

    CASE sc-status WHEN 406 THEN 1 ELSE 0 END AS [_406],

    CASE sc-status WHEN 500 THEN 1 ELSE 0 END AS [_500],

    CASE sc-status WHEN 501 THEN 1 ELSE 0 END AS [_501]

 

 

FROM %logfilename%

 

GROUP BY Minute

 

ORDER BY Minute

 

You will notice that during 9:39 through 9:41 this server was not performing well.  From our earlier Logparser query the problem URI / application to focus on is “/obadapp/errorpages/error.aspx”.   

 

Now that we have identified the problem URI / application you can engage the team that created the content / application and debug this issue further.

 

Using Logparser and the samples files above you can analyze the performance of your site as you roll out new content, applications or a new Operation System and ensure that you quickly identify issues during the rollout process.

Power Parsing...some days you just need more power for your parser

Obviously the title can mean a bunch of different things, in this case we are merging two fun technologies: Microsoft PowerShell and Microsoft Log Parser.  Here in MSCOM Operations, we are in the process of introducing Microsoft PowerShell  into our environment and as a serious user of Log Parser I wondered how to get the two to interact nicely, hopefully in a way that makes each a little more valuable. 

 

The first solution was to simply call logparser.exe from the PowerShell .  This did what I expected, returned a bunch of lines to the window... not real exciting, but at least we determined that you can do it.  Looking for something better, I stumbled across the fact that PowerShell  can manipulate COM objects directly.  Great, we can use Log Parser's COM object model from PowerShell pretty much exactly like it has always been used from script. 

 

I started with some simple JScript:

 

//Create the main Log Parser Query object

var myQuery=new ActiveXObject("MSUtil.LogQuery");

 

//Create the text of the query

var szQuery =    "Select top 10 * from r:\ex07011210.log";

var recordSet=myQuery.Execute(szQuery);

 

for(; !recordSet.atEnd(); recordSet.moveNext())

{

            record=recordSet.getRecord();

            WScript.Echo(record.GetValue(0) + ","+record.GetValue(1));

}

recordSet.Close();

 

Which I ported to PowerShell :

 

$myQuery = new-object -com MSUtil.LogQuery

$szQuery = "Select top 10 * from r:\ex07011210.log";

$recordSet = $myQuery.Execute($szQuery)

for(; !$recordSet.atEnd(); $recordSet.moveNext())

{

            $record=$recordSet.getRecord();

            write-host ($record.GetValue(0) + ","+ $record.GetValue(1));

}

$recordSet.Close();

 

Great!  Easy! No Problems! Of course we really haven't added much value, and accessing $rec.getValue(0) seems so ugly. The next big breakthrough in investigating the integration was finding the PowerShell  command import-csv.  We could now have Log Parser output to CSV and then use import-csv to suck the data back into PowerShell :

 

PSv1.0 R:\> LogParser "select top 10 * into temp.csv from r:\ex07011210.log"

 

Statistics:

-----------

Elements processed: 10

Elements output:    10

Execution time:     0.05 seconds

 

PSv1.0 R:\> $recordSet = import-csv temp.csv

PSv1.0 R:\> $recordSet[0].LogFileName

r:\ex07011210.log

PSv1.0 R:\> $recordSet[0].LogRow

5

PSv1.0 R:\> $recordSet[1].LogRow

6

 

This is nifty because import-csv creates objects that have the field names as properties... VERY cool.  Of course there is the minor drawback of creating a temp.csv file (and possibly conflicting over it, leaking it, etc).  I Started thinking, "since import-csv can dynamically generate objects, I bet I can do the same", and after some internet searching I was able to produce:

 

function LPExecute([string] $query)

{

            $lp = new-object -com MSUtil.LogQuery

            $rs = $lp.Execute($query)

            $LPResult= new-object System.Management.Automation.PSObject[] 0;

            if(!$rs.atEnd())

            {

                        do {

                                    $rec = $rs.getRecord();

                                    $LPResult += new-Object System.Management.Automation.PSObject;

                                    for($i = 0; $i -lt $rs.getColumnCount();$i++)

                                    {          

                                                $LPResult[$LPResult.length-1] | add-member NoteProperty $rs.getColumnName($i) -value $rec.getValue($i);

                                    }

                                    $rs.moveNext();

                        } until ($rs.atEnd())

            }

            $rs.close();

            return $LPResult;

}

 

And after adding it to my profile I can do stuff like:

 

PSv1.0 R:\> $recordSet = LPExecute("select top 10 * from r:\ex07011210.log")

PSv1.0 R:\> $recordSet[0].LogFileName

r:\ex07011210.log

PSv1.0 R:\> $recordSet[0].LogRow

5

PSv1.0 R:\> $recordSet[1].LogRow

6

 

Without the use of a temp.csv (though I fully admit it might be a little faster to have one - I haven't bothered to find out).  The important piece of this puzzle is the use of the System.Management.Automation.PSOject, and its add-member functionality.  This is what allows us to create dynamically extensible objects (something JScript does by default).

 

To round things out (and come around full circle), I thought someone might want to walk through the record set instead of getting back the full array, so I added the functions to allow you to do it this way as well:

 

function LPExecuteOnly([string] $query)

{

            $lp = new-object -com MSUtil.LogQuery

            $rs = $lp.Execute($query)

            return $rs

}

 

function LPGetRecord ($rs)

{

            $LPR = new-Object System.Management.Automation.PSObject;

            if(!$rs.atEnd())

            {

                        $rec = $rs.getRecord();

                        for($i = 0; $i -lt $rs.getColumnCount();$i++)

                        {          

                                    $LPR | add-member NoteProperty $rs.getColumnName($i) -value $rec.getValue($i);

                        }

            }

            return $LPR;

}

 

With these functions we can do something that looks a lot like our original, but with interesting properties instead of calls to GetValue():

 

$recordSet = LPExecuteOnly("select top 10 * from r:\ex07011210.log")

for(; !$recordSet.atEnd(); $recordSet.moveNext())

{

        $record=LPGetRecord($recordSet);

        write-host ($record.LogFileName + ","+ $record.LogRow);

}

$recordSet.Close();

 

So there you have it.  Simply copy those functions into your profile and you can access Log Parser in a clean, nifty way (assuming Log Parser is registered properly). Obviously we haven't added a lot of impact over Log Parser + Script (got rid of (GetValue(#) calls), however for those of you interested in and already using Microsoft PowerShell  these could be a good functions to have in your toolbox.

 

At some point I may have to investigate connecting the two going the other direction, by creating a Log Parser custom input type using PowerShell ... assuming I find something that Log Parser can't already parse.

 

Have fun Power Parsing !

The Tasty Morsels Found In Dogfood… MSCOM OPS Top 10 Changes In IIS7.0

Dogfood….yummmm! Yes it is true, Microsoft.com has been running Beta 3 of Windows Server 2008 in production since June 12, 2007. What does that mean? 78 of 80 servers that host the www.microsoft.com website are running W2K8 Beta 3 and IIS7.0. Why only 78? We keep a couple of servers running our previous build of W2k3 and IIS6.0 as a reference. The move from W2K3 to W2K8 while very slick, is a topic for another blog. This is about the top 10 changes that we encountered in IIS7.0

 

1.     Simple, Configurable Command Line Setup

·         Install only the IIS components needed to run your site
Example:
start /w pkgmgr /l:log.etw /iu:IIS-WebServerRole;IIS-WebServer;IIS-CommonHttpFeatures;IIS-StaticContent;IIS-DefaultDocument;IIS-DirectoryBrowsing;IIS-HttpErrors;IIS-HttpRedirect;IIS-ApplicationDevelopment;IIS-ASPNET;IIS-NetFxExtensibility;IIS-ASP;IIS-ISAPIExtensions;IIS-ISAPIFilter;IIS-ServerSideIncludes;IIS-HealthAndDiagnostics;IIS-HttpLogging;IIS-LoggingLibraries;IIS-RequestMonitor;IIS-HttpTracing;IIS-Security;IIS-ClientCertificateMappingAuthentication;IIS-IISCertificateMappingAuthentication;IIS-RequestFiltering;IIS-IPSecurity;IIS-Performance;IIS-HttpCompressionStatic;IIS-HttpCompressionDynamic;IIS-WebServerManagementTools;IIS-ManagementConsole;IIS-ManagementScriptingTools;IIS-ManagementService;IIS-IIS6ManagementCompatibility;IIS-Metabase;IIS-WMICompatibility;IIS-LegacyScripts;IIS-LegacySnapIn;WAS-WindowsActivationService;WAS-ProcessModel;WAS-NetFxEnvironment;WAS-ConfigurationAPI

2.     Great Compatibility Story

·         Most (99%+) ASP and ASP.NET applications just worked.

ü  One application encountered breaking change

ü  Handful of applications required config migration to run in Integrated
(We have about 260 applications running on www.microsoft.com as defined by IIS, there are thousands of pages of code that could have broken but didn’t.)

·         Integrated Pipeline is the new unified request processing pipeline. Benefits include:

ü  Allowing services provided by both native and managed modules to apply to all requests, regardless of handler. For example, managed Forms Authentication can be used for all content, including ASP pages, CGIs, and static files.

ü  Empowering ASP.NET components to provide functionality that was previously unavailable to them due to their placement in the server pipeline. For example, a managed module providing request rewriting functionality can rewrite the request prior to any server processing, including authentication, takes place.

ü  A single place to implement, configure, monitor and support server features. For example, single module and handler mapping configuration, single custom errors configuration, single url authorization configuration.

·         Classic ASP mode allows for easy app migration

ü  ASP.NET Setup provides a “Classic .NET AppPool”

ü  For more information on check out the article ASP.Net Integration With IIS7

·         Use AppCmd to migrate apps to Integrated mode

ü  %windir%\system32\inetsrv\APPCMD.EXE migrate config <Application Path>

ü  For more information about AppCmd.exe see Getting Started With AppCmd.exe

·         IIS 6.0 Metabase compatibility layer

ü  Allows you the run old ADSI scripts

ü  IIS6.0 Metabase Compatibility module must be installed

3.     No More Metabase! 

·         Clean clear-text schema

·         IIS settings stored in XML configuration file (applicationHost.config)

ü  Metabase exists for SMTP/NNTP/FTP only

·         Site-wide changes made easily

ü  Update central applicationHost.config and copy to all web servers

ü  Replaces our bulky ADSI based script solution for metabase changes

·         Microsoft.com considerations

ü  Careful copying to production servers under load:
(Know Thy Environment! When you push out a new applicationHost.config those affected worker process need to reload the new configuration. It comes down to the scope of the change. For example, if you are making a global change that that affectes all the worker processes, and you are heavily dependent on caching then you could cause some grief in your environment as those new configurations are reloaded by the worker processes.)

4.     Centralized Configuration

·         applicationHost.config stored on UNC share

·         Allows us to copy to two (maybe four) servers rather than 80

ü  Potential gotcha - managing password changes for account used to connect to config store
(This is because that currently you cannot use the UNC share that is running under the Network service, which we use heavily. It currently requires a domain account, which our security policy mandates a periodic password change.)

5.     Delegated Configuration

·         Admin can now delegate IIS settings to application owner

·         Settings defined in web.config file in application directory

·         Example of setting to delegate include:

ü     System.webServer section of applicationHost.config

ü     Caching,  defaultDocument, httpErrors, security

6.     AppCmd and Other New Management Options

·         Managing via the UI

ü  New modular, task-based look and feel

ü  Moving away from the right-click/properties paradigm

·         Managing via the Command Line

ü  AppCmd

§  Command line utility which replaces adsutil.vbs, iisapp.vbs, and others

§  Allows command line management of sites, applications, vdirs, apppools, modules, tracing, and more

ü  Powershell

§  IIS community creating IIS-specific Powershell cmdlets

·         MSCOM Considerations

ü  AppCmd limitations – no remote

ü  No IIS provider for Powershell

 

7.     Failed Request Tracing

·         Buffers the trace events for requests and flushes them to disk if they meet your failure criteria

·         Captures trace data while you’re sleeping

·         Very little perf impact when targeting failing requests

·         Quick test: Enabling tracing for all file extensions and errors results in approx 5% fewer requests/sec at full stress load (please don’t do this in production)

·         View Currently Executing Requests via AppCmd

ü  appcmd list requests (for all request)

ü  appcmd list requests /apppool.name:DefaultAppPool

REQUEST "3e00000080012675" (url:GET /casestudies/casestudy.aspx?casestudyid=201269, time:2954 msec, client:127.0.0.1)

·         New Task Scheduler

ü  Trigger tasks on events

8.     Request Filtering

·         No more URLScan

·         </requestFiltering> settings in applicationHost.config

·         Gotcha for Microsoft.com: If filename includes “+” then allowDoubleEscaping must be set to “true”

ü  <requestFiltering allowDoubleEscaping="true">

·         Allow or disallow specific file extensions and verbs

ü  <add fileExtension=".exe" allowed="false" />

·         DenyURLSequences

ü  <add sequence="./" />

ü  <add sequence="/." />

·         RequestLimits

ü  maxAllowedContentLength="1000000“

ü  maxUrl="260“

ü  maxQueryString="2048"

9.     UNC Content

·         Simplified content synchronization

·         Reduced H/W footprint (potentially less cost)

ü  Common industry pain point

10.  Output Caching of Dynamic Content

·         Fewer off-box calls to backend dependencies

·         Significant performance gains

·         Simple WCAT (Web Capacity Analysis Tool) Stress Test against www.microsoft.com/en/us/default.aspx
Not appropriate for all applications (e.g. not effective for those with very personalized output)

 

Well that is our Top 10. We are making new discoveries every week and are looking forward to the next builds that we can update to. We are filing bugs when we find them and will continue to push information to the product teams and we will try and keep this blog updated as new, juicy morsels are discovered in the Dogfood.

Using SQL 2005 Snapshots As A Rollback Procedure

One of the great new features of SQL 2005 is database snapshots.  With one simple transact SQL statement, you can provide your users with a read-only, static copy of their data.  For example, you can take a snapshot on the last day of the month and users can do month-end reporting without worrying about changes to the underlying data.  That’s pretty powerful.  But did you know that snapshots can also be used to recover to a point in time?  That is the subject of this article – how to use database snapshots to significantly reduce your recovery time in certain emergency situations.

As DBA’s, we spend a lot of time worrying about how to protect our data from disaster.  There are really 2 goals here:

1.     Amount of acceptable data loss.

2.     Amount of acceptable downtime.

On the higher end of acceptable data loss we have full and differential backups.  On the low end we have log backups, log shipping and finally mirroring where the potential loss is mere seconds.  Pretty cut and dried. 

But what about the down TIME involved?  With mirroring you can fail over to a hot standby rapidly and automatically, thus achieving the smallest possible recovery time.  This protects your data against physical disaster but does nothing to protect you from data corruption, because the mirror simply replicates the data “as is”. 

To roll back bad data, you’re back to good old full, differential and log backups.  For small databases this can be pretty simple.  But it is often meticulous work, and if you restore one too many logs, you’ll have to start all over.  For large databases, the recovery time can be very long.  If you’re in this business long enough you’ll experience losing the online backups and have to restore from tape.  I’ve seen situations where this took 1-2 weeks!

This is where snapshots can really help.  Since the database can be recovered DIRECTLY from a snapshot, without going through the full, differential and log backups, this gives us a way to be very granular with point-in-time recoveries.  And the recovery is almost instantaneous.  So changes can be rolled back with almost no downtime.

Here is how we use it in MSCOM.  Let’s say I’m doing an application upgrade.  Before starting, I create a snapshot.  Now something goes wrong.  The deployment doesn't pass "smoke" testing, for what ever reason you can imagine -- and we decide to roll back and start again.  I just recover from the snapshot, and in seconds all the changes are gone.   It’s like having an undo file, or running the whole deployment in a transaction.  And it works for both DDL and DML changes.

I think you can see how powerful this can be.  We recommend making this a regular part of your internal change controls.  Having said that, there are a few points to consider:

• Snapshots do not back up your full text indexes, so don’t stop doing database backups. 
• Recovering from a snapshot breaks the log “chain”, so follow that up with a new full backup. 
• Recovering from a snapshot (like any other restore) breaks mirroring and transactional replication.  So plan on breaking the mirror beforehand and taking new replication snapshots afterwards. 
• Recovering from a snapshot will undo everything, so take steps to stop taking changes from your users or point them to a redundant server. 

Now for some sample code to get you started:

1. Run this in the database you wish to snapshot.   If you uncomment the “exec” statement it will print a script and then create the snapshot in one step:

set nocount on

declare @sql_string varchar (4000)

set @sql_string = 'Create Database ' + db_name() +  '_Recovery_SS ON '

 

select @sql_string = @sql_string +

'(Name = ''' + name + '''' + ', FILENAME = ''' +

substring(filename, 1, len(filename)-3) + 'ss''),'

from sys.sysfiles

where groupid <> 0

 

select @sql_string = substring(@sql_string, 1, len(@sql_string)-1) + ' AS SNAPSHOT OF ' + db_name()

 

print @sql_string

 

-- exec (@sql_string) 

2. Drop the snapshot once you're sure you don’t need to roll back:

drop database <db-name_Recovery_SS>

 

 

3. Recover changes, Undo:

RESTORE DATABASE <db-name> from

DATABASE_SNAPSHOT = <db-name_Recovery_SS>'

 

Here’s a test I did.  Try it for yourself.  First create a snapshot.  Then in your original database, create a table, add 1 million rows, run sp_spaceused  to prove they exist.  Then restore from the snapshot and run sp_spaceused again to prove both schema changes and data are rolled back.  On my system, the restore took 5 seconds:

create table Lots_Inserts (

Recno   int identity,

Time     datetime,

Data     int

)

go

 

set nocount on

go

 

declare @counter int

set @counter = 0

while @counter < 1000000

begin

            insert Lots_Inserts (Time, Data)

            values ( getdate(), DATEPART(ms, GETDATE()) )

            select @counter = @counter +1

end

go

-- Took 3 minutes, 30 seconds

 

sp_spaceused lots_inserts

Release Management: Building the Bridge

In the last post from the release management team, we talked about the strategic topic of developing the philosophy of RM being a bridge, and not a barrier, to the deployment and support of applications into the production environment. One of the items listed as necessary to achieving that goal was having a solid understanding of when Operations needs to be engaged in a project. For this post, I wanted to provide a specific example of how we’re delivering on that in the MSCOM world.

 

As the RM team began to evangelize the updated release criteria checklist, and in particular to identify the right roles with respect to responsible, accountable, consulted, and informed (RACI) parties as they related to each release criteria deliverable, we determined in working with the MSCOM Support Services team that the checklist did not reference them appropriately. As a result, many of the project teams did not know to engage those teams early on in the project, and the Support Services teams didn’t have enough advance notice of those applications to be able to provide the necessary level of support. Once the issue was identified, the Support Services leads were engaged and worked with RM to identify the items for which they needed to be aware in order to provide support in production. We also teamed with them to create a mapping of the appropriate Support Service contact for each application. Armed with this information, we can now work with project PMs to ensure that their Support Service contact is engaged early on in the project, and can help to coordinate the delivery of support information in a timely manner, giving both sides the opportunity to set, and meet, support expectations.

 

As I said in the last post, the MSCOM RM team still has a ways to go to be the bridge we really want to be, but using the release criteria checklist as a single touchpoint for all of the teams involved in developing, testing, and supporting an application gives us the opportunity to consistently and systematically build the bridges between those teams that makes it easier for everyone to do their job well, and to provide the highest availability and world class support of our applications in the production environment

Posted by MSCOM | 0 Comments

Filed under: Release Management

MSCOM At TedchEd 2007 in Orlando...On Demand Viewing of Breakout Session

If you were unabel to make it to Orlando and were jsut dying to catch the MSCOM Operations Presentation

Microsoft.com: Employing Windows Server 2008 and Internet Information Services 7 As A Next Generation Platform for SaaS Web Applications, here is another chance for you to view this session. As the abstract states:

 

Get the latest technology and architectural information about Microsoft’s corporate Web site and many other hosted business services such as Microsoft Update, MSDN, TechNet, and Communities. The system engineers share how they are leveraging the next generation Web platform stack including Windows Server 2008/Internet Information Services 7, SQL Server 2005, SharePoint/Microsoft Office SharePoint Services, and System Center Operations Manager.