Microsoft.com Operations

We are the operations team that runs the Microsoft.com sites.

Scripting Patch Management of Enterprise Web Clusters on Microsoft.com

Scripting Patch Management of Enterprise Web Clusters on Microsoft.com

  • Comments 3
  • Likes

One of the most common questions I am asked when meeting with customers is, How does MSCOM patch their Enterprise Web servers? I will cover exactly what our approach is, and give you a little background into some of the challenges we are faced with when patching our production Web servers.  Also, I have included a sample script we currently use on the Microsoft.com Web team to patch our production Web servers. 

 

To give you an overview of our environment, Microsoft.com is comprised of over 120 Web properties hosted on over 1000 Web servers.   Our Web team supports sites that range from our corporate business presence of www.microsoft.com, to Developer/ITPro sites such as MSDN, and TechNet, including Download distribution sites such as Windows Update, and Download.microsoft.com.  To maintain a high level of availability, each of our sites is comprised of multiple web clusters in multiple datacenters each running NLB.   As I mentioned high availability is extremely important to our customers, and is invaluable in showcasing the Microsoft products and services we utilize here on Microsoft.com.  Unfortunately, we are faced with the same challenges of having to apply hotfixes and service packs while making sure there are no service interruptions. This is a tough challenge.  For example, on the Download.microsoft.com site it can take up to six hours to drain all the active connection from each server.   We have architected each of these sites with added server capacity to ensure we can handle peak capacity and also handle any unplanned or planned outages such as a patch event. 

 

Whether we are configuring new Web servers, maintaining configuration control or in this case patching our servers, we leverage admin scripting.  Every engineer should have a working knowledge of the basics of a scripting language. This skill set is invaluable in managing enterprise Web servers.  Scripting our deployments allows us the flexibility to perform controlled patching.  The advantages of scripting your patch deployments are zero service interruption, and the ability to have coordinated customer deployments.  For the advantages listed above we leverage the use of admin scripting during our deployments.

 

Listed below is a sample vbscript we have recently used to patch our Web servers. Copy the script below and save it as SrvPatch.vbs. You will need to create a text file named ServerList.txt containing all the servers you want patched.  To execute the script from the command line type “SrvPatch.vbs ServerList.txt”.  You will then be prompted to type in your password.  The basic flow is of this script is to drain the live connections, run the appropriate patch and associated switches, place the server back into rotation, and loop thru the next server in your ServerList file.

 

‘ Script example: (Save the below script as SrvPatch.vbs)

‘ Script requires the SysInternals tool PSEXEC.  http://www.sysinternals.com/Utilities/PsExec.html

‘ Syntax to execute script “SrvPatch.vbs ServerList.txt”

Dim oFSO

Set oFSO = CreateObject("Scripting.FileSystemObject")

Dim oFile

Dim sServer

Dim sPass

dim sSystemRoot

Set WSHShell = WScript.CreateObject("WScript.Shell")

 

If Wscript.arguments.count > 1 Then

                Wscript.Echo "Syntax:  SrvPatch.vbs <Server List> [Password]"

                WScript.Echo "Example: SrvPatch.vbs ServerList.txt Password"

                WScript.Quit (0)

END If

 

set oFile = oFSO.OpenTextFile(WScript.arguments(0))

 

'sPass = WScript.arguments(1)

Wscript.Echo "Please enter password:"

sPass = wscript.stdin.readline

 

sSystemRoot = wshShell.ExpandEnvironmentStrings("%systemroot%")

 

Do while oFile.AtEndOfStream =false

                sServer = oFile.ReadLine

                GetInfo sServer, CountConnections

 

              Draining traffic from server         

                Set BeginDrain = CreateObject("wscript.Shell")

                BeginDrain.Run "psexec \\" & sServer & " wlbs drain all",0,"true"

 

                Do While CountConnections > 20

                                GetInfo sServer, CountConnections

                                wscript.echo sServer & ": " & CountConnections

                                wscript.echo "Sleeping..Draining.."

                                WScript.Sleep 10000

                Loop

                                Set WLBSSuspend = CreateObject("wscript.Shell")

                                WLBSSuspend.run "psexec \\" & sServer & " wlbs suspend",0,"true"

 

                                wscript.echo sServer & ": Drained..WLBS Suspend..Patching to Begin"

 

                                Launching your patch or executable file          

Set LaunchPatch = CreateObject("wscript.Shell")

                                LaunchPatch.run sSystemRoot & "\system32\cmd.exe /c echo " &  sPass & "| \\Server\share\patchfile.exe", 1,"true"

 

                                wscript.echo sServer & ": Patched......."

 

                                WScript.Sleep 10000

 

                               Adding server back into rotation                          

                                Set WLBSResumeIntoRotation = CreateObject("wscript.Shell")

                                WLBSResumeIntoRotation.run "psexec \\" & sServer & " wlbs Resume",0,"true"

                                wscript.echo sServer & ": Resumed"

                               

                                Set WLBSStartIntoRotation = CreateObject("wscript.Shell")

                                WLBSStartIntoRotation.run"psexec \\" & sServer & " wlbs Start",0,"true"

                                wscript.echo sServer & ": Started and taking traffic"

 

                                wscript.echo "------------------------------------"

               

Loop

oFile.Close

 

Function GetInfo(Computer, CountConnections)

                REM On Error Resume Next

                strComputer = Computer

                Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")

                Set colItems = objWMIService.ExecQuery("Select * from Win32_PerfFormattedData_Tcpip_TCPV4",,48)

 

               

                                For Each objItem in colItems

                                                CountConnections = objItem.ConnectionsEstablished

                                               

                                Next

                                                               

rem wscript.echo "Connections at: " & CountConnections

End Function

Comments
  • The SysInternals licensing has been updated on the SysInternals website.
    The new licensing is something...

  • While the MSCom Operations team manages 1800+ production servers, hosting over 120 different web properties...

  • While the MSCom Operations team manages 1800+ production servers, hosting over 120 different web properties

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment