Avoid Overload by Scaling and Queuing PowerShell Background Jobs

Avoid Overload by Scaling and Queuing PowerShell Background Jobs

  • Comments 6
  • Likes

Summary: Use scaling and queuing Windows PowerShell background jobs to avoid system overload.

 

Microsoft Scripting Guy Ed Wilson here. Today I am proud to announce the return of Boe Prox to the blog.

 

Photo of Boe Prox

Boe Prox is currently a senior systems administrator with BAE Systems. He has been in the IT industry since 2003 and has spent the past three years working with VBScript and Windows PowerShell. Boe looks to script whatever he can, whenever he can. He is also a moderator on the Hey, Scripting Guy! Forum. You can check out his blog at http://boeprox.wordpress.com and also see his current project, the WSUS Administrator module, published on CodePlex.

Take it away Boe!

 

Ed asked us (the Hey, Scripting Guys! Forum moderators) what one of our favorite Windows PowerShell tricks is. After a little bit of time thinking, I decided that scaling and queuing background jobs to accomplish a task is one of my favorite tricks (and something that I have used quite a bit in the past few months).

The Windows PowerShell team blogged (link to blog post) about this topic earlier this year and explained what needs to be done in order to configure a script or function to process a number of items within a certain threshold.

I have used this technique (modified for my own use) on several occasions and it has worked like a champ each time. In fact, this is one of the key components in my project, PoshPAIG. By using this, I was able to eliminate most of the UI freeze-up that occurs when you attempt to run a Windows PowerShell command under the same thread as the UI.

Some uses for this that I have personally used are for performing a data migration where I only want to have so many copy jobs running at a time. As one job finishes up, another job begins to copy the next set of folders that I have queued. The example that I will show you uses this technique to perform a monitored reboot of a number of systems with a specific threshold of how many systems can be rebooted at a time. In this case, I will track five systems at a time and a warning will appear if a machine does not come up within five minutes of being rebooted. The script I am using is available on the TechNet Script Gallery, and I will go through it in chunks to show what is going on.

I start by running my script, named Restart-ComputerJob:

.\Restart-ComputerJob –MaxJobs  5 –InputObject (Get-Content hosts.txt)

#Define report

$Data = @()

$Start = Get-Date

#Queue the items up

$queue = [System.Collections.Queue]::Synchronized( (New-Object System.Collections.Queue) )

foreach($item in $InputObject) {

    Write-Verbose "Adding $item to queue"

    $queue.Enqueue($item)

}

 

Here, I am defining an empty collection that will be used later to store the data from each job that has finished. I have my collection of computers defined from the $InputObject variable. Each item is added to the $Queue, which was created using the System.Collections.Queue class. Using the Synchronized method allows only one job to access the queue at a time:

# Start up to the max number of concurrent jobs

# Each job will take care of running the rest

For( $i = 0; $i -lt $MaxJobs; $i++ ) {

    Restart-ServerFromQueue

}

Now that we have the collection queued, we can now begin creating jobs to start rebooting the systems in the Restart-ServerFromQueue function. I use a For statement with the $MaxJobs variable that is defined by either the user, or sticks with the default value of five to limit the number of jobs that will be run at any given time.

Function  Global:Restart-ServerFromQueue {

    $server = $queue.Dequeue()

    $j = Start-Job -Name $server -ScriptBlock {

            param($server,$location)

            $i=0

            If (Test-Connection -Computer $server -count 1 -Quiet) {

                Try {

                    Restart-Computer -ComputerName $server -Force -ea stop

                    Do {

                        Start-Sleep -Seconds 2

                        Write-Verbose "Waiting for $server to shutdown..."

                        }

                    While ((Test-Connection -ComputerName $server -Count 1 -Quiet))  

                    Do {

                        Start-Sleep -Seconds 5

                        $i++      

                        Write-Verbose "$server down...$($i)"

                        #5 minute threshold (5*60)

                        If($i -eq 60) {

                            Write-Warning "$server did not come back online from reboot!"

                            Write-Output $False

                            }

                        }

                    While (-NOT(Test-Connection -ComputerName $server -Count 1 -Quiet))

                    Write-Verbose "$Server is back up"

                    Write-Output $True

                } Catch {

                    Write-Warning "$($Error[0])"

                    Write-Output $False

                }

            } Else {

                Write-Output $False

            }

    } -ArgumentList $server

 

In the beginning part of the Restart-ServerFromQueue function, I first get the system name by using the $Queue.dequeue() method and saving it to the $Server variable that removes the system from the queue. From there, I create the new job and save the job object to a variable that will be used later. The job performs a reboot of the system and then goes into a monitoring phase until the system comes back online. If it doesn’t come back online after five minutes, the system is deemed Offline and a Boolean value of $False is returned; otherwise, if the system is online, $True is returned.

    Register-ObjectEvent -InputObject $j -EventName StateChanged -Action {

        #Set verbose to continue to see the output on the screen

        $VerbosePreference = 'continue'

        $serverupdate = $eventsubscriber.sourceobject.name 

        $results = Receive-Job -Job $eventsubscriber.sourceobject

        Write-Verbose "[$(Get-Date)]::Removing Job: $($eventsubscriber.sourceobject.Name)"          

        Remove-Job -Job $eventsubscriber.sourceobject

        Write-Verbose "[$(Get-Date)]::Unregistering Event: $($eventsubscriber.SourceIdentifier)"

        Unregister-Event $eventsubscriber.SourceIdentifier

        Write-Verbose "[$(Get-Date)]::Removing Event Job: $($eventsubscriber.SourceIdentifier)"

        Remove-Job -Name $eventsubscriber.SourceIdentifier

        If ($results) {

            Write-Verbose "[$(Get-Date)]::$serverupdate is online"

            $temp = "" | Select Computer, IsOnline

            $temp.computer = $serverupdate

            $temp.IsOnline = $True

            } Else {

            Write-Verbose "[$(Get-Date)]::$serverupdate is offline"

            $temp = "" | Select Computer, IsOnline

            $temp.computer = $serverupdate

            $temp.IsOnline = $False

            }

        $Global:Data += $temp

        If ($queue.count -gt 0 -OR (Get-Job)) {

            Write-Verbose "[$(Get-Date)]::Running Restart-ServerFromQueue"

            Restart-ServerFromQueue

        } ElseIf (@(Get-Job).count -eq 0) {

            $End = New-Timespan $Start (Get-Date)                   

            Write-Host "$('Completed in: {0}' -f $end)"

            Write-Host "Check the `$Data variable for report of online/offline systems"

            Remove-Variable Queue -Scope Global

            Remove-Variable Start -Scope Global

        }          

    } | Out-Null

    Write-Verbose "[$(Get-Date)]::Created Event for $($J.Name)"

}

The last piece of the function holds the event information that is used to track each job. I use the $j variable, which holds the job object for the most recently started job along with using the Register-ObjectEvent cmdlet and checking for the StateChanged status of the job. The job changing the status from Running to anything will prompt the registered event to perform the action defined in the –Action parameter. Because this parameter takes a script block, I can set up a series of commands to run to gather the results of the job and save it to a report. Also, I have added to this action block some commands to perform cleanup on both the job that finished and the associated event subscription. By default, I have $VerbosePreference set to Continue, which will display some extra messages after each job finishes up. You can set this to SilentlyContinue, if you do not wish to see these messages.

The following figure shows this in action.

Image of script in action

As you can see, most of the systems are offline. DC1 is the one server that does get rebooted and the job continued to monitor the server until it came back online. By the way, did I mention I like to use Write-Verbose? (Another nice tip is to use Write-Verbose in your code to track your script in action). Here you see where each system is added into the queue and also where the first five are added into a job while the sixth system patiently waits until the first job has completed. You can also see where each job finishes and another begins, which includes removing of job, event job, and the event itself. After the last job is finished, a message is displayed showing how long it took to complete all of the jobs and to check the $Data variable for a report of systems that are either offline or came back up after the reboot.

So there you have it. You can harness the power of background jobs and events to create a set of jobs that updates itself in the background without any user interaction. And it also frees up your console to perform other work while the jobs run in the background. I hope everyone enjoyed this article and can use this technique in their daily tasks or for some other project. Thanks again also to the Windows PowerShell team for their excellent article that helped pave the way in making this work!

 

I want to thank Boe Prox for taking the time to share this really cool Windows PowerShell tip.

I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.

Ed Wilson, Microsoft Scripting Guy

 

 

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Hi Boe,

    thanks for sharing this very well explained article with us!

    The stuff is not that easy, but as the previous HSG articles already explained,

    working with job and events is a bit of an art of it's own ...

    It makes perfectly sense to queue the jobs and try to limit parallelism to some

    reasonable degree that depends on the power of our workstation or server.

    I think that we can reuse the idea of this script as a prototype for similar job

    related tasks or we might even encapsulate it into a more general function or module

    that performs job scheduling on this basis.

    Thanks,

    Klaus.

  • Hi Klaus!

    Glad you enjoyed the article! This technique has saved me a lot of time and pain on several projects lately. I like the idea of building this into a function that can be re-used more easily. Definitely ran into some gotchas while working with the events jobs, but it was well worth the time to learn and utilize this powerful feature of PowerShell.

    Boe

  • Hey Boe,

    One question about this script. I found out it doesn't run when you run it as a script. Like explained here ...

    blogs.msdn.com/.../scaling-and-queuing-powershell-background-jobs.aspx

    Any idea how to fix this on your posted script? I am trying to dive into the whole job/event thing but it's a bit above my head for now ;)

  • Hi Marcel!

    Glad you like the blog article! As far as the script is concerned,  could you list how you are running the script and also if you are receiving any Verbose messages and list those out as well. Also, what OS are you running the script from?

    Thanks again!

    Boe

  • Hi Boe, Great article! I am having trouble with it though. First, there is a discrepancy between the downloaded script and the code in your article. In the actual script the Function is not declared globally as you do in the article. Does this make a difference? Second, when I run the script either with the function globally declared or not, it doesn’t give the same results. For example, on my Windows 7 machine it doesn’t tell me it is adding servers to the queue, it doesn’t tell me that it created any events, and there is no date/time on the verbose, “removing” and “unregistering”, comments it does give. Do you have any idea why my results would be so far apart from yours?

  • Hi Keith!

    You are correct, the Restart-ServerFromQueue function should actually be a global function so it can be called during the Action block of the object event. There were also a couple of other areas where a global variable needed to be used: queue and data. Also, is your $verbosepreference set to 'continue'? If not, then you will not see a majority of the messages about the queue and creating events. I also modified the script to allow the use of -Verbose parameter to better see what is going on with the script. I also added the datetime stamps on the verbose messages as well.

    Hope this helps!