One of the cool things about Orchestrator is that it can automatically handle multiple-instance threading of runbooks or queue them up depending on your need. For instance, you may have a runbook that processes new users to be added to Active Directory, and for a large company, you might have several concurrent requests going, so you want the runbook to run with multiple instances. Or, you make have a process that makes transactional-type changes to a system, and you want to ensure that when multiple users request changes, they are queued and run serially instead of potentially conflicting with each other. This runbook behavior is easily controlled via the “Job Concurrency” tab in the runbook’s properties dialog:
By default, it’s set to 1, which means a runbook will run one instance and queue all other requests. If you want to increase that number to allow multiple concurrent instances, you just change the number to something higher. Easy enough, right?
But what if you have something that needs to run as a single instance, but you want to ignore all other requests until the current job is done? An example of this might be an automatic scale out of a cloud service. Let’s say you’re monitoring a service using Operations Manager, and a performance alert triggers a runbook to start scaling out the service. That scale out process might take some time, and it might also take a while after it’s done to have the performance numbers average down into an acceptable range (we call this the “cool down” period). For whatever reason, the alert might have fired again telling Orchestrator to scale up the service. Well, if you already have a runbook in progress to scale up the service, you don’t want to accept any more requests, and you certainly don’t want to just queue them up. You know that you just want to ignore or drop any requests until after this runbook is completed. So how do you do that?
Note: As with any example, I’m showing you a relatively simplistic view, and in the real world you’d probably have some more detailed logic. However, I think that with this basic example, you can get the idea and transform it as you need to in your own environment.
First, create a new runbook. Now, if we want to avoid queuing up requests for a runbook job, the first thing we’ll need to do is increase the number of simultaneous jobs. In the dialog shown above, increase the number to something like 10 (just in case we get a rapid-fire succession of requests, we can handle them instead of queuing).
Next, drag and drop an “Initialize Data” activity and a “Query Database” activity into the runbook and link them together.
Basically what we’ll do here is query the Orchestrator database for any active jobs (TimeEnded = NULL) that contain the activity ID of the “Initialize Data” activity in this runbook. We kind of have to do that in a round-about way though. We don’t have access to the runbook’s ID as it’s running, so we have to infer that from the activity ID by using a query that joins two tables as shown below:
You might notice there’s some fancy string replacement going on in the query. The problem is that the Activity ID property from every activity is formatted as a GUID with curly braces, like this:
However, the data inside the database is formatted without the braces, so the ID has to be reformatted to strip off the braces. Luckily, SQL provides some simple ways to format strings, so we don’t have to put another activity in between these two in order to reformat the string.
I’ll add a few more activities to the runbook so that it has something to do.
I put a “sleep” activity in there to keep the job running for 60 seconds while I activate more instances to test the functionality. I also added a “Send Platform Event” activity to let me know when I’ve intentionally dropped new jobs. I’ve also modified the link conditions from the Query Database activity to route the process flow accordingly. For the link to the Sleep 60 activity (and to the operational part of a “real” runbook), I set the link condition to go if there is only one erunning instance of this runbook (which would include this one).
For the Send Platform Event activity link, I set it to go that route if there are more than one current instance of this runbook.
After checking in the runbook, I switch over to the Orchestration console and browse to that runbook. Then over in the Actions pane, I click “Start Runbook” and then the Start button, repeating the same actions a few more times. After refreshing the view, I can see that my original instance is still running, but the other two I started have already finished.
Clicking on “Events” in the left pane, I see that the two runbooks that finished early generated the events they were supposed to.
Down at the bottom of the page I see a description of the message:
Now personally, if I stop a runbook before its intended full operation is complete, I don’t necessarily want the runbook to show up as completed successfully. I might want it to show up as a Warning so that it draws my attention more than an event notification might (of course every circumstance is different). So in that case I might create a custom activity that does nothing but throws a warning from the runbook, causing it to stop with a warning status. I’ll give an example of that code in a future post.
did you ever get around to blogging about forcing a runbook to display a warning or error? would be good to hear how you achieve this..