Understanding Sequential vs. Parallel Processing of Runbook Activities

Understanding Sequential vs. Parallel Processing of Runbook Activities

  • Comments 10
  • Likes

I saw a question in the forums the other day by a person who asked how you should expect activities in a workflow to be executed when one of the activities generates multi-value data. In other words, one of my activities returns a list of things. How do the next activities react? Will they run in parallel or will they run sequentially? The answer, like so many things, is that it depends. It depends on how you have the activities in your runbook structured.

Let’s start out with the simplest case. I have a Run .Net Script activity that runs the following command:

$collection = gci C:\ | Select Name

It then publishes the variable Collection to the data bus so that subsequent activities can access the data. When this activity runs, it returns a list of files and directory names, which is returned as multi-value data (a collection or list of data items). In the simplest scenario, there is just one activity following the Run .Net Script activity. I’ll use a “Send Event Log Message” activity because I won’t have to deal with any file locking issues if I was writing to a file and activities are running in parallel.

image

Now let’s assume that the script returns a list of 10 items. This causes the next activity to run 10 times, but does it run sequentially or in parallel? The answer is that they run sequentially.

Within a single runbook, you can think of the process flow like passing a token from activity to activity. Generically, an activity can only handle one token at a time. Some activities take in one token and generate multiple tokens, as in the above example, the Initialize Data activity sends one token to the Run .Net Script activity, which generates 10 tokens from the result of the script (or it could generate just 1 token if you had the “flatten” setting checked). However, it doesn’t pass all 10 tokens off to the next activity at once, it has to pass them one at a time. When the next activity passed on the token to the activity after that (or it “falls off the end of the runbook”), it can accept another token.

So what happens when you have even more activities following the one that generates multiple values? Take the following example:

image

What happens here is not quite what you might expect from the previous description. When these activities run, the Run .Net Script activity generates 10 tokens and passes them off, one by one to the first Send Event Log Message activity as that activity processes them. The interesting thing is that the first Send Event Log Message activity doesn’t immediately pass off the token to the next activity. It holds on to all of them until it’s done receiving them from the Run .Net Script activity and after it processes the last one, then it starts sending to the next activity. So the process looks like this.

  1. Initialize Data
  2. Run .Net Script
  3. Send Event Log Message
  4. Send Event Log Message
  5. Send Event Log Message
  6. Send Event Log Message
  7. Send Event Log Message
  8. Send Event Log Message
  9. Send Event Log Message
  10. Send Event Log Message
  11. Send Event Log Message
  12. Send Event Log Message
  13. Send Event Log Message (2)
  14. Send Event Log Message (2)
  15. Send Event Log Message (2)
  16. Send Event Log Message (2)
  17. Send Event Log Message (2)
  18. Send Event Log Message (2)
  19. Send Event Log Message (2)
  20. Send Event Log Message (2)
  21. Send Event Log Message (2)
  22. Send Event Log Message (2)

As long as you understand that this is the way it works, then this isn’t a surprise to you and you can build your runbooks appropriately around this functionality. What about branching in multiple directions from an activity at the same time? Do those operate in parallel? Well, because they’re still in the same runbook, the answer is no. Take the following runbook:

image

When this runbook executes, the activities run in this order:

  1. Initialize Data
  2. Run .Net Script
  3. Send Event Log Message
  4. Send Event Log Message (2)
  5. Send Event Log Message
  6. Send Event Log Message (2)
  7. Send Event Log Message
  8. Send Event Log Message (2)
  9. Send Event Log Message
  10. Send Event Log Message (2)
  11. Send Event Log Message
  12. Send Event Log Message (2)
  13. Send Event Log Message
  14. Send Event Log Message (2)
  15. Send Event Log Message
  16. Send Event Log Message (2)
  17. Send Event Log Message
  18. Send Event Log Message (2)
  19. Send Event Log Message
  20. Send Event Log Message (2)
  21. Send Event Log Message
  22. Send Event Log Message (2)

In this instance, the Run .Net Script activity alternates the tokens it sends out to each direction in the outbound links. Ok, you might be asking what happens when you combine the two versions and have a branch structure with multiple subsequent activities? Something like this:

image

In this example, I am using Run .Net Script activities to write to the event log because I can control the source name better and provide more readable output when looking at a lot of events. So what happens when I run this? Here’s what the event log looks like:

image

As you can see, the pattern follows the original branched runbook model I showed before, alternating back and forth from branch 1 to branch 2 until it finishes sending tokens out from the main script activity. It then combines the functionality of the previous multiple following activities example in that each “A” activity (1a and 2a) collects all the tokens until there are no more being received, and then sends them to the next activity, and the branches again alternate so that activities 1b and 2b are run in alternating fashion until the entire runbook is done.

So what you can expect from this is that when you have a non-exclusive branch (also called a split) where both sides are run without conditions or where both conditions are true, you will end up running activities in an alternating manner up until they rejoin via a Junction activity.

Just to take it one step further (because I wanted to know), what about when you have multiple branches in a runbook? How does that work? Here’s an example:

image

Now look at the event log:

image

The same pattern emerges that all the “A” activities are done first, then the “B” activities are done. However, note how the second branch is handled. When it hits the second branch, “branch 1” now does two activities for every one activity “branch 2 runs”. This means that when an activity runs and passes on its token(s) to the next item in the runbook, it will actually evaluate all the outgoing link conditions and process those as a set before relinquishing command to the other branch.

Parallel Processing

So how do you get the runbook process things in parallel? Well, within a single runbook, you can’t do it. It takes a combination of a parent and child runbook, and even then, it’s not *truly* parallel, it’s *mostly* parallel. I’ll explain…What you have to do is place the activities you want to run in parallel in a separate runbook, and then use the Invoke Runbook activity to call it.

image

image

You’ll then need to do two more things. First, you need to make sure that the Invoke Runbook activity has the “Wait for Completion” box unchecked.

image

Next, you’ll need to set the properties of the child runbook to allow multiple concurrent runbooks to be executed. Right-click on the runbook tab and select Properties, and then click on the Job Concurrency tab and set the number of concurrent runbooks allowed to a higher number.

image

Note: the number you set here can still be limited to a maximum number of overall running runbooks on the Runbook Server as determined by the throttling limit. For more info on that setting, see How to Configure Runbook Throttling.

When you leave the “Wait for completion” box unchecked, it basically triggers the other runbook to start, and as soon as it does, it considers the action complete and starts the next one, regardless of where the previous one is in its execution. So while the starting of the child runbooks happens sequentially, it happens so fast that it’s nearly parallel, and because each of the child runbooks runs in its own policymodule process, it can run independently of the others, and can run in parallel. Here is the event log output from the above runbooks:

image

As you can see, every instance of the child runbook ran and created an event log entry within the same second. If that’s not parallel execution, I don’t know what is Smile

So hopefully this has given you a better understanding of how runbook activities flow and how data passes across them in a runbook. Feel free to create your own test runbooks and try out your own scenarios.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Robert, this is one of the best SCO related post in ages! Very good description of the activity behavior.

    cheers,

    Stefan

  • Thanks! I think you'll see a lot more of this in-depth stuff in the future. I have a bunch of cool topics in the queue to wite about!

  • Excellent job on this. I have a feeling I'll be comming back to this from time to time to work out more complex activity sequencing.

  • thanks robert - nice post! is it still possible to process the return value of the "child" runbook?

  • Great post.

    I think I understand Sequential processing but still have a query caused but this behaviour.

    I am using the TFS IP in connection with SCSM. I have fairly lengthy RunBooks to allow incidents to be processed from the SCSM portal, changed to SCSM Change request’s and then TFS bug or Tasks are created in TFS, Orchestrator is the glue to link this process.

    My issue; When my Runbook gets to the Create Work Item activity this is created correctly but I have activity’s after this to update the SCSM change request with for example with the TFS work item ID. But when moving to the next stage I have a long Sequential processing issue because the Create Work Item activity has 50 fields, so any activity after the Create Work Item step get sequentially process 50 times.

    How can I overcome this issue?  I have thought If I have a Stop runbook activity, I could create a parallel step to allow processing to occur once, them stop the whole RunBook. Is there a way to stop a Runbook with an activity from within the Runbook?

    Thanks.

  • This isn't working for my runbook. Any help appreciated. I created two runbooks Parent/Child as suggested, Job Concurrency on Child set to 10, wait for completion on Parent unchecked. One of my child activities includes copying files. I can't get the copy files activity to run in parallel. Here's a quick run down - Initialize Data (Enter OU)>Get Users>Invoke Child Runbook>>CMD (Take Owner of Files)>Clear DACL>xcopy files>... etc. It grabs all the users in that OU (I've tested this to ensure), starts the first users file copy but will not start the second until the first has finished. If there were 10 users in that OU then why doesn't it send all 10 users to the Child runbook simultaneously and execute the Child runbook 10 times concurrently or in parallel? Where am I going wrong?

  • Here's something I stumbled upon recently: concerning activity "Send Event Log Message" receiving data from "Get Monitor," when it receives more than one item, it doesn't execute. In my case, the runbook flows as this:

    1. Read Line [the text file contains the FQDNs of only two servers].

    2. Get Monitor [to ensure they aren't already in maintenance mode in order to proceed].

    3. Start Maintenance Mode.

    4. Run Program [to restart Web service].

    5. Get Service Status [to ensure W3SVC is running].

    6. If #5 is OK, then Stop Maintenance Mode.

    7. Get Monitor [to ensure they exited maintenance mode in order to proceed].

    8. Send Event Log Message [to post message in Application Log reporting successful Web service restart].

    The last activity doesn't even appear in the Log History as failed; it's completely missing.  Weird, but true.

    The nearly identical runbook is running for a dev server; the difference is that it lacks the Read Line activity.  This runbook kicks off at step #2 above.  So, its "Get Monitor" points to the explicit FullName of the server.  Anyone else notice this behavior?  Is there a workaround?

  • Jonathan, to overcome your problem I'd suggest placing the Create Work Item Activity into a child Runbook. The Create Work Item Activity can run the 50 required steps and return success/fail results to the parent. The parent can then determine whether or not to continue.

    You asked if there was a way to stop a Runbook from an activity within the Runbook. You can Link to any activity based on success or failure.

  • nice post and explanation. thanks.