Kevin Holman's System Center Blog

Posts in this blog are provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified in the Terms of UseAre you interested in having a dedicated engineer that will be your Mic

Using the “sync time” property in workflows and overrides

Using the “sync time” property in workflows and overrides

  • Comments 10
  • Likes

 

We use the scheduler datasource in SCOM for workflows for all kinds of purposes.  Most workflows have some concept of a schedule in the datasource to tell how often to inspect for something, like a service running, or to look at perfmon data, or to run a script.

The most common property would be the Interval, which is how often we want the workflow to run.

SyncTime is another common property.  This is used to tell the workflow to run “synchronized” on a very specific point in time.  From:  http://msdn.microsoft.com/en-us/library/jj130319.aspx

“Specifying a synchronization time forces the module to output a data item at the specified time, and it executes on that frequency based on that synchronization point. “

If we do not provide a sync time, and only provide an interval (say, 5 minutes for example), the behavior of the workflow will be to run the workflow IMMEDIATELY upon agent startup (or as soon as the agent is capable of running workflows) and then on a 5 minute interval schedule from that point forward.  This is generally a good thing, as it will randomize when agents are running workflows on common intervals, assuming all your agents don’t start up at the same time.  This has two good effects:  It controls us from flooding the network sending data from the agents to MS in bursts, all at the same time, and it keeps us from overloading a Hypervisor host, having lots of monitored servers running on it.  If all agents ran a script that consumed 50% CPU at the exact same time, it could exhaust the resources of the host during script executions.

If we DO provide a sync time for the workflow, then the behavior is to NOT run the workflow immediately on agent startup.  Instead, we will run the workflow based on the set interval, and start running it as soon as possible where it will run “synchronized” to the given sync time.  When the workflow or agent is initialized, the full schedule is calculated, and the module is output at the next closest time. This might be confusing, so let me provide some examples:

Scenario 1: Daily execution  I have a script based rule, where the Interval is set to 86400 seconds (once per day) and the sync time is set to 05:15.  Sync time is based on a 24 hour clock, so 05:15 is 5:15am, local agent time.  Lets say the monitored agent gets rebooted at 11:03 PM.  The workflow will wait, to run on its defined interval, and execute at exactly 5:15 am.

Scenario 2: Infrequent execution  I have a script based rule, where the Interval is set to every 4 hours and the sync time is set to 05:15.  Sync time is based on a 24 hour clock, so 05:15 is 5:15am, local agent time.  Lets say the monitored agent gets rebooted at 11:03 PM.  The workflow will wait, to run on its defined interval, and execute based on its interval, that also must fall on 5:15 AM.  Therefore, since the agent started up at 11:03 PM, the next possible interval on the 05:15 sync time would be 1:15 AM.  This will be the first execution of the script, and then it will continue repeating every 4 hours, staying in sync with always having one of the executions running at 05:15.

Scenario 3: Moderate frequency  I have a script based rule, where the Interval is set to every 15 minutes and the sync time is set to 05:15.  Sync time is based on a 24 hour clock, so 05:15 is 5:15am, local agent time.  Lets say the monitored agent gets rebooted at 11:03 PM.  The workflow will wait, to run on its defined interval, and execute based on its interval, that also must fall on 5:15 AM.  Therefore, since the agent started up at 11:03 PM, the next possible interval on the 05:15 sync time would be 11:15 PM.  This will be the first execution of the script, and then it will continue repeating every 15 minutes, staying in sync with always having one of the executions running at 05:15.

Scenario 4: Very frequent  I have a script based rule, where the Interval is set to every 1 minute and the sync time is set to 05:15.  Sync time is based on a 24 hour clock, so 05:15 is 5:15am, local agent time.  Lets say the monitored agent gets rebooted at 11:03 PM.  The workflow will wait, to run on its defined interval, and execute based on its interval, that also must fall on 5:15 AM.  Therefore, since the agent started up at 11:03 PM, the next possible interval on the 05:15 sync time would be 11:04 PM.  This will be the first execution of the script (assuming the agent has fully started and is ready to run workflows), and then it will continue repeating every 1 minute, staying in sync with always having one of the executions running at 05:15.  As you can see, there is VERY LITTLE value to ever using a sync time with a very frequent execution.

What we can see, is that synch time bring the most value, when we want to control when workflows execute at a specific point in time.  This works really well, if I have a MP with very expensive discovery scripts, and I need to ensure they don’t all run at the same time, or we wish to control which discoveries runs first as subsequent discoveries might depend on previous discovered data.  It can also be useful if we DON’T want the agent to run the workflow upon agent startup.  Beyond that, you should take care in leveraging any sync time value, as you can accidentally create unnecessary load on the network, or virtualization hosts.

 

Raphael Burri mentioned a potentially better solution for controlling workflow initialization and intervals, using the “SpreadInitializationOverInterval” element in the Scheduler datasource.  This property is shown in the schema here:  http://msdn.microsoft.com/en-us/library/ee692976.aspx and an example in XML here:  http://msdn.microsoft.com/en-us/library/hh442320.aspx   It is not well documented however.  The best explanation of how it works and how to use it is written by Kris Bash, here:  http://operatingquadrant.com/2009/10/12/scom-distributing-workflows-with-the-spreadinitializationoverinterval-scheduler-parameter/

Essentially, using “SpreadInitializationOverInterval” will keep your workflow from executing immediately after agent startup, and will initialize randomly over a set interval of time that you provide.  This will help randomize execution across multiple agents, and will keep all agents from running your workflow immediately upon startup.

Comments
  • SyncTime provides almost no value, in my opinion. The only time it may actually help reduce load is when a single server provides several different role - this is where a discover sync time may help to conserve cpu resource minimally. When it comes to virtualized agent restarts, sync time doesn't provide value there. Whether or not the host system or any agents restart on that host, sync time will have the same impact on cpu as queuing the workflows to run when possible. In fact, I would argue that sync time would have a greater impact in the virtualized world than not having sync time, because it's a rare occasion to have a ESX or Hyper-V host go down unexpectedly. Even still, this would mean the hosted agents will still run on a theoretical sync time - until any one agent is restarted, and then you have a real spread of workflows being triggered at different times. I believe sync time provides value when your pack has a discovery race issue, for whatever reason. This is a very uncommon scenario, but I have see certain services and distributed applications where it wasn't exactly necessary to have a hosting relationships or seed class for a particular discovery to run. In this case, it is recommended to use sync time, in order to force a cascaded discovery. Good to see you blogging again, and congrats on the twins! Jonathan

  • I will clarify on one thing - if you have multiple scripts in your pack, it is a good idea to use sync time. 5 scripts, for example, would be sync time of 00:01, 00:02, 00:03, 00:04, 00:05. If a script runs for more than a minute, then there are other problems :)

  • Using "SpreadInitializationOverInterval" instead of "SyncTime" on the schedulers is another option to assure that the workflows do not run immediately after the agent starts. To run a workflow hourly, while letting the agent place the initial run randomly during the first hour, the following configuration example may be used: <DataSource ID="Scheduler" TypeID="System!System.Scheduler"> <Scheduler> <SimpleReccuringSchedule> <Interval>3600</Interval> <SpreadInitializationOverInterval>3600</SpreadInitializationOverInterval> </SimpleReccuringSchedule> <ExcludeDates /> </Scheduler> </DataSource> Obviously it is either SyncTime or SpreadInitializationOverInterval. And it sometimes means more work since many library MP datasources have wired parameters for SyncTime overrides but not usually SpreadInitializationOverInterval overrides. Raphael

  • wow ! Good to know about this internal SCOM control which can help minimize impact on infrastructure and assets in the context of resource choke.Thanks much.

  • Jonathan - I agree on all counts. Raphael - that's interesting - I might add some details around that option to this post.

  • Using sync times with infrequent/daily intervals can lead to situations where it just never runs. e.g. due to a planned reboot at that specific time.

  • Raphael - I can't even find SpreadInitializationOverInterval mention on MSDN for the scheduler module, other than this page here: http://msdn.microsoft.com/en-us/library/ee692976.aspx. I see Daniele Grandini ad actually wrote about this a while back for SCOM 2007 R2, and I must have missed it - but he offers no MSDN references. Any MSDN resources regarding this configuration element for System.Scheduler? -Jonathan

  • Jonathan: Indeed: It isn't very clearly written up. However; you can find the reference in the schema definition of the PublicSchedulerType on http://msdn.microsoft.com/en-us/library/jj130319.aspx There you see that you actually have a choice between SpreadInitializationOverInterval and SyncTime.

  • Rapael - yes, I see it now :)

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Search Blogs