hotfixHere’s a new Knowledge Base article we published today. This one talks about troubleshooting an issue where configuration doesn’t update in System Center Operations Manager 2007:

=====

Symptoms

You may experience one or more of the following symptoms in a System Center Operations Manager 2007 Management Group:

  • Newly installed agents display as "Not Monitored" in the Operations Console, yet existing agents are monitored.
  • One or more monitors on one or more agents may not change state when healthy or unhealthy conditions are met.
  • Agents show as being in maintenance mode in the Operations Console, yet the workflows are not actually unloaded by the System Center Management service on the monitored computer.
  • Configuration changes, new rules or monitors, or overrides are not applied to some agents.
  • The Operations Manager event log on one or more agents will display event 21026, indicating that the current configuration is still valid, even though the configuration for these agents should have been updated.
  • The file "OpsMgrConnector.Config.xml" in the management group folder under "Health Service State"\"Connector Configuration Cache" does not update for long periods of time relative to the rest of the management group on one or more agents.

In addition, the Operations Manager event log may display one or more event with an ID 29106 when the System Center Configuration Management service restarts. For example

Log Name: Operations Manager
Source: OpsMgr Config Service
Event ID: 29106
Level: Warning
Description:
The request to synchronize state for OpsMgr Health Service identified by "da4d36df-ce22-8930-e6d4-45b783e9fdb1" failed due to the following exception "System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

Log Name: Operations Manager
Source: OpsMgr Config Service
Event ID: 29106
Level: Warning
Description:
The request to synchronize state for OpsMgr Health Service identified by "fc1c815b-c0c4-242d-ae27-30db4ef99b54" failed due to the following exception "Microsoft.EnterpriseManagement.Common.DataItemDoesNotExistException: TypedManagedEntityId = 'ac8f3d08-ee2a-ae21-0e46-19c3da794183' is deleted.

Collecting ETL logs against the Configuration Service at INF level might reveal lines similar to that below:

3326 [ConfigurationChangeSetProvider.UpdateQueryTimestampFromResults] [configurationchangesetprovider_cs595]( 000000000343A92F )Timestamp = 04/11/2074 08:57:09.
3327 [DatabaseAccessor.NotifyOnChanges] [databasenotification_cs329]( 0000000002E4BD4E )Firing change notification.
3328 [ConfigurationEngine.DatabaseHelper.OnConfigurationChange] [configurationengine_cs499]( 00000000023546E1 )IsIncremental=True, NumberOfChanges=0
3329 [StateManager.CollectDirty] [statemanager_cs39]( 00000000035D75A8 )State=274cda45-6031-c0e2-3659-0072251f5655 is dirty
< large number of additional GUIDS >
3432 [StateManager.CollectDirty] [statemanager_cs39]( 00000000035D75A8 )State=6ec4fb2d-d1c1-72a8-32e6-fe26df42aba8 is dirty
3433 [StateManager.CollectDirty] [statemanager_cs45]( 00000000035D75A8 )NumberOfDirtyStates=104
3434 [ConfigurationEngine.CommunicationHelper.NotifyDirtyStatesTask.Run] [configurationengine_cs869]Completed successfully
3435 [DatabaseAccessor.GetPollingIntervalMillisecondsTimeSpan] [databaseaccessor_cs126]Database polling interval 0 milliseconds

 

Note the timestamp in line 3326 is set to 04/11/2074. If this appears in ETL logging, use the SQL queries in the "More Information" section to confirm the condition listed in the "cause" section exists.

Cause

The System Center Management Configuration service uses a timestamp to determine when new configuration data needs to be calculated for agents and management servers. If the system clock on an agent is faster than the system clock on the RMS, discovery data from this agent will set the timestamp for one or more managed instances hosted by that agent to the current agent system clock time. The System Center Configuration Management service will delay calculating configuration updates for the instances on that agent until the system clock on the RMS is current with the timestamp for that discovery data. If the agent system clock was significantly faster than RMS system time when discovery data was sent, or the agent continues to send data with a future timestamp, then it is possible that the management group would experience the symptoms listed above.

Setting the agent system clock time to match the RMS system clock time will not reset the timestamp for the existing discovery data and the issue will remain until the RMS system clock time exceeds the discovery data by the grooming interval, when the obsolete discovery data will be groomed normally.

Resolution

1) The system clocks for all agents and management servers in the management group must not significantly exceed the system clock on the RMS when submitting ANY data. If any agents or management servers have system clocks more than a few minutes faster than the RMS, they should be corrected first to avoid any additional data with future timestamps being added to the database.

2) The future timestamps for the discovery data that has already been submitted must be modified in the OperationsManager database to reflect the current time.

3) The System Center Configuration Management service and System Center Management service on the RMS must be restarted after both the above conditions are met.

More Information

1) Use the following three queries to determine if this condition exists. The queries must be run against the OperationsManager database. If the timestamp with the greatest value in the table is greater than the current time (in UTC format), then the condition exists.

Select GetUTCDate()as 'Current Time',
MAX(TimeGeneratedOfLastSnapshot) as 'DiscoverySource Timestamp' from DiscoverySource

Select GetUTCDate()as 'Current Time',
MAX(timegenerated) as 'DiscoverySourceToTypedManagedEntity Timestamp' from DiscoverySourceToTypedManagedEntity

Select GetUTCDate()as 'Current Time',
MAX(timegenerated) as 'DiscoverySourceToRelationship Timestamp' from DiscoverySourceToRelationship

2) The following three queries can be used to determine which computers may have submitted discovery data with a future timestamp. If the system clocks on these agents are not current, set them to current time before taking any additional action.

-- Find all computers with DiscoverySource Timestamp more than one day in future --
Select DisplayName, *
from BaseManagedEntity
where BaseManagedEntityID in
(select BaseManagedEntityId from BaseManagedEntity BME
join DiscoverySource DS on DS.BoundManagedEntityId = BME.BaseManagedEntityId
where DS.TimeGeneratedOfLastSnapshot > DATEADD (d, 1, GETUTCDATE())
and FullName like 'Microsoft.Windows.Computer%')

-- Find all computers with DiscoverySourceToTypedManagedEntity Timestamp more than one day in future --
Select DisplayName, *
from BaseManagedEntity
where BaseManagedEntityID in
(select BaseManagedEntityId from BaseManagedEntity BME
join DiscoverySourceToTypedManagedEntity DSTME on DSTME.TypedManagedEntityId = BME.BaseManagedEntityId
where DSTME.TimeGenerated > DATEADD (d, 1, GETUTCDATE())
and FullName like 'Microsoft.Windows.Computer%')

-- Find all computers with DiscoverySourceToRelationship Timestamp more than one day in future --
Select DisplayName, *
from BaseManagedEntity
where BaseManagedEntityID in
(select BaseManagedEntityId from BaseManagedEntity BME
join DiscoverySource DS on DS.BoundManagedEntityId = BME.BaseManagedEntityId
join DiscoverySourceToRelationship DSR on DSR.DiscoverySourceId = DS.DiscoverySourceId
where DSR.TimeGenerated > DATEADD (d, 1, GETUTCDATE())
and FullName like 'Microsoft.Windows.Computer%')

3) To correct the existing data, run the following commands against the affected tables.

Update DiscoverySource
Set TimeGeneratedOfLastSnapshot = GETUTCDATE()
where TimeGeneratedOfLastSnapshot > GETUTCDATE()

Update DiscoverySourceToTypedManagedEntity
Set TimeGenerated = GETUTCDATE()
where TimeGenerated > GETUTCDATE()

Update DiscoverySourceToRelationship
Set TimeGenerated = GETUTCDATE()
where TimeGenerated > GETUTCDATE()

4) The following query can be used to see what additional data has been submitted to the database with a timestamp in the future. The tables related to maintenance mode should have several rows, assuming there are agents currently in maintenance mode which is scheduled to end at some time. All other tables should have timestamps with the current time, or in the past.

/* */
/* The following query will search all tables in the database */
/* for columns with datetime datatypes. It will then return */
/* the total number of rows in each table that have values */
/* greater than the configured number of days from present. */
/* Times are all in UTC format. The default increment is */
/* 3 days, but can be adjusted as needed. */
/* */
DECLARE @tabname AS sysname;
DECLARE @colname AS sysname;
DECLARE @fcontin AS tinyint;
DECLARE @query AS nvarchar(max);

CREATE TABLE #work
(
TableName sysname,
ColumnName sysname,
NumRows int,
);

DECLARE cur_meta CURSOR FOR
SELECT t.Name 'Table',
c.Name 'Column'
FROM sys.columns c
INNER JOIN sys.tables t ON c.object_id = t.object_id
INNER JOIN sys.types y ON c.system_type_id = y.system_type_id
WHERE y.Name = 'datetime';

/* Change the increment in the DATEADD(dd,3,GETUTCDATE()) function */
/* as needed from the default of +3 days from current time */
OPEN cur_meta;
SET @fcontin = 1;
WHILE (@fcontin > 0)
BEGIN
FETCH cur_meta INTO @tabname, @colname;
IF (@@FETCH_STATUS < 0)
BREAK;
PRINT 'Table = '+ @tabname + ', Column = ' + @colname;
SET @query = 'SELECT ''' + @tabname
+ ''', ''' + @colname
+ ''', COUNT(*) FROM ' + QUOTENAME(@tabname)
+ ' WHERE ' + QUOTENAME(@colname) + ' > DATEADD(dd,3,GETUTCDATE())';
INSERT INTO #work
EXECUTE ( @query );
END
CLOSE cur_meta;
DEALLOCATE cur_meta;

SELECT *
FROM #work
ORDER BY 3 DESC;
DROP TABLE #work;

=====

For the most current version of this article please see the following:

2635742 : Configuration may not update in System Center Operations Manager 2007

J.C. Hornbeck | System Center & Security Knowledge Engineer

Get the latest System Center news on Facebook and Twitter:

clip_image001 clip_image002

App-V Team blog: http://blogs.technet.com/appv/
ConfigMgr Support Team blog: http://blogs.technet.com/configurationmgr/
DPM Team blog: http://blogs.technet.com/dpm/
MED-V Team blog: http://blogs.technet.com/medv/
Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog: http://blogs.technet.com/momteam/
SCVMM Team blog: http://blogs.technet.com/scvmm
Server App-V Team blog: http://blogs.technet.com/b/serverappv
Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center Essentials Team blog: http://blogs.technet.com/b/systemcenteressentials
WSUS Support Team blog: http://blogs.technet.com/sus/

The Forefront Server Protection blog: http://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/