Kevin Holman's OpsMgr Blog

Making groups of logical disks – an example from simple to advanced

kevinhol — Thu, 05 Nov 2009 06:00:37 GMT

I have been seeing this question come up a lot lately – as customers try and create groups of their disks – in order to create overrides for “certain” disks. So – I am creating this post to give some real world examples.

Well – I will start this simply. Say we want to create a group of all logical disks, with the drive letters of C: and D:?

I would start with creating a new group – and adding the “Windows Server 2003 Logical Disk” class. Now – I could just use the parent class of “Logical Disk” instead of the OS specific class if I wanted. The only issue with that is that most monitors targeting a disk – are OS specific – and duplicated three times. So it is best to create specific groups for these – but totally not required.

Ok – so in the Dynamic Members query builder – I click add, and pick a property. Since I know “Device Name” contains the drive letter – this will do nicely. I select device name “Equals” “C:”.

Now – I want to also include D:. There are many way to do – this – and I will go through them. First – I could simply Insert a new line for Windows Server 2003 Logical Disk – and replicate the line I have – adding one for D:

Only one problem – this is an “AND” grouping – I really need this to be an “OR” grouping to include both C: and D: drives. You can switch this grouping the in UI, just right click the word “AND” and change it to an OR grouping:

Voila!

This formula now looks like:

( Object is Windows Server 2003 Logical Disk AND ( Device Name Equals C: ) OR ( Device Name Equals D: ) )

Save your group – then right click it – and choose “View Group Members”. This will ensure we are cooking with gas. It should contain all your Windows 2003 based C: and D: volumes.

So far – so good.

Now – what if I ONLY want C: and D: disks, that are HOSTED by specific Windows Computers? I can do that too! Lets say I want a group – of all the C: and D: logical disks, on servers that begin with the name “SR______”

If you look at the bottom of the list of properties for Logical Disks – you will see (Host=Windows Computer). From here – we can pick any attribute of the Windows Computer class as well to add to our expression – to limit our logical disks in our group to very specific Computers.

Go back to the properties of your group, edit the Dynamic Members, and you can construct something like this:

Which translates to the following formula:

( Object is Windows Server 2003 Logical Disk AND ( Windows Computer.NetBIOS Computer Name Matches wildcard sr* ) AND ( ( Device Name Equals C: ) OR ( Device Name Equals D: ) ) )

Now – I will be honest – getting all the “ands” and “ors” in the right place using the UI is a big pain. It is very easy to screw it up. I like to simplify this to the fewest lines possible – using Regex.

Using Regular Expressions – we can use modifiers to create very advanced expressions. my favorites are ^ which means the beginning of a new line or word, and | which is the “pipe” symbol – which means “or”.

So a simple way to accomplish the same example above – without all the complexity – is this:

WAY simpler!

However – you might notice – this doesn't work right. This is because Regex is case sensitive. If the Server NetBIOS name is detected in all CAPS, this expression wont match. I talk a little about this issue in this post: http://blogs.technet.com/kevinholman/archive/2009/04/21/quick-tip-using-regular-expressions-in-a-dynamic-group.aspx

So – based on that posts example – there is a simple way to make a RegEx case insensitive: (?i:blah)

Using that as an example – we can now make very advanced groupings, quite easily:

(?i: to make it case insensitive. ^ to signify the beginning of the word/line match. Here is the formula now:

( Object is Windows Server 2003 Logical Disk AND ( Device Name Matches regular expression (?i:^C|^D) ) AND ( Windows Computer.NetBIOS Computer Name Matches regular expression (?i:^sr) ) )

Check it out:

Victory!

What if I wanted all logical disks that we NOT hosted by a Virtual Machine? Easy!

( Object is Logical Disk AND ( Windows Computer.Virtual Machine Equals False ) AND True )

This reveals a group of ALL logical disks hosted by a Windows Computer with the attribute of Virtual Machine = False:

As you can see – using the Hosting relationship of the disk – to the Windows Computer object, there is much more you can do with groups.

Updated Active Directory ADMP Management Pack released – Version 6.0.7065.0

kevinhol — Wed, 04 Nov 2009 19:53:34 GMT

This is now available on the catalog.

http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Changes in this update:

Support for monitoring Windows Server® 2008 R2 server operating systems as well as Windows® 7 client operating systems
Support for monitoring the Active Directory Web Service (ADWS) in Windows Server 2008 R2 as well as the Active Directory Management Gateway Service in Windows Server 2008 and Windows Server 2003

There were a few other minor fixes as well, such as the AD LSASS CPU overload monitor. This release also replaces a recently released ADMP version 6.0.7050.0 that was pulled to fix the aforementioned monitor.

Why do my group memberships for Windows Computers have machines that don't belong there?

kevinhol — Tue, 03 Nov 2009 05:35:50 GMT

Here is a little tip if you find that your Windows Computer Groups (and state views scoped by groups) contain computers that should not be there.

Have you noticed that you have state views or Windows Computer Groups that contain servers that you don't expect? Like Exchange Servers in your SQL Computers Group? Or SQL Servers in your Exchange 2007 computer group? Or maybe Hyper-V host servers in a LOT of your groups? If so – you are probably running Hyper-V, and using the Hyper-V MP version 6.0.6633.0.

Have a look at the below example: My SQL Management Pack “Computers” view – contains domain controllers, exchange servers… even XP clients… plus it also includes the Hyper-V host (VS3).

This can wreak havoc on your management group…. because we use groups of Windows Computers for Overrides, and for scoping console views.

The good news is – there is a very simple workaround: There is a relationship in this MP that we need to disable. This relationship attempts to associate the Windows Computer objects of a guest to its host – however it doesn't work properly, and isn't necessary.

Open Authoring in the console. Select “Object Discoveries”. Scope to “Hyper-V Virtual Machine”. Find the discovery named: “Hyper-V 2008 Guest Computer Relationship Discovery”

Create an override for this – “for all objects of class: Root Management Server”. Set this discovery to disabled.

Now that that is disabled – we need to run a little cleanup on aisle 7. We have a nifty little cmdlet in the OpsMgr command shell – named remove-disabledmonitoringobject. This cmdlet will basically remove any discovered objects – for any situation where they are explicitly disabled with an override on a discovery. Since that is what we just did (disabled a discovery) this will quickly delete any discovered relationships which previously existed.

Now – when I look at my state views scoped to SQL Computers group – I only see SQL servers, AND – the Hyper-V host. We don't want the Hyper-V host tho….. apparently the cmdlet cleanup doesn't take care of those. To resolve that membership – I generally bounce the HealthService on the Hyper-V hosts, and then the HealthService on the RMS, and in a few minutes they will be gone.

Voila!

Holy Management Packs Batman – we are surrounded!

kevinhol — Wed, 14 Oct 2009 00:00:19 GMT

Wow – what a plethora of new MP’s to look at.

Just in the past week – we have gotten a bunch of updated MP. I wont take any time to go into details on the changes in this post – but be aware! You probably need to get these in your evaluation and test cycles… looks like lots of good stuff in these updates.

From the MP catalog at: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Exchange Server 2003
10/12/2009
(6.0.6702.0)
Exchange Server 2003 Management Pack for System Center Operations Manager 2007
The Exchange Server 2003 Management Pack monitors the performance and availability of Exchange Server 2003 systems. It can also issue alerts for possible configuration problems Availability and performance monitoring is done using synthetic transactions. In addition, the Management Pack collects Event Log alerts and provides associated knowledge articles with additional user details, possible causes, and suggested resolutions. The Management Pack discovers and monitors the individual server components and functional areas on an Exchange Server 2003 server.

Exchange Server 2007
10/12/2009
(6.0.6702.0)
Microsoft Exchange Server 2007 Management Pack for System Center Operations Manager 2007 R2
The Exchange Server 2007 Management Pack is designed to monitor Exchange 2007 key health indicators, collect Exchange component-specific performance counters in one central location, and raise alerts for operator intervention as necessary. By detecting, sending alerts, and automatically responding to critical events, this Management Pack helps indicate, correct, and prevent possible service outages or configuration problems, allowing you to proactively manage Exchange servers and identify issues before they become critical. The Management Pack monitors and provides alerts for automatic notification of events indicating service outages, performance degradation, health monitoring, and centralized management.

Operations Manager 2007 R2 Management Pack
10/8/2009
(6.1.7533.0)
Operations Manager 2007 R2 Management Pack
The Operations Manager 2007 R2 Management Pack alerts you to problems with components such as agents, management servers, the Operations Manager database, agents, modules used by workflows and services so that you can continue to monitor the servers and clients that your business depends on. The management pack includes tasks that you can automate to get easy access to common diagnostic tools, such as restarting a health service or reloading an agent configuration.

Windows Server Operating System
9/18/2009
(6.0.6667.0)
Windows Server Operating System Management Pack for Operations Manager 2007
The Microsoft Windows Server management packs monitor the performance, health, and availability of Windows Server 2008 R2, 2008, 2003 R2, 2003, and Windows 2000 Server. By detecting, alerting on, and automatically responding to critical events and performance indicators, management packs reduce resolution times for issues and increase the overall availability and performance of your Windows Server operating systems, thereby helping to reduce the total cost of ownership.
(Chinese (Taiwan), German, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Chinese (PRC), Spanish)

Dude. Got R2? Then go get this update!

kevinhol — Fri, 09 Oct 2009 06:17:23 GMT

Seriously.

Operations Manager 2007 R2 Management Pack version 6.1.7533.0 from the MP CATALOG.

This is basically a set of the core MP’s in OpsMgr R2. You should expect to see these get updated on a fairly regular basis now. These are the MP’s that are built in to OpsMgr, that monitor the health of the Management group, from the server role, and agent role perspective. There should be a SP1 version of this update available soon to follow. These are a series of updates to OpsMgr, based on Community, Customer, and internal feedback…. and it’s good stuff.

Everyone running R2 should run these through your standard testing cycle and get these implemented. This is a simple MP import-update.

The MOMTEAM Blog has some more details here: http://blogs.technet.com/momteam/archive/2009/10/07/opsmgr-2007-r2-mp-version-6-1-7553-0-is-released.aspx

This includes a long list of updates (available in the guide in the download)… but I will hit a few high points.

You might have seen some of my blog posts on agents restarting all the time, in SP1 and R2, and the significant impact that can have:

http://blogs.technet.com/kevinholman/archive/2009/06/22/health-service-and-monitoringhost-thresholds-in-r2-how-this-has-changed-and-what-you-should-know.aspx

Well, this is mostly resolved in this update…. as the default threshold for HealthService and MonitoringHost has been changed from 100MB to 300MB. This will stop the majority of your agents from hitting this limit, and restarting. Once you import this MP update – you should review any overrids you made on these Monitors… and make sure you dont have any conflicts. The only other override I like to recommend is to generate an alert on these monitors – so that you will know when there is a problem that will cause the agent to get restarted – specifically on HealthService or MonitoringHost, and PrivateBytes or HandleCount:

You might find that there are still some agents that need a much higher threshold…. but those should be fairly rare, and limited to very large servers with a very high instance count of objects (large SQL clusters, large Exchange servers, etc) and also possibly for agents that are a watcher node or proxy for a large number of devices (network, VMWare, etc…)

In THIS blog post I discussed adding a custom perf counter collection rule and creating a view to see how many consoles or SDK connections you have. This is now built-in:

Lots of Knowledge and descriptions added to help troubleshoot alerts. For instance – on the common “Script or Executable Failed to run”

Summary

The Health Service was attempting to run an executable and was unable to create the process. This may affect some monitoring or discovery.

Causes

This can be caused by:

The executable could not be found.

The computer does not have enough resources (for example; memory) to run the executable.

The antivirus software on the computer blocking Visual Basic scripts or Java scripts. The following link is a link to the KB article regarding this issue.

Antivirus software blocking script execution

Resolutions

The alert description and context has information indicating which rule or monitor failed. The following link will display all events indicating a failure to run the executable:

View Batch Response Events

After reviewing the error in the context, check:

That the path to the executable exists on the computer.

The antivirus software is not blocking scripts from running.

That the computer is not over utilized.

Check Task Manager to see if there is enough free memory.

Check Task Manager to see if there are any processes consuming all the CPU.

Check the error information in the event or alert context for the script path and name. There could be a problem with the script not handling an error correctly and exiting. If the script exits without outputting the data that is expected (e.g. property bag data), this error is raised.

Another error can be caused by the misconfiguration of the workflow executing this script. The configuration (script params, policy, timeout) could be wrong causing no output or timeout.

There is a big list of other enhancements or fixes in the MP guide. Don't take this lightly – this is good stuff…. get it updated.

What is config churn?

kevinhol — Mon, 05 Oct 2009 08:29:49 GMT

There have been a couple good articles briefly covering this topic…. you might have read them. I will reference some below. Config churn is basically, when your RMS is in an almost never-ending loop of generating config. This can be caused by “less than optimized” management packs, pushing agents all the time, or injecting major changes into a management group, such as overrides or custom rules and monitors, or importing updated management packs. By examining this topic in depth – we will re-state some already known best practices with maintaining a healthy management group, and get some deeper knowledge as to why they are best practices in the first place.

Any time you push agents, or create rules and monitors, or overrides for widespread classes….. you can create a config update on the RMS that must be sent down to ALL agents in the management group. For small management groups (under 500 agents) this is generally not a big deal and processes rather quickly. For large management groups over 1000 agents, this can cause high resource utilization of the RMS and SQL Database, in terms of CPU, Memory, and Disk I/O. This can impact data insertion, and console performance during these times. For these reasons, we like to keep those activities down to a minimum during working hours, and schedule these major changes in an off-hours maintenance window.

What about “less than optimized” management packs? What does that mean? Well, this means management packs that you might be using, that have poorly written discoveries.

We have long known that a worst practice in Management Pack development, is to have a discovery that discovers instances of a class, that has properties for those instances that are likely to change frequently. Here is a write-up from OpsManJam on the topic: LINK

Ok… wait… Whaaaaat?

Let me put that in English:

Say we have a discovery for a Logical Disk. This will discover any logical disk, like C:, D:, E:, Q:, etc…. When we write the discovery for a logical disk, we can add properties to that discovery. These are attributes of the discovered instances. So – in this case – lets say we decided to add “Size” of the disk as a property, and “Free Space” as a property. And for the discovery frequency – we will run this discovery every hour, looking for new disks.

“Size” is an excellent property for the Logical Disk class. We like to know the size of the disks…. we can use this property group them if needed. “Size” of a logical disk is not something that we would expect to change very often.

“Free Space” is a horrible property for the Logical Disk class. Free space is something that will likely change, just a small amount even, between each run of the discovery. Free space is a property that is likely to change frequently, therefore – it should NOT be used in a discovery.

Make sense?

Ok – so… what's the big deal?

Well, the agent will run almost all discoveries that it knows about when the health service starts up (like when you bounce the service, or after a reboot). It will always send this discovery data to the management server. (this is another reason why agents restarting all the time is very bad) Then, it will run then based on the “Interval” frequency specified on the discovery. Sometimes this is as frequent as once per hour, sometimes as long as once per day. When the discovery runs, the agent will inspect the discovery data that it gets, and compare it to the last discovery data it sent to the management server. If nothing changed – the agent drops the discovery data and does nothing. IF anything changed in the values of the discovery data – it will re-submit the new data to the management server, which will submit this data to the database. The RMS will detect the change, and will have to recalculate (regenerate) configuration. You will see this on the RMS as a 21025 event:

Log Name:      Operations Manager
Source:        OpsMgr Connector
Date:          9/27/2009 11:51:49 PM
Event ID:      21025
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
OpsMgr has received new configuration for management group PROD1 from the Configuration Service. The new state cookie is "D7 9B A4 BE 00 90 CF 13 35 B5 9B 5F 3B 14 FF 78 D6 13 9A 2D "

The 21025 event isn’t really “bad”… it simply means the config service did its job. It re-generated its configuration file from the database data, and wrote it to: \Program Files\System Center Operations Manager 2007\Health Service State\Connector Configuration Cache\<MGNAME>\OpsMgrConnector.Config.xml The problem is – when this config file gets large (like in large agent count environments) and when the “Config Instance Space” is large (number of discovered objects in total). Recalculating this config can have a significant impact on the disk where the file exists on the RMS, use lots of memory and CPU on the RMS for the config service, and use significant disk I/O on the SQL database.

If the RMS is in a perpetual cycle of recalculating config, and sending these config updates to all agents…. the performance of the management group is impacted.

Daniele Grandini of Quaue Nocent Docent is pretty much the “godfather” of good information researching the 21025 event. Read his 3 part series on config churn here:

http://nocentdocent.wordpress.com/2009/07/09/troubleshooting-21025-events-wrap-up/

So – what can I do if I think I have too much config churn?

The biggest problem causing the most frequent config updates is management packs with noisy discoveries. However, lets wrap up all the issues that can cause it, and what you can do:

New agents. Discover/install/approve new agents in bulk and off-hours.
Overrides. Set overrides during off-hours, or create override MP’s in a lab, then synch to production management groups during schedule off-hours times.
Custom rules and monitors. Create these during off-hours, or create using the authoring console, test in a lab, then import to production during off-hours.
Newly discovered instances. For instance – someone adds a new disk, or SQL database, or DNS zone, to an existing agent. Not much we can do about this, except the expectation that this would be done during off hours.
Management packs with noisy discovery properties. See below.

Ok – the remainder of this article will touch on #5.

How can I tell which discoveries are noisy?

Danilele Grandini has put together a good query on this, from his link: http://nocentdocent.wordpress.com/2009/05/23/how-to-get-noisy-discovery-rules/

I will repost these (slightly modified) below:

/* Top Noisy Rules in the last 24 hours */

select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As 'Changes'
from
(select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
) As #T
group by ManagedEntityTypeSystemName, DiscoverySystemName
order by count(*) DESC

and

/* Modified properties in the last 24 hours */

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName', MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
ORDER BY MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName

Wow – that returned a LOT of discoveries running all the time! What can I do?

Don't import too many MP’s! The FIRST line of defense – is NOT to import ANY management packs into a management group that you don't absolutely need RIGHT THEN. Management packs are constantly updated, and by the time you have an actual SLA in a technology area – there will likely be a newer, better MP available for it. The biggest mistake many customers make is to import any available MP for a technology that they have internally. They end up with a FLOOD of alerts, big fat databases, slow consoles, and lots of weird errors. MP’s should be transitioned slowly, one at a time – tuning and resolving as you go.

Disable the noisy discoveries. Probably not a great solution, unless they discover objects that you really don't care about – but there are other objects in the MP that you DO want to monitor.

Increase the interval of the discovery frequency. This means… essentially – change any “bad” discoveries to run only once per day.

Add a “synch time” override to the discovery – if possible. This option is not available unless the MP author of the discovery exposed it. What this will do – it cause all the agents to ONLY run the discovery at a distinct and specified time every day (say…. 1AM). This might cause too much discovery data to flood in at one time… but since it will all come in at the same time – it wont cause constant config churn all throughout the day.

Re-write the discovery. If this is a custom MP – rewrite the discovery/MP, and remove that property which changes too often.

Make sure your hardware and software is optimized for scalability. On your RMS – it is good to place your config file on fast disks, especially in large environments. I have worked with very large customers who were experiencing config churn, but had zero ill effects, because their RMS disk I/O was on a 4 spindle RAID10 with 15K spindles, CPU and memory were really good, and their SQL database disk I/O for the OpsDB was excellent with plenty of breathing room. I have also worked with smaller agent counts, where config churn has a serious impact…. mostly due to the RMS config file being places on the same RAID spindle set as that OS and pagefile, using only 2 older 10,000 RPM disks. The SQL disk I/O was also just borderline for their agent count.

Re-run the queries periodically – especially after importing/upgrading to a new management pack in your management group. This “instance space change” report should be part of your testing and evaluation of a new MP when brought into your lab…. if you have a large agent count environment.

Some very common discoveries I have seen – that have properties that change very frequently – are listed below. I often recommend these be overridden to run once per day (86,400 seconds)

Discovery Display Name	Discovery Target Class	Discovered Type	Default frequency	Modified frequency
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File	7260	86400
Discover File Groups and Files	SQL 2005 DB Engine	SQL 2005 DB File Group	7260	86400
Discover SQL 2000 Databases	SQL 2000 DB Engine	SQL 2000 DB	1800	86400
Discover Databases for a Database Engine	SQL 2005 DB Engine	SQL 2005 DB	7200	86400
DNS 2003 Component Discovery	DNS 2003 Server	DNS 2003 Zone	21600	86400
DNS 2008 Component Discovery	DNS 2008 Server	DNS 2008 Zone	21600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2003 Server Role	IIS FTP Site	3600	86400
Windows Internet Information Services Base Classes Discovery Rule	IIS 2000 Server Role	IIS NNTP Virtual Server	3600	86400

The above is just a sample – you should examine the query output of the query above and see what is impacting your management group the most.

Keep your management pack names SHORT in SP1!

kevinhol — Fri, 02 Oct 2009 23:31:08 GMT

I have seen this twice now… so I will blog about it. It seems to be rare in the wild, but it will completely cripple a management group when this occurs. So beware SP1 users!

This article does not apply to R2. This is only an issue in OpsMgr 2007 SP1.

When you create your custom management packs – and especially your override management packs – keep the names as simple and short as possible. There is an issue in OpsMgr SP1 – when an agent tries to download management packs – where it will fail if the MP ID (derived from the MP Name) is too long. The worst part about this problem is there there WONT be an error logged. What will happen – is that the agent will keep trying to re-download the MP in question, and it will block ALL other MP’s from being downloaded from that point forward.

There is no simple way to know this condition is impacting you. What will happen – is that an agent will continue to work just fine… but will not get any NEW management packs. So… you might think all is well, but not so. The symptoms thatmight lead you to notice that something is wrong:

Performance data not collected for newer MP’s.

Objects not being discovered for newer MP’s

Alerts not generating as expected from newer MP’s.

The root problem is an age old Windows issue…. file paths over 255 characters not supported well. This has been resolved in R2 in how the agent copies the files over.

In both cases I have seen – someone was creating an override MP for the “IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems” management pack. So – when they created their override MP – they named it something like: “Overrides - IBM Hardware Management Pack for IBM System x and BladeCenter x86 Blade Systems”

This equated to a Management Pack ID of: Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems

That MP ID is 86 characters!

What happens…. is when this management pack is created, and an override is placed in it…. (or custom rule)… the agents that require it will:

Get contacted by the RMS to update their config, and then issue a config change request (21024 in the event log)

Receive new config from the RMS (event 21025)

Process new config – and realize they need a new MP, and request that MP (event 1200)

Where this process breaks…. is the next step in the chain should be that the agent RECEIVES the MP (event 1201) and then issues a statement that the new config has become active (event 1210). The never happen.

Behind the scenes…. from looking at a ETL tracelog, we can see this is failing, when we try to move the file from the “downloaded files” folder to the “management packs” folder:

Error CMPFileManager::MoveManagementPackFile(MPFileManager_cpp383)MoveFile from '\\?\C:\Program Files\System Center Operations Manager 2007\Health Service State\Downloaded Files\MGNAME\1\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' to 'C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml' failed with code 3(ERROR_PATH_NOT_FOUND).

In the example above – the bad path is:

C:\Program Files\System Center Operations Manager 2007\Health Service State\Management Packs\Override.IBM.Hardware.Management.Pack.for.IBM.System.x.and.BladeCenter.x.Blade.Systems.{26504FED-2FF4-4AC4-A63D-59BF8C09F51F}.{7136257C-1791-7BAB-7072-2FA24284C102}.xml

Which is 261 characters. The limit is 255.

Therefore – I recommend you keep your Management Pack *ID* to less than 60 characters. You can examine your long management packs by looking in the console – generally your longest display names will be the longest ID’s:

Even some Microsoft MP’s are dangerously close to the limit…. such as: Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.VMWare.HostPerformance with 76 characters. In most environments you can squeak by at 79 characters…. more or less depending on where you installed your agent path.

Here is a SQL query you can run against the OpsDB to also detect this condition…. and quick check all your potentially long MP’s:

select MPName from managementpack
WHERE len(MPName) > 60

Just change 60 to whatever character count you want.

DONT freak out if you have some more than 60. Just be aware.

DO freak out if you have some more than 80!

If someone had the time and was really handy – you could write a monitor – that runs against the RMS – that would in turn query the OpsDB, and run this query, and change the RMS to unhealthy when over a threshold. THAT would be cool…. and alert you when some author makes a really long MP that has the potential to break all your agents.

Another idea I had… was to create a correlated missing event monitor…. and when you get a 1200, but do NOT get a 1201 within, say, 15 minutes….. that might be a problem. Of course if you wrote this and were already impacted…. the bad agents would never get your new MP to tell you. :-)

Fixing troubled agents

kevinhol — Fri, 02 Oct 2009 00:23:46 GMT

Sometimes agents either will not “talk” to the management server upon initial installation, and sometimes an agent can get unhealthy long after working fine. Agent health is an ongoing task of any OpsMgr Admin’s life.

This post in NOT an “end to end” manual of all the factors that influence agent health…. but that is something I am working on for a later time. There are so many factors in an agent’s ability to communicate and work as expected. A few key areas that commonly affect this are:

DNS name resolution (Agent to MS, and MS to Agent)
DNS domain membership (disjointed)
DNS suffix search order
Kerberos connectivity
Kerberos SPN’s accessible
Firewalls blocking 5723
Firewalls blocking access to AD for authentication
Packet loss
Invalid or old registry entries
Missing registry entries
Corrupt registry
Default agent action accounts locked down/out (HSLockdown)
HealthService Certificate configuration issues.
Hotfixes required for OS Compatibility
Management Server rejecting the agent

How do you detect agent issues from the console? The problem might be that they are not showing up in the console at all! Perhaps they might be a manual install that never shows up in Pending Actions? Or a push deployment, that stays stuck in Pending actions and never shows up under “Agent Managed”. Or even one that does show up under “Agent Managed” but never shows as being monitored… returning agent version data, etc.

One of the BEST things you can do when faced with an agent health issue… if to look on the agent, in the OperationsManager event log. This is a fairly verbose log that will almost always give you a good hint as to the trouble with the agent. That is ALWAYS one of my first steps in troubleshooting.

Another way of examining Agent health – is by the built in views in OpsMgr. In the console – there is a view – Located at the following:

This view is important – because it gives us a perspective of the agent from two different points:

1. The perspective of the agent monitors running on the agent, measuring its own “health”.

2. The perspective of the “Health Service Watcher” which is the agent being monitored from a Management Server".

If any of these are red or yellow – that is an excellent place to start. This should be an area that your level 1 support for Operations manager checks DAILY. We should never have a high number of agents that are not green here. If they aren't – this is indicative of an unhealthy environment, or the admin team not adhering to best practices (such as keeping up with hotfixes, using maintenance mode correctly, etc…

Use Health Explorer on these views – to drill down into exactly what is causing the Agent, or Health Service Watcher state to be unhealthy.

Now…. the following are some general steps to take to “fix” broken agents. These are not in definitive order. The order of steps really comes down to what you find when looking at the logs after taking these steps.

Start the HealthService on the agent. You might find the HealthService is just not running. This should not be common or systemic. Consider enabling the recovery for this condition to restart the HealthService on Heartbeat failure. However – if this is systemic – it is indicative of something causing your HealthService to restart too frequently, or administrators stopping SCOM. Look in the OpsMgr event log for verification.

Bounce the HealthService on the agent. Sometimes this is all that is needed to resolve an agent issue. Look in the OpsMgr event log after a HealthService restart, to make sure it is clean with no errors.

Clear the HealthService queue and config (manually). This is done by stopping the HealthService. Then deleting the “\Program Files\System Center Operations Manager 2007\Health Service State” folder. Then start the HealthService. This removes the agent config file, and the agent queue files. The agent starts up with no configuration, so it will resort to the registry to determine what management server to talk to. From the registry – it will find out if it is AD integrated, or a fixed management server to talk to if not. This is located at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Agent Management Groups\PROD1\Parent Health Services\ location, in the \<#>\NetworkName string value. The agent will contact the management server – request config, receive config, download the appropriate management packs, apply them, run the discoveries, send up discovery data, and repeat the cycle for a little while. This is very much what happens on a new agent during initial deployment.

Clear the HealthService queue and config (from the console). When looking at the above view (or any state view or discovered inventory view which targets the HealthService or Agent class) there is a task in the actions pane - “Flush Health Service State and Cache”. This will perform a very similar action to that above…. as a console task. This will only work on an agent that is somewhat responsive…. if it does not work you need to perform this manually as the agent is really broken from communication with the management server. This task will never complete, and will not return success – because the task breaks off from itself as the queue is flushed.

“Repair” the agent from the console. This is done from the Administration pane – Agent Managed. You should not run a repair on any AD-integrated agent – as this will break the AD integration and assign it to the management server that ran the repair action. A “repair” technically just reinstalls the agent in a push fashion, just like an initial agent deployment. It will also apply/reapply any agent related hotfixes in the management server’s \Program Files\System Center Operations Manager 2007\AgentManagement\ directories.

Reinstall the agent (manually). This would be for manual installs or when push/repair is not possible. This section is where the combination of options gets a little tricky. When you are at this point… where you have given up, I find just going all the way with a brute force reinstall is the best way. This means performing the following steps:

Uninstall the agent via add/remove programs.
Run the Operations Manager Cleanup Tool CleanMom.exe or CleanMOM64.exe. This is designed to make sure that the service, files, and all registry entires are removed.
Ensure that the agent’s folder is removed at: \Program Files\System Center Operations Manager 2007\
Ensure that the following registry keys are deleted:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService

Reboot the agent machine (if possible)
Delete the agent from Agent Managed in the OpsMgr console. This will allow a new HealthService ID to be detected and is sometimes a required step to get an agent to work properly, although not always required.
Now that the agent is gone cleanly from both OpsMgr console and the agent Operating System…. manually reinstall the agent. Keep it simple – install it using a named management server/management group, and use Local System for the agent action account (these will remove any common issues with a low priv domain account, and AD integration if used) If it works correctly – you can always reinstall again using low priv or AD integration.
Remember to import certificats at this point if you are using those on the individual agent.
As always – look in the OperationsManager event log…. this will tell you if it connected, and is working, or if there is a connectivity issue.

To summarize…. there are many things that can cause an agent issue, and many methods to troubleshoot. However – to summarize at a very general level, my typical steps are:

Review OpsMgr event log on agent
Bounce HealthService
Bounce HealthService clearing \Health Service State folder.
Complete brute force reinstall of the agent.

If it an external issue is causing the issue (DNS, Kerberos, Firewall) then these steps likely will not help you…. but those should be available from the OpsMgr event log.

Also – make sure you see my other posts on agent health and troubleshooting during deployment:

Console based Agent Deployment Troubleshooting table

Agent discovery and push troubleshooting in OpsMgr 2007

Getting lots of Script Failed To Run alerts- WMI Probe Failed Execution- Backward Compatibility

Agent Pending Actions can get out of synch between the Console, and the database

Which hotfixes should I apply-

New Base OS MP 6.0.6667.0 adds file fragmentation monitor to all Logical Disks

kevinhol — Mon, 28 Sep 2009 19:09:03 GMT

I recently blogged about the new Base OS MP that was recently released: HERE

One of the things you will notice RIGHT off the bat… is that a huge percentage of your logical disks will go into a warning state, if you don't already have some sort of scheduled defragmentation set up. This will be true for virtual machines and physical machines…. anything over 10 percent file fragmentation (or the OS recommended setting) will get hit:

You will also get many warning alerts on this monitor…. the first time the condition is detected and the state changes for this monitor. This monitor checks status every Saturday, at 3:00AM by default, for all logical disks discovered.

If you don't care about this monitoring in SCOM – disable this monitor using overrides.

If you do care about seeing the state change – but don't want the alerts – turn the “Generates Alert” property to False, using overrides.

You can adjust the threshold from 10% to some other number…. but make sure you take note – this monitor will ignore the “File Percent Fragmentation” property by default, and always use the OS recommended setting. If you want to control this – you also need to set “Use OS Recommendation” to FALSE.

Here is an example of hard coding the frag percentage to 20% from the OS default:

“Use OS Recommendation” property description:

Lastly – one thing of interest…. If you want SCOM to “fix” the fragmentation issue…. it can. There is a recovery on this very monitor that can run a VBScript that will run a defrag job against your logical disks. It is disabled by default.

Keep in mind – if you turn on this defrag…. on your physical boxes – it wont be a big deal… it will simply fix the fragmentation issue. However – this will also run on ALL yours VM’s. If this was triggered all at the same time – Saturday at 3:00AM by default – this can kill the disk I/O on the disk subsystem hosting your VM/VHD files. Keep this in mind if you decide to enable this…. This recovery will only run when the state change is detected… as a recovery to the condition, so any disks that are already in a warning state will not run this recovery should you enable it. This defrag has a timeout of 1 hour…. so it should kill the defrag if it cannot complete within an hour.

Another cool thing to do – is to use the recovery action as a single run-time task. You can do this right from health explorer, to fix the disks on your own schedule:

Just click the link, and run the task:

Minimize this…. and just let it run – you can come back in 1 hour – and see if it completed, or timed out.

You can also monitor for task status in the Task Status list in the console:

On the agent – you will see the following events logged in the OpsMgr event log:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          9/28/2009 10:50:04 AM
Event ID:      4002
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
Microsoft.Windows.Server.LogicalDisk.Defrag.vbs : Perform Defragmentation (disk: C:; computer: OMDW.opsmgr.net).

And when completed:

Log Name:      Operations Manager
Source:        Health Service Script
Date:          9/28/2009 11:03:44 AM
Event ID:      4002
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OMDW.opsmgr.net
Description:
Microsoft.Windows.Server.LogicalDisk.Defrag.vbs : Defragmentation completed (disk: C:; computer: OMDW.opsmgr.net): FilePercentFragmentation = 0.

New KMS MP released version 6.0.7234.0 and some things you should know

kevinhol — Thu, 24 Sep 2009 01:37:33 GMT

A new KMS (Key Management Services) MP is released. I don't expect this is a widely used MP out there…. but there are some things you should know if you ever imported the previous versions.

First – the new MP is available on the catalog: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

DO NOT download this MP directly via the R2 direct download mechanism. You need to get this one from the web catalog. As in all MP’s – you need to read the guide FIRST… because the “Before you import” section is critical.

The new MP is not backward compatible with the previous MP. You MUST delete any previous version of the KMS MP you have installed.

If you used the previous version of the KMS MP – you might have noticed some issues. For instance, the old KMS MP added a unique dataset type to the warehouse, with a grooming procedure just for the KMS dataset. These didn't work. When you download the new MP, it comes with instructions and a cleanup script to fix this. If you ever imported the old MP – you are affected. You will see the following events on your RMS every 10-11 minutes:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          9/23/2009 4:20:53 PM
Event ID:      31552
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMRMS.opsmgr.net
Description:
Failed to store data in the Data Warehouse.
Exception 'SqlException': Sql execution failed. Error 777971002, Level 16, State 1, Procedure StandardDatasetGroom, Line 303, Message: Sql execution failed. Error 2812, Level 16, State 62, Procedure StandardDatasetGroom, Line 145, Message: Could not find stored procedure 'KMS_EventGroom'.

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.DataWarehouse.StandardDataSetMaintenance
Instance name: KMS Activation Event Data Set
Instance ID: {35CE8740-3FCF-FA7F-D733-1ECD0FBE0D7F}
Management group: PROD1

So – here are the necessary steps to get this new MP working (from reading the MP guide):

Delete the old MP (typically going to be version 6.0.6278.9)
Run the cleanup SQL script against the OperationsManagerDW (Warehouse) database.
Import the new MP.

If you had previously imported the old KMS MP – and are getting this event on your RMS, and you no longer wish to use the KMS MP anymore….. you can clean this up using the script that is included with the new MP. So for a cleanup ONLY – where you do not wish to use the KMS MP:

Make sure the old KMS MP is deleted.
Run the cleanup SQL script against the OperationsManagerDW (Warehouse) database.

The script output simply displays this when complete:

A cleaned up environment will no longer have any KMS datasets, or stored procedures for grooming this dataset. You can inspect for these with the following queries against the warehouse:

select * from standarddataset

select * from sysobjects where name like ('%kms%')

Once the cleanup script has been run – these should return no results.

***Note – the new MP does not contain any reports. The old MP had reports the leveraged the KMS Dataset, but the new MP does not.

Alert Notification Subscription Variables, and linking that to the console, database, and SDK

kevinhol — Wed, 23 Sep 2009 20:54:00 GMT

Attached you will find a spreadsheet, with all the possible alert notification subscription variables that I am aware of. In this spreadsheet, I link these to the same values in the Alert table of the DB, the alert view of the DB, the Console alert view, the SDK (Get-Alert), and lastly the new R2 Connector Key pairs.

My thought was to better understand each data property type of an alert, and what you can managed, from each area. Hope this is beneficial.

Most of these are listed at my other blog post on alert description and notification variables: http://blogs.technet.com/kevinholman/archive/2007/12/12/adding-custom-information-to-alert-descriptions-and-notifications.aspx

Here is a sample shot of the spreadsheet tool:

See attached:

New Base OS MP released 9-18-2009 version 6.0.6667.0 to add support for Windows Server 2008R2 and other fixes

kevinhol — Tue, 22 Sep 2009 15:40:59 GMT

As you can see from the “Changes in this update section” of the guide – this isn't just about supporting Server 2008 R2 – there are many fixes and updates, so this would be a good one to get through your testing cycles and plan for production.

Available on the catalog: http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx

Changes in This Update

The September 2009 release (version 6.0.6667.0) of the Windows Server Operating System Management Pack includes the following changes:

Support for Windows Server 2008 R2, including new classes to represent Windows Server 2008 R2 systems specifically.
The intervals for some discoveries have been changed to reduce CPU usage. For details, see Objects the Windows Server Operating System Management Pack Discovers.
Added Logical Disk Fragmentation Level monitors for Windows Server 2003 and Windows Server 2008.
Disabled the following rules and monitors for Windows Server 2008 because the events needed are in Windows Server 2003 only.
- Duplicate computer name was detected
- Windows Activation State
- Disk Group Failed
- Disk Group Auto Import Failed
- Volume Not Started
- Run WMIAdap
Fixed the summary details for the Performance History (Percent Processor Time) report to indicate that the object type is Windows Server 2003 operating system rather than Windows Server 2003 processor.
Fixed the Performance History (Percent Interrupt Time) report to use a rule that targets the operating system rather than the processor.
Changed the data source for discoveries from System.Scheduler to System.Discovery.Scheduler to avoid performance issues.
Changed the Logical Disk Availability monitor to use a Run As profile with administrative privileges to enable it to run in a low-privilege environment.
Modified logical disk discovery to exclude mount points, because numerous rules use performance counters that are not applicable to mount points.
Fixed an issue in which an alert could be generated for Windows Activation too early.
Addressed an issue that was causing some discoveries to not show up correctly in the authoring section of the console.
Improved the product knowledge for the alert generated by the “A Service is Mis-configured” rule.
Fixed issue in which logical disk health check could have performance problems when run against clustered servers.
Fixed links to views in product knowledge articles.
Added Active Alerts views for Windows Server 2003 and Windows Server 2008.
Added retry logic to the "Probe Module: Is Feature Installed" module to avoid failures due to periodic time-outs.
Added a new integer property named "Size (MBytes) (Numeric)" to the Logical Disk class and updated the existing property "Size (Bytes) (String)" to clarify that it is a string. The new property can now be used in formulas and views as expected.

Make sure you also check out the new supported config statement and known issues regarding Server 2008 R2 and Windows 7 support:

http://support.microsoft.com/kb/974722

Community Cisco Network Hardware Management pack available

kevinhol — Wed, 16 Sep 2009 01:08:43 GMT

This guy has some serious talent.

http://operatingquadrant.com/2009/09/14/scom-advanced-snmp-monitoring-part-iii-the-completed-cisco-management-pack/

Make sure you check out all of Kris’s blog posts – amazingly good stuff there on using SNMP to accomplish monitoring that is otherwise very difficult, among other good posts.

Hard to believe its been two years

kevinhol — Sat, 12 Sep 2009 00:12:50 GMT

As I was looking at my blog today…. I noticed my first blog post was Sept of 2007. Hard to believe I have been blogging about OpsMgr for two years now. Even harder to believe (to me) is that I have been working at Microsoft for nearly 3 and a half years now. In this Company… time literally flies. I still sometimes feel like I am the new guy here…. like I just started. I am still cutting my baby teeth.

On blogging…. I never thought I would ever blog. I saw all the good SCOM blogs out there, and thought…. what do I possibly have to write about? I also used to see blogging in general as narcissistic. Sort of a “hey look at me!” approach to the industry or technology. I was wrong on that one. Technical blogs add a new dimension to the products we use, and the community that uses them. It helps us share information and neat ideas faster and simpler than we ever could before. It is also very good self-promotion for those whom make their living as consultants.

How I started: What happened was… I had this big list of SQL queries, that I used for MOM 2005. I used to host them on a personal website…. so that anytime I was onsite with a customer – we could pull them up, copy, and paste them, depending on what we were working on. I HATED writing SQL statements back then, because I was terrible at it….. so I cheated. I kept them all in a big notepad document, that I would upload to my website from time to time. As I shared my list of queries (mostly stolen or collected from hundreds of sources) I decided to start a MS blog – and post those queries there. My peers were always using them – so why not post them publicly? And the rest just took on a life of its own.

When OpsMgr 2007 hit RTM…. the need for real world examples and guidance was staggering. What a challenging product to learn from scratch. So from that point on, anytime I learned something from working with a customer that I thought would be valuable…. up on the blog it went.

I appreciate all the feedback I have gotten from you all, and your topics, issues, and questions. I enjoy blogging more than I ever thought I would… and I plan on continuing every chance I get.

Removing an old product connector

kevinhol — Fri, 11 Sep 2009 00:01:57 GMT

In OpsMgr – there is no simple way to remove a product connector once installed. There is no delete function in the UI:

What if you have an old connector that you no longer use? Or, if you quit using an Engyro connector, and started using a MS branded R2 connector?

It may be necessary to forcibly remove a connector from OpsMgr 2007.

*** Note – this process is not officially supported. There is no officially supported way to remove an old connector. It is only being demonstrated here for example purposes. I have used these steps with customers, and thus far no ill effects were discovered. These steps were gleaned from a MS newsgroup posting.

Open SQL Server Management Studio from the Start menu and run the following queries on the Operations Manager database.

Step 1: Find the ConnectorID:

select DisplayName,IsInitialized,ConnectorID from Connector,BaseManagedEntity

where Connector.BaseManagedEntityID=BaseManagedEntity.BaseManagedEntityID

This will return 3 Columns: The Display Name, Initialized Flag, and ConnectorID.

Find the ConnectorID of the connector you want to remove. Copy it to Notepad for safekeeping.

Step 2: Un-Initialize the Connector

If the Connector field "IsInitialized" is 1, then you will need to uninitialize the connector before deleting it.

If IsInitialized is 0, skip to step 3.

Use the p_ConnectorUpdate SP to uninitialize the connector:

The first column is YOUR connectorID, 2nd column is the bookmark (this should be NULL), and the 3rd column is the initialized state (0 in our case)

EXEC p_ConnectorUpdate '5fabdf4c-a1f8-43bf-ac18-e46e20bd470b',NULL,0

Make sure this worked – by re-running the initial query, and ensure the connector initialized state is now 0.

Step 3: Delete the Connector

Use the p_ConnectorDelete SP to delete the connector.

The first parameter is YOUR ConnectorID, the 2nd Parameter is the comments you want added to the alert history when the connector is deleted, and the 3rd parameter is the Modified By field you want added to the alert history. (I use NULL for the optional fields to keep it simple.)

EXEC p_ConnectorDelete '5fabdf4c-a1f8-43bf-ac18-e46e20bd470b',NULL,NULL

This procedure can take a long time if there is a lot of data still associated with the connector. Be Patient.

Now – I need to mention – there is also a very interesting community tool – which was written to manage product connectors. I have not used it – but it is worth a look:

http://www.systemcentercentral.com/Downloads/DownloadsDetails/tabid/144/IndexID/12581/Default.aspx