Here is an interesting issue I came across. I did not encounter this too often but still, it's very interesting and it was very fun to troubleshoot! :D

In case you get the same error I suggest opening a case with Microsoft Support because it will be easier. Still, let me tell the story just for fun.

Some Alerts were not appearing in Reports and when we checked these were definitely not written to the Data Warehouse database - the AlertStage table for example where it ends up in the first place when Alerts are synched from the operational database to the data warehouse database. Ok then, so first thing to check is the Operations Manager event log on all Management Servers in case you have OM 2012 because any Management Server from the All Management Server Resource Pool can get the role of Data Warehouse Synchronization Server *OR* if you are on OM 2007 still, then *only* the Root Management Server will have this role.

Interestingly enough, we were seeing error events 31551 in the Operations Manager event log on the Server which had the Data Warehouse Synchronization Server role:

  • please keep in mind that the error event ID (31551) is pretty general and it is not the event ID that matters, it is its description that tells us if we have the same issue or not

  Failed to store data in the Data Warehouse. The operation will be retried.
  Exception 'InvalidOperationException': The given value of type String from the data source cannot be converted to type nvarchar of the specified target column.

 

So what does this actually mean?!

Well the way that the Alert data is inserted into the Data Warehouse when synchronized from the operational database is by using SQL Bulk Insert because of the amount of data needed to be inserted. Well when Bulk Insert is used the class used will check the values of the data being inserted against the database tables field types and limitations. It will perform a check in this case so that all the values of an Alert (Item) will "fit" into the fields of the AlertStage table of the data warehouse database. If any of the fields (like Alert Title) are bigger than the MAX value of the table (AlertStage) then it will fail with that error message. The AlertStage Title field has NVARCHAR(255) as maximum length and if we have an Alert which has a title bigger than 255, then we will encounter this issue - as of course any other field which does not fit - like the CustomProperty fields of an Alert.

 

Here is a screenshot as FYI for the field types and definitions of the AlertStage table from the data warehouse database:

 

So now the question is ... how do we figure out *which* of the Alerts has a longer string *than* *which* property (field) ?! Well because we are doing a Bulk Insert here from the .NET SQL Class we cannot see this by tracing the SQL Server there ... it never ends up trying to insert this into SQL because it fails locally in the code on the Management Server which is trying to do this action. Right ... so how to figure this one out?!

Debugging my friends, the answer to everything, hehe! :D

Ok so we need to look at ... what process here? Well as you know (or will find out now) the Management Servers as well as the Agents have the "core" service called HealthService.exe. This process (service) however, does *NOT* do the heavy-lifting. It will spawn MonitoringHost.exe processes which will do the work for it after loading the necessary modules (DLL's) in them which are needed for the specific tasks they are assigned to do.

So which MonitoringHost.exe process will it be which runs the Data Warehouse Synchronization workflow? Well this would have to be a MonitoringHost.exe process which is running under the Data Warehouse Writer Account would it not? ;)

  • in case you have the same account used for more roles or something you can use Process Explorer to check out the loaded DLLs in the MonitoringHost.exe processes and find out which has the Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.dll DLL loaded

Because of the fact that the process is *NOT* crashing when throwing this Exception - as it is handled - we cannot simply get a memory dump of that process. So we either attach the debugger (WinDbg) live to the environment *OR* we can use another nifty tool which is called Time Travel Tracing to get the "dynamic" memory dump of that process so that we can "step" through the process execution. We could also use a tool like DebugDiag which can be set to write a memory dump of a process on a custom .NET Exception like in this example.

Either way, whatever method we will use here, we end up in the "same" scenario where we need to set up a breakpoint of the Exception we are interested in (if not using DebugDiag which writes the memory dump on exactly that Exception). So I will continue with this further on with the example of attaching to the MonitoringHost.exe process in question directly in the live environment.

Let's start shall we? :D We are in the debugger now and let's make sure we have loaded the correct SOS .NET Debugger Extension version:    .cordll -ve -u -l

Now we can continue by creating the breakpoint to stop when hitting the exception we are interested in which in this case is System.InvalidOperationException:    !StopOnException -create System.InvalidOperationException 1

And then we enter g in the debugger so that the process continues execution until we hit the breakpoint (Exception). When it stops and the debugger breaks in, we use !PrintException to see the exception details:

0:008> !PrintException
Exception object: 0000000201f1dc38
Exception type:   System.InvalidOperationException
Message:          The given value of type String from the data source cannot be converted to type nvarchar of the specified target column.
InnerException:   System.InvalidOperationException, Use !PrintException 0000000201f1d6d8 to see more.
StackTrace (generated):
    SP               IP               Function
    00000000075FC7A0 000007FEF338D5A8 system_data_ni!System.Data.SqlClient.SqlBulkCopy.ConvertValue(System.Object, System.Data.SqlClient._SqlMetaData)+0x19ec38
    00000000075FED10 000007FEF31EDD39 system_data_ni!System.Data.SqlClient.SqlBulkCopy.WriteToServerInternal()+0x989
    00000000075FEE00 000007FEF31EE263 system_data_ni!System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServer(Int32)+0x463
    00000000075FEED0 000007FEF31EE945 system_data_ni!System.Data.SqlClient.SqlBulkCopy.WriteToServer(System.Data.IDataReader)+0x125
    00000000075FEF40 000007FF0018DA89 microsoft_enterprisemanagement_datawarehouse_dataaccess!Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.Commands.DataWarehouseSqlBulkInsertCommand.Execute()+0x99

StackTraceString: <none>
HResult: 80131509

As we can see we have a nested Exception here, so let's have a look at this as well:

0:008> !PrintException 0000000201f1d6d8
Exception object: 0000000201f1d6d8
Exception type:   System.InvalidOperationException
Message:          String or binary data would be truncated.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    00000000075FEBB0 000007FEF31EEF5A system_data_ni!System.Data.SqlClient.SqlBulkCopy.ConvertValue(System.Object, System.Data.SqlClient._SqlMetaData)+0x5ea

StackTraceString: <none>
HResult: 80131509

Ok, so this is interning - now how do we find out which table field is affected here? It is going to be a System.Data.SqlClient._SqlMetaData object and should be in the Managed Heap here of this Thread we stopped on. Thus we can use !DumpStackObjects to see all the objects:

0:008> !DumpStackObjects
OS Thread Id: 0x3cb4 (8)
RSP/REG          Object           Name

[...SNIPPED...]

00000000075FA218 00000001c01beca0 System.Data.SqlClient.TdsParserStateObject
00000000075FA370 00000001c01beb90 System.Data.SqlClient.TdsParser
00000000075FA490 0000000201f1dc38 System.InvalidOperationException
00000000075FA8B0 0000000201f12240 System.Data.SqlClient._SqlMetaData
00000000075FA8C8 0000000201f1d400 System.Int32[]
00000000075FA8F8 0000000201f1d2c0 System.Object[]    (System.Object[])
00000000075FABD0 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075FABF0 0000000201ee0be8 System.Data.SqlClient.SqlBulkCopyColumnMappingCollection
00000000075FAE00 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075FAE20 0000000201ee0be8 System.Data.SqlClient.SqlBulkCopyColumnMappingCollection
00000000075FB128 0000000201f1dc38 System.InvalidOperationException
00000000075FB410 00000001bfe40488 System.String   
00000000075FB4D0 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075FB4F0 0000000201ee0be8 System.Data.SqlClient.SqlBulkCopyColumnMappingCollection
00000000075FB880 0000000201f1eaf0 System.Environment+ResourceHelper+GetResourceStringUserData
00000000075FBA70 0000000201f1eb68 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
00000000075FBA78 0000000201f1eaf0 System.Environment+ResourceHelper+GetResourceStringUserData
00000000075FBAA0 0000000201f1eb68 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
00000000075FBAB0 0000000201f1eb68 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
00000000075FBB08 0000000201f1eb28 System.Runtime.CompilerServices.RuntimeHelpers+TryCode
00000000075FBB10 0000000201f1eaf0 System.Environment+ResourceHelper+GetResourceStringUserData

[...SNIPPED...]

Ok, from that big list, let's grab the object ID of the object we are interested in (System.Data.SqlClient._SqlMetaData) - there will be more entries, but it should have the same Object ID: !DumpObj 0000000201f12240

0:008> !DumpObj 0000000201f12240
Name:        System.Data.SqlClient._SqlMetaData
MethodTable: 000007fef2d98b90
EEClass:     000007fef2c36e60
Size:        224(0xe0) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef2d91c50  4001bc9       80         System.Int32  1 instance               12 type
000007fef880c158  4001bca       8c          System.Byte  1 instance              231 tdsType
000007fef880c158  4001bcb       8d          System.Byte  1 instance              255 precision
000007fef880c158  4001bcc       8e          System.Byte  1 instance              255 scale
000007fef880c7d8  4001bcd       84         System.Int32  1 instance              512 length
000007fef2d925a0  4001bce        8 ...ient.SqlCollation  0 instance 0000000201f139c0 collation
000007fef880c7d8  4001bcf       88         System.Int32  1 instance             1251 codePage
0000000000000000  4001bd0       10                       0 instance 000000017fe6eb28 encoding
000007fef880d608  4001bd1       8f       System.Boolean  1 instance                1 isNullable
000007fef880d608  4001bd2       90       System.Boolean  1 instance                0 isMultiValued
000007fef88068f0  4001bd3       18        System.String  0 instance 0000000000000000 udtDatabaseName
000007fef88068f0  4001bd4       20        System.String  0 instance 0000000000000000 udtSchemaName
000007fef88068f0  4001bd5       28        System.String  0 instance 0000000000000000 udtTypeName
000007fef88068f0  4001bd6       30        System.String  0 instance 0000000000000000 udtAssemblyQualifiedName
000007fef8808278  4001bd7       38          System.Type  0 instance 0000000000000000 udtType
000007fef88068f0  4001bd8       40        System.String  0 instance 0000000000000000 xmlSchemaCollectionDatabase
000007fef88068f0  4001bd9       48        System.String  0 instance 0000000000000000 xmlSchemaCollectionOwningSchema
000007fef88068f0  4001bda       50        System.String  0 instance 0000000000000000 xmlSchemaCollectionName
000007fef2d928b8  4001bdb       58 ...qlClient.MetaType  0 instance 000000017fe701c0 metaType
000007fef88068f0  4001bdc       60        System.String  0 instance 0000000000000000 structuredTypeDatabaseName
000007fef88068f0  4001bdd       68        System.String  0 instance 0000000000000000 structuredTypeSchemaName
000007fef88068f0  4001bde       70        System.String  0 instance 0000000000000000 structuredTypeName
0000000000000000  4001bdf       78                       0 instance 0000000000000000 structuredFields
000007fef88068f0  4001be0       98        System.String  0 instance 0000000201f139f0 column
000007fef88068f0  4001be1       a0        System.String  0 instance 0000000000000000 baseColumn
000007fef32c3cb0  4001be2       b0 ...ultiPartTableName  1 instance 0000000201f122f0 multiPartTableName
000007fef880c7d8  4001be3       94         System.Int32  1 instance               25 ordinal
000007fef880c158  4001be4       91          System.Byte  1 instance                2 updatability
000007fef880c158  4001be5       a8          System.Byte  1 instance                0 tableNum
000007fef880d608  4001be6       a9       System.Boolean  1 instance                0 isDifferentName
000007fef880d608  4001be7       aa       System.Boolean  1 instance                0 isKey
000007fef880d608  4001be8       ab       System.Boolean  1 instance                0 isHidden
000007fef880d608  4001be9       ac       System.Boolean  1 instance                0 isExpression
000007fef880d608  4001bea       ad       System.Boolean  1 instance                0 isIdentity
000007fef880d608  4001beb       ae       System.Boolean  1 instance                0 isColumnSet
000007fef880c158  4001bec       af          System.Byte  1 instance                0 op
000007fef88142d0  4001bed       92        System.UInt16  1 instance                0 operand

Ok, so using the column field of this object we can see that it is the CustomProperty9 column which we know has a max 255 character limitation - here we need to dump the pointer address of the object + the offset where the column table is found:

0:008> !DumpObj poi(0000000201f12240+98)
Name:        System.String
MethodTable: 000007fef88068f0
EEClass:     000007fef838ed78
Size:        50(0x32) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      CustomField9
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef880c7d8  4000103        8         System.Int32  1 instance               12 m_stringLength
000007fef880b318  4000104        c          System.Char  1 instance               43 m_firstChar
000007fef88068f0  4000105       10        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000003e9470:00000001bfe40488 <<

So now we need to get a list of Alerts which the process is trying to insert and check the CustomProperty9 of all of them and find which is bigger than 255. We need to dump the object which is having all the alerts which is System.Data.SqlClient.SqlBulkCopy - just like before !DumpStackObjects and look for the object address:

0:008> !DumpStackObjects
OS Thread Id: 0x3cb4 (8)
RSP/REG          Object           Name
00000000075F5908 0000000201f1dc38 System.InvalidOperationException
00000000075F59A8 0000000201f1dc38 System.InvalidOperationException
00000000075F5A30 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F5A38 0000000201f1dc38 System.InvalidOperationException
00000000075F5A48 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F5A50 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F5C90 0000000201f1dc38 System.InvalidOperationException
00000000075F60B0 00000002000b13d8 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.Commands.DataWarehouseSqlBulkInsertCommand
00000000075F60C8 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F60D0 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F63D0 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F63E8 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F63F0 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F6600 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F6618 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F6620 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F6928 0000000201f1dc38 System.InvalidOperationException
00000000075F6CD0 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F6CE8 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F6CF0 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F7640 0000000201f12240 System.Data.SqlClient._SqlMetaData
00000000075F7968 0000000201f1dc38 System.InvalidOperationException
00000000075F7B10 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075F7B40 0000000201f1dc38 System.InvalidOperationException
00000000075F7B58 000000014007fa40 System.RuntimeType

[...SNIPPED...]

Ok now let's have a look at the list shall we - dumping the object first as before using !DumpObj and the object address:

0:008> !DumpObj 0000000201ee0948
<Note: this object has an invalid CLASS field>
Name:        System.Data.SqlClient.SqlBulkCopy
MethodTable: 000007fef32c3868
EEClass:     000007fef2c4eae0
Size:        176(0xb0) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef880c7d8  40016e9       78         System.Int32  1 instance                0 _batchSize
000007fef880d608  40016ea       a0       System.Boolean  1 instance                0 _ownConnection
000007fef32a4370  40016eb       7c         System.Int32  1 instance                0 _copyOptions
000007fef880c7d8  40016ec       80         System.Int32  1 instance               30 _timeout
000007fef88068f0  40016ed        8        System.String  0 instance 000000013fe4ea28 _destinationTableName
000007fef880c7d8  40016ee       84         System.Int32  1 instance                0 _rowsCopied
000007fef880c7d8  40016ef       88         System.Int32  1 instance                0 _notifyAfter
000007fef880c7d8  40016f0       8c         System.Int32  1 instance                0 _rowsUntilNotification
000007fef880d608  40016f1       a1       System.Boolean  1 instance               32 _insideRowsCopiedEvent
000007fef8805a48  40016f2       10        System.Object  0 instance 00000002000b4c10 _rowSource
000007fef2d91038  40016f3       18 ...ent.SqlDataReader  0 instance 0000000000000000 _SqlDataReaderRowSource
000007fef32c3a48  40016f4       20 ...MappingCollection  0 instance 0000000201ee09f8 _columnMappings
000007fef32c3a48  40016f5       28 ...MappingCollection  0 instance 0000000201ee0be8 _localColumnMappings
000007fef2d8fba8  40016f6       30 ...ent.SqlConnection  0 instance 0000000201ee0610 _connection
000007fef32c3b70  40016f7       38 ...nt.SqlTransaction  0 instance 0000000000000000 _internalTransaction
000007fef32c3b70  40016f8       40 ...nt.SqlTransaction  0 instance 0000000000000000 _externalTransaction
000007fef32a4058  40016f9       90         System.Int32  1 instance                1 _rowSourceType
000007fef2d9b880  40016fa       48  System.Data.DataRow  0 instance 74042e00b98a6449 _currentRow
000007fef880c7d8  40016fb       94         System.Int32  1 instance      -1738182535 _currentRowLength
000007fef3290ab0  40016fc       98         System.Int32  1 instance       1765434373 _rowState
000007fef880f240  40016fd       50 ...tions.IEnumerator  0 instance 693a640598657079 _rowEnumerator
000007fef2d973e8  40016fe       58 ...lClient.TdsParser  0 instance 00000001c01beb90 _parser
000007fef2d97ac8  40016ff       60 ...ParserStateObject  0 instance 0000000000000000 _stateObj
000007fef8810158  4001700       68 ...ections.ArrayList  0 instance 0000000201f18ca8 _sortedColumnMappings
000007fef32c3630  4001701       70 ...opiedEventHandler  0 instance 4c4d582f31303032 _rowsCopiedEventHandler
000007fef880c7d8  4001703       9c         System.Int32  1 instance           212855 _objectID
000007fef880c7d8  4001702      9d8         System.Int32  1   static           212855 _objectTypeCount

So the _rowSource is what we are interested in - let's dump this shall we? Again using !DumpObj with the object address + field offset:

0:008> !DumpObj poi(0000000201ee0948+10)
Name:        Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.Commands.BulkDataReader
MethodTable: 000007ff002c4338
EEClass:     000007ff002a4d90
Size:        64(0x40) bytes
File:        C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef880d608  40000e5       10       System.Boolean  1 instance                0 isClosed
000007ff002c3b70  40000e6        8 ...DataReaderAdaptor  0 instance 00000002000b1480 dataAdaptor
000007fef8828658  40000e7       18          System.Guid  1 instance 00000002000b4c28 dataSetId
000007fef8828658  40000e8       28          System.Guid  1 instance 00000002000b4c38 managementGroupId

Ok what now? Now we need to look at the dataAdapter, like before ;) But wait ... we used poi() before - well, we'll use it twice this time: 

0:008> !DumpObj poi(poi(0000000201ee0948+10)+8)
Name:        Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.AlertDataReaderAdaptor
MethodTable: 000007ff002cfa98
EEClass:     000007ff00301318
Size:        48(0x30) bytes
File:        C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef2d9a548  4000035        8 ...em.Data.DataTable  0 instance 00000002000b1928 schemaTable
000007fef880adf8  4000036       10      System.Object[]  0 instance 00000002001d89e8 dataItems
000007fef880c7d8  4000037       18         System.Int32  1 instance                0 currentIndex
000007fef880c7d8  4000038       1c         System.Int32  1 instance                0 startIndex
000007fef880c7d8  4000039       20         System.Int32  1 instance             1000 itemsToProcessCount

We are interested in the dataItems object - but wait ... it is an array as we can see from the [] characters in System.Object[] so we can use a nifty other command called !DumpArray here - again with triple pointer function:

0:008> !DumpArray poi(poi(poi(0000000201ee0948+10)+8)+10)
Name:        Microsoft.EnterpriseManagement.HealthService.DataItemBase[]
MethodTable: 000007fef880adf8
EEClass:     000007fef838fc68
Size:        8032(0x1f60) bytes
Array:       Rank 1, Number of elements 1000, Type CLASS
Element Methodtable: 000007ff000474b8
[0] 0000000140b851a0
[1] 0000000140b8a950
[2] 0000000140b8d420
[3] 0000000140b8feb8
[4] 0000000140b92950
[5] 0000000140b94eb0
[6] 0000000140b97428
[7] 0000000140b999a0
[8] 0000000140b9aba8
[9] 0000000140b9d6b8
[10] 0000000140ba0150
[11] 0000000140ba2be8
[12] 0000000140ba5648

[...SNIPPED...]

Ok interesting, so this is an Array which is holding items of type Microsoft.EnterpriseManagement.HealthService.DataItemBase but when we dump the array we only get the address of each such object, so we would need to dump each of them using !DumpObj and check them out - wow! ... really? there are a looot of them ... Ok, let's worry about that a little later and now just dump one of the items (first one as example) just to see what we need to check there for each one:

0:008> !DumpObj 0000000140b851a0
Name:        Microsoft.EnterpriseManagement.Mom.AlertSubscriptionModule.DataItemAlertSubscription
MethodTable: 000007ff002cdc70
EEClass:     000007ff002fe738
Size:        568(0x238) bytes
File:        C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.AlertSubscriptionDataSourceModule.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef88296c8  400001a       18      System.DateTime  1 instance 0000000140b851b8 dateCreated
000007fef8828658  400001b       20          System.Guid  1 instance 0000000140b851c0 sourceHealthServiceId
000007fef88068f0  400001c        8        System.String  0 instance 00000001400a54d8 itemXml
000007fef88068f0  400001d       10        System.String  0 instance 0000000140b862b8 dataTypeName
000007fef8828658  4000019        8          System.Guid  1   static 000000023fe46128 localHealthServiceId
000007ff002cd6f8  400009b      118         System.Int32  1 instance                0 subscriptionType
000007ff002cd850  400009c      11c         System.Int32  1 instance                0 subscriptionProperty
000007fef8828658  400009d      138          System.Guid  1 instance 0000000140b852d8 id
000007fef88068f0  400009e       30        System.String  0 instance 0000000140b864b8 name
000007fef88068f0  400009f       38        System.String  0 instance 0000000140b86520 description
000007fef8828658  40000a0      148          System.Guid  1 instance 0000000140b852e8 baseManagedEntityId
000007fef8828658  40000a1      158          System.Guid  1 instance 0000000140b852f8 problemId
000007fef880d608  40000a2      130       System.Boolean  1 instance                0 createdByMonitor
000007fef8828658  40000a3      168          System.Guid  1 instance 0000000140b85308 workflowId
000007fef8827af0  40000a4      120        System.UInt32  1 instance                0 resolutionState
000007ff002cd9a8  40000a5      124         System.Int32  1 instance                2 priority
000007ff002cdb00  40000a6      128         System.Int32  1 instance                1 severity
000007fef88068f0  40000a7       40        System.String  0 instance 0000000140b86778 category
000007fef88068f0  40000a8       48        System.String  0 instance 0000000000000000 owner
000007fef88068f0  40000a9       50        System.String  0 instance 0000000000000000 resolvedBy
000007fef88296c8  40000aa      178      System.DateTime  1 instance 0000000140b85318 timeRaised
000007fef88296c8  40000ab      180      System.DateTime  1 instance 0000000140b85320 timeAdded
000007fef88296c8  40000ac      188      System.DateTime  1 instance 0000000140b85328 lastModified
000007fef88296c8  40000ad      190      System.DateTime  1 instance 0000000140b85330 lastModifiedExceptRepeatCount
000007fef88068f0  40000ae       58        System.String  0 instance 0000000140b86f70 lastModifiedBy
000007fef88296c8  40000af      198      System.DateTime  1 instance 0000000140b85338 timeResolved
000007fef88296c8  40000b0      1a0      System.DateTime  1 instance 0000000140b85340 timeResolutionStateLastModified
000007fef88296c8  40000b1      1a8      System.DateTime  1 instance 0000000140b85348 timeResolutionStateLastModifiedInDB
000007fef88068f0  40000b2       60        System.String  0 instance 0000000140b86f98 customField1
000007fef88068f0  40000b3       68        System.String  0 instance 0000000140b86fe0 customField2
000007fef88068f0  40000b4       70        System.String  0 instance 0000000140b87020 customField3
000007fef88068f0  40000b5       78        System.String  0 instance 0000000140b87060 customField4
000007fef88068f0  40000b6       80        System.String  0 instance 0000000140b870a0 customField5
000007fef88068f0  40000b7       88        System.String  0 instance 0000000140b870f8 customField6
000007fef88068f0  40000b8       90        System.String  0 instance 0000000140b87128 customField7
000007fef88068f0  40000b9       98        System.String  0 instance 0000000140b87160 customField8
000007fef88068f0  40000ba       a0        System.String  0 instance 0000000140b871a0 customField9
000007fef88068f0  40000bb       a8        System.String  0 instance 0000000140b873c0 customField10
000007fef88068f0  40000bc       b0        System.String  0 instance 0000000000000000 ticketId
000007fef88068f0  40000bd       b8        System.String  0 instance 0000000140b87438 context
000007fef88296c8  40000be      1b0      System.DateTime  1 instance 0000000140b85350 lastModifiedByNonConnector
000007fef8828658  40000bf      1b8          System.Guid  1 instance 0000000140b85358 connectorId
000007fef880c7d8  40000c0      12c         System.Int32  1 instance                0 repeatCount
000007fef8828658  40000c1      1c8          System.Guid  1 instance 0000000140b85368 alertStringId
000007fef88068f0  40000c2       c0        System.String  0 instance 0000000140b88848 alertParams
000007fef88068f0  40000c3       c8        System.String  0 instance 0000000000000000 siteName
000007fef88068f0  40000c4       d0        System.String  0 instance 0000000140b88b30 baseManagedEntityFullName
000007fef88068f0  40000c5       d8        System.String  0 instance 0000000000000000 baseManagedEntityPath
000007fef88068f0  40000c6       e0        System.String  0 instance 0000000140b88bb8 baseManagedEntityName
000007fef88068f0  40000c7       e8        System.String  0 instance 0000000140b88bf8 baseManagedEntityDisplayName
000007fef88068f0  40000c8       f0        System.String  0 instance 0000000140b88c30 resolutionStateName
000007fef88068f0  40000c9       f8        System.String  0 instance 0000000000000000 timeZone
000007fef88068f0  40000ca      100        System.String  0 instance 0000000140b88c50 languageCode
000007fef88296c8  40000cb      1d8      System.DateTime  1 instance 0000000140b85378 timeRaisedLocal
000007fef88296c8  40000cc      1e0      System.DateTime  1 instance 0000000140b85380 timeAddedLocal
000007fef88296c8  40000cd      1e8      System.DateTime  1 instance 0000000140b85388 lastModifiedLocal
000007fef88296c8  40000ce      1f0      System.DateTime  1 instance 0000000140b85390 lastModifiedExceptRepeatCountLocal
000007fef88296c8  40000cf      1f8      System.DateTime  1 instance 0000000140b85398 timeResolvedLocal
000007fef88296c8  40000d0      200      System.DateTime  1 instance 0000000140b853a0 timeResolutionStateLastModifiedLocal
000007fef88296c8  40000d1      208      System.DateTime  1 instance 0000000140b853a8 timeResolutionStateLastModifiedInDBLocal
000007fef88296c8  40000d2      210      System.DateTime  1 instance 0000000140b853b0 lastModifiedByNonConnectorLocal
000007fef88292a0  40000d3      218      System.TimeSpan  1 instance 0000000140b853b8 queryExecutionTimeSpan
000007fef88296c8  40000d4      220      System.DateTime  1 instance 0000000140b853c0 dataItemCreateTime
000007fef88296c8  40000d5      228      System.DateTime  1 instance 0000000140b853c8 dataItemCreateTimeLocal
000007fef88068f0  40000d6      108        System.String  0 instance 0000000000000000 tfsWorkItemId
000007fef88068f0  40000d7      110        System.String  0 instance 0000000000000000 tfsWorkItemOwner

Alright, so we know that the CustomField9 which we are interested in had offset a0 - so now we can dump it directly:

0:008> !DumpObj poi(0000000140b851a0+a0)
Name:        System.String
MethodTable: 000007fef88068f0
EEClass:     000007fef838ed78
Size:        540(0x21c) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      SOME_STRING_HERE_WON'T_TELL_YOU_THE_VALUE_HA!
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef880c7d8  4000103        8         System.Int32  1 instance              257 m_stringLength
000007fef880b318  4000104        c          System.Char  1 instance               57 m_firstChar
000007fef88068f0  4000105       10        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000003e9470:00000001bfe40488 <<

Cool, so now we know what we want and need to dump here - so what's next? Well we have this nifty .foreach and .shell command to use the Windows CMD Find command on the output as we can see in these examples WinDbg Scripting :D

  • because the list all object addresses command will also output other crap when it gets to object index and/or Alert DataItems with EMPTY CustomProperty9 fields, we need to exclude "Invalid parameter" from the output

0:008> .shell -ci ".foreach (obj { !DumpArray poi(poi(poi(0000000201ee0948+10)+8)+10) }) { !DumpObj poi(${obj}+a0) }" FIND /V "Invalid parameter"
<Note: this object has an invalid CLASS field>
Invalid object
Integrated managed debugging does not support enumeration of symbols.
See http://dbg/managed.htm for more details.
<Note: this object has an invalid CLASS field>
Invalid object
Name:        System.String
MethodTable: 000007fef88068f0
EEClass:     000007fef838ed78
Size:        540(0x21c) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      SOME_STRING_HERE_WON'T_TELL_YOU_THE_VALUE_HA!
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef880c7d8  4000103        8         System.Int32  1 instance              257 m_stringLength
000007fef880b318  4000104        c          System.Char  1 instance               57 m_firstChar
000007fef88068f0  4000105       10        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000003e9470:00000001bfe40488 <<
Name:        System.String
MethodTable: 000007fef88068f0
EEClass:     000007fef838ed78
Size:        540(0x21c) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      SOME_STRING_HERE_WON'T_TELL_YOU_THE_VALUE_HA!
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef880c7d8  4000103        8         System.Int32  1 instance              257 m_stringLength
000007fef880b318  4000104        c          System.Char  1 instance               57 m_firstChar
000007fef88068f0  4000105       10        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000003e9470:00000001bfe40488 <<
.shell: Process exited

---

You do know the FIND command in CMD right? ;)

Now from the output there check the length of the strings and when you found the one which has a size bigger than 255 that's your hit there!

Well it turns out that we had a 3rd Party Management Pack there which was using the CustomField9 for ... well something whatever ... and thus we either remove this MP or we engage the Vendor.

Problem solved! :D

By the way - there are some cool WinDbg extensions out there which may help you a lot with debugging stuff in general so that you don't need to use WinDbg scripting. These are more than awesome help for .NET Debugging.

A good example of such an extension would be SOSEx.

Have fun debugging!!! :)