Today I was troubleshooting an OpsMgr Issue what seemed to be caused by using the wrong MOMIISModules.dll.

On some servers we saw the next OpsMgr EventID’s popping up:

Log Name: Operations Manager
Source: HealthService
Date: 28-9-2010 10:12:55
Event ID: 4507
Task Category: Health Service
Level: Error
Keywords: Classic
User: N/A
Computer: servername1.customer.corp
Description:Creation of module with CLSID "{64D5DB86-322D-4A18-AAA9-4BFE063624A5}" failed with error "0x800700C1" in rule "Microsoft.Windows.InternetInformationServices.2008.Discover24To32WebSites" running for instance "IIS Web Server" with id:"{8CD0918D-53F7-8B5A-9655-523CFD38BE5E}" in management group "MG1".

 

Followed by:

Log Name: Operations Manager
Source: HealthService
Date: 28-9-2010 10:12:55
Event ID: 1103
Task Category: Health Service
Level: Warning
Keywords: Classic
User: N/A
Computer: servername1.customer..corp
Description:
Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload. Management group "MG1". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s).

All the event errors are generated by the IIS 7.0 MP.

Background info:
OpsMgr Environment: OpsMgr 2007 SP1
IIS 7.0 MP and version: Windows Server 2008 Internet Information Services 7.0, version: 6.0.6539.0
Hotfixes installed: KB 954049; KB 957123 QFE; KB971541-ENU;

 

Troubleshooting steps:

1. Checking error info about 0x800700C1.
You can use the tool ERR.exe for this purpose. ERR.exe helps to determine error values from decimal and hexadecimal error codes in Microsoft Windows® operating systems.

E:\Err>err 0x800700C1
# as an HRESULT: Severity: FAILURE (1), Facility: 0x7, Code 0xc1 # for hex 0xc1 / decimal 193 :
SPECIAL_POOL_DETECTED_MEMORY_CORRUPTION bugcodes.h
SQL_193_severity_15 sql_err
# The object or column name starting with '%.*ls' is too # long. The maximum length is %d characters.
ERROR_BAD_EXE_FORMAT winerror.h
# %1 is not a valid Win32 application.
# 3 matches found for "0x800700C1"

This could indicate that there is something wrong one of the *Module.dlls.

2. Check KB articles for the correct File Versions and File Sizes.
First you need to check which dll versions are upgraded after the hotfix installations. The easiest way to do this is by going to the KB article and check the files names table.
Example:

According to hotfix (http://support.microsoft.com/kb/957123)

The next file versions should be updated for x-64 bit:

File name

File version

File size

Date

Time

Platform

Momiismodules.dll

6.0.6278.43

665,464

08-Dec-2008

21:17

x64

Mommodulemsgs.dll

6.0.6278.43

313,720

08-Dec-2008

21:17

x64

Mommodules.dll

6.0.6278.43

3,281,272

08-Dec-2008

21:17

x64

Mommodules2.dll

6.0.6278.43

351,608

08-Dec-2008

21:17

x64

Momagentinstaller.exe

6.0.6278.56

259,456

26-Mar-2009

15:45

x64

 

According to hotfix http://support.microsoft.com/kb/971541 (SP1 Update) the next file versions should be updated:

x-64:

File name

File version

File size

Date

Time

Platform

Momiismodules.dll

6.0.6278.100

685,952

2-Nov-09

23:08

x64

Mommodulemsgs.dll

6.0.6278.100

312,688

29-Sep-09

15:09

x64

Mommodules.dll

6.0.6278.100

3,352,448

2-Nov-09

23:08

x64

Mommodules2.dll

6.0.6278.100

363,392

2-Nov-09

23:08

x64

x-86:

File name

File version

File size

Date

Time

Platform

Momiismodules.dll

6.0.6278.100

363,888

2-Nov-09

23:16

x86

Mommodulemsgs.dll

6.0.6278.100

313,200

29-Sep-09

15:08

x86

Mommodules.dll

6.0.6278.100

1,860,464

2-Nov-09

23:16

x86

Mommodules2.dll

6.0.6278.100

253,296

2-Nov-09

23:16

x86

So we should expect to see a Momiismodules.dll file with a version of 6.0.6278.100 and size of 685,952 on our Windows 2008 64-bit IIS 7.0 Server.

3. The next steps is comparing above File Versions and File Sizes with the file information on the agents (having issues).

For this I created a vbscript and PowerShell script that queries WMI and uses a servers.txt file as input.

'Vbscript to find OpsMgr dll info for troubleshooting hotfix update issue
'Author: Stefan Stranger
'Date:   10-4-2010

On error resume next

Const strInputFile = "d:\temp\servers.txt" 
Const ForReading = 1
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objTextFile = objFSO.OpenTextFile _
    (strInputFile, ForReading)
Do Until objTextFile.AtEndOfStream
    strComputer = objTextFile.Readline
    Wscript.Echo "Server to be queried: " & strComputer


Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")

Set colFiles = objWMIService.ExecQuery _
     ("Select * from CIM_DataFile where Drive = 'C:' AND Path = '\\Program Files\\System Center Operations Manager 2007\\' and Extension='dll'")

For Each objFile in colFiles
    Wscript.Echo "****************************************"
    Wscript.Echo "Creation date: " & mid(objfile.CreationDate,5,2) & "/" &_
                mid(objfile.CreationDate,7,2) & "/" & mid(objfile.CreationDate,1,4)
    Wscript.Echo "Computer system name: " & objFile.CSName
    Wscript.Echo "Extension: " & objFile.Extension
    Wscript.Echo "File name: " & objFile.FileName
    Wscript.Echo "File size: " & objFile.FileSize
    Wscript.Echo "File type: " & objFile.FileType
    Wscript.Echo "Last accessed: " & mid(objfile.LastAccessed,5,2) & "/" &_
                mid(objfile.LastAccessed,7,2) & "/" & mid(objfile.LastAccessed,1,4)
    Wscript.Echo "Last modified: " & mid(objfile.LastModified,5,2) & "/" &_
                mid(objfile.LastModified,7,2) & "/" & mid(objfile.LastModified,1,4)
    Wscript.Echo "Manufacturer: " & objFile.Manufacturer
    Wscript.Echo "Name: " & objFile.Name
    Wscript.Echo "Path: " & objFile.Path
    Wscript.Echo "Version: " & objFile.Version
    Wscript.Echo "****************************************"
    Wscript.Echo
Next

loop

 

Save above vbscript to OpsMgrFileInfo.vbs and create a servers.txt file with on each line a server name.

Example (servers.txt):

server1.contoso.com
server2.contoso.com

 

Or you can use the PowerShell version:

###############################################################################            
# Get MOM dll's using PowerShell            
# This script retrieves the version of the mom dll's using WMI            
# Authors: Stefan Stranger (Microsoft)            
# Date 04-10-2010            
# Name: Get-OpsMgrFileVersion.ps1            
# v1.000 - 04/20/2010 - stefstr - initial sstranger's release            
###############################################################################            
            
#Get the servers from the servers.txt file            
$servers = get-content -path C:\Scripts\OpsMgr\servers.txt            
            
foreach ($server in $servers)            
{            
    get-wmiobject -query "Select * from CIM_DataFile where Drive = 'C:' AND Path = '\\Program Files\\System Center Operations Manager 2007\\' and Extension='dll'" -computer $server | select -property @{n='ComputerName';e={$_.__SERVER}}, FileName, Version, FileSize            
}

 

You can use out-gridview for even better viewing and filtering. You can even use the export-csv cmdlet to export the results to a csv file.

image

image

For our Windows 2008 64-bit IIS 7.0 server we found the next File Version info for the Momiismoduls.dll.

C:\Temp>cscript getfileversioninfo.vbs
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

Creation date: 2009-9-29
Computer system name: server1
Extension: dll
File name: MomIISModules
File size: 357888
File type: Application Extension
Last accessed: 2010-03-14
Last modified: 2009-09-29
Manufacturer: Microsoft Corporation
Name: c:\program files\system center operations manager 2007\momiismodules.dll
Path: \program files\system center operations manager 2007\
Version: 6.0.6278.100

 

As stated before we should expect to see a Momiismodules.dll file with a version of 6.0.6278.100 and size of 685,952 on our Windows 2008 64-bit IIS 7.0 Server.

So it seems that the File Size is incorrect and this indicates that instead of the 64-bit version of the Momiismodules.dll the 32-bit version is being used.

To confirm if I was correct I used the EXE Explorer from Mitec. It reads and displays executable file properties and structure.

clip_image002

As you see the MomIISModules.dll is 32-bit instead of 64-bit.

After copying a 64-bit momiismodules.dll to the server discoveries started to work again.

 

No idea though how it was possible to have a 32-bit momiismodules.dll on a 64-bit OpsMgr Agent and OS. If you have any ideas please let me know.

 

Have fun troubleshooting!