Although Microsoft Operations Manager 2005 and Operations Manager 2007 both have the concept of targeting, the way targeting works in Operations Manager 2007 has completely changed from the previous version. To effectively deploy new rules and monitors, it is critical to understand the difference. Consider the following scenario:
In Microsoft Operations Manager 2005, to monitor the three SQL Server databases shown in the preceding figure, you would create a computer group that includes Server 1 and Server 2. You then would create a specific rule, put it into a Processing Rule Group (PRG), and then target the rule to the computer group. However, this method has some shortcomings:
In Operations Manager 2007, you no longer target computer groups. Instead, you target specific classes. To monitor the SQL Server objects shown in the following figure, you create a monitor and then target it to the SQL Database class.
This has the following advantages:
A couple of people asked me questions on how data flowed in OpsMgr 2007 this weekend so I thought this would be a good topic for this week’s blog. Unfortunately this is not documented anywhere.
Data Type Flow:
Events/Perf/State data is sent from OpsMgr 2007 agents to their primary Management Server which then write to the Operational database and data warehouse simultaneously.
Heartbeats are sent to the Root Management Server from the management servers in the environment. If an OpsMgr 2007 agent goes down for some reason and therefore does not report to its primary or secondary management servers the root management server will change the state of the agent from healthy to unavailable. Only these state changes are registered in the Operational Database, otherwise heartbeats data is not written in the Operational Database.
Alert/Discovery Agents send alert and discovery data to their primary management servers which then write directly to the Operational Database and the SDK service on the root management server then pulls the data from the Operational Database and displays in your console. The sync module on the root management server then reflects the alerts and discovery data changes into the data warehouse.
- Satya
Network Bandwidth Utilization for the various OpsMgr 2007 Roles
I recently started owning performance and scale for OpsMgr and it is definitely one of the most interesting and challenging areas I have ever worked on. I know the first question that is popping up in most of your minds is why is console performance so darn slow in OpsMgr 2007? There are various reasons for this which I will divulge at another time but the one thing I will assure you is that the console performance with Service Pack 1 is a lot faster (Geo Metro to BMW M3 faster, if that is a valid comparison). But I wanted to dedicate today’s blog to talk about the network bandwidth utilization as it seems to be a question a lot of customers have been asking. There are essentially three sections to discuss a) Agent to Root Management Servers\ Management Server\ Gateway Server b) Root Management Server\MS to Database c) Audit Forwarders to Audit Collectors.
a) Agent to Root Management Servers\ Management Server (MS)\ Gateway Servers
The amount of data sent to MS is based on the kind of Management Packs (MPs) you have in your environment and how you as the end user have tuned these MPs. Some MPs by default send more data to MSs than others and to get an idea of what these MPs are you can view the table attached in the doc. Data sent between agents and MSs is always compressed, in our test environment with the Active Directory, Base OS, DNS and OpsMgr MPs we noticed that the estimated bandwidth utilization was about 500 bytes per sec on a single agent and about 75 kilobytes per sec on server for 150 agents. We also did a test with all the MPs that are there out of the box and with an additional stress test MPs (simulating real world load) and we noticed that about 200Kbytes per sec received by a Management Server for 2000 agents. So as you can see from the numbers that the data sent between the agent and MS is compressed. The one common question that we get is the minimum bandwidth requirement for agent to MS in the supported configurations document is 64Kbps but my customer has a few servers that only have 56Kbps. Can we support this?. If your agent data packet size is small enough for the bandwidth you have then it should not be a problem. Our Microsoft product support folks (PSS) have always been very accommodating so while you maybe outside the support requirements they will not stop from helping you troubleshoot your issues. This is definitely one of the big perks of using Microsoft products and having Microsoft support. Gateway server are basically proxy agents that tunnel data from multiple agents to a Management Server. We have seen bandwidth utilization of about 22Kbytes per sec received by Gateway Servers for 400 agents with all the out of the box Management Packs and some stress Management Packs.
b) Root Management Server (RMS) \ Management Server(MS) to Database(DB) and Data Warehouse (DW)
In OpsMgr 2007 the RMS and MS both write directly to the DB and DW. There is no DTS jobs like we had in MOM 2005. Since, the RMS and MS write directly to the DB and DW the data is not compressed and the size of data is larger as well. What we recommend many customers to do is to have their RMS\ MS close to the DB and DW and have fast links between them. It is much better to have the agents in remote geographic locations report to an MS than to have management server in remote geographic locations write to a DB and DW.
c) Audit Forwarders to Audit Collectors
While I am no expert in Audit Collection as a feature I will share with you the data we have collected for ACS so far. My colleague Joseph is the definitive ACS guru and should be the ultimate source of information on this topic.
ACS forward events in near real time, rather than batching them together. This is different from how MOM2005 sends events. Therefore when we say ACS bandwidth utilization is 100 bytes per events and a system generates ~27 events/sec, you can literally translate that to 2.7KB per sec
If there’s a loss of network connectivity, the forwarder will resend all security events that are not confirmed to be written to the DB by the collector. The forwarder sends a heartbeat (in the form of an event) to the collector every 45 seconds. If the collector does not receive the default of 3 heartbeats from the forwarder, the collector will drop the connection and the forwarder will automatically re-initiate the connection (if it is alive)
The size of a typical security event when it is being transmitted from the managed system to the ACS collector is usually less than 100 bytes. The size of a typical security event when it has been recorded in the ACS SQL database is less than 0.5KB. Typical CPU and memory utilization be for an agent assuming that ACS functionality is enabled also on the managed system CPU is typically is less than 1% and memory is about 4-6 MB.
Joseph in one of his mail threads to a OpsMgr discussion alias had written a quick and dirty script which is attached to the blog(rename it back to .vbs) that helps you count the number of events generated per sec on the local computer.
(It can run against remote computer, just supply the computer name as an argument, but it seems to be slow…) Run it like “CScript SecurityEventPerSecond.vbs >> NumOfEvtsGenPerSec.csv” and just load the csv in Excel to calculate the average. This can be useful in situation where it is not possible to install a pilot ACS collector in order to measure incoming event rate (by looking at the perf counter ACS Collector\Incoming Event per Sec)
Satya Vel | Program Manager | System Center |
I have attached the OpsMgr 2007 Database and Data Warehouse excel calculator I have been using the last couple of months to calculate what I think the size of customers databases would be. This tool is put together based on data we got from MSIT DB and a couple of TAP customers. While the tool has been pretty accurate so far somethings to consider before using the tool are a) I gave a lot of buffer in the tool b) there maybe things I did not account for (such as AEM data) c) no one outside MS has verified the accuracy of the data with any other data sample. The average data size per data type took into consideration raw and aggregated data and the fudge factor is the overhead that we assume that SQL may need.
Enjoy
Satya Vel
The Microsoft Learning team is developing certifications for Systems Center Configuration Manager 2007 (SCCM) and Systems Center Operations Manager 2007 (SCOM). Live Meetings discussing them are scheduled for late October.
Check out Trika's blog at http://blogs.msdn.com/trika/archive/2007/09/19/certifications-for-tsfkasms-mom.aspx for more information.
Some customers have asked how they can obtain a certificate for use with Operations Manager 2007 when are using a Certificate Authority (CA) other than Certificate Services on Windows Server. Use the template below:
[Version]
Signature= "$Windows NT$"
[NewRequest]
Subject = "CN=agent.contoso.com"
KeySpec=1
KeyLength = 1024
KeyUsage = 0xa0
ProviderName = "Microsoft RSA Schannel Cryptographic Provider"
ProviderType = 12
RequestType = PKCS10
Exportable = TRUE
MachineKeySet = TRUE
UseExistingKeySet = FALSE
[EnhancedKeyUsageExtension]
OID = 1.3.6.1.5.5.7.3.1
OID = 1.3.6.1.5.5.7.3.2
In the last couple of years, virtualization technology has become the “hot thing” in the IT management space because it is seen as reducing overall IT costs which is every CTO’s dream. And the fact that you create backups of your environments is a life saver for most organizations by itself. I thought I would take sometime today to talk about virtualization with relation to Operations Manager 2007 and share with you some of the thoughts Rob Scott (senior PM) and myself had on this topic .
The Operations Manager 2007 team officially supports the Database, Data Warehouse, Root Management Server, Management Server, Reporting Server, Gateway Servers and Agents on virtual machines in particular Virtual Server 2005. It should work on VMware but nobody I know has tried it so I cannot say for sure it works on VMware.
OpsMgr DB and DW performance on VMs is directly related to SQL Server 2005 performance on VMs. I read this really good white paper on SQL Server 2005 in a virtual environment which I think is a must read for anyone who is planning to virtualize any of the OpsMgr 2007 components. To give you the “skinny” on what’s written “while virtualization has many benefits, it is not the right solution for every case. For very high throughput applications and database applications that must be highly scalable, virtualization may not be the best choice. In these scenarios, running multiple instances of SQL Server 2005 would be a better choice.” Link to the doc: http://download.microsoft.com/download/a/c/d/acd8e043-d69b-4f09-bc9e-4168b65aaa71/SQLVirtualization.doc
My thoughts: If you have a small or medium environment of say 500 or less agents you should be OK virtualizing your DB and DW but anything above that you should look into getting dedicated hardware.
The Root Management Server or RMS as we call it in ‘sunny’ Redmond J is a central component is OpsMgr 2007. Since it is a core component in OpsMgr 2007 and all console and SDK connections link to it the obvious thought is to virtualize the RMS role. On the Root Management Server, the most critical resource is RAM followed by CPU. Many of the operations performed on the Root Management Server are memory-intensive. Performance problems can occur if the Operations Manager services on the Root Management Server are paged. If you feel that the virtual server can handle the load after reading the perf and scale guide then go for it. The most important resource on a Management Server is the CPU, so if you have a decent CPU then the MS can be virtualized.
My thoughts: If you have a decent budget and can buy dedicated hardware then that is the way to go. Root Management Server is not a good candidate to virtualize but the Management Server is.
People seems to think that the Gateway Server (GWS) is essentially another Management Server which is incorrect. The GWS is basically a proxy agent which directs traffic from multiple agents through a single channel to the Management Server.
My thoughts: The GWS is another good candidate to have virtualized.
Key Points:
Few points I would like to emphasize:
1. Virtualizing a server role makes it much easier to quickly back out changes made to the system and get back to a previous known good state. This includes OS-level changes like applying patches, SPs, making registry setting changes, file system changes etc.
2. Virtualizing a server role makes it much easier to move that server role from one physical server to another. Moving the OpsMgr DB or DW DB from one machine to another or to move an RMS there is a lot of complexity in moving things around with Ops Mgr. If these server roles are virtualized then moving the server roles from one machine to another is almost as easy as copying files since everything necessary is encapsulated in VHD files.
3. Memory gets dedicated to a given VM at startup so as long as you’ve got an extra 32 MB or so of RAM to cover the overhead of virtualization memory shouldn’t be a big issue.
4. A virtual machine can only use one processor in the current version of VS. If you’ve got a multi-threaded application this can be a limiting factor. This is probably one of the more limiting scale factors in running a virtualized RMS since there are three major processes running and each is multi-threaded.
Many of the current restrictions of the current Virtual Server product are expected to be alleviated by Veridian when it ships post-WS2008. The VHD format will still be supported so it “should” be fairly easy to migrate currently virtualized server roles to new servers running Windows Server 2008 with Veridian.