Shawn Gibbs on Application Management

System Center .NET Application Alerts vs Events

Shawn Gibbs — Fri, 13 Apr 2012 14:26:13 GMT

Formulating a process to solve production application issues quickly requires a well thought out process for alerting, triaging and diagnosis issues. Whether you currently use System Center Operations Manager now or not these are still the same three buckets used to help develop a process. The eventual goal in the process should be to automate as much of the resolution process as possible but that is beyond the scope of this article, here I simply want to discuss what an APM Alert is versus what an APM Event is. Each has a place in the problem resolution process and certainly they are related and interconnected in many ways. It’s easiest to understand alerts for those familiar with Operations Manager but of course not everyone is, and in most cases events are also not well understood.

So to set up the conversation it’s probably worth a quick review of APM in general. I usually start explaining that APM is simply automating the process of instrumenting .NET code with performance timings and exception collections. Doing something that developers would, and in some case have done to applications if given enough time and budget to do so. A bigger challenge is that if this has been done what to do with the data and did they create a method to understand this data by all the team members that need to know. In many cases it has then been left up to the operations team to filter and collect in Operations Manager via custom built Management Packs the data that is important to them and create alerts accordingly. In addition it hasn’t always included enough root cause details to solve problems are even steer diagnostics better to speed resolution. How many of us have seen event logs get over populated with events we not only don’t understand fully but tend to ignore because of the flood of information is too much. APM is much more than this but for the Alerts versus Events conversations this is enough to get use started. Let’s look at events; these are simply the detailed collection of application specific performance problems and exception events in the code. You tell the monitoring template to collect an event when the performance exceeds X milliseconds and in doing so it will include details about the performance challenge inside the event. You can also tell the template to collect exception events or what we call application failures in the code based on whether they are considered critical or not. Again depending on configurations this will include some level of detail about the code exception including parameters if so desired. Like a developer logging their own performance problems and exception events into an event log or their own database APM logs these events in an easy to use web based console for further analysis and use. But a single slow page or code failure doesn’t always mean an availability problem is arising so these events might not always create alerts. To highlight the difference here is an event log entry created by the developer of a web based application when rendering a shopping cart failed. Below is the APM event collected for the exact same failure being rendered from the AppDiagnostics console. You can see a fairly different approach to rendering the data with some similarities. The detail collected in the APM can be controlled and is collected without rewriting code, and it doesn’t rely on the developer to create the event or do anything.

So this actually is a great example of a developer collecting and the automated collection from Operations Manager APM. But now how and why would we alert on this event. First a couple of things around application failures that is worth noting. First is that APM will distinguish between different types of exceptions in the code and categorize them with what is referred to as Aspects. This error for example is not Connection related or Security related which are 2 of the aspects we distinguish so the event gets the aspect of ‘Application Failure’. This is important since we can alert differently based on this aspect. As an example operations personnel should almost always be alerted when security in a database changes, this may cause the applications database commands to fail for security reasons like not able to insert permissions. When creating an APM monitoring template we can distinguish between creating individual event alerts or only based on aspect of failure or event quantities and increases in percentage of events.

As you can see we have options as to what and to what depth to collect events but also how to use these events in alerting. We can even go to the level of transactions which simply base similar event collection and alerting on specific web pages or code functions. For example don’t trigger alerts on application failures at the site level but do trigger them for the cart page. At this point we can monitor for alerting the number of events and base an overall alert for the percent of increase since a large number of failures impacts performances. So why would you set this up this way? Imagine a public site that changes regularly which is cataloged by a search engine. The search engine would regularly refresh content by retrying the urls in its database, and the site may no longer have the url. This will actually cause a .NET exception at the site level for ‘File not found’ which is an aspect of application failure. We don’t need an alert for ‘File not found’ at the site level, which could be hundreds or thousands. At this point count and track them and alert if they drastically increase in a time interval. So disabling alerting on application failure at the site level prevents this potential flood of data but then what about the ‘applications failures we care about. We do this by adding transactions and configuring alerts for application failures at that level. So now cart.aspx is a transaction, it can’t get 404 file not found type failures because it is a page and now any aspect of application failure is important so create the alert.

Application Dependencies and Chained Events

Shawn Gibbs — Wed, 04 Apr 2012 18:57:08 GMT

The question seems to come often enough around application dependency discovery with AVIcode and or SCOM APM so it’s worth a quick discussion and walk through. First let’s define for the purpose of this discussion what we mean by application discovery and application dependency. This would then allow us to talk about how to use them for monitoring as well as application diagnostics. An application dependency is simply what does the application depend on from a .NET application resource point of view. For example does this application call a soap service or does it communicate to a database? These are application function calls to recourses and can be collected and reported fairly easily. Application discovery is a lot more complex discussion and in some circles called a black art. Also for the purpose of this discussion I will be referring to System Center 2012 Operations Manager APM. Outside of ASP.NET applications application discovery requires many things like permissions and application ‘finger prints’ for discovery to happen, Inside ASP.NET it is slightly easier to list what is installed in the IIS instance to run under what app pools.

So let’s first talk about application discovery in Operations Manager APM, specifically .NET applications. Before monitoring can happen we need to select an application to monitor. This selection is based on the list of applications discovered using the IIS 7.0 APM management pack in Operations Manager. Operations Manager can also allow you to discover applications of any type based on a custom management pack and custom discovery scripts, and some third party vendors have already created these for other applications. Once the discovery happens you can use the APM template wizard to create a template for application monitoring. This new template manages what application, where it is running and how we will alert and collect information from it. After applications are being monitored we will use the monitoring to collect events and performance counters. Events are made of exceptions in the code and functions performing slower than a set threshold.

Now the application dependency. This is a great diagnostics tool and even can be a great discovery tool for understanding the application. Using the monitoring that is running on the application we will listen for certain functions to be called. These functions are database calls and soap service calls. To us these are dependencies the application has. Now what do we do with these? First we treat them as resources inside of a collected event. So if a database is called inside of a poor performing web response it will be timed and counted even if it doesn’t break collection levels called sensitivity settings. See figure 1.

Figure 1

If they break the collection level called sensitivity level we then collect some additional detail about the call. See figure 2.

Figure 2

Now for some magic. Since we know the calls are happening and when we shouldn’t waste this knowledge. So let’s do 2 things with it. First let’s take the fact that a function call was made and register it somewhere so we know we depend on that resource, even if there is no event collected that the call is part of. Helps map and understand the application and is useful in diagnosing problems. For example if 10 databases and data servers are used and only one is consistently part of events we can assume a problem exist with that one and not the code that calls databases. See figure 3 for an example of dependency listing.

Figure 3

But what else can we do with this knowledge? This is a great question and a better answer. We can look at these function calls that make the calls to these dependencies and we can also look at function calls that are receive calls from dependencies in the event detail. When we see a ‘caller’ we can then look for events that could potentially be the ‘callee’ if we find something or bunches of multiples of something we can run a simple logical set of “if then” type statements against them to suggest they are related in some way. Well how could they relate? So if we monitor application ‘A’ and we see it calls a web service application ‘B’ which is also being monitored we may get events from both that either do or don’t correlate to each other. To confirm we see ‘A’ called at X time and date and waited Y time period with a set of parameters used, we then see that ‘B’ has an event similar in ‘X’, ‘Y’ and parameters maybe they are related. If we determine they are exactly related we can set this to a single transaction distributed chain and if not we can tell you these all are calls being made and show a multi transaction distributed chain. See figure 4 as the calling applications event details and it’s rendering in figure 5 of the single transaction distributed chain.

Figure 4

Figure 5

System Center 2012 Operation Manager APM

Shawn Gibbs — Thu, 01 Mar 2012 21:48:00 GMT

Increasing depth of details in performance events. Collecting variables on method calls.

Okay, what are all the knobs and buttons for in APM configuration? In this article I want to discuss some of the reasons for making changes to depth of collection but also some of the cost. Not cost in terms of dollar value but cost in terms of impact of the monitor on what it is monitoring. It’s very difficult to explain exactly what is this cost and/or how does it impact my application, especially since there is an endless amount of applications and scenarios. The best analogy I can give is a cars fuel tank. In the very early days of cars it was enough to simply let the driver physically look into the tank to see how much fuel they have. This worked well but for ease of use it was decided that a gauge and a sending unit will be physically installed into the tank. What was the cost of this, well the tank held just a little less fuel because of the space the new sending unit took up and this cost was well worth the convenience. What if we wanted to know more about the fuel, like the quality and octane level? We would have to add a lot more to the tank and the car and the overall benefit is now not worth the cost. With a loss in capacity and miles per gallon from extra weight we start cost more than the returned value. Would a scientist like this extra detail and information? Most likely they would, but that doesn’t make the cost any better. How does this translate to APM monitoring? Great question. Simply put the knobs and buttons are designed to control the depth of information we collect when an event triggers collection. For performance events this is when an ‘Entry point’ into the application exceeds the ‘Performance event threshold’ set in the template creation. The goal is to be as small of a footprint but yet still have enough information to diagnose issues.

Enough philosophy and let’s talk about a real world requirement to increase depth of collection. For the purposes of this article I’ll use a SharePoint application as an example which turns out to be a great example since it acts somewhat as a black box. If I write my own .NET application I am using namespaces for my code that are most likely specific to me and/or my company. This is not entirely the case in SharePoint since it not only uses custom namespaces when it is customized but also has a set of important classes and internal namespaces that fall into the Microsoft namespaces. For information on enabling custom namespaces and monitoring custom code in SharePoint visit the blog entry from Chris Childers by clicking here. By default the Microsoft namespace is a disabled namespace, and should remain that way. There are several namespaces automatically disabled, including the System namespace. This is the first line of defense in preventing too much collection of detail. But sometimes we need more and within this ‘System’ namespace there are methods or application functions that are interesting. These are already configured by default to collect extra details like variables. As an example in the System namespace, which is the main .NET namespaces there is a class for working with SQL. In that class there is a function that executes the SQL called “ExecuteReader”. This functions full name is System.Data.SqlClient.SqlCommand.ExecuteReader and it is preconfigured in APM as a resource and APM collects variables like the SQL command text being performed which is important to diagnose SQL performance problems.

So then the question comes up, how do we know what methods in the code are important enough to grab details like variables? For most custom applications it can be rare to enable this depth. It becomes more important to expand the execution tree to see the application flow and to increase this we add namespaces to be included. Typically the execution tree will highlight challenges and usually they step down into methods already being collected like the SQL example we used earlier. So in that case nothing else from a variable point of view is helpful. For SharePoint and other Microsoft applications it becomes slightly more difficult to know what will be helpful and where is the information stored in the form of variables. One could take a test system and enable namespaces like Microsoft.SharePoint which would give more detail of application code flow, but still hides where is the variables I should care about and also may even trigger additional depth collection protection which would kick in and prevent the collection anyway. Another method is to talk to developers, but also to just collect at the higher level the events and decide as you go the depth you need. Here for example is a default configuration of a SharePoint 2010 portal page.

Notice that SQL calls and WCF are interesting resources so they are by default being collected in the break the sensitivity we configured in the template, default is 100ms. These are resources because if I clicked ‘Resource Group View’ they will be show as a total count and percentage of the overall page load. This would show a total count of SQL calls and WCF calls including the ones under the sensitivity level.

Good information if I’m trying to rule out SQL and WCF calls as the problem, but there is 16,982 ms of time not accounted for. So if I received this event form my production systems I would need to get more detail. For SharePoint we can cheat a little and use the knowledge AVIcode’s SharePoint monitoring to learn what is important to gather. Below you will find a list of AVIcode’s SharePoint 2007 methods that should translate into SharePoint 2010 for the most part. It may be a little time consuming you would simply add these to the template by select ‘add Methods …’. I would even go as far as saying some may be important enough to capture at any sensitivity when part of an event so setting the sensitivity to zero for these will force the collection.

By adding the above LoadWebParts method we now see variables we can use to diagnose further but we can also see things like which web parts and list are part of this page load. Is one causing the issue?

So we have effectively increased the depth on a set of methods that will help diagnose and recreate issues so that further action can be taken if needed. We’ve done so without adding unneeded burden to the system by collecting to much information that wouldn’t speed the process up anyway.

SharePoint 2007 Methods to collect variables.

Microsoft.SharePoint.WebPartPages.XmlWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.XmlWebPart.OnPreRender

Microsoft.SharePoint.WebPartPages.TitleBarWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.SimpleFormWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.SimpleFormWebPart.OnPreRender

Microsoft.SharePoint.WebPartPages.SPWebPartManager.LoadWebParts

Microsoft.SharePoint.WebPartPages.PageViewerWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.MembersWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.ListViewWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.ListViewWebPart.CreateChildControls

Microsoft.SharePoint.WebPartPages.ListFormWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.ListFormWebPart.OnPreRender

Microsoft.SharePoint.WebPartPages.ImageWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.DataViewWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.DataViewWebPart.OnPreRender

Microsoft.SharePoint.WebPartPages.CustomPropertyToolPart.OnPreRender

Microsoft.SharePoint.WebPartPages.ContentEditorWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.ChartViewWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.ChartViewWebPart.OnPreRender

Microsoft.SharePoint.WebPartPages.BlogAdminWebPart.RenderWebPart

Microsoft.SharePoint.WebPartPages.AggregationWebPart.RenderWebPart

Microsoft.SharePoint.WebControls.TopologyViewWebPart.RenderWebPart

Microsoft.SharePoint.Search.Internal.WebControls.SearchSummaryWebPart.RenderWebPart

Microsoft.SharePoint.Search.Internal.WebControls.SearchSummaryWebPart.OnPreRender

Microsoft.SharePoint.Search.Internal.WebControls.SearchStatsWebPart.RenderWebPart

Microsoft.SharePoint.Search.Internal.WebControls.SearchStatsWebPart.OnPreRender

Microsoft.SharePoint.Search.Internal.WebControls.SearchPagingWebPart.RenderWebPart

Microsoft.SharePoint.Search.Internal.WebControls.SearchPagingWebPart.OnPreRender

Microsoft.SharePoint.Portal.WebControls.WebPartLoc.RenderWebPart

Microsoft.SharePoint.Portal.WebControls.WebPartLoc.OnPreRender

Microsoft.SharePoint.Portal.WebControls.WSRPConsumerWebPart.Render

Microsoft.SharePoint.Portal.WebControls.ThisWeekInPicturesWebPart.RenderWebPart

Microsoft.SharePoint.Portal.WebControls.ThisWeekInPicturesWebPart.OnPreRender

Microsoft.SharePoint.Portal.WebControls.SPSlicerChoicesWebPart.OnPreRender

Microsoft.SharePoint.Portal.WebControls.RelatedGroupsWebPart.RenderWebPart

Microsoft.SharePoint.Portal.WebControls.PrivacyItemViewer.OnPreRender

Microsoft.SharePoint.Portal.WebControls.KpiRenderer.getKpiData

Microsoft.SharePoint.Portal.WebControls.KPIListWebPart.OnPreRender

Microsoft.SharePoint.Portal.WebControls.KPIListWebPart.GetCallbackResult

Microsoft.SharePoint.Portal.WebControls.IndicatorWebpart.OnPreRender

Microsoft.SharePoint.Portal.WebControls.IndicatorWebpart.GetCallbackResult

Microsoft.SharePoint.Portal.WebControls.GroupDetailWebPart.RenderWebPart

Microsoft.SharePoint.Portal.WebControls.FilterValuesPickerDialog.OnLoad

Microsoft.SharePoint.Portal.WebControls.ContactFieldControl.OnPreRender

Microsoft.SharePoint.Portal.WebControls.BusinessDataWebPart.GetCallbackResult

Microsoft.SharePoint.Portal.WebControls.BusinessDataListWebPart.CreateChildControls

Microsoft.SharePoint.Portal.WebControls.BusinessDataItemBuilder.OnPreRender

Microsoft.SharePoint.Portal.WebControls.BusinessDataDetailsWebPart.CreateChildControls

Microsoft.SharePoint.Portal.WebControls.BusinessDataActionsWebPart.RenderWebPart

Microsoft.SharePoint.Portal.WebControls.ApplyFiltersWebPart.RenderWebPart

Microsoft.SharePoint.Portal.WebControls.ApplyFiltersWebPart.OnPreRender

Microsoft.SharePoint.Meetings.PageTabsWebPart.RenderWebPart

Microsoft.SharePoint.Meetings.CustomToolPaneManager.RenderWebPart

Microsoft.Office.Server.Search.WebControls.SearchSummaryWebPart.RenderWebPart

Microsoft.Office.Server.Search.WebControls.SearchSummaryWebPart.OnPreRender

Microsoft.Office.Server.Search.WebControls.SearchStatsWebPart.RenderWebPart

Microsoft.Office.Server.Search.WebControls.SearchStatsWebPart.OnPreRender

Microsoft.Office.Server.Search.WebControls.SearchPagingWebPart.RenderWebPart

Microsoft.Office.Server.Search.WebControls.SearchPagingWebPart.OnPreRender

Microsoft.Office.Server.Search.Query.KeywordQuery.Execute

Microsoft.Office.Server.Search.Query.FullTextSqlQuery.Execute

Microsoft.Office.Server.ApplicationRegistry.SystemSpecific.WebService.WebServiceSystemUtility.ExecuteStatic

Microsoft.Office.Server.ApplicationRegistry.SystemSpecific.Db.DbSystemUtility.ExecuteStatic

Microsoft.Office.Server.ApplicationRegistry.MetadataModel.Entity.ExecuteInternal

Microsoft.Office.Excel.WebUI.InternalEwr.OnPreRender

Microsoft.Office.Excel.WebUI.ExcelWebRenderer.OnPreRender

New AVIcode version released and the top new features

Shawn Gibbs — Tue, 01 Feb 2011 18:44:00 GMT

In December 2010 AVIcode released version 5.7 of Intercept Studio. Some of the hot new features are:
-added built in support for .NET 4.0 framework and MVC style applciations
-the ability to manage event state and create rules to manage events
-and a web application scanner which analyzes your site for the gotchas that may happen when using AVICode uX user health monitoring.

...(read more)