Skip to main content

scom: overloading the consolidation module (and how to avoid it)

in a previous post titled using repeat count to detect a problem in a window of time I described a process whereby you can using consolidation settings, you can detect something happening in a window of time.  for example, event id 529 equals "bad password" basically.  if we alerted on every bad password, that'd be problematic.  however, if we looked at every one and then alerted whenever the count of bad passwords for a single user exceeded a threshold, that might be useful.

apparently there's this concept called a "consolidation module".  this module has a limit of 128k.  if you go beyond this limit, you tend to overload the module and cause the event scraping to backlog.  on very active domain controllers, using a large sliding window, it's very easy to overrun this limit.  it results in odd errors like this:

(event id 11105)

The Microsoft Operations Manager Condolidator Module failed to save the state after processing and might loose data. 
Error: 0x80070057
One or more workflows were affected by this.
 
 
The Windows Event Log Provider monitoring the System Event Log is 317 minutes behind in processing events. This can occur when the provider is restarted after being offline for some time, or there are too many events to be handled by the workflow.

One or more workflows were affected by this.

before you try to correct me, I copied and pasted that.  I know how to spell "consolidator" and "lose data".  I mentioned fixing this.  there are two ways.  the most obvious is by reducing your sliding window time frame so that you're not collecting as many events in a given period of time.  the second way is to simply set the storestate value to false.  the first one, you should be able to derive quite easily.  modifying the storestate value tells the agent not to store the internal state.  the problem here, albeit a small exchange, is that the state does not survive healthservice restarts.

as for the second method, it's not available in the console (surprise, surprise!) and must be done by editing the xml (surprise, surprise!).  so, you'll need to export your rule, modify your xml, and import it again.  once you have your xml, locate an area of the xml that should look suspiciously like this:

<Consolidator>
<ConsolidationProperties>
<PropertyXPathQuery>EventDisplayNumber</PropertyXPathQuery>
<PropertyXPathQuery>PublisherName</PropertyXPathQuery>
<PropertyXPathQuery>LoggingComputer</PropertyXPathQuery>
<PropertyXPathQuery>Params/Param[1]</PropertyXPathQuery>
<PropertyXPathQuery>Params/Param[2]</PropertyXPathQuery>
</ConsolidationProperties>
<TimeControl>
<WithinTimeSchedule>
<Interval>1800</Interval>
</WithinTimeSchedule>
</TimeControl>
<CountingCondition>
<Count>20</Count>
<CountMode>OnNewItemTestOutputRestart_OnTimerSlideByOne</CountMode>
</CountingCondition>
</Consolidator>
 
 
okay, now modify it to add this one entry noted below.
 
<Consolidator>
<ConsolidationProperties>
<PropertyXPathQuery>EventDisplayNumber</PropertyXPathQuery>
<PropertyXPathQuery>PublisherName</PropertyXPathQuery>
<PropertyXPathQuery>LoggingComputer</PropertyXPathQuery>
<PropertyXPathQuery>Params/Param[1]</PropertyXPathQuery>
<PropertyXPathQuery>Params/Param[2]</PropertyXPathQuery>
</ConsolidationProperties>
<StoreState>false</StoreState>
<TimeControl>
<WithinTimeSchedule>
<Interval>1800</Interval>
</WithinTimeSchedule>
</TimeControl>
<CountingCondition>
<Count>20</Count>
<CountMode>OnNewItemTestOutputRestart_OnTimerSlideByOne</CountMode>
</CountingCondition>
</Consolidator>

after making the modification, import it back in and the problems should go away.

Comments

Popular posts from this blog

using preloadpkgonsite.exe to stage compressed copies to child site distribution points

UPDATE: john marcum sent me a kind email to let me know about a problem he ran into with preloadpkgonsite.exe in the new SCCM Toolkit V2 where under certain conditions, packages will not uncompress.  if you are using the v2 toolkit, PLEASE read this blog post before proceeding.   here’s a scenario that came up on the mssms@lists.myitforum.com mailing list. when confronted with a situation of large packages and wan links, it’s generally best to get the data to the other location without going over the wire. in this case, 75gb. :/ the “how” you get the files there is really not the most important thing to worry about. once they’re there and moved to the appropriate location, preloadpkgonsite.exe is required to install the compressed source files. once done, a status message goes back to the parent server which should stop the upstream server from copying the package source files over the wan to the child site. anyway, if it’s a relatively small amount of packages, you can

How to Identify Applications Using Your Domain Controller

Problem Everyone has been through it. We've all had to retire or replace a domain controller at some point in our checkered collective experiences. While AD provides very intelligent high availability, some applications are just plain dumb. They do not observe site awareness or participate in locating a domain controller. All they want is the name or IP of one domain controller which gets hardcoded in a configuration file somewhere, deeply embedded in some file folder or setting that you are never going to find. How do you look at a DC and decide which applications might be doing it? Packet trace? Logs? Shut it down and wait for screaming? It seems very tedious and nearly impossible. Potential Solution Obviously I wouldn't even bother posting this if I hadn't run across something interesting. :) I ran across something in draftcalled Domain Controller Isolation. Since it's in draft, I don't know that it's published yet. HOWEVER, the concept is based off

sccm: content hash fails to match

back in 2008, I wrote up a little thing about how distribution manager fails to send a package to a distribution point . even though a lot of what I wrote that for was the failure of packages to get delivered to child sites, the result was pretty much the same. when the client tries to run the advertisement with an old package, the result was a failure because of content mismatch. I went through an ordeal recently capturing these exact kinds of failures and corrected quite a number of problems with these packages. the resulting blog post is my effort to capture how these problems were resolved. if nothing else, it's a basic checklist of things you can use.   DETECTION status messages take a look at your status messages. this has to be the easiest way to determine where these problems exist. unfortunately, it requires that a client is already experiencing problems. there are client logs you can examine as well such as cas, but I wasn't even sure I was going to have enough m