When we left off on this issue, the app pool was crashing at semi-random intervals. After much discussion with Microsoft’s IIS forum, Microsoft’s ASP.Net forum (not much help there), and peer development groups (AZGroups.com primarily), I downloaded and installed IISState.exe, which is a command line monitor (basically) that is placed on the IIS server and listens for app pool issues. It’s far more detailed than Event Log or App Pool logging, providing codes at every POST/GET. Additionally, I tried turning app pool recycling completely off with no success, and also tried scheduling a single recycle in the middle of the night, also without success.
I decided to run IISState and see what happened. Upon running I noticed the expected logging, with typical “Ok, that worked” codes being thrown. After almost 2 days of running fine, the app pool finally crashed and IISState captured it. The interesting thing is, the error code it threw was a “C++ EH exception – code e06d7363 (first chance)”. Initial research shows this to be a problem internal to IIS. So far, additional research proves this is either not well known or not well documented. I also researched a forum about the error, and their recommendation was to uninstall and reinstall IIS. Yeah…..not happening. In fact, that solution is insulting. Similar to when your computer is slow, blow away the hard drive and reinstall the OS. Please, I thought we were professionals here!
So, the issue remains unresolved at this time. I had to put it aside temporarily to work on other projects, but it’s very much a priority to discover root cause and resolution. As soon as I get to that point, or if I discover anything else I’ll post it here.
While working on a website that’s designed to run 24 hours a day with no user interaction (an automatically-updating display for users to walk past and see), I came across an interesting phenomenon.
Now, the website is designed in ASP.Net 3.5 with AJAX. There’s an UpdatePanel and a Timer, and that’s it for the AJAX portion. The Timer control fires every 10 seconds and fires the UpdatePanel. Simple enough, right? Well, at random instances the website would stop updating and throw a ScriptResource.axd related error, but only visible in the status bar at the bottom of the browser. After some exhaustive research on this error and finding very little documented on the Internet, I received some feedback from some colleagues that it might be an app pool recycling problem in IIS.
So, my next step was to modify the IIS settings on that server. I removed everything and only enabled recycling once a day. I also turned on app pool logging in the Event Log as described in this article. The next day everything was still working, so far so good! However, the following day (2nd day) the website stopped updating again. This time I checked the Event Log and found the following warning:
I looked up the error code 8007006d and it came back to “App Pool Crashed”. Well, I think that was fairly obvious. Now to figure out why the thing crashed, when all it’s doing is updating a lousy panel every 10 seconds!
So, at this time I enabled healthMonitoring in the app’s web.config file to attempt to capture more events when the app pool crashes again. This, combined with the IIS log and extended app pool logging in Event Viewer, I hopefully will glean something valuable about this event. Stay tuned for Part 2, which should hopefully be the root cause and solution to this issue.