+972 52-548-6969
Taking full dumps for big IIS apps
If your application is running on IIS, you are getting quite a lot for free. To start with, monitoring and management tools are right there out of the box. You are also getting some… other effects.
In particular, we had RavenDB running inside IIS that exhibit a set of performance problems in a couple of nodes (and just on those nodes). We suspected that this might be related to memory usage, and we wanted to take a full process dump so we can analyze this offline.
Whenever we tried doing that, however, the process would just restart. The problem was that to reproduce this we had to wait for a particular load pattern to happen after the database was live for about 24 hours. So taking the dump at the right time was crucial. Initially we thought we used the wrong command, or something like that. The maddening this was, when we tried it on the same machine, using the same command, without the performance issue present, it just worked (and told us nothing).
Eventually we figured out that the problem was in IIS. Or, to be rather more exact, IIS was doing its job.
When the performance problem happened, the memory usage was high. We then needed to take a full process dump, which meant that we had to write a lot. IIS didn’t hear from the worker process during that time (since it was currently being dumped), and it killed it, creating a new one.
The solution was to ask IIS to not do that, the configuration is available in the advanced settings for application pool. Note that just changing that would force IIS to restart the process, which was another major annoyance.
Comments
Did you use procdump to use process reflection (procdump -r ...) to minimize the downtime of the process? This should be so fast that even a ping or other e.g. WCF timeouts are not happening.
Procdump -r has been giving corrupt memory dumps, not sure if this was ever improved.
For example: http://forum.sysinternals.com/procdump-r-not-mapping-all-memory-regions-in-dmp_topic29014.html
Comment preview