I have a strange situation on a production server. Connection for asp.net get queued but the CPU is only at 40%. Also the database runs fine at 30% CPU.
Some more history as requested in the comments:
My conclusion is that something else is stopping the server from handling the requests faster. Possible suspects
To find out what the proces is doing I created to minidumps.
I managed to create two MemoryDumps 20 seconds apart. This is the output of the first:
and the output of the second:
As you can see there are a lot of Request in Queue.
Question 1: what does it mean that there are 1589 requests in queue. Does it mean something is blocking?
The !threadpool list contains mostly these entries: Unknown Function: 6a2aa293 Context: 01cd1558 AsyncTimerCallbackCompletion TimerInfo@023a2cb0
If I you into depth with the AsyncTimerCallbackCompletion
Then I look at the objects in the TimerCallback and most of them are of types:
Question 2: Does it make any sense that those Objects hava a timer, and so much? Should I prevent this. And how?
Main Question do I miss any obvious problems why I’m queueing connections and not maxing out the CPU?
I succeeded in making a crashdump during a peak. Analyzing it with debugdiag gave me this warning:
A quick google search doesn’t give me any results. Does somebody has a clue?
Too many ASP.NET queued requests will destroy performance. There are a very limited number of request threads.
Try to free up those threads by processing slow parts of your pages asynchronously or do anything else you can to bring down page execution times.
I’m with realworldcoder: IIS works by having Worker Processes handle the incoming requests. If the requests get stacked up, as it appears is happening, then performance takes a nose dive.
There are several possible things to do/check for.
The above are just a couple things to look at. You basically need to get further into the details to find out exactly what is going on and most of the regular performance counters aren’t going to give you that clarity.
The worker processes handling the queue was the real dealbreaker. Probably connected with the website calling webservices on the same host. Thus creating a kind of deadlock.
I changed the machine.config to to following:
Standard this processModel is set to autoConfig=”true”
With the new config the webserver is handling the requests fast enough to not get queued.