Long communication time of WCF Web Services within the same server

Question

Even if this question is a year old, I am still searching a good answer for this question. I appreciate any information that will lead me to fully understand this issue regarding low performances of communicating web services hosted on the same machine.

I am currently developing a system with several WCF Web Services that communicate intensively. They are running under IIS7, on the same machine, each service being in a different Application Pool, with multiple workers in the Web Garden.

During the individual evaluation of each Web Service, I can serve 10000-20000 requests per minute, quickly and without any issues for the resource consumption (processor and memory).

When I test the whole system or just a subsystem formed by two Web Services I can't serve more than 2000 requests/minute. I also observed that communication time between Web Service is a big issue (sometimes more than 10 seconds). But when testing with only 1000 requests per minute everything goes smoothly (connection time of no more than 60 ms). I have tested the system both with SOAPUI and JMETER, but the times were computed based on system logs, not from the testing tools. Memory and network aren't an issue (they are used very little).

Later on, I have tested the performance of 2 communicating WCF web services, hosted on two server and on the same server. It again seems that there is a bottleneck when the services are on the same machine, lowering the number of connection with from ten thousands to thousands; again, no memory or processor limiting.

As a note, I am working with quite big data in some cases and some of the operations needed are long ones.

I used perf.mon to see what's going on, for memory, processes, webservice, aspnet, etc. but I didn't see anything that could indicate what it's going wrong. I also tried all the performance settings and tuning options I could find on the Internet.

Does someone know what can be wrong? Why the communication between Web Services could last so long? Why the Web Service which serves as an entry point in the system can accept 10000 requests/minute when is tested alone, but when communicating with another Web Service barely accepts 2000? It's an IIS7 problem? Could my system perform better if each Web Service will be deployed on a different server?

I want to understand better how things internally function (IIS and WCF services) to improve performances for current and future systems.

Answer 1

You could try to collect data from WCF performance counters : concurrent calls, instances, duration, ... In addition, WCF throttling provides some properties that you can use to limit how many instances or sessions are created at the application level. Performance of the WCF service can be improved by creating proper instance. Finally, in load testing, there are many configuations to apply to different component : max concurrent http connection, IIS limits, having many load clients... You load test is invalidated because of this.