Hi mikezopa,

Sorry to hear this, I'm inclined to believe this is something that happens in the bowels of WebRequest itself. googling webrequest and 21 seconds yields numerous results such as this one

WebRequest notoriously measures timeouts from time to send first byte to last byte read. Anything that needs to happen prior (DNS setup, enqueueing connections) uses the default timeout of 20 seconds. So even if you set the Timeout to 1second on the request, a slow DNS lookup can cause the overal request to fail only after 20 seconds.

Are you able to tell us more about your connectionsettings setup, the exception that you see when this intermittent spikes happen?

AFAIK we do properly dispose of all the request streams we start and have extensive unit tests of all the code paths to make sure we close the responsestreams.

We also do extensive Wireshark sessions every now and then to make sure we are reusing TCP connections. To hear you still face these sporadic spikes is saddening but I would love to find out the root cause of this.

Hi @Mpdreamz,

Sorry for my delayed response, I hadn't noticed that the issue had been updated! Thank you for the fast initial response.

With regards to the connectionsettings setup, we initialise the client like this:

var settings =
                new ConnectionSettings(new SniffingConnectionPool(nodeUris), this.indexName)
                    .ThrowOnElasticsearchServerExceptions();

            this.client = new ElasticClient(settings);

Thanks for the information regarding the WebRequest. We are making a larger number of requests which we have not been doing before. We used to be making a request once every few seconds, but during times of loading, one ElasticClient instance might be making a couple of requests within a second. Could this be upsetting any reuse of TCP connections? Is it wise for us to be using one ElasticClient instance for the entire micro-service?

I don't believe there are any exceptions thrown when this error occurs, the request completes successfully, just very delayed. Unfortunately, the caller of this micro-service gives up after 5 seconds (as we can't wait longer than that for various reasons). Sometimes we see multiples of this delay on our micro-service, so we see spikes in lower counts at 42, 63 seconds etc.

We might have a go at implementing some of these suggestions and let you know what we find!

You can new ElasticClient as many times as you like but if you are using connection pooling make sure each client gets passed the same instance of the SniffingConnectionPool. This should not have bearing on socket reeuse and open TCP connections this is handled by the servicepoint(s) for the urls in the cluster in the .NET framework already. If you do not share IConnectionPool then you might be doing more bookkeeping (sniffiing/ping) calls then necessary though.

We try to follow most of the suggestions in that thread already :) request.Proxy = null; can be a silver bullet for some (and can be controlled with .DisableAutomaticProxyDetection() on ConnectionSettings but usually this is making the very first call on a system very slow not intermittent ones. We also had reports from users where request.Proxy = null actually introduces a slow down on some systems so there really is no sane default there.

If you are running on Azure we typically recommend setting SetTcpKeepAlive() on connectionsettings. This will cause actual keep alive packets to be send periodically on the reused tcp connections. (http://blogs.msdn.com/b/cie/archive/2014/02/14/connection-timeout-for-windows-azure-cloud-service-roles-web-worker.aspx)

A little involved but the get a bit more clarity you could run your microservice with trace enabled on system.net: http://mikehadlow.blogspot.nl/2012/07/tracing-systemnet-to-debug-http-clients.html

This might give some insights into peeks.

With regards to some more investigation work, I have been unable to reproduce the problem by creating various integration tests which hammer requests to elasticsearch, even when using multiple threads (run via the NUnit test runner). This is leading me to suspect it might be something to do with hosting within IIS and how connections are managed there.

I was hoping to reproduce on my local machine allowing me to use Wireshark and various tracing tools a bit more easily then when located on a server. I will see if I can reliably reproduce using load testing against the micro-service when its running under IIS.

None of the suggestions in the thread helped us unfortunately! :)

Hi @mikezopa closing this due to inactivity but please feel free to report back! This issue for us is never really closed (due to its heisenbug nature) and something we revisit often.

I know I'm kicking a dead horse here, but I was having a similar issue with the first query in an automated test taking ~20 seconds. Turns out, in my case, it was probably a DNS issue. Switching from localhost to 127.0.0.1 solved the problem.

If anyone else happens upon this, maybe that info will be helpful for them.