StackExchange.Redis timeout StackExchange.Redis timeout azure azure

StackExchange.Redis timeout


There are 3 scenarios that can cause timeouts, and it is hard to know which is in play:

  1. the library is tripping over; in particular, there are known issues relating to the TLS implementation and how we handle the read loop in the v1.* version of the library - something that we have invested a lot of time working on for v2.* (however: it is not always trivial to update to v2, especially if you're using the library as part of other code that depend on a specific version)
  2. the server/network is tripping over; this is a very real possibility - looking at "slowlog" can help if it is server-side, but I don't have any visibility of that
  3. the server and network are fine, and the library is doing what it can, but there are some huge blobs flying between client and server that are delaying other operations; this is something that I'm making changes to help identify right now, and if this shows itself to be a common problem, we'll perhaps look at making better use of concurrent connections (which doesn't increase bandwidth, but can reduce latency for blocked operations) - this would be a v2 only change, note


Lazy Connection

As a best practice make sure you are using the following pattern to connect to the StackExchange Redis client:

private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => {    return ConnectionMultiplexer.Connect("cachename.redis.cache.windows.net,ssl=true,abortConnect=false,password=password");});public static ConnectionMultiplexer Connection {    get {        return lazyConnection.Value;    }}

If the above does not work, there are some more debugging routes described in Source 1, regarding region, bandwidth and NuGet package versions among others.

IO Threads

Another option could be to increase the minimum IO threads. It’s often recommend to set the minimum configuration value for IOCP and WORKER threads to something larger than the default value. There is no one-size-fits-all guidance on what this value should be because the right value for one application will be too high/low for another application. A good starting place is 200 or 300, then test and tweak as needed.

How to configure this setting:

  • In ASP.NET, use the minIoThreads configuration setting under the <processModel> configuration element in machine.config. According to Microsoft, you can’t change this value per site by editing your web.config (even when you could do it in the past), so the value that you choose here is the value that all your .NET sites will use. Please note that you don’t need to add every property if you have autoConfig set to false, just putting autoConfig="false" and overriding the value is enough:<processModel autoConfig="false" minIoThreads="250" />

Important Note: the value specified in this configuration element is a per-core setting. For example, if you have a 4 core machine and want your minIOThreads setting to be 200 at runtime, you would use <processModel minIoThreads="50"/>.

Sources:

  1. Microsoft Azure - Investigating timeout exceptions in StackExchange.Redis for Azure Redis Cache
  2. StackExchange.Redis


Have the network traffic monitor switched on to confirm/deny the blip.have a solution to the issue but a crude one. Option 1 - try restarting the managed redis instamce in azure.