Every had a latent bug go undetected and then jump up and bite you?
Yeah, not the nicest feeling.
I’ve been working on a pretty interesting project that now makes use of Redis to provide a caching layer as the system uses multiple servers and does some reasonably heavy computation to prepare the data for use in the front end. For reference there are numerous REST calls to get the basic data and then a raft of search queries that are executed (again via REST) to build up a data payload of about 1MB. Anyhow given this complexity we cache this. previously we just used the old school System.Web.HttpRuntime.Cache. While this worked it had a few limitations, most notably ensuring consistency of cached data across multiple servers is all but impossible. So we elected to change out implementation to use a Redis cache server, which we can easily provision in Azure, WIN!
So we implemented a cache wrapper object to abstract away the complexity of connecting to the cache etc. Actually we borrowed the wrapper object we’d implemented on another project…
If you read the How to Use Azure Redis Cache article there is some great guidance in there on how to set up a connect to your cache. There’s a single line in that article that is EXTREMELY important; “The connection to the Azure Redis Cache is managed by the ConnectionMultiplexer class. This class is designed to be shared and reused throughout your client application, and does not need to be created on a per operation basis.” In fact this class MUST be shared in a single instance manner. Now if you’re using an IOC container, such as Unity or Ninject, all you need do is ensure that your IOC container has an instance of your wrapper class to treat as a singleton and then this criteria is met.
Now, the project that we borrowed that wrapper class into isn’t using a IOC container, unlike where it came from. The net result was that we wound up creating a ConnectionMultiplexer instance every time our RedisCache wrapper object was instantiated meaning that we slowly added more and more open client connections to the Redis server. Being that this code was running on IIS the app pool was recycling nightly, as they do, and closing all of those open connections…
So we didn’t notice our problem, until the number of calls into the code that talked to Redis reached a certain level, at which point the Redis server came to a grinding halt with a load metric of 100% 😐
Full credit to the Azure Support team, they have been super responsive and helped me resolve the issue we had with our code. Personally I’d love the Azure team to include a full class listing in that article, or linked from it, that handles the connections properly. But until such time as they do so I’m going to provide one here which they came up with in the thread I work with them.
public class RedisCache : IRepositoryCache
private static ConfigurationOptions _configurationOptions;
private readonly CachePrefix _prefix;
public RedisCache(ConfigurationOptions configurationOptions, CachePrefix prefix)
if (configurationOptions == null) throw new ArgumentNullException("configurationOptions");
_configurationOptions = configurationOptions;
_prefix = prefix;
private static IDatabase Cache
private static readonly Lazy<ConnectionMultiplexer> LazyConnection
= new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(_configurationOptions));
public static ConnectionMultiplexer Connection
public void ClearItem(string key)
key = _prefix + key;
if (key == null) throw new ArgumentNullException("key");
// Other cache access methods ommited for brevity
The key things that make this implementation work is that the _configurationOptions member and the wrappers around the ConnectionMultiplexer are static and therefore shared among all instances of this class.
Once I got this version of the code up into production then the number of open connections to the Redis server dropped right off and hasn’t grown out of control since 🙂
Anyway, hopefully this helps someone else avoid making the same mistake we did.