Don’t get burned by Redis ConnectionMultiplexer; A sample wrapper

Every had a latent bug go undetected and then jump up and bite you?
Yeah, not the nicest feeling.

I’ve been working on a pretty interesting project that now makes use of Redis to provide a caching layer as the system uses multiple servers and does some reasonably heavy computation to prepare the data for use in the front end. For reference there are numerous REST calls to get the basic data and then a raft of search queries that are executed (again via REST) to build up a data payload of about 1MB. Anyhow given this complexity we cache this. previously we just used the old school System.Web.HttpRuntime.Cache. While this worked it had a few limitations, most notably ensuring consistency of cached data across multiple servers is all but impossible. So we elected to change out implementation to use a Redis cache server, which we can easily provision in Azure, WIN!

So we implemented a cache wrapper object to abstract away the complexity of connecting to the cache etc. Actually we borrowed the wrapper object we’d implemented on another project…

If you read the How to Use Azure Redis Cache article there is some great guidance in there on how to set up a connect to your cache. There’s a single line in that article that is EXTREMELY important; “The connection to the Azure Redis Cache is managed by the ConnectionMultiplexer class. This class is designed to be shared and reused throughout your client application, and does not need to be created on a per operation basis.” In fact this class MUST be shared in a single instance manner. Now if you’re using an IOC container, such as Unity or Ninject, all you need do is ensure that your IOC container has an instance of your wrapper class to treat as a singleton and then this criteria is met.

Now, the project that we borrowed that wrapper class into isn’t using a IOC container, unlike where it came from. The net result was that we wound up creating a ConnectionMultiplexer instance every time our RedisCache wrapper object was instantiated meaning that we slowly added more and more open client connections to the Redis server. Being that this code was running on IIS the app pool was recycling nightly, as they do, and closing all of those open connections…
So we didn’t notice our problem, until the number of calls into the code that talked to Redis reached a certain level, at which point the Redis server came to a grinding halt with a load metric of 100% 😐

Full credit to the Azure Support team, they have been super responsive and helped me resolve the issue we had with our code. Personally I’d love the Azure team to include a full class listing in that article, or linked from it, that handles the connections properly. But until such time as they do so I’m going to provide one here which they came up with in the thread I work with them.

public class RedisCache : IRepositoryCache
{
  private static ConfigurationOptions _configurationOptions;
  private readonly CachePrefix _prefix;

  public RedisCache(ConfigurationOptions configurationOptions, CachePrefix prefix)
  {
    if (configurationOptions == null) throw new ArgumentNullException("configurationOptions");
    _configurationOptions = configurationOptions;
    _prefix = prefix;
  }


  private static IDatabase Cache
  {
    get
    {
      return Connection.GetDatabase();
    }
  }

  private static readonly Lazy<ConnectionMultiplexer> LazyConnection 
    = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(_configurationOptions));

  public static ConnectionMultiplexer Connection
  {
    get
    {
      return LazyConnection.Value;
    }
  }
  public void ClearItem(string key)
   {
     key = _prefix + key;
     if (key == null) throw new ArgumentNullException("key");
     Cache.KeyDelete(key);
   }

  // Other cache access methods ommited for brevity
}

The key things that make this implementation work is that the _configurationOptions member and the wrappers around the ConnectionMultiplexer are static and therefore shared among all instances of this class.

Once I got this version of the code up into production then the number of open connections to the Redis server dropped right off and hasn’t grown out of control since 🙂

Anyway, hopefully this helps someone else avoid making the same mistake we did.

Advertisements
This entry was posted in Azure, Best Practice, Development. Bookmark the permalink.

14 Responses to Don’t get burned by Redis ConnectionMultiplexer; A sample wrapper

  1. Will says:

    You saved my day!

  2. Whizz Coder says:

    Thanks much. Saved lot of research time

  3. Anthony says:

    since the connection is static can’t redirect to different cache server (at the same time) with this Cache class right?. Please advice

  4. TheJmack says:

    THANK YOU for posting this! We were having the same issue – our connections were capping out super quickly due to some changes to our DataCache to make it more IoC friendly.

  5. james wolf says:

    can you show a sample of how you are using the RedisCache in your code?

  6. Ben Mehr says:

    Great article, however can you point how do you dispose the connection object?

  7. jstawski says:

    What scope do you suggest we use? Per request or per application?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s