Don’t get burned by Redis ConnectionMultiplexer; A sample wrapper

Every had a latent bug go undetected and then jump up and bite you?
Yeah, not the nicest feeling.

I’ve been working on a pretty interesting project that now makes use of Redis to provide a caching layer as the system uses multiple servers and does some reasonably heavy computation to prepare the data for use in the front end. For reference there are numerous REST calls to get the basic data and then a raft of search queries that are executed (again via REST) to build up a data payload of about 1MB. Anyhow given this complexity we cache this. previously we just used the old school System.Web.HttpRuntime.Cache. While this worked it had a few limitations, most notably ensuring consistency of cached data across multiple servers is all but impossible. So we elected to change out implementation to use a Redis cache server, which we can easily provision in Azure, WIN!

So we implemented a cache wrapper object to abstract away the complexity of connecting to the cache etc. Actually we borrowed the wrapper object we’d implemented on another project…

If you read the How to Use Azure Redis Cache article there is some great guidance in there on how to set up a connect to your cache. There’s a single line in that article that is EXTREMELY important; “The connection to the Azure Redis Cache is managed by the ConnectionMultiplexer class. This class is designed to be shared and reused throughout your client application, and does not need to be created on a per operation basis.” In fact this class MUST be shared in a single instance manner. Now if you’re using an IOC container, such as Unity or Ninject, all you need do is ensure that your IOC container has an instance of your wrapper class to treat as a singleton and then this criteria is met.

Now, the project that we borrowed that wrapper class into isn’t using a IOC container, unlike where it came from. The net result was that we wound up creating a ConnectionMultiplexer instance every time our RedisCache wrapper object was instantiated meaning that we slowly added more and more open client connections to the Redis server. Being that this code was running on IIS the app pool was recycling nightly, as they do, and closing all of those open connections…
So we didn’t notice our problem, until the number of calls into the code that talked to Redis reached a certain level, at which point the Redis server came to a grinding halt with a load metric of 100% 😐

Full credit to the Azure Support team, they have been super responsive and helped me resolve the issue we had with our code. Personally I’d love the Azure team to include a full class listing in that article, or linked from it, that handles the connections properly. But until such time as they do so I’m going to provide one here which they came up with in the thread I work with them.

public class RedisCache : IRepositoryCache
{
  private static ConfigurationOptions _configurationOptions;
  private readonly CachePrefix _prefix;

  public RedisCache(ConfigurationOptions configurationOptions, CachePrefix prefix)
  {
    if (configurationOptions == null) throw new ArgumentNullException("configurationOptions");
    _configurationOptions = configurationOptions;
    _prefix = prefix;
  }


  private static IDatabase Cache
  {
    get
    {
      return Connection.GetDatabase();
    }
  }

  private static readonly Lazy<ConnectionMultiplexer> LazyConnection 
    = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(_configurationOptions));

  public static ConnectionMultiplexer Connection
  {
    get
    {
      return LazyConnection.Value;
    }
  }
  public void ClearItem(string key)
   {
     key = _prefix + key;
     if (key == null) throw new ArgumentNullException("key");
     Cache.KeyDelete(key);
   }

  // Other cache access methods ommited for brevity
}

The key things that make this implementation work is that the _configurationOptions member and the wrappers around the ConnectionMultiplexer are static and therefore shared among all instances of this class.

Once I got this version of the code up into production then the number of open connections to the Redis server dropped right off and hasn’t grown out of control since 🙂

Anyway, hopefully this helps someone else avoid making the same mistake we did.

Advertisements
This entry was posted in Azure, Best Practice, Development. Bookmark the permalink.

23 Responses to Don’t get burned by Redis ConnectionMultiplexer; A sample wrapper

  1. Will says:

    You saved my day!

  2. Whizz Coder says:

    Thanks much. Saved lot of research time

  3. Anthony says:

    since the connection is static can’t redirect to different cache server (at the same time) with this Cache class right?. Please advice

  4. TheJmack says:

    THANK YOU for posting this! We were having the same issue – our connections were capping out super quickly due to some changes to our DataCache to make it more IoC friendly.

  5. james wolf says:

    can you show a sample of how you are using the RedisCache in your code?

  6. Ben Mehr says:

    Great article, however can you point how do you dispose the connection object?

  7. jstawski says:

    What scope do you suggest we use? Per request or per application?

  8. Jmac says:

    Do you have a sample of how you are using this compared to the sample that you started with? I did notice the connections going up testing and thought that was odd. Just not sure how to use the code you supplied. We have a simple MVC application so were accessing this stuff from a controller.

    • gavinbarron says:

      The key thing to ensure is that the field used to store the ConfigurationOptions object and the property used to get the ConnectionMultiplexer object are marked static.

  9. Sreejith MS says:

    I’m using the same configuration.
    When my application starts and the initial cache hit the connection will be open. From next request onward it will serve from that connection. But after few hours with no requests (I think IIS recycle) the connection initialization gets called again.
    Is this normal?

  10. Leslie says:

    I’m not sure if I’d still get a reply from this question but will still try. 🙂

    What if you already have a cached data in Redis but there has been an update in the sql server db, how do you tell Redis about these changes and how do you update the cache? Thanks in advance. 🙂

    • gavinbarron says:

      Ahhh Cache management/invalidation.

      You know there’s only two hard things in coding?
      Naming, Cache Management and Off by one errors.

      But I digress, assuming that you’re in control of the code sending updates to your database you could ensure that any calls to the update method either clear the associated cache key or update the data held in the cache.
      On the other hand you might need to consider a background job to update teh cache on a periodic basis. This can also have benefits in ensuring that the items in your cache don’t expire due to you expiry policies.
      This is certainly a non-trivial problem and there’s many schools of thought in this space.

  11. Hi Gavin, why wouldn’t you set an instance of RedisCache : ICache as a singleton in your IoC? This way you can alleviate all the static methods and still provide the benefits of mocking your ICache interface and more importantly the IDatabase interface when Unit Testing.

    Great article.

    • gavinbarron says:

      Great question!
      The issue were only occurring in a system that wasn’t using an IoC container
      The system in question wasn’t using an IoC Container for a variety of reasons and when we added Redis to the mix we didn’t want throw that level of change onto that project.

  12. Pingback: Cache Manager : Agiliza tus desarrollos en Azure - Piensa en software, desarrolla en colores

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s