Every had a latent bug go undetected and then jump up and bite you?
Yeah, not the nicest feeling.
I’ve been working on a pretty interesting project that now makes use of Redis to provide a caching layer as the system uses multiple servers and does some reasonably heavy computation to prepare the data for use in the front end. For reference there are numerous REST calls to get the basic data and then a raft of search queries that are executed (again via REST) to build up a data payload of about 1MB. Anyhow given this complexity we cache this. previously we just used the old school System.Web.HttpRuntime.Cache. While this worked it had a few limitations, most notably ensuring consistency of cached data across multiple servers is all but impossible. So we elected to change out implementation to use a Redis cache server, which we can easily provision in Azure, WIN!
So we implemented a cache wrapper object to abstract away the complexity of connecting to the cache etc. Actually we borrowed the wrapper object we’d implemented on another project…
If you read the How to Use Azure Redis Cache article there is some great guidance in there on how to set up a connect to your cache. There’s a single line in that article that is EXTREMELY important; “The connection to the Azure Redis Cache is managed by the ConnectionMultiplexer class. This class is designed to be shared and reused throughout your client application, and does not need to be created on a per operation basis.” In fact this class MUST be shared in a single instance manner. Now if you’re using an IOC container, such as Unity or Ninject, all you need do is ensure that your IOC container has an instance of your wrapper class to treat as a singleton and then this criteria is met.
Now, the project that we borrowed that wrapper class into isn’t using a IOC container, unlike where it came from. The net result was that we wound up creating a ConnectionMultiplexer instance every time our RedisCache wrapper object was instantiated meaning that we slowly added more and more open client connections to the Redis server. Being that this code was running on IIS the app pool was recycling nightly, as they do, and closing all of those open connections…
So we didn’t notice our problem, until the number of calls into the code that talked to Redis reached a certain level, at which point the Redis server came to a grinding halt with a load metric of 100% 😐
Full credit to the Azure Support team, they have been super responsive and helped me resolve the issue we had with our code. Personally I’d love the Azure team to include a full class listing in that article, or linked from it, that handles the connections properly. But until such time as they do so I’m going to provide one here which they came up with in the thread I work with them.
public class RedisCache : IRepositoryCache
{
private static ConfigurationOptions _configurationOptions;
private readonly CachePrefix _prefix;
public RedisCache(ConfigurationOptions configurationOptions, CachePrefix prefix)
{
if (configurationOptions == null) throw new ArgumentNullException("configurationOptions");
_configurationOptions = configurationOptions;
_prefix = prefix;
}
private static IDatabase Cache
{
get
{
return Connection.GetDatabase();
}
}
private static readonly Lazy<ConnectionMultiplexer> LazyConnection
= new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(_configurationOptions));
public static ConnectionMultiplexer Connection
{
get
{
return LazyConnection.Value;
}
}
public void ClearItem(string key)
{
key = _prefix + key;
if (key == null) throw new ArgumentNullException("key");
Cache.KeyDelete(key);
}
// Other cache access methods ommited for brevity
}
The key things that make this implementation work is that the _configurationOptions member and the wrappers around the ConnectionMultiplexer are static and therefore shared among all instances of this class.
Once I got this version of the code up into production then the number of open connections to the Redis server dropped right off and hasn’t grown out of control since 🙂
Anyway, hopefully this helps someone else avoid making the same mistake we did.
Gav's Blog - Tech Adventures 
You saved my day!
Thanks much. Saved lot of research time
since the connection is static can’t redirect to different cache server (at the same time) with this Cache class right?. Please advice
As it stands, correct.
You would need to have a means of setting a new set of ConfigurationOptions and opening a new connection exposed publicly.
THANK YOU for posting this! We were having the same issue – our connections were capping out super quickly due to some changes to our DataCache to make it more IoC friendly.
can you show a sample of how you are using the RedisCache in your code?
What are you interested in here? How we’re reading and writing to/from Redis or something else?
I am trying to use redis in a node azure function and am a little unsure of where to put the connection logic, I don’t want to be accidentally opening too many connections.
Sorry, I’m not familiar with the NodeJS libraries
great post btw
Great article, however can you point how do you dispose the connection object?
We’re not explicitly disposing the connection and we’ve not detected this being an issue.
What scope do you suggest we use? Per request or per application?
We’ve been using per application scope on the projects where we have DI
Do you have a sample of how you are using this compared to the sample that you started with? I did notice the connections going up testing and thought that was odd. Just not sure how to use the code you supplied. We have a simple MVC application so were accessing this stuff from a controller.
The key thing to ensure is that the field used to store the ConfigurationOptions object and the property used to get the ConnectionMultiplexer object are marked static.
I’m using the same configuration.
When my application starts and the initial cache hit the connection will be open. From next request onward it will serve from that connection. But after few hours with no requests (I think IIS recycle) the connection initialization gets called again.
Is this normal?
Yes. If IIS is recycling the app pool then the code will be re-executed. This is perfectly normal.
I’m not sure if I’d still get a reply from this question but will still try. 🙂
What if you already have a cached data in Redis but there has been an update in the sql server db, how do you tell Redis about these changes and how do you update the cache? Thanks in advance. 🙂
Ahhh Cache management/invalidation.
You know there’s only two hard things in coding?
Naming, Cache Management and Off by one errors.
But I digress, assuming that you’re in control of the code sending updates to your database you could ensure that any calls to the update method either clear the associated cache key or update the data held in the cache.
On the other hand you might need to consider a background job to update teh cache on a periodic basis. This can also have benefits in ensuring that the items in your cache don’t expire due to you expiry policies.
This is certainly a non-trivial problem and there’s many schools of thought in this space.
Hi Gavin, why wouldn’t you set an instance of RedisCache : ICache as a singleton in your IoC? This way you can alleviate all the static methods and still provide the benefits of mocking your ICache interface and more importantly the IDatabase interface when Unit Testing.
Great article.
Great question!
The issue were only occurring in a system that wasn’t using an IoC container
The system in question wasn’t using an IoC Container for a variety of reasons and when we added Redis to the mix we didn’t want throw that level of change onto that project.
Pingback: Cache Manager : Agiliza tus desarrollos en Azure - Piensa en software, desarrolla en colores