Implementing retry and circuit breaker pattern using Polly

In a highly distributed cloud based application infrastructure a single application depends on many other application and services.In this kind of environment it is important to have special focus on stability and robustness of the application.What that means is that one of the dependent service failing due to a transient failure should not fail whole application. A transient failure is a failure which is short term and unpredictable and corrects itself automatically (e.g. application restarts or network connectivity fluctuations or timeouts).It becomes worse when there is chain of dependency between applications and one application failing due to a transient failure results into cascading failures.

On the other hand there are certain scenarios where failures can be long term and might take couple of hours or even days to fix .In such scenarios it is not practical to keep calling dependent service resulting in failed requests as this would consume unnecessary resources (memory ,threads, db connections etc.) and bandwidth.

There are specific patterns which can deal with both of these situations.

Retry pattern

Retry pattern deals with the first situation where we don’t want our application to fail because of transient faults.In such case a retry pattern can just retry (either endlessly or for a particular number of times) and overcome the transient fault scenario.Dependent applications in such cases wont even be aware that any such faults occurred and user experience would be similar to a normal application flow.

Circuit Breaker pattern

This pattern deals with the second situation where you don’t want to keep calling a service which is down and it will take significant time to bring it up. Such a pattern can be configured to allow certain number of exceptions to occur before it breaks the circuit i.e. it does not allow the dependent service to be called any more.This is called as circuit is in open state.You can further configure that how long the circuit will be in open state before trying again to call the service. If this call succeeds the circuit will be closed and everything goes normal .If this calls fail circuit will again go in the open state.

In an actual production application to meet availability and stability demands it is essential that we implement both these patterns in combination. So you implement a retry pattern to deal with transient failures but in case problem is not solved on multiple retries circuit breaker comes into play and breaks the circuit.

Lets see an implementation of this using Polly library which saves you from implementing all this logic by yourself.

Implementing combination of retry and circuit breaker pattern

For demonstrating this I have a simple web api which returns a list of string values from the GET method (this is default implementation when you create a web api project using visual studio).This web api will be called by a console application which will be calling this API in a while loop simulating a scenario where you are continuously calling external services for some information (let say weather data,traffic ,stock prices etc).

For using Polly you need to have Polly nuget package installed.

No retry and circuit breaker

Lets see a scenario where there is no pattern applied .I will run the console application and in between restart the web api which can be considered as a transient fault.

In this case if you see the moment web api was unavailable application stopped. Below is the code which I am using

namespace PollyDemo.Console

{
class Program
{
static HttpClient _client = new HttpClient();

static Policy _circuitBreakerPolicy;
static Policy _retryPolicy;
static void Main(string[] args)
{

InitClient();
NoPolicy();

}

static void InitClient()
{
_client.BaseAddress = new Uri("<a href="http://localhost:2891/&quot;);">http://localhost:2891/");</a>
_client.DefaultRequestHeaders.Accept.Clear();
_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}

static void NoPolicy()
{
while (true)
{
try
{
System.Console.WriteLine("Calling api to get values...");
//Calling web api
GetValuesAsync("api/values").Result.ForEach(x =>
{
System.Console.WriteLine($"Value : {x}");

});
Thread.Sleep(1000);
}
catch (Exception ex)
{

System.Console.WriteLine("Error occured in application : " + ex.Message);
}
}
}
static async Task<List<string>> GetValuesAsync(string path)
{
List<string> values = null;
HttpResponseMessage response = await _client.GetAsync(path);
if (response.IsSuccessStatusCode)
{
values = await response.Content.ReadAsAsync<List<string>>();
}
return values;
}
}

}

With retry policy applied

Lets add a retry policy now so that we can handle this transient failure.Before that lets see how Polly works.

When you define a retry policy in Polly you need to follow the sequence shown below.

Below code adds a RetryForever policy for our code.Also I have highlighted each of the above steps in the code below.

We are using RetryForever policy where a code is retried forever.But Polly also provides other policies like Retry (you can give number of times to retry),WaitAndRetry (Wait before next attempt to retry e.g. for implementing exponential retries) and CircuitBreaker.

Lets see how code behaves now.

Applying both Retry and Circuit breaker.

In above example we were looking at restart scenario but if the API / service does not come back in timely fashion our application will keep trying forever using resources and bandwidth.For this we will now add  combination of Retry and circuit breaker policy.

In this case we will define two instances of policies (one retry and one circuit breaker) and then wrap one into another as shown below.

1. Define a retry and circuit breaker policy.

//Make sure you create single instance of this policy and use it in code
// since each of the failure should occur on the same instance
_circuitBreakerPolicy = Policy.Handle<AggregateException>(x =>
{
var result = x.InnerException is HttpRequestException;
return result;
})
.CircuitBreaker(5, TimeSpan.FromSeconds(10));
_retryPolicy = Policy.Handle<AggregateException>(x =>
{
var result = x.InnerException is HttpRequestException;
return result;
}).RetryForever();

What the circuit breaker policy indicates is that circuit will break if 5 HttpRequest exceptions occur and it will remain open for 10 seconds  i.e. No api call will go through for these 10 seconds.

After 10 seconds if the call again fails the circuit will again go in open state else it will work as usual.

2. Wrap circuit breaker in retry policy and execute the desired code.

retryPolicy.Wrap(_circuitBreakerPolicy)
.Execute(() =>
{
GetValuesAsync("api/values").Result.ForEach(x =>
{
System.Console.WriteLine($"Value : {x}");
});

});

Below is how application behaves now.

This is how we can deal with both transient and long term downtime in dependent services and keep application stable and robust. Below is the whole code with Retry and circuit breaker policy.

using Polly;

using System;

using System.Collections.Generic;

using System.Linq;

using System.Net;

using System.Net.Http;

using System.Net.Http.Headers;

using System.Text;

using System.Threading;

using System.Threading.Tasks;

namespace PollyDemo.Console

{
class Program
{
static HttpClient _client = new HttpClient();

static Policy _circuitBreakerPolicy;
static Policy _retryPolicy;
static void Main(string[] args)
{

InitClient();
//Make sure you create single instance of this policy and use it in code since each of the failure should occur on the same instance
_circuitBreakerPolicy = Policy.Handle<AggregateException>(x =>
{
var result = x.InnerException is HttpRequestException;
return result;
})
.CircuitBreaker(5, TimeSpan.FromSeconds(10));
_retryPolicy = Policy.Handle<AggregateException>(x =>
{
var result = x.InnerException is HttpRequestException;
return result;
}).RetryForever(ex=>System.Console.WriteLine("Retrying..."));
RetryAndCircuitBreakerPolicy();

System.Console.ReadLine();
}

static void InitClient()
{
//Address for you API. You can create a default web api project in visual studio and use that for this application to work.
_client.BaseAddress = new Uri("<a href="http://localhost:2891/&quot;);">http://localhost:2891/");</a>
_client.DefaultRequestHeaders.Accept.Clear();
_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}

static void NoPolicy()
{
while (true)
{
try
{
System.Console.WriteLine("Calling api to get values...");
//Calling web api
GetValuesAsync("api/values").Result.ForEach(x =>
{
System.Console.WriteLine($"Value : {x}");

});
Thread.Sleep(1000);
}
catch (Exception ex)
{

System.Console.WriteLine("Error occured in application : " + ex.Message);
}
}
}
static void RetryAndCircuitBreakerPolicy()
{
while (true)
{
try
{
System.Console.WriteLine("Calling api to get values...");

_retryPolicy.Wrap(_circuitBreakerPolicy)
.Execute(() =>
{
GetValuesAsync("api/values").Result.ForEach(x =>
{
System.Console.WriteLine($"Value : {x}");
});

});
Thread.Sleep(2000);//Just for inserting some delay in console

}
catch (Exception ex)
{

System.Console.WriteLine("Error occured in application : " + ex.Message);
Thread.Sleep(1000);//Just for inserting some delay in console
}
}

}

static async Task<List<string>> GetValuesAsync(string path)
{
List<string> values = null;
HttpResponseMessage response = await _client.GetAsync(path);
if (response.IsSuccessStatusCode)
{
values = await response.Content.ReadAsAsync<List<string>>();
}
return values;
}
}

}

Retry and circuit breaker comes under cloud design patterns for availability and resilience.
For other patterns in this category refer this msdn page .Also there is an excellent pluralsight course for the same.