Introduction
Circuit breaker logic is an excessive situation the place one must cease the applying mechanically if one thing is flawed. This downside could be with our utility itself or with the third-party APIs on which our utility depends. To implement circuit breaker logic you need to have an excellent understanding of retry insurance policies and transient exceptions. in digital phrases circuit breaking or opening a circuit means chopping the wire at a single level to cease the circulation of present so it doesn’t hurt the wire or elements linked to that wire.
Construction
We have now an present eCommerce system say Contoso which will get one million hits per second. In Contoso, now we have a buyer module hosted as an internet service. We have now an order processing module which is a inside module which signifies the warehouse to start out processing the objects within the order. We wished to make each the modules decoupled and async so we launched a serverless layer that employes azure capabilities and occasion hubs.
We used occasion hubs as a result of it’s able to handeling thousands and thousands of requests per second till the brink is reached i.e. 1000 requests per second or 1 MB of request information per second and now we have extra management on retries in instances of occasion hub then service bus or queue storage. When a buyer makes an order we retailer the order in any DB and push the information within the occasion hub. The azure operate is triggered by the occasion hub which sends the order information to the backend course of through Refit.
Requirement
After this new implementation, a brand new downside arose. What if the backend order processing app is down for jiffy? On this case, the azure operate will begin giving errors for each request filling the dear logs with errors. You additionally know that if the backend API is down that operate is working purposeless and simply growing compute items and value. Additionally due to this now we have to implement a lookup service to retrieve the misplaced information which was not processed due to backend pai downtime.
So our requirement is to know if our service is giving a lot of error in a restricted time in order that we are able to cease the operate app and repair the difficulty. We additionally wish to retry one message virtually 5 occasions if it generates a transient error
Now comes the advantage of the occasion hub. Occasions hubs keep a pointer which signifies that as much as this location occasions have been processed. When the occasion hub delivers a message to the operate app it expects a response, if the response is successful then the occasion hub will increase the pointer by one in any other case it doesn’t enhance the counter and the identical message could be retried.
Answer
We are able to both use [FixedDelayRetry] or [ExponentialBackOffRetry] attributes to start out the retry for extra particulars see right here.
[ExponentialBackoffRetry(5, "00:00:05", "00:00:50")]
The above attribute will retry the identical request 5 occasions after the primary request. It’ll delay the primary request by 5 seconds and the final request by 50 seconds. Now now we have made one change in our code. in catch blocks as a substitute of simply logging the error, we have to throw in order that the occasion hub is aware of that and it doesn’t enhance the pointer Now earlier than stopping the operate app now we have to create the azure logic app.
You may see within the logic app designer I’ve added one set off and one motion. You should utilize HTTP Set off or occasion grid set off however I already had an occasion hub employed for this so I used occasion hub set off.
Configure the set off with eventhub occasion title, shopper group (default or any), and different particulars set connection to eventhub.
Hook up with eventhub utilizing APIConnection.
Right here choose the eventhub namespace and press create. After this add motion as the following step and use Azure useful resource supervisor to cease the operate app with the next configuration,
Right here you may filter the information in accordance with your want or earlier than this HTTPAction, you may introduce a situation to filter the information. and it’s also possible to add extra electronic mail ship motion to inform the DevOps saying “one thing is flawed stopping operate app”
It’s good to make it possible for this logic app has acceptable entry to the operate app which it’ll cease.
Now the ultimate circuit breaker logic and find out how to name this logic app.
Create a Redis cache occasion in Azure the place we’ll retailer our error rely. I wished to cease the operate app I acquired 40 errors inside 100 seconds at any two-point in a timeframe so I arrange a cache with sliding expiry.
personal async Process OpenCircuit() {
ConnectionMultiplexer redis = ConnectionMultiplexer.Join("Redis Connection String");
IDatabase db = redis.GetDatabase();
var transaction = db.CreateTransaction();
transaction.SortedSetRemoveRangeByScoreAsync("Redis key title", double.NegativeInfinity, DateTime.Now.AddSeconds(100 * -1).Ticks);
transaction.SortedSetAddAsync("Redis key title", DateTime.Now.Ticks, DateTime.Now.Ticks);
transaction.KeyExpireAsync("Redis key title", new TimeSpan(0, 0, 100));
var sliding_failures = transaction.SortedSetLengthAsync("Redis key title");
if (await transaction.ExecuteAsync()) {
var failures = await sliding_failures;
if (failures >= 40) {
transaction = db.CreateTransaction();
if (await transaction.ExecuteAsync()) {
var eventHubClient = EventHubClient.CreateFromConnectionString("Eventhub connectionstring");
var physique = JsonConvert.SerializeObject("Cease app");
var eventData = new EventData(Encoding.UTF8.GetBytes(physique));
await eventHubClient.SendAsync(eventData);
}
}
}
}
As quickly as we ship the “cease app” message on the given eventhub to which the logic app is listening it would cease our operate app and save us valuable sources, cash, and logs.
One benefit of that is each time the operate app is restarted manually or mechanically it once more begins listening to eventhub message from the place it left. Additionally, consider every partition of eventhub has a separate pointer so.
Hope this helps.
For additional dialogue be at liberty to attach with me.
Thanks.
