Using Python Retry Decorator to Implement Retry Mechanism

Advertisement: Click here to learn how to Generate Art From Text

I’m sure there will be a time in your learning or working journey when you will need to implement a retry mechanism to carry out a specific request or function as usual. A Retry mechanism is a decent approach to giving it one or more than one chance to succeed, if things go south. You can find retry mechanisms in various applications such as retrying an API request in case a server is down or something and much more.

In this blog post, I will be taking you through 2 approaches for implementing Python retry decorator which will give us an important element of our retry mechanism: One is by using Python’s inbuilt library and the other approach is by building a simple function to implement retry mechanism by ourselves. I’m sure you’ll love either of them, so without further ado, let’s get into the approaches and start implementing the python retry decorator.

What are Python Decorators?

Before implementing the Python Retry Decorator, first, let’s look at what Python decorators are exactly. Well, there are some other ways for building a retry mechanism as well, but we are choosing Python decorators here to do it.

Python decorators provide a syntax for calling higher-order functions that take and return functions. They allow “wrapping” functions to extend behavior without permanently modifying them.

Decorators in Python are constructed using @syntax:

Python

@my_decorator
def my_func():
  # Function body here

This applies @my_decorator to my_func() during definition. Decorators can modify a function’s behavior and even replace it entirely.

Here’s a list of some of the well-known Python decorators that Python programmers often use:

  1. @classmethod
  2. @abstractmethod
  3. @staticmethod
  4. @property
  5. @singledispatch
  6. @lru_cache
  7. @authenticated 
  8. @validate_input

Similar Read – Python SHA256: Secure Hashing Implementation

1. Using Python Retry Decorator Library

Python offers an amazing retry library that offers a Python Retry Decorator that can save your life. This Python Retry decorator provides us with numerous functionalities under the hood that make the task of implementing the retry mechanism much easier. Let’s try to use it step by step:

To Begin with, we need to install Python’s retry package using pip. Below is the command for installing Python’s Retry Package:

pip install retry

Now to use that, create a new Python file and import the retry package in the file to start using it.

Python

from retry import retry

Here’s how you can use Python retry decorator with a function that gets the email address for us:

Python

@retry((ValueError, TypeError), tries=3, delay=2)
def get_email():
  """ It triggers retry if there's a ValueError or TypeError after 3 attempts and sleep 2 seconds between attempts."""

According to the documentation of Retry, it takes the following parameters:

Python

@retry(f, fargs=None, fkwargs=None, exceptions=Exception, tries=-1, delay=0, 
        max_delay=None, backoff=1, jitter=0, logger=logging_logger)

Here:

  • Exceptions is a tuple that takes the exceptions as input on which we need to trigger the retry mechanism.
  • tries is another parameter that represents the number of attempts.
  • delay is the initial delay in seconds between attempts.
  • max_delay is the maximum delay value.
  • backoff is the multiplier that is applied to delay between attempts to increase the delay after each cycle.

These were the parameters that you’ll mostly use, if you want to know about the other ones also, i suggest you visit the documentation of retry.

Similar Read – Sliding Window Algorithm: Explained with Example

2. Creating Python Retry Decorator Function

Now in this second approach, we will create a Python Retry Decorator Function that executes the same thing as our retry library did. Here’s how we can build a retry function:

Python

import time
import random


def retry(f, exceptions, tries=4, delay=3, max_delay=None, backoff=2):
    """ 
    Retry calling the decorated function using an exponential backoff.

    Args:
        f: The function to decorate and retry
        exceptions: An exception or tuple of exceptions to catch.
        tries: Number of times to try (not retry) before giving up. 
        delay: Initial delay between retries in seconds.
        max_delay: Max delay in seconds (or None for no max).
        backoff: Backoff multiplier e.g. value of 2 will double the delay each retry.
    """

    def wrapper(*args, **kwargs):
        mtries, mdelay = tries, delay

        while mtries > 1:
            try:
                return f(*args, **kwargs)
            except exceptions as e:
                msg = "%s, Retrying in %d seconds..." % (str(e), mdelay)
                print(msg)

                time.sleep(mdelay)
                mtries -= 1
                mdelay *= backoff

                if max_delay and mdelay > max_delay:
                    mdelay = max_delay

        return f(*args, **kwargs)

    return wrapper  
    
@retry(exceptions=ValueError, tries=4, delay=5, backoff=2)
def flaky_function():
    # Some logic here that might raise ValueError randomly
    if random.randint(1,3) != 1:
        raise ValueError("Broken")

    print("Success!")

flaky_function()

Here we created a retry function that will act as a python retry decorator which we will place on top of flaky_function() with some arguments. This will exactly perform how our retry python library does. You can try implementing this retry function locally and see if it works the way you want it to.

That’s it for using Python Retry Decorator !

Conclusion

To Conclude, we first understood what a python decorator is i.e, used calling higher-order functions that take and return functions. Then, we looked at two approaches to implement python retry decorator which are: By using a retry library in python and another approach is by writing our own small code for python retry decorator.

Both the approaches will help you to implement the python retry decorator mechanism to trigger retry whenever necessary.

Read More:

FAQs

  1. What problem do python retry decorator solve?

    Retry Decorator automate repeating function calls that fail intermittently until success, greatly simplifying the error handling code needed for fault-tolerant systems.

  2. What logic do retries encapsulate behind the scenes?

    Intricate details like handling transient exceptions, timing backoffs between retries, setting stop conditions, abort handlers, etc are embedded inside retrying decorators.

  3. What exceptions should trigger a retry vs bubble up to user?

    Ideally, retry transient issues like temporary connection losses or network blips. But pass through permanent failures caused by bad input or unsupported scenarios to user.

  4. How do you prevent endless futile retries?

    Set sane defaults like max 5 attempts overall or 2 minutes total retrying. Exponentially backoff retry intervals to ease load. Use conditional retries only for specific exceptions.

  5. When should I avoid using retry decorators in Python?

    Avoid retrying for non-idempotent operations like saving data or sending duplicate emails. Also if multiple retries make output non-deterministic.

  6. What is a good starting retry delay and backoff multiplier?

    Starting at 1-5 seconds with a backoff multiplier of 2x is reasonable for APIs. Tune values based on tolerance for delays vs risk of transient faults in system.

  7. Can Python retry decorator be applied selectively only on parts of a program?

    Yes, Python decorators integrate non-invasively by wrapping only chosen functions. No need to blanket apply retries globally.

Leave a Reply

Your email address will not be published. Required fields are marked *