Recover from failures
Use bounded retries and separate temporary failures from permanent problems. This module has 3 lessons and takes about 1h 50m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 13 in the Learning LabPlan a backoff delay
For a non-negative integer attempt, return a capped exponential delay of min(2 ** attempt, 30) seconds.
Immediate repeated retries can overload a struggling service. Exponential backoff increases the wait between attempts. A cap prevents very long waits.
For attempt numbers starting at zero, this example uses min(2 ** attempt, 30) seconds. It calculates a delay but does not sleep. Real clients may add jitter and respect Retry-After.
A backoff schedule needs an attempt limit as well. Waiting longer forever is still an unbounded retry loop.
See it in action
def solve(attempt):
return min(2 ** attempt, 30)
print(solve(0))
A backoff schedule needs an attempt limit as well. Waiting longer forever is still an unbounded retry loop.
Classify retryable errors
Return retry only for rate_limit or temporary when replay_safe is exactly True. Return stop otherwise.
Some failures can improve on a later attempt; others need a changed request or permission. Blind retries waste time and can repeat side effects.
Our simplified policy retries rate_limit and temporary only when replay_safe is True. Authentication, validation, and unknown errors stop for investigation.
A timeout can leave the outcome unknown. Before replaying a write, use provider idempotency support or reconcile whether it already succeeded.
See it in action
def solve(error, replay_safe):
return "retry" if error in ("rate_limit", "temporary") and replay_safe is True else "stop"
print(solve("temporary", True))
A timeout can leave the outcome unknown. Before replaying a write, use provider idempotency support or reconcile whether it already succeeded.
Practical: Finish or escalate a retry run
Return done on success; retry a replay-safe temporary or rate_limit outcome below three attempts; then dead letter. Return review otherwise.
A retry policy must define success, stop, wait, and escalation. Attempts here count calls already made, including the latest failure.
Success returns done. An unsafe replay returns review. A safe temporary failure returns retry while attempts are below three, then dead letter. Other failures return review.
A dead-letter queue stores exhausted work for an owner to inspect. It is not a place to forget failed requests.
Your experiment plan
Use the browser editor or the notebook. All records and AI responses are fictional. No network calls, credentials, or real messages are used.
- Simulate a temporary failure followed by success and three repeated failures.
- Run the example and predict its result before changing it.
- Complete the function in Practice and run the checks. Add your own normal, empty, and failure cases.
- Mark replay_safe False and confirm a retry is not scheduled.
- Save the notebook with your observations and explain one limitation of the simulation.
What to hand in
- A retry state table with delays, limits, and escalation ownership.
- A test table with inputs, expected outputs, actual outputs, and one revision.
Reflect on your results
How would you reconcile an unknown external result?
The automatic exercise checks the core code behaviour. Review observations and written deliverables yourself or with a mentor.
See it in action
def solve(outcome, attempts, replay_safe):
if outcome == "success":
return "done"
if outcome in ("rate_limit", "temporary") and replay_safe is True:
return "retry" if attempts < 3 else "dead letter"
return "review"
print(solve("temporary", 1, True))
A dead-letter queue stores exhausted work for an owner to inspect. It is not a place to forget failed requests.
Free, open and run by a nonprofit.
This page is part of AI Automation, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.