Skip to content

ExponentialBackoff overflows timedelta above attempt 46 #47

Description

@codingjoe

ExponentialBackoff.__call__ computes

delay = min(self.base_delay * (self.factor**context.attempt), self.max_delay)

so the product is evaluated before min() clamps it. With the default shape (base delay 1 s, factor 2.0, max delay 1 h) the product exceeds the timedelta limit at attempt 47:

OverflowError: days=1628906115; must have magnitude <= 999999999

Repro (threadmill 0.7.1)

import datetime
from types import SimpleNamespace
from threadmill.retry import ExponentialBackoff

policy = ExponentialBackoff(
    base_delay=datetime.timedelta(seconds=1),
    max_delay=datetime.timedelta(hours=1),
    factor=2.0,
    max_retries=720,
)

def context(attempt):
    error = SimpleNamespace(exception_class=ValueError)
    return SimpleNamespace(attempt=attempt, task_result=SimpleNamespace(errors=[error]))

for attempt in range(1, 60):
    try:
        print(attempt, policy(context(attempt)))
    except Exception as exc:
        print(attempt, type(exc).__name__, exc)
        break

Attempts 1 to 46 return a delay (2 s doubling to the 1 h cap at attempt 12), attempt 47 raises.

Why it matters

Executor.retry_delay catches the exception, logs Retry callback failed, and returns None, so the backend acknowledges the result and the retry chain ends. The task looks like it exhausted its policy, but max_retries was never reached: a policy of 720 attempts really stops after 46 retries.

Suggested fix

Clamp in seconds before building the timedelta, and derive the capped attempt count from max_delay so the exponential is never evaluated past the cap:

seconds = min(
    self.base_delay.total_seconds() * self.factor**context.attempt,
    self.max_delay.total_seconds(),
)
return datetime.timedelta(seconds=seconds)

A plain min() on the two timedelta values does not help on its own, because the product still overflows before the comparison.

Found while bounding the spam scan retry budget in codingjoe/relay#230.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions