ExponentialBackoff.__call__ computes
delay = min(self.base_delay * (self.factor**context.attempt), self.max_delay)
so the product is evaluated before min() clamps it. With the default shape (base delay 1 s, factor 2.0, max delay 1 h) the product exceeds the timedelta limit at attempt 47:
OverflowError: days=1628906115; must have magnitude <= 999999999
Repro (threadmill 0.7.1)
import datetime
from types import SimpleNamespace
from threadmill.retry import ExponentialBackoff
policy = ExponentialBackoff(
base_delay=datetime.timedelta(seconds=1),
max_delay=datetime.timedelta(hours=1),
factor=2.0,
max_retries=720,
)
def context(attempt):
error = SimpleNamespace(exception_class=ValueError)
return SimpleNamespace(attempt=attempt, task_result=SimpleNamespace(errors=[error]))
for attempt in range(1, 60):
try:
print(attempt, policy(context(attempt)))
except Exception as exc:
print(attempt, type(exc).__name__, exc)
break
Attempts 1 to 46 return a delay (2 s doubling to the 1 h cap at attempt 12), attempt 47 raises.
Why it matters
Executor.retry_delay catches the exception, logs Retry callback failed, and returns None, so the backend acknowledges the result and the retry chain ends. The task looks like it exhausted its policy, but max_retries was never reached: a policy of 720 attempts really stops after 46 retries.
Suggested fix
Clamp in seconds before building the timedelta, and derive the capped attempt count from max_delay so the exponential is never evaluated past the cap:
seconds = min(
self.base_delay.total_seconds() * self.factor**context.attempt,
self.max_delay.total_seconds(),
)
return datetime.timedelta(seconds=seconds)
A plain min() on the two timedelta values does not help on its own, because the product still overflows before the comparison.
Found while bounding the spam scan retry budget in codingjoe/relay#230.
ExponentialBackoff.__call__computesso the product is evaluated before
min()clamps it. With the default shape (base delay 1 s, factor 2.0, max delay 1 h) the product exceeds thetimedeltalimit at attempt 47:Repro (threadmill 0.7.1)
Attempts 1 to 46 return a delay (2 s doubling to the 1 h cap at attempt 12), attempt 47 raises.
Why it matters
Executor.retry_delaycatches the exception, logsRetry callback failed, and returnsNone, so the backend acknowledges the result and the retry chain ends. The task looks like it exhausted its policy, butmax_retrieswas never reached: a policy of 720 attempts really stops after 46 retries.Suggested fix
Clamp in seconds before building the
timedelta, and derive the capped attempt count frommax_delayso the exponential is never evaluated past the cap:A plain
min()on the twotimedeltavalues does not help on its own, because the product still overflows before the comparison.Found while bounding the spam scan retry budget in codingjoe/relay#230.