In BasicCrawler._should_retry_request, we always return False for status codes 400-499.
def _should_retry_request(self, context: BasicCrawlingContext, error: Exception) -> bool:
if context.request.no_retry:
return False
# Do not retry on client errors.
if isinstance(error, HttpStatusCodeError) and is_status_code_client_error(error.status_code):
return FalseThe is_status_code_client_error function enforces this behavior:
def is_status_code_client_error(value: int) -> bool:
"""Returns `True` for 4xx status codes, `False` otherwise."""
return 400 <= value <= 499 # noqa: PLR2004This makes it impossible to configure retry behavior for these codes.
In
BasicCrawler._should_retry_request, we always returnFalsefor status codes 400-499.The is_status_code_client_error function enforces this behavior:
This makes it impossible to configure retry behavior for these codes.