Skip to content

fix: run fallback fetch after a single failed attempt - #2388

Open
JackyHsu-X wants to merge 1 commit into
unclecode:developfrom
JackyHsu-X:fix/fallback-fetch-without-retries
Open

JackyHsu-X wants to merge 1 commit into
unclecode:developfrom
JackyHsu-X:fix/fallback-fetch-without-retries

Conversation

@JackyHsu-X

Copy link
Copy Markdown

Summary

A fetch exception skips fallback_fetch_function when max_retries=0 and there is only one proxy/direct attempt. For example, a failed direct fetch returns an error without calling an explicitly configured fallback, while the same configuration with one retry reaches the fallback.

Let that exception reach the existing fallback path when a fallback is available. This uses a directly constructed configuration and is separate from the config-cloning fix in #2385.

List of files changed and why

  • crawl4ai/async_webcrawler.py: preserve the early error path when neither another attempt nor a fetch fallback is available.
  • tests/unit/test_fallback_fetch_without_retries.py: cover exception/403 recovery with zero and one retry, empty/raising fallbacks, no fallback, raw HTML, and successful fetches.
  • docs/md_v2/advanced/anti-bot-and-fallback.md: clarify that a fallback does not require an extra retry after an exception.

How Has This Been Tested?

  • The final nine regression cases produce 3 failed / 6 passed against the original arun() from develop at 2efcccc5d1838da1f7718813e9be5177ce579b49; 9 passed with the fix.
  • Related configuration, deep-crawl, and anti-bot-placeholder tests: 209 passed, 2 skipped, 1 deselected on Windows / Python 3.12.8. The skipped tests require the optional PDF dependency. The deselected BFS timing check divides by zero on the unchanged source as well (Windows monotonic clock resolution: 15.625 ms).
  • Black passes for the new tests; git diff --check and installed dependency checks pass.

The fetch strategy is mocked; HTML processing and Markdown generation run through the real AsyncWebCrawler.arun() path. No browser or live model call is needed. The full repository/browser integration suite was not run. The production module has existing formatting drift outside this change; its changed condition is unchanged by Black's range formatter.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes (related scope and exclusions above)

AI assistance was used for investigation, implementation, and validation. No independent human review of the full diff is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant