Skip to content

fix: keep fallback_fetch_function and other fields in CrawlerRunConfig.clone() - #2385

Open
Parsiffall1 wants to merge 1 commit into
unclecode:developfrom
Parsiffall1:fix/clone-fallback-fetch-function-2379
Open

Parsiffall1 wants to merge 1 commit into
unclecode:developfrom
Parsiffall1:fix/clone-fallback-fetch-function-2379

Conversation

@Parsiffall1

Copy link
Copy Markdown

Summary

Fixes #2379

CrawlerRunConfig.clone() rebuilds the config from to_dict(), and to_dict() did not include fallback_fetch_function. BFS, DFS and BestFirst deep crawls clone the run config for every batch (config.clone(deep_crawl_strategy=None, stream=...)), so the fallback fetcher was dropped and never used for deep-crawled pages, while the same page fetched with arun() / arun_many() did use it.

check_cache_freshness, cache_validation_timeout and virtual_scroll_config were missing from to_dict() for the same reason and were also reset to their defaults on clone. This adds the four keys to to_dict(); clone() keeps working as before and still lets callers override any of them through kwargs.

List of files changed and why

  • crawl4ai/async_configs.py: add fallback_fetch_function, check_cache_freshness, cache_validation_timeout and virtual_scroll_config to CrawlerRunConfig.to_dict().
  • tests/unit/test_run_config_clone_fields.py: new tests that clone a config the way deep crawl strategies do and check these fields survive, and that fallback_fetch_function can still be overridden.

How Has This Been Tested?

  • pytest tests/unit/test_run_config_clone_fields.py: 3 passed; 2 of them fail on develop without the change.
  • pytest tests/unit: the same set of failures before and after the change (they need network/browser access locally); no new failures.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

…g.clone()

CrawlerRunConfig.clone() rebuilds the config from to_dict(), which left out
fallback_fetch_function, check_cache_freshness, cache_validation_timeout and
virtual_scroll_config. Deep crawl strategies clone the run config for every
batch, so these settings were silently reset and the fallback fetcher was
never called during deep crawls.

Add the missing keys to to_dict() and cover them with unit tests.

Fixes unclecode#2379
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant