Skip to content

Add retry to coursebook scraper#155

Open
justinschwerdtfeger wants to merge 23 commits intodevelopfrom
coursebook-retry
Open

Add retry to coursebook scraper#155
justinschwerdtfeger wants to merge 23 commits intodevelopfrom
coursebook-retry

Conversation

@justinschwerdtfeger
Copy link
Copy Markdown
Contributor

@justinschwerdtfeger justinschwerdtfeger commented Apr 3, 2026

This PR was moved from a fork to a branch

This PR improves coursebook scraper's ability to retry and recover. Here's the main changes

  • log.Fatals have been replaced with simply returning an error or log.panic where the error shouldn't be recovered. log.Fatal runs Os.Exit, which is not recoverable. We should consider this for other tools as well.
  • Improve error handling and behavior during network timeouts or non-200 server response codes adding more exponential retry.
  • Retry flag will fully restart coursebook scraper upon error or panic. With the other changes, this is unlikely to fix any issues that went wrong during scraping, but it could. It acts as mostly a last resort, so the flag is optional
  • Wrong parameters end scraping early with log.Fatal and a clear error.

@justinschwerdtfeger justinschwerdtfeger marked this pull request as ready for review April 20, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant