Conversation
The publisso jq script failed with 'string and array cannot be added' when the doi field was an array instead of a string. This caused the entire publisso repo to report a jq error and lose data. Fixed by checking type before string concatenation in two places: - Line 8: @id construction - Line 23: identifier value and url construction
The edal repository (335 URLs) had many URLs returning 500/503/timeout errors from the IPK server. Previously, HttpSessionTechnicalError was not caught in _extract_metadata, causing the entire repo to be skipped. Now catches HttpSessionTechnicalError alongside HttpSessionResponseError and HttpSessionDecodeError, treating server errors as recoverable per-dataset failures instead of unrecoverable repo-wide errors. Result: edal now returns ~299 valid entries instead of 0 (skipped).
Root cause: IPK server (doi.ipk-gatersleben.de) rate-limits under concurrent requests, returning transient 503s. With connection_limit 20 and no retry logic, ~54% of edal URLs failed per run. Changes: - get_decoded_url: retry up to 3x with exponential backoff + jitter on 5xx/network/timeout errors (transient). 4xx permanent, no retry. - Set raise_for_status=False so explicit status checks are effective. - edal per-repo http_client: connection_limit 4 (gentler on server). - Added 3 new tests: retry-then-succeed, retry-exhausts, no-retry-404. Verified: sequential requests all succeed; 20 concurrent = 25% fail; 4 concurrent = 0% fail. Retry covers transient spikes.
Set default User-Agent header to 'FAIRagro-Harvester/1.0 (harvestmaster@fairagro.net)'. All HTTP requests now identify as FAIRagro bot instead of default aiohttp UA. User-Agent is configurable via HttpSessionConfig.user_agent field.
jorgitogb
changed the title
- Add encoding='utf-8' to open() call (W1514) - Wrap long lines in metadata_scraper (C0301) - Remove trailing whitespace in openagrar.py (C0303) - Extract _fetch_http and _read_file helpers from get_decoded_url (R0912, R0915) - Remove redundant except HttpSessionResponseError: raise (W0706)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.