Repository navigation
Conversation
* Add tests to cover Finnish dates with 'klo' * Add Finnish "klo" (o'clock) to skip tokens and add translation test - Add "klo" to skip tokens in Finnish language data files - Modified: dateparser/data/date_translation_data/fi.py - Modified: dateparser_data/supplementary_language_data/date_translation_data/fi.yaml - Add translation test case for Finnish dates with "klo" - Modified: tests/test_languages.py This fixes parsing of Finnish dates like "28 maalis klo 9:37" (March 28 at 9:37). The word "klo" (abbreviation for "kello", meaning "o'clock") is now properly skipped during tokenization, allowing correct date parsing.
* Add tests for relative date time offset parsing * Fix explicit +/- signs in relative date offsets * Update tests/test_date_parser.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The demo has been broken for a long time scrapinghub#1173 and scrapinghub#1188 I played around with GitHub Spark throw and put together a in-browser demo, and then used claude to decouple it from Spark. It doesn't require any python backend but runs it in the browser with Pyodide, thus should be easier and cheaper to host. It could even be brought into this repository to make it a official demo.
…ear-biased `DATE_ORDER` (scrapinghub#1298)
It's soon to be removed from setuptools
…#1316) * Fix word-number phrases parsing (e.g., 'two days later') Close scrapinghub#1314 * Fix word-number phrases parsing (e.g., "two days later", "two days from now") (scrapinghub#1317) * Initial plan * Fix word-number phrases parsing for 'later' and 'from now' patterns Co-authored-by: serhii73 <24910277+serhii73@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: serhii73 <24910277+serhii73@users.noreply.github.com> Co-authored-by: Serhii A <aserhii@protonmail.com> * Update tests/test_date_parser.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply pre-commit * Fix approximate day counts in word number tests - Update 'four months later' from 120 to 122 days (June 15 to Oct 15, 2025) - Update 'six years later' from 2190 to 2191 days (accounts for leap year in 2028) - These values now match the actual calendar arithmetic performed by the parser --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: serhii73 <24910277+serhii73@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ub#1315) * Remove fastText support and enable numpy 2.x compatibility FastText library has been archived since March 2024 and is no longer maintained. This dependency was blocking numpy 2.x adoption. Changes: - Remove fasttext extra from setup.py dependencies - Add deprecation warnings to fasttext.py module - Update documentation to recommend langdetect as primary option - Update CLI to deprecate --fasttext flag - Add comprehensive test suite for langdetect integration - Remove fasttext from tox.ini test configurations Users should migrate to langdetect: pip install dateparser[langdetect] Closes scrapinghub#1287 Closes scrapinghub#1251 * Update dateparser_cli/cli.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update dateparser/custom_language_detection/fasttext.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Simplify fasttext deprecation to always error The previous approach was confusing: - If fasttext was installed, it showed a warning but still worked - If fasttext wasn't installed, it raised an ImportError This creates inconsistent deprecation - users don't know if the feature is truly deprecated or not. Now the module: - Always shows deprecation warning on import - Always raises ImportError when detect_languages() is called - Provides clear migration path to langdetect - No functional fasttext code remains This makes the deprecation clear and consistent. * Remove unused module-level mock in test_language_detect The module-level 'detect_languages' mock was defined but never used. The MockLangDetectTest class defines its own mock as a class attribute, and other test classes use the actual lang_detect_detect_languages function. This cleanup removes the unused mock to avoid confusion. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…nghub#1302) (scrapinghub#1324) * fix: preserve whitespace when removing translation tokens (fix scrapinghub#1302) - Fixed whitespace collapsing when skip tokens (like 'klo' in Finnish) are removed - When a skip token is removed between spaces, preserve the maximum of surrounding spaces - Added comprehensive tests in test_whitespace_preservation.py - Updated existing affected test expectations to match corrected behavior - Fixes issue where double spaces were created when skip tokens were removed * fix: update remaining test expectations for whitespace preservation - Fixed 6 edge case tests related to _clear_future_words method - Most cases now correctly expect single spaces (he, nl, pl, vi-lúc, da) - Vietnamese pipe case ('|') correctly expects double space preservation - All 1,342 language and whitespace tests now passing * Fix the issue
* tests(tests/freshness_date_parser.py): add tests for operations with spaces * fix(dateparser/freshness_date_parser.py): handle spaces around operator
…b#1332) * Document region usage for supported locales * Use parser-relative month for incomplete dates
* Fix search_dates language fallback * Apply ruff formatting
Bumps the github-actions group with 5 updates: | Package | From | To | | --- | --- | --- | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.2` | `7.0.1` | | [actions/checkout](https://github.com/actions/checkout) | `3.7.0` | `7.0.1` | | [actions/setup-python](https://github.com/actions/setup-python) | `5.6.0` | `7.0.0` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5.5.5` | `7.1.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | Updates `actions/upload-artifact` from 4.6.2 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4.6.2...043fb46) Updates `actions/checkout` from 3.7.0 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3.7.0...3d3c42e) Updates `actions/setup-python` from 5.6.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@a26af69...5fda3b9) Updates `codecov/codecov-action` from 5.5.5 to 7.1.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@0fb7174...303a32d) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@37930b1...3e5f45b) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-python dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: codecov/codecov-action dependency-version: 7.1.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…hole DATE_ORDER (scrapinghub#868) (scrapinghub#1443) * Add a STRICT_DATE_ORDER setting so that an invalid day is not read as the year (scrapinghub#868) DATE_ORDER is a preference: any number that is not a valid day or month is read as a two-digit year, so "32 DEC 10" with DATE_ORDER="DMY" is 2032-12-10. With the new setting, which defaults to False, a two-digit year is read only where DATE_ORDER puts the year, and such a date string gives None instead. The rejection is final: no other locale may guess another reading, and the no-spaces parser, which would read the same numbers in a date order again, is skipped. The other parsers of the locale still get their turn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Make STRICT_DATE_ORDER a string that can also enforce the day and month The setting is now "none" (the default), "year" or "all", so that it can name all three behaviours: DATE_ORDER as a preference, the position of a two-digit year enforced, or the whole order enforced. With "all", a day and a month that do not fit are no longer swapped, neither in the token loop nor through the retry with the swapped order: "13/12/10" with DATE_ORDER="MDY" is None. A four-digit year is still read wherever it is, and a date that starts with one is still read as year, month and day. With "all", a date string that cannot be read in DATE_ORDER at all is final too, so that no other locale reads it differently: with auto-detected languages, "10年20月00日" with DMY would otherwise come back from zh as a relative date, 10 years and 20 months ago. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Make STRICT_DATE_ORDER="year" reject a misplaced year in both directions A two-digit year is now read only where DATE_ORDER puts the year: the day and month read before it must be the ones the order puts before it. Until now only a day or month that the order puts before the year, but is read after it, was rejected, so "32 DEC 10" with DMY was None but "10 DEC 32" with MYD was still 2032-12-10, and "year" did nothing at all for YMD and YDM, where no part comes before the year. This also rejects scrapinghub#519's last-resort "4-99" with YMD, which is acceptable for an opt-in setting. DMY and MDY are not affected, since nothing comes after the year in them. The day and month swap is now also tested under DYM and MYD, where the year is between them and the swap changes which of them is read before it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Raise SettingValidationError when STRICT_DATE_ORDER has no DATE_ORDER to enforce "year" and "all" only enforce a DATE_ORDER that the caller sets, since the default one is a preference, so without one they silently did nothing. Raise a SettingValidationError from check_settings instead, which only looks at the settings the caller passed, so the rebuilt settings of search_dates, which forget those, cannot trigger it. "none" does not need a DATE_ORDER. The cases that pinned "no effect without DATE_ORDER" are replaced by tests of the error, of "none", and of search_dates and a reused parser, which keep working. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Shorten the STRICT_DATE_ORDER docs to the year rule and the two caveats "year" now reads as one rule: a two-digit year is read only from the position DATE_ORDER gives the year. Of the details after the examples, only two matter to users, that "year" and "all" need DATE_ORDER, and that '95年12月10日' with MDY is rejected. The rest, such as that a rejection is final across locales, which parsers still run, and that a month name or a missing part is not checked, is in the code comments. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Drop the right day when a month name follows a day range 05f2737 records in auto_order that the end of a day range is dropped when a month name follows it, but it always deleted the entry after the month. In a day-first order, as in "12-12 June 2021" with DMY or "5-5 juin 2021" in French, the day comes before the month, so the deletion raised IndexError, parse_alpha swallowed it and the month name was not read: the date was None even without STRICT_DATE_ORDER, unlike master. Remove the day entry itself instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test that a two-digit year is still the last resort under STRICT_DATE_ORDER Moving "4-99" with YMD and "10 DEC 32" with MYD to the rejected cases left no test of scrapinghub#519's two-digit year fallback under STRICT_DATE_ORDER, and left its comment on "10年20月00日", which goes through the day and month swap instead. Also say what the search_dates test checks: that it does not raise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say in the STRICT_DATE_ORDER docs which parts are checked Without it, "read a two-digit year only from the position DATE_ORDER gives the year" suggests that "32 DEC" with DMY is rejected too, but a month name and a missing part are not checked, and a four-digit year is read wherever it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop issue numbers from two comments and fix the STRICT_DATE_ORDER examples Quote the values of the setting in the docs, since they are strings, and drop the None lines from its examples, since the REPL prints nothing for None. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…crapinghub#351) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ghub#351) Add simplifications so a bare unit after the "ago" word or before the "ago" suffix means "1 unit": Arabic, Belarusian, Bulgarian, Croatian, Finnish, Hebrew, Latvian, Lithuanian, Macedonian, Polish, Slovenian and Ukrainian. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Owner
Author
|
Superseded by scrapinghub#1540 (this one was opened against the fork's stale master). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs scrapinghub#351
Problem
Many languages allow dropping the numeral "1" in relative dates: Czech "před týden", Polish "tydzień temu", Finnish "viikko sitten", Hebrew "לפני שבוע" all mean "1 week ago". dateparser only handled the numeric form, so these returned
None.Change
sl.yaml,lt.yaml,lv.yaml).write_complete_data.py.test_freshness_date_parser.py.Not covered: Slovak, Hungarian, Persian, Indonesian (bare form is already handled, ambiguous, or not natural), and Estonian and Serbian (the numeric form itself does not parse yet).
Testing
Full test suite and data-integrity test pass; ruff check and format are clean.
🤖 Generated with Claude Code