Skip to content

Parse relative dates without the numeral 1, like "před týden" (#351) - #1

Closed
yamilmaud wants to merge 107 commits into
masterfrom
fix_DC-1344_Handling_optional_1_day_ago_for_some_languages
Closed

yamilmaud wants to merge 107 commits into
masterfrom
fix_DC-1344_Handling_optional_1_day_ago_for_some_languages

Conversation

@yamilmaud

@yamilmaud yamilmaud commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Refs scrapinghub#351

Problem

Many languages allow dropping the numeral "1" in relative dates: Czech "před týden", Polish "tydzień temu", Finnish "viikko sitten", Hebrew "לפני שבוע" all mean "1 week ago". dateparser only handled the numeric form, so these returned None.

Change

  • Czech: bare "týden", "měsíc", "rok" and "den" after "před"; "před dnem"; accusative "minutu", "sekundu", "vteřinu" word forms.
  • Other languages: simplifications that turn a bare unit into "1 unit" next to the "ago" word. Added for Arabic, Belarusian, Bulgarian, Croatian, Finnish, Hebrew, Latvian, Lithuanian, Macedonian, Polish, Slovenian and Ukrainian (new sl.yaml, lt.yaml, lv.yaml).
  • Regenerated the language data with write_complete_data.py.
  • Tests for the new forms in test_freshness_date_parser.py.

Not covered: Slovak, Hungarian, Persian, Indonesian (bare form is already handled, ambiguous, or not natural), and Estonian and Serbian (the numeric form itself does not parse yet).

Testing

Full test suite and data-integrity test pass; ruff check and format are clean.

🤖 Generated with Claude Code

serhii73 and others added 30 commits January 23, 2026 14:08
* Add tests to cover Finnish dates with 'klo'

* Add Finnish "klo" (o'clock) to skip tokens and add translation test

- Add "klo" to skip tokens in Finnish language data files
  - Modified: dateparser/data/date_translation_data/fi.py
  - Modified: dateparser_data/supplementary_language_data/date_translation_data/fi.yaml
- Add translation test case for Finnish dates with "klo"
  - Modified: tests/test_languages.py

This fixes parsing of Finnish dates like "28 maalis klo 9:37" (March 28 at 9:37).
The word "klo" (abbreviation for "kello", meaning "o'clock") is now properly
skipped during tokenization, allowing correct date parsing.
* Add tests for relative date time offset parsing

* Fix explicit +/- signs in relative date offsets

* Update tests/test_date_parser.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The demo has been broken for a long time scrapinghub#1173 and scrapinghub#1188

I played around with GitHub Spark throw and put together a in-browser demo, and then used claude to decouple it from Spark.

It doesn't require any python backend but runs it in the browser with Pyodide, thus should be easier and cheaper to host. It could even be brought into this repository to make it a official demo.
It's soon to be removed from setuptools
…#1316)

* Fix word-number phrases parsing (e.g., 'two days later')

Close scrapinghub#1314

* Fix word-number phrases parsing (e.g., "two days later", "two days from now") (scrapinghub#1317)

* Initial plan

* Fix word-number phrases parsing for 'later' and 'from now' patterns

Co-authored-by: serhii73 <24910277+serhii73@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: serhii73 <24910277+serhii73@users.noreply.github.com>
Co-authored-by: Serhii A <aserhii@protonmail.com>

* Update tests/test_date_parser.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply pre-commit

* Fix approximate day counts in word number tests

- Update 'four months later' from 120 to 122 days (June 15 to Oct 15, 2025)
- Update 'six years later' from 2190 to 2191 days (accounts for leap year in 2028)
- These values now match the actual calendar arithmetic performed by the parser

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: serhii73 <24910277+serhii73@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ub#1315)

* Remove fastText support and enable numpy 2.x compatibility

FastText library has been archived since March 2024 and is no longer
maintained. This dependency was blocking numpy 2.x adoption.

Changes:
- Remove fasttext extra from setup.py dependencies
- Add deprecation warnings to fasttext.py module
- Update documentation to recommend langdetect as primary option
- Update CLI to deprecate --fasttext flag
- Add comprehensive test suite for langdetect integration
- Remove fasttext from tox.ini test configurations

Users should migrate to langdetect:
  pip install dateparser[langdetect]

Closes scrapinghub#1287
Closes scrapinghub#1251

* Update dateparser_cli/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update dateparser/custom_language_detection/fasttext.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Simplify fasttext deprecation to always error

The previous approach was confusing:
- If fasttext was installed, it showed a warning but still worked
- If fasttext wasn't installed, it raised an ImportError

This creates inconsistent deprecation - users don't know if the
feature is truly deprecated or not.

Now the module:
- Always shows deprecation warning on import
- Always raises ImportError when detect_languages() is called
- Provides clear migration path to langdetect
- No functional fasttext code remains

This makes the deprecation clear and consistent.

* Remove unused module-level mock in test_language_detect

The module-level 'detect_languages' mock was defined but never used.
The MockLangDetectTest class defines its own mock as a class attribute,
and other test classes use the actual lang_detect_detect_languages function.

This cleanup removes the unused mock to avoid confusion.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…nghub#1302) (scrapinghub#1324)

* fix: preserve whitespace when removing translation tokens (fix scrapinghub#1302)

- Fixed whitespace collapsing when skip tokens (like 'klo' in Finnish) are removed
- When a skip token is removed between spaces, preserve the maximum of surrounding spaces
- Added comprehensive tests in test_whitespace_preservation.py
- Updated existing affected test expectations to match corrected behavior
- Fixes issue where double spaces were created when skip tokens were removed

* fix: update remaining test expectations for whitespace preservation

- Fixed 6 edge case tests related to _clear_future_words method
- Most cases now correctly expect single spaces (he, nl, pl, vi-lúc, da)
- Vietnamese pipe case ('|') correctly expects double space preservation
- All 1,342 language and whitespace tests now passing

* Fix the issue
* tests(tests/freshness_date_parser.py): add tests for operations with spaces

* fix(dateparser/freshness_date_parser.py): handle spaces around operator
…b#1332)

* Document region usage for supported locales

* Use parser-relative month for incomplete dates
* Fix search_dates language fallback

* Apply ruff formatting
dependabot Bot and others added 27 commits September 30, 2026 13:45
Bumps the github-actions group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.2` | `7.0.1` |
| [actions/checkout](https://github.com/actions/checkout) | `3.7.0` | `7.0.1` |
| [actions/setup-python](https://github.com/actions/setup-python) | `5.6.0` | `7.0.0` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5.5.5` | `7.1.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |


Updates `actions/upload-artifact` from 4.6.2 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v4.6.2...043fb46)

Updates `actions/checkout` from 3.7.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v3.7.0...3d3c42e)

Updates `actions/setup-python` from 5.6.0 to 7.0.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@a26af69...5fda3b9)

Updates `codecov/codecov-action` from 5.5.5 to 7.1.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@0fb7174...303a32d)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@37930b1...3e5f45b)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/checkout
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/setup-python
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: codecov/codecov-action
  dependency-version: 7.1.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…hole DATE_ORDER (scrapinghub#868) (scrapinghub#1443)

* Add a STRICT_DATE_ORDER setting so that an invalid day is not read as the year (scrapinghub#868)

DATE_ORDER is a preference: any number that is not a valid day or month
is read as a two-digit year, so "32 DEC 10" with DATE_ORDER="DMY" is
2032-12-10. With the new setting, which defaults to False, a two-digit
year is read only where DATE_ORDER puts the year, and such a date string
gives None instead. The rejection is final: no other locale may guess
another reading, and the no-spaces parser, which would read the same
numbers in a date order again, is skipped. The other parsers of the
locale still get their turn.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Make STRICT_DATE_ORDER a string that can also enforce the day and month

The setting is now "none" (the default), "year" or "all", so that it can
name all three behaviours: DATE_ORDER as a preference, the position of a
two-digit year enforced, or the whole order enforced. With "all", a day
and a month that do not fit are no longer swapped, neither in the token
loop nor through the retry with the swapped order: "13/12/10" with
DATE_ORDER="MDY" is None. A four-digit year is still read wherever it is,
and a date that starts with one is still read as year, month and day.

With "all", a date string that cannot be read in DATE_ORDER at all is
final too, so that no other locale reads it differently: with
auto-detected languages, "10年20月00日" with DMY would otherwise come
back from zh as a relative date, 10 years and 20 months ago.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Make STRICT_DATE_ORDER="year" reject a misplaced year in both directions

A two-digit year is now read only where DATE_ORDER puts the year: the day and
month read before it must be the ones the order puts before it. Until now only
a day or month that the order puts before the year, but is read after it, was
rejected, so "32 DEC 10" with DMY was None but "10 DEC 32" with MYD was still
2032-12-10, and "year" did nothing at all for YMD and YDM, where no part comes
before the year.

This also rejects scrapinghub#519's last-resort "4-99" with YMD, which is acceptable for an
opt-in setting. DMY and MDY are not affected, since nothing comes after the
year in them.

The day and month swap is now also tested under DYM and MYD, where the year is
between them and the swap changes which of them is read before it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Raise SettingValidationError when STRICT_DATE_ORDER has no DATE_ORDER to enforce

"year" and "all" only enforce a DATE_ORDER that the caller sets, since the
default one is a preference, so without one they silently did nothing. Raise a
SettingValidationError from check_settings instead, which only looks at the
settings the caller passed, so the rebuilt settings of search_dates, which
forget those, cannot trigger it. "none" does not need a DATE_ORDER.

The cases that pinned "no effect without DATE_ORDER" are replaced by tests of
the error, of "none", and of search_dates and a reused parser, which keep working.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Shorten the STRICT_DATE_ORDER docs to the year rule and the two caveats

"year" now reads as one rule: a two-digit year is read only from the position
DATE_ORDER gives the year. Of the details after the examples, only two matter
to users, that "year" and "all" need DATE_ORDER, and that '95年12月10日' with
MDY is rejected. The rest, such as that a rejection is final across locales,
which parsers still run, and that a month name or a missing part is not
checked, is in the code comments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Drop the right day when a month name follows a day range

05f2737 records in auto_order that the end of a day range is dropped when a
month name follows it, but it always deleted the entry after the month. In a
day-first order, as in "12-12 June 2021" with DMY or "5-5 juin 2021" in French,
the day comes before the month, so the deletion raised IndexError, parse_alpha
swallowed it and the month name was not read: the date was None even without
STRICT_DATE_ORDER, unlike master. Remove the day entry itself instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test that a two-digit year is still the last resort under STRICT_DATE_ORDER

Moving "4-99" with YMD and "10 DEC 32" with MYD to the rejected cases left no
test of scrapinghub#519's two-digit year fallback under STRICT_DATE_ORDER, and left its
comment on "10年20月00日", which goes through the day and month swap instead.
Also say what the search_dates test checks: that it does not raise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say in the STRICT_DATE_ORDER docs which parts are checked

Without it, "read a two-digit year only from the position DATE_ORDER gives the
year" suggests that "32 DEC" with DMY is rejected too, but a month name and a
missing part are not checked, and a four-digit year is read wherever it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Drop issue numbers from two comments and fix the STRICT_DATE_ORDER examples

Quote the values of the setting in the docs, since they are strings, and drop
the None lines from its examples, since the REPL prints nothing for None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ghub#351)

Add simplifications so a bare unit after the "ago" word or before the "ago"
suffix means "1 unit": Arabic, Belarusian, Bulgarian, Croatian, Finnish,
Hebrew, Latvian, Lithuanian, Macedonian, Polish, Slovenian and Ukrainian.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@yamilmaud yamilmaud changed the title Parse Czech relative dates without the numeral 1, like "před týden" Parse relative dates without the numeral 1, like "před týden" (#351) Oct 6, 2026
@yamilmaud

Copy link
Copy Markdown
Owner Author

Superseded by scrapinghub#1540 (this one was opened against the fork's stale master).

@yamilmaud yamilmaud closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.