From f2a60cbd27b58ee864bc84fb9df779d5a77a686a Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Thu, 10 Sep 2026 13:23:46 +0200 Subject: [PATCH 01/18] fix: correcting compatibility claims, sharpening inline doc I'm currently working on a releas of the caldav-server-tester project, and I've found some bugs and mistakes, behaviour has been altered a bit in that project. This again caused the compatibility test to break for multiple servers. It also caused me to flip back on some `fragile`-observations. Some servers do updates asyncronously. A PUT/DELETE/MKCALENDAR etc is sent to the calendar server, and we get "2xx OK" in return - but the change has been queued up rather than processed. Sometimes it's processed within a fraction of a second, sometimes it can take minutes. More often than not this doesn't matter at all in real life scenarioes - but it does matter a lot for the caldav-server-tester as well as for test code in this project! It's possible to configure delays, but it's not possible to probe directly what the delay should be set to. We already did some polling for delays on calendar creation and calendar deletion, now it's also doing polling for delays on event creation. Earlier `fragile` was used to mark delays in calendar creation and deletion - now `quirk` is used. One thing here, it's not possible to probe the difference between a very fast asynchronous server and a server that returns "200 OK" only after all databases are in sync. None of the docker-based test servers showed up as asynchronous in the first few test rounds. I flipped them from `quirk` to `full` _as for now_. Cyrus later had to be flipped back to `quirk`. Sometimes a server returns a 4xx or a 5xx error due to a feature not supported. This may arguably be better than silently dropping data - but still, the support level is to be considered "ungraceful". This was not quite consistently probed in the caldav-server-tester. It's now more consistently returning "ungraceful" on such return values, and the feature support matrix needed some upgrades. There are some definitions of the various support levels in comments in the compatibility_hints.py - those should eventually be moved out to proper documentation - but as for now I've gone through and improved the existing documentation. Many of the changes have been done manually, but I've also had Claude doing things for me, so ref the AI guidelines the prompts should be included: Prompt: (hand-edited changes in the file) Followup-Prompt: (comment from the review process, pasted test output abridged) Test results are in. The posteo stable-url thing [pre-existing 'unsupported' in the configuration, apparently put there by me without AI-assistance and without any documentation] seems to be hallucinated. Prompt: (in the caldav-server-tester project) It's not possible to separate async-but-very-fast from synchronous - it could be that things will break later, but still - please move support level to "full" for cyrus, nextcloud and zimbra - and then I can move it back to "quirk" first time we observe async behaviour. Followup-Prompt: (comment from the review process) Is this right? In the async quirk case, the calendar only needs to be deleted once. Now, with the async case, maybe a client that immediately would look for the calendar will see that the delete-operation failed ... but this is a corner case that probably only exists in the caldav-feature-checker! Yank out this commit? [about a commit that made 'quirk' enter Calendar.delete()'s retry loop; it was yanked] Followup-Prompt: (test output pasted, abridged) This is interesting: (server-tester run against Cyrus: testCheckCompatibility failed, and the write-delay probe reported behaviour 'delete-calendar takes ~1s') Followup-Prompt: [it should be] `quirk`, not `fragile`. The delete always goes through, it's just delayed, while `fragile` could cause the delete-operation to be retried Prompt: (None - changes hand-edited by tobixen) Followup-Prompt: (comment from the review process) I've tried to rewrite [the inline documentation in the comments about support levels] myself, do a quick QA on it and fix up the commit with my changes Followup-Prompt: (While going through AI-generated code review comments, user agreed to let Claude Opus include a definition for "unknown" to the list) Prompt: (test output pasted, abridged) And OX also is unhappy: (server-tester run, reproduced from both clones: save-load.event.recurrences.exception.reschedule expected 'unsupported', observed 'ungraceful' with "PutError at '409 Conflict'") Prompt: Now there are four commits touching one file and some docstrings in test code - it would make sense to squash it into one, wouldn't it? Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/compatibility_hints.py | 74 +++++++++++++++++++++++++++-------- tests/fixture_helpers.py | 13 +++--- tests/test_fixture_helpers.py | 17 ++++---- 3 files changed, 76 insertions(+), 28 deletions(-) diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 7f420632..b076a081 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -47,9 +47,16 @@ class FeatureSet: TODO: use enums? TODO: describe the different types TODO: think more through the different types, consolidate? type -> "client-feature", "client-hints", "server-peculiarity", "tests-behaviour", "server-observation", "server-feature" (last is default) - support -> "full" (default), "unsupported", "fragile", "quirk", "broken", "ungraceful" + support -> "full" (default), "unsupported", "fragile", "quirk", "broken", "ungraceful", "unknown" - unsupported means that attempts to use the feature will be silently ignored (this may actually be the worst option, as it may cause data loss). quirk means that the feature is suppored, but special handling needs to be done towards the server. fragile means that it sometimes works and sometimes not - either it's arbitrary, or we didn't spend enough time doing research into the patterns. My idea behind broken was that the server should do completely unexpected things. Probably a lot of things classified as "unsupported" today should rather be classified as "broken". Some AI-generated code is using"broken". TODO: look through and clean up. "ungraceful" means the server will throw some error (this may indeed be the most graceful, as the client may catch the error and handle it in the best possible way). + unsupported means that attempts to use the feature will be silently ignored (this may actually be the worst option, as it may cause data loss). + quirk means that the feature is supported, but not entirely as expected, and special handling may need to be done towards the server. + fragile means that it sometimes works and sometimes not - possibly it's non-deterministic, more likely we need better probes. + broken means the server does unexpected things - apparently supporting the feature, but in reality doing things wrongly. Possibly some of the things classified as "unsupported" today should rather be classified as "broken" (and possibly vice-versa). TODO: look more into this and clean up. + ungraceful means the server will come up with an error (which usually causes the library to raise an error). ("ungraceful" may in some cases be the best handling as the client may catch the error and handle it in the best possible way - while support level "unsupported", "broken" and "fragile" often may involve data loss). + unknown means nobody has probed this yet. It is the absence of a claim, not a claim that the feature is missing. + + For a server-feature, is_supported(feature) returning a bool is True for "full" and "quirk" only. "fragile" is True as well when called with accept_fragile=True; "unsupported", "broken", "ungraceful" and "unknown" are all False. Note in particular that "ungraceful" is False even though the server does respond - the response is an error. types: * client-feature means the client is supposed to do special things (like, rate-limiting). While the need for rate-limiting may be set by the server, it may not be possible to reliably establish it by probling the server, and the value may differ for different clients. @@ -241,7 +248,7 @@ class FeatureSet: "description": "Server honours the supported-calendar-component-set restriction set at MKCALENDAR time. When 'full', the server both advertises (or enforces) the restriction; when 'unsupported', the restriction is silently ignored (wrong-type objects can be saved to the calendar). When 'ungraceful', the MKCALENDAR request itself fails when a component set is specified.", }, "calendar-color": { - "description": "Server stores the nonstandard Apple/Mozilla {http://apple.com/ns/ical/}calendar-color property (set with a colour name like 'blue') on a calendar collection. 'full' covers servers that normalise the name to a hex value (the set value still tracks the input); 'broken' is a read-only property (the same value comes back regardless of what is set). Not described by RFC4791/RFC5545, so a server that rejects or ignores it ('unsupported') is not breaching any RFC. The default is 'fragile' because the behaviour varies a lot between servers and is rarely worth asserting on.", + "description": "Server stores the nonstandard Apple/Mozilla {http://apple.com/ns/ical/}calendar-color property (set with a colour name like 'blue') on a calendar collection. 'full' covers servers that normalise the name to a hex value (the set value still tracks the input); 'broken' is a read-only property (the same value comes back regardless of what is set). Not described by RFC4791/RFC5545, so a server that rejects or ignores it ('unsupported') is not breaching any RFC. The default is 'fragile' because the behaviour varies a lot between servers and is rarely worth asserting on (TODO: wouldn't unknown be better?).", "default": {"support": "fragile"}, "note": """The real default ought to be False because this is not a part @@ -330,7 +337,7 @@ class FeatureSet: "links": ["https://datatracker.ietf.org/doc/html/rfc4918#section-15.2"], }, "delete-calendar": { - "description": "RFC4791 says nothing about deletion of calendars, so the server implementation is free to choose weather this should be supported or not. Section 3.2.3.2 in RFC 6638 says that if a calendar is deleted, all the calendarobjectresources on the calendar should also be deleted - but it's a bit unclear if this only applies to scheduling objects or not. Some calendar servers moves the object to a trashcan rather than deleting it", + "description": "RFC4791 says nothing about deletion of calendars, so the server implementation is free to choose weather this should be supported or not. Section 3.2.3.2 in RFC 6638 says that if a calendar is deleted, all the calendarobjectresources on the calendar should also be deleted - but it's a bit unclear if this only applies to scheduling objects or not. Some calendar servers moves the object to a trashcan rather than deleting it. 'quirk' is the right grade for a delete that always goes through but takes a measurable time; 'fragile' is a negative status and additionally switches on Calendar.delete()'s retry-and-poll loop, which re-issues the DELETE", ## Independent feature (directly probed): the default marks it so the ## node uses its own probed value rather than being derived from ## .free-namespace. @@ -364,7 +371,7 @@ class FeatureSet: ## information was simply discarded, and the current search behaviour would in ## such a case be incorrect if the exception is simply discarded. "save-load.event.recurrences.exception": {"description": "When a VCALENDAR containing a master VEVENT (with RRULE) and exception VEVENT(s) (with RECURRENCE-ID) is stored, the server keeps them together as a single calendar object resource. When unsupported, the server splits exception VEVENTs into separate calendar objects, making client-side expansion unreliable (the master expands without knowing about its exceptions)."}, - "save-load.event.recurrences.exception.reschedule": {"description": "The server accepts a PUT that reschedules an entire recurring event - changing the master VEVENT's DTSTART (re-anchoring the whole series) while detached exception VEVENT(s) (with RECURRENCE-ID) are present and their RECURRENCE-IDs are shifted to line up with the new series. This is unsupported for Ox, the server rejects such a PUT with 409 Conflict even when a matching If-Match etag is supplied. Rescheduling a recurring event that has no exceptions still works. Exercised by save(all_recurrences=True) after changing dtstart/dtend.", "default": {"support": "full"}}, + "save-load.event.recurrences.exception.reschedule": {"description": "The server accepts a PUT that reschedules an entire recurring event - changing the master VEVENT's DTSTART (re-anchoring the whole series) while detached exception VEVENT(s) (with RECURRENCE-ID) are present and their RECURRENCE-IDs are shifted to line up with the new series. This is 'ungraceful' for Ox, the server rejects such a PUT with 409 Conflict even when a matching If-Match etag is supplied. Rescheduling a recurring event that has no exceptions still works. Exercised by save(all_recurrences=True) after changing dtstart/dtend.", "default": {"support": "full"}}, "save-load.todo": { "description": "it's possible to save and load tasks to the calendar", "default": { "support": "full" } @@ -1326,9 +1333,12 @@ def compare(self, observed): 'search.comp-type.optional': {'support': 'full'}, 'search.recurrences.expanded.todo': {'support': 'unsupported'}, "search.recurrences.includes-implicit.infinite-scope": False, - 'delete-calendar': { - 'support': 'fragile', - 'behaviour': 'Deleting a recently created calendar fails'}, + ## Re-verified 2026-09-08 against the docker test server: creating and + ## deleting a calendar works, immediately and without an error, and the + ## former 'fragile' verdict ('Deleting a recently created calendar fails') + ## could not be reproduced. No delay observed either, unlike Cyrus, so + ## 'full' rather than the 'quirk' recorded there. + 'delete-calendar': {'support': 'full'}, 'delete-calendar.free-namespace': { ## TODO: not caught by server-tester 'behaviour': "deleting a calendar moves it to a trashbin, thrashbin has to be manually 'emptied' from the web-ui before the namespace is freed up", 'support': 'fragile', @@ -1374,7 +1384,17 @@ def compare(self, observed): ## Genuinely returns matching objects for a comp-type-less query that carries ## a time-range (verified: the event is returned, not just "no error"). 'search.time-range.comp-type-optional': {'support': 'full'}, - 'delete-calendar': {'support': 'fragile', 'behaviour': 'may move to trashbin instead of deleting immediately'}, + ## Re-verified 2026-09-08 against the docker test server: the calendar is + ## deleted immediately and the id is free for re-use afterwards; the former + ## 'may move to trashbin instead of deleting immediately' could not be + ## reproduced. Unlike Cyrus, no delay has been observed here yet: 'full' + ## rather than 'quirk', since a delay too small to observe cannot be told + ## from none at all - an actual observation is what should put a 'quirk' + ## here, as it did for Cyrus. + 'delete-calendar': {'support': 'full'}, + ## The re-use half of the same observation, recorded rather than left to the + ## implicit default. + 'delete-calendar.free-namespace': {'support': 'full'}, ## This is a zimbra bug when creating calendars with a display ## name. Now mitigated in the calendar creation code. #'save-load.get-by-url': {'support': 'fragile', 'behaviour': '404 most of the time - but sometimes 200. Weird, should be investigated more'}, @@ -1554,9 +1574,24 @@ def compare(self, observed): "save.duplicate-uid.cross-calendar": {"support": "ungraceful"}, # Ephemeral Docker container: wipe objects but keep calendar (avoids UID conflicts) "test-calendar": {"cleanup-regime": "wipe-calendar"}, + ## Re-probed against the docker test server. The former 'fragile' verdict + ## ('Deleting a recently created calendar fails') could not be reproduced - + ## the DELETE is accepted without an error. It is not synchronous, though: + ## a run on 2026-09-09 measured ~1s before the calendar stopped answering, + ## which is why this is 'quirk' and not 'full'. 'quirk' and not 'fragile' + ## either - the delete deterministically goes through and only the wait + ## varies, whereas 'fragile' is a negative status and would make + ## is_supported('delete-calendar') False, silently skipping the + ## free-namespace probe below. 'delete-calendar': { - 'support': 'fragile', - 'behaviour': 'Deleting a recently created calendar fails'}, + 'support': 'quirk', + 'behaviour': 'delayed deletion - the calendar stays queryable for ~1s', + 'delay': 1, + }, + ## Pinned rather than inherited: the parent is 'quirk' now, and the id is + ## observed to free up cleanly, so leaving this derived would declare a + ## delay on the re-use half that nobody measured. + 'delete-calendar.free-namespace': {'support': 'full'}, # Cyrus changes the Schedule-Tag even on attendee PARTSTAT-only updates, # violating RFC6638 section 3.2 which requires the tag to remain stable. "scheduling.schedule-tag.stable-partstat": {"support": "unsupported"}, @@ -1852,9 +1887,11 @@ def compare(self, observed): "search.time-range.alarm": {"support": "unsupported"}, ## Recurrence expansion actually works within the (near-future) search window; ## this was previously reported "unsupported" only because the test fixtures - ## lived in year 2000, which CCS's min-date-time restriction hid. Only infinite - ## scope (far-future) remains unsupported. - "search.recurrences.includes-implicit.infinite-scope": {"support": "unsupported"}, + ## lived in year 2000, which CCS's min-date-time restriction hid. Only the + ## far-future (infinite-scope) probe still fails, and CCS rejects it outright + ## with a 403 max-date-time - an error rather than a silent non-answer, so + ## "ungraceful", the same grading its old-dates entries already carry. + "search.recurrences.includes-implicit.infinite-scope": {"support": "ungraceful"}, ## search.recurrences.expanded.todo was 'unsupported'; 'full' observed ## 2026-08-26. The declaration dated from when the probe searched the ## *event* calendar for the recurring todo, so a server that keeps tasks @@ -2111,7 +2148,9 @@ def compare(self, observed): ## 409 Conflict once detached exceptions exist - even with a matching If-Match ## etag. Shifting the DTSTART of an exception-free recurring event still works. ## Confirmed by direct probe 2026-06-14. - 'save-load.event.recurrences.exception.reschedule': {'support': 'unsupported'}, + ## 409 Conflict on the PUT - an error the caller can catch, not a silent + ## non-answer, so 'ungraceful' rather than 'unsupported'. + 'save-load.event.recurrences.exception.reschedule': {'support': 'ungraceful', 'behaviour': "PutError at '409 Conflict'"}, ## OX ignores the time-range on VTODO queries and returns every task 'search.time-range.todo.strict': {'support': 'broken'}, ## OX silently ignores the is-not-defined prop-filter and returns the whole @@ -2208,7 +2247,10 @@ def compare(self, observed): ## be that the behaviour has changed at the server side. 418 was originally an ## April joke and may mean anything ... but it's sometimes used as a rate-limit ## response. However, it seems to consistently break exactly here. - 'sync-token': {'support': 'ungraceful', 'behaviour': "418 I'm a teapot"}, + ## The removed member comes back with status 418 inside the multistatus, which + ## caldav's _validate_status turns into a ResponseError - the sync raises rather + ## than silently coming back wrong, hence "ungraceful". + 'sync-token.delete': {'support': 'ungraceful', 'behaviour': "418 I'm a teapot"}, } # fmt: on diff --git a/tests/fixture_helpers.py b/tests/fixture_helpers.py index 62825de3..ad7cd10d 100644 --- a/tests/fixture_helpers.py +++ b/tests/fixture_helpers.py @@ -273,11 +273,14 @@ async def afix_calendar( own calendar used to repeat inline: * a leftover calendar from an interrupted run is deleted first - but only - on servers where deleting actually frees the URL. On servers where it - does not (``delete-calendar`` unsupported: Synology, Nextcloud), a - ``delete()`` is a no-op wipe, the MKCALENDAR that follows would 405 with - "a collection already exists at that location", and the correct move is - to reuse the calendar instead. + on servers where deleting actually frees the URL, which is what + ``delete-calendar.free-namespace`` records. Two different servers fail + that: Synology refuses the DELETE outright (``delete-calendar`` + unsupported), while Nextcloud accepts it but moves the calendar to a + trashbin, so the id stays taken. Either way a ``delete()`` is a no-op + wipe, the MKCALENDAR that follows would 405 with "a collection already + exists at that location", and the correct move is to reuse the calendar + instead. * the display name is dropped on servers that cannot set one, or that move the calendar to a server-chosen URL when one is set, and on component-restricted calendars - same three-legged rule as diff --git a/tests/test_fixture_helpers.py b/tests/test_fixture_helpers.py index d8937b68..3796b7ee 100644 --- a/tests/test_fixture_helpers.py +++ b/tests/test_fixture_helpers.py @@ -61,9 +61,10 @@ class FakePrincipal: """Principal that only lets a calendar be created once, like a real server. A second MKCALENDAR at the same cal_id fails with ``MkcalendarError``, which - is what a server whose calendars cannot be deleted (Synology, Nextcloud) - replies with on the second run of a test - 405 "a collection already exists - at that location". + is what a server that does not free the cal_id on delete (Synology, which + refuses the DELETE; Nextcloud, which trashbins the calendar) replies with on + the second run of a test - 405 "a collection already exists at that + location". """ def __init__(self, existing: dict[str, FakeCalendar] | None = None) -> None: @@ -130,10 +131,12 @@ async def test_afix_calendar_creates_and_names() -> None: async def test_afix_calendar_reuses_and_wipes_when_calendar_cannot_be_deleted() -> None: """A leftover calendar on a no-delete server is reused and emptied. - This is the Synology/Nextcloud (and jeanes) case: ``delete()`` degrades to a - no-op wipe, so the leftover calendar survives and the MKCALENDAR that - follows 405s. The helper must hand back that calendar, emptied, rather than - letting the MkcalendarError escape. + This is the Synology/Nextcloud (and jeanes) case - ``delete-calendar + .free-namespace`` false, whether because the DELETE is refused or because + the calendar only moves to a trashbin: ``delete()`` degrades to a no-op + wipe, so the leftover calendar survives and the MKCALENDAR that follows + 405s. The helper must hand back that calendar, emptied, rather than letting + the MkcalendarError escape. """ leftover = FakeCalendar(url="http://dav.example.com/testcal/", n_objects=3) client = FakeClient({"delete-calendar": False}) From 576d29fb3430043152e9d9ccc8355905fb72ea17 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Thu, 10 Sep 2026 14:49:01 +0200 Subject: [PATCH 02/18] fix: Cyrus delete-calendar back to 'fragile' 2d4d360b regraded it to 'quirk' on the tester's verdict, which made is_supported() true and so switched off Calendar.delete()'s retry-and-poll loop - the loop is keyed on 'fragile' alone. Cyrus needs it: a calendar re-created on a just-deleted cal_id answers 500 to DELETE for about a second (17 of 20 attempts on the docker server, each accepted a second later), while a fresh cal_id deletes cleanly. The cause is filed upstream as https://github.com/cyrusimap/cyrus-imapd/issues/6383: the DELETED.* mailbox name carries a whole-second timestamp, so two deletes of one name inside the same second collide. testCreateDeleteCalendar and test_principal_make_calendar hit it because they tear a calendar down and make it again. Prompt: (test output pasted, abridged) $ pytest --last-failed (TestForServerCyrus::testCreateDeleteCalendar and TestAsyncForCyrus::test_principal_make_calendar both DeleteError at '500 Internal Server Error'; ) Followup-Prompt: It sounds like "fragile" with that behaviour note is the correct thing to have in the compatibility matrix? Delete fails hard if the calendar was freshly created? Or perhaps the calendar creation should be tagged as quirky and asynchronous, and all test code depending on calendars created should sleep a second? Followup-Prompt: [do research on the cyrus issue] Followup-Prompt: The issue should be referenced either from compatibility_hints.py, the check code or both [the Cyrus bug filed as https://github.com/cyrusimap/cyrus-imapd/issues/6383 earlier in this session; done in both repos] Followup-Prompt: [hand edit: the AI-written note condensed into human text] Followup-Prompt: (comment from the review process) Can it be folded into 08e74967 ? Followup-Prompt: (comment from the review process) Can this be folded into another commit? We're on a feature branch, so force-push is allowed Followup-Prompt: (comment from the review process) Can this be folded into the earlier cyrus issues? AI-assisted: yes Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/compatibility_hints.py | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index b076a081..07c5547f 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -1574,23 +1574,21 @@ def compare(self, observed): "save.duplicate-uid.cross-calendar": {"support": "ungraceful"}, # Ephemeral Docker container: wipe objects but keep calendar (avoids UID conflicts) "test-calendar": {"cleanup-regime": "wipe-calendar"}, - ## Re-probed against the docker test server. The former 'fragile' verdict - ## ('Deleting a recently created calendar fails') could not be reproduced - - ## the DELETE is accepted without an error. It is not synchronous, though: - ## a run on 2026-09-09 measured ~1s before the calendar stopped answering, - ## which is why this is 'quirk' and not 'full'. 'quirk' and not 'fragile' - ## either - the delete deterministically goes through and only the wait - ## varies, whereas 'fragile' is a negative status and would make - ## is_supported('delete-calendar') False, silently skipping the - ## free-namespace probe below. + ## Calendar deletion has a very small fragility on Cyrus, one that + ## does not matter for ordinary users, but it matters when running + ## tests - if a calendar is deleted, recreated under the same URL + ## and then deleted again within a very short timeframe - then the + ## server gives 500 internal server error. Due to this it's + ## flagged as 'fragile'. Retry after one second (on the docker + ## test server on my laptop) and it works. + ## Reported upstream, with the root cause (the DELETED.* mailbox name + ## carries a whole-second timestamp, so two deletes of one name inside + ## the same second collide): https://github.com/cyrusimap/cyrus-imapd/issues/6383 'delete-calendar': { - 'support': 'quirk', - 'behaviour': 'delayed deletion - the calendar stays queryable for ~1s', + 'support': 'fragile', + 'behaviour': 'deleting a calendar re-created on a just-deleted cal_id answers 500 for ~1s before it succeeds', 'delay': 1, }, - ## Pinned rather than inherited: the parent is 'quirk' now, and the id is - ## observed to free up cleanly, so leaving this derived would declare a - ## delay on the re-use half that nobody measured. 'delete-calendar.free-namespace': {'support': 'full'}, # Cyrus changes the Schedule-Tag even on attendee PARTSTAT-only updates, # violating RFC6638 section 3.2 which requires the tag to remain stable. From 96426471c65ae66702d983313a599c31061589a2 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Thu, 10 Sep 2026 15:07:06 +0200 Subject: [PATCH 03/18] docs: spell out what 'fragile' asks of a client The support-level prose described "quirk" and "fragile" by how certain the behaviour is, which left the operational difference implicit. What "fragile" asks for now depends on what is fragile. A fragile write, such as delete-calendar, may not have gone through and may be re-issued, which is what Calendar.delete() implements. A fragile synchronous-write is a server that may be asynchronous but settles too fast to probe: writes are never re-issued, and a read right after one waits or retries. A write that always goes through but settles slowly is a "quirk". Prompt: `fragile` means a retry is needed, `quirk` means a retry is not needed. Keep it like that. Followup-Prompt: (comment from the review process) For me, this is crystal clear, let me try to rephrase it: One shouldn't retry an async operations (sic), one should wait it out. If "synchronous-write" is declared fragile, then it's the "synchronous"-part of it that is fragile, the "write" part of it is considered to be deterministic, so no retries on the write-ops are needed. A _read_-ops right after may need to be retried if it doesn't contain the changed/added item. "Fragile" means it may be asynchronous under the hood, but it may not be possible to probe this (deterministically) as the write-operations settle very fast. If calendar creation or calendar deletion is considered to be "fragile", then it's the write operation that is fragile - the creation/deletion may or may not have gone through, and if it didn't go through, then in some cases a retry may help. If everything except calendar creation and/or calendar deletion is observed to be async, then "synchronous-write" may be considered supported, while the async nature of calendar creation/deletion is to be considered a "quirk". For cyrus, it's well-known that the calendar delete is "fragile", it's just one oddball corner case where it may be needed to retry after 1s. [please] make sure this is crystal clear for anyone reading the doc. If the actual code deviates from my writing above, it needs to be fixed. AI-assisted: yes Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/compatibility_hints.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 07c5547f..dddde0e9 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -51,11 +51,14 @@ class FeatureSet: unsupported means that attempts to use the feature will be silently ignored (this may actually be the worst option, as it may cause data loss). quirk means that the feature is supported, but not entirely as expected, and special handling may need to be done towards the server. - fragile means that it sometimes works and sometimes not - possibly it's non-deterministic, more likely we need better probes. + fragile means that it sometimes works and sometimes not (or works in some cases and not in others) - possibly it's non-deterministic, more likely we need better probes. broken means the server does unexpected things - apparently supporting the feature, but in reality doing things wrongly. Possibly some of the things classified as "unsupported" today should rather be classified as "broken" (and possibly vice-versa). TODO: look more into this and clean up. ungraceful means the server will come up with an error (which usually causes the library to raise an error). ("ungraceful" may in some cases be the best handling as the client may catch the error and handle it in the best possible way - while support level "unsupported", "broken" and "fragile" often may involve data loss). unknown means nobody has probed this yet. It is the absence of a claim, not a claim that the feature is missing. + What "fragile" asks of a client depends on what is fragile. An asynchronous operation is never retried, it is waited out: + * On a write operation such as create-calendar or delete-calendar, the write itself is fragile: it may or may not have gone through, and re-issuing it may help. Calendar.delete() keys its retry-and-poll loop on exactly that. A create or delete that always goes through but takes a measurable time to settle is a "quirk", not "fragile". + * On synchronous-write, it is the "synchronous" part that is fragile, not the write: the server may be asynchronous under the hood but settle too fast to be probed deterministically. Writes are never re-issued; a read right after one may have to wait (the configured delay) or be retried if it does not show the change yet. A server where only calendar creation and/or deletion is asynchronous supports synchronous-write, with the async part graded as a "quirk" on create-calendar or delete-calendar. For a server-feature, is_supported(feature) returning a bool is True for "full" and "quirk" only. "fragile" is True as well when called with accept_fragile=True; "unsupported", "broken", "ungraceful" and "unknown" are all False. Note in particular that "ungraceful" is False even though the server does respond - the response is an error. types: From 700442e6b44a9719a8da21461250a3fe79c568d9 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Thu, 10 Sep 2026 16:28:10 +0200 Subject: [PATCH 04/18] fix: no random UID on the VCALENDAR wrapper A bare icalendar component assigned to a CalendarObjectResource gets wrapped in a VCALENDAR, and that wrapper was built with icalendar.Calendar.new(), which adds a random RFC 7986 UID. Stalwart takes the calendar-level UID to be the identity of the calendar object resource, so a second save of the same object arrived with a new UID and was rejected with 412 no-uid-conflict. The wrapper is now built explicitly, without a UID. The async test also PUT the same event twice to one URL, first from an icalendar.Calendar (carrying its own calendar-level UID) and then from a bare Event - a UID change on an existing resource, which a server may legitimately refuse. It is now parametrized like its sync counterpart. Prompt: It's still needed to figure out of this error (sic), it's reproducible and I've also tried to restart the epheremal stalward docker container (sic) to make sure no old state remains on the server: (pytest --last-failed --pdb output pasted, TestAsyncForStalwart::test_create_event_from_ical failing with PutError 412 no-uid-conflict) Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- CHANGELOG.md | 6 ++++++ caldav/calendarobjectresource.py | 15 +++++++------- tests/test_async_integration.py | 21 ++++++++++++-------- tests/test_caldav_unit.py | 34 ++++++++++++++++++++++++++++++++ 4 files changed, 61 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82deff3e..ce6d6ba0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,12 @@ Changelogs prior to v3.0 are pruned, but are available in the v3.1 release This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html), though for pre-releases PEP 440 takes precedence. +## [Unreleased] + +### Fixed + +* A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. + ## [3.3.0] - 2026-09-03 3.3.0 is mostly a maintenance and QA release. The major news here is that we've done an AI-based (Claude Fable) review of all the code, this has resulted in quite a lot of hammering on the code to get all the issues found smoothened out. diff --git a/caldav/calendarobjectresource.py b/caldav/calendarobjectresource.py index 8903bbdf..ddf1a5e3 100644 --- a/caldav/calendarobjectresource.py +++ b/caldav/calendarobjectresource.py @@ -1696,13 +1696,14 @@ def _set_icalendar_instance(self, inst): if not isinstance(inst, icalendar.Calendar): ## assume inst is an Event, Journal or Todo. ## TODO: perhaps a bit better sanity checking here? - try: ## DEPRECATION TODO: remove this try/except the future - ## icalendar 7.x behaviour (not released yet as of 2025-09 - cal = icalendar.Calendar.new() - except AttributeError: - cal = icalendar.Calendar() - cal.add("prodid", "-//python-caldav//caldav//en_DK") - cal.add("version", "2.0") + ## Deliberately not using icalendar.Calendar.new() here - it adds a + ## random RFC 7986 UID to the VCALENDAR wrapper, and some servers + ## (i.e. Stalwart) take that UID to be the identity of the calendar + ## object resource. A fresh random UID on every wrap then makes + ## the second save of the same object fail with 412 no-uid-conflict. + cal = icalendar.Calendar() + cal.add("prodid", "-//python-caldav//caldav//en_DK") + cal.add("version", "2.0") cal.add_component(inst) inst = cal self._icalendar_instance = inst diff --git a/tests/test_async_integration.py b/tests/test_async_integration.py index d07448bb..489c0544 100644 --- a/tests/test_async_integration.py +++ b/tests/test_async_integration.py @@ -2299,9 +2299,10 @@ async def test_create_calendar_and_event_from_vobject(self, async_calendar: Any) events = await c.get_events() assert len(events) == cnt + @pytest.mark.parametrize("klass", ["Calendar", "Event"]) @pytest.mark.asyncio - async def test_create_event_from_ical(self, async_calendar: Any) -> None: - """Add event from icalendar.Calendar and icalendar.Event objects.""" + async def test_create_event_from_ical(self, async_calendar: Any, klass: str) -> None: + """Add event from an icalendar.Calendar or an icalendar.Event object.""" self.skip_unless_support("save-load.event") c = async_calendar try: @@ -2319,12 +2320,16 @@ async def test_create_event_from_ical(self, async_calendar: Any) -> None: ) icalcal.add_component(icalevent) - for obj in [icalcal, icalevent]: - await c.add_event(obj) - events = await c.get_events() - assert any(e.icalendar_component["uid"] == "ctuid1" for e in events), ( - f"Event with uid ctuid1 not found after adding {type(obj).__name__}" - ) + ## Both the Calendar object and the Event object should be accepted. + ## They are tested one at a time, on a fresh calendar - putting both to + ## the same URL would change the VCALENDAR-level UID of an existing + ## calendar object resource, which some servers refuse (i.e. Stalwart). + obj = {"Calendar": icalcal, "Event": icalevent}[klass] + await c.add_event(obj) + events = await c.get_events() + assert any(e.icalendar_component["uid"] == "ctuid1" for e in events), ( + f"Event with uid ctuid1 not found after adding {klass}" + ) @pytest.mark.asyncio async def test_set_due(self, async_task_list: Any) -> None: diff --git a/tests/test_caldav_unit.py b/tests/test_caldav_unit.py index 5e955974..c552898d 100755 --- a/tests/test_caldav_unit.py +++ b/tests/test_caldav_unit.py @@ -4282,3 +4282,37 @@ def test_async_ambiguous_name_keeps_requested_url(self) -> None: ) asyncio.run(calendar._async_adopt_canonical_url("My Calendar")) assert str(calendar.url) == self.REQUESTED + + +class TestWrappedComponentHasNoCalendarUid: + """A bare icalendar component handed to caldav is wrapped in a VCALENDAR. + That wrapper must not carry an RFC 7986 UID of its own: servers that treat + the calendar-level UID as the identity of the calendar object resource + (Stalwart does) will see a randomly generated new UID on every save and + reject the PUT with 412 no-uid-conflict.""" + + def test_setting_a_bare_component_adds_no_calendar_uid(self) -> None: + ievent = icalendar.Event() + ievent.add("uid", "ctuid1") + ievent.add("dtstart", datetime(2026, 10, 10, 15, 15)) + + event = Event() + event.icalendar_instance = ievent + + assert "UID" not in event.icalendar_instance + assert event.icalendar_component["UID"] == "ctuid1" + + def test_two_wraps_yield_identical_data(self) -> None: + """Saving the same component twice must produce byte-identical data -- + a fresh random calendar-level UID per wrap is what breaks the second + PUT.""" + ievent = icalendar.Event() + ievent.add("uid", "ctuid1") + ievent.add("dtstart", datetime(2026, 10, 10, 15, 15)) + + first = Event() + first.icalendar_instance = ievent + second = Event() + second.icalendar_instance = ievent + + assert first.data == second.data From a5cea87d4a96f05a6aa9f8b3bf229ba232fbe9a7 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Thu, 10 Sep 2026 22:44:51 +0200 Subject: [PATCH 05/18] feat!: support Bedework 5 Every Bedework measurement we had came from a 2018 quickstart-3.10.3 image, while upstream is alive and at 5.0. That profile is now `bedework_3_10_3`, a breaking rename: `features: bedework` raises a ValueError naming both profiles. The `bedework` test server is now a locally built 5.0.0 image, installed from upstream's galleon feature pack with three upstream startup bugs patched at build time; the 2018 image moved to bedework3. The new `bedework_5_0_0` profile comes from several caldav-server-tester runs: writes are asynchronous, a client cannot create a collection that holds tasks (analysis upstream in https://github.com/Bedework/bedework/issues/5), text search matches nothing, an expanded search repeats the href per instance, and a VEVENT without SUMMARY is refused. Task and journal storage are graded unknown, and tests skip where no task calendar can be had. Prompt: The bedework docker image we're using is 8 years old - but bedework seems to be a an active and alive project. (sic) Followup-Prompt: A cheap "fix" would be to redefine "bedework" to "bedework_3_10". Then at least the compatibility_hints is not giving misinformation. Followup-Prompt: ok [approving the plan: version-stamp to the real 3.10.3, add a rename map rather than a bare rename, and document the provenance] Followup-Prompt: like with ox, it seems like we need to build our own docker-image for newer versions of bedework. Try to build a container based on the main branch. The old bedework docker image should be remaned (sic) bedework3. [approved to go for the released 5.0.0 feature pack rather than "main branch"] Followup-Prompt: New bedework5 container should be running and working now. Please probe it and populate the new bedework profile in ~/caldav/compatibility_hints.py Followup-Prompt: (comment from the review process) s/four/several/ and s/two/several/ and all is good. [on the message saying four runs and the profile comment saying two] Followup-Prompt: `pytest --last-failed` is giving lots of bedework errors. Please investigate. Followup-Prompt: Either todos work or they don't. If the server does not accept tasks, then the caldav-server-tester should report on that. If the server does accept task (sic), the tests should pass. If the truth is somewhere in between, we need to work out something. Followup-Prompt: fix the profile and add the missing skips. Make sure caldav-server-tester reports tasks as not supported for bedework. Make sure all task-related test code skips for bedework. Followup-Prompt: We're working on getting tests passing on bedework. Followup-Prompt: 1) Apparently all other servers supports (sic) tasks - though many of them require tasks to be on a separate task-list. Bedework is the only server which is declared not to support `save-load.todo`. Tests that puts (sic) tasks on a calendar should bail out when `save-load.todo` is not supported - but apparently there are some such tests that do not bail out. Please investigate. Tests are slow towards bedework, but pytest keeps some database over failing tests, dosn't (sic) it? Look into it. Followup-Prompt: 2) comments in compatibility_hints.py writes (sic) something about a "Qproperty". What is a Qproperty? Followup-Prompt: 3) Actually save-load.todo should probably be unknown and not unsupported for bedework5 - it's the combo of unsupported `create-calendar.with-supported-component-types` and `save-load.todo.mixed-calendar` that should disable tests. While being at it, look into caldav-server-tester and consider if it should report unknown rather than unsupported for save-load.todo for bedework5. Followup-Prompt: I've modified the comments in compatibility_hints.py - all those details [that was removed] seem to belong to an issue report to the bedework project rather than in the compatibility_hints.py. Please check if it's possible to file an issue, as well as if there already exists an issue on this Followup-Prompt: Please follow up on issue #5 [https://github.com/Bedework/bedework/issues/5] with details on the failure. Followup-Prompt: post [the drafted comment] Followup-Prompt: yes, fix the typo, add the link and commit. (...) Followup-Prompt: add the probe to the server tester. support level "quirk" is probably OK, and a behaviour note that it's not following the RFC. Followup-Prompt: (pytest -k bede output pasted: 2 failed, testChangeAttendeeStatusWithEmailGiven sync and async, PutError 500 missingeventproperty summary on Bedework) Followup-Prompt: Feature + probe + test fix (Recommended) [answering: a save-load.event.no-summary feature, a tester probe, and a SUMMARY in the two test events] Followup-Prompt: (comment from the review process) I consider "support bedework 5" as one feature. Most of the 29 commits below is (sic) related to the task "support bedework 5". Please squash some commits. Followup-Prompt: Feature + separate fixes (Recommended) [answering how to squash: the Bedework commits into this one, the library fixes kept as commits of their own] AI-assisted: Claude Opus 5 via Claude Code Co-Authored-By: Claude Opus 5 --- .github/workflows/tests.yaml | 6 +- .lycheeignore | 2 + CHANGELOG.md | 9 ++ caldav/compatibility_hints.py | 147 +++++++++++++++++- caldav/config.py | 47 ++++-- tests/README.md | 3 +- tests/caldav_test_servers.yaml.example | 21 ++- tests/docker-test-servers/bedework/Dockerfile | 76 +++++++++ tests/docker-test-servers/bedework/README.md | 115 ++++++++++++-- .../bedework/apacheds-setenv.sh | 13 ++ tests/docker-test-servers/bedework/build.sh | 18 +++ .../bedework/docker-compose.yml | 29 +++- .../bedework/migrate-h2.sh | 42 +++++ .../bedework/patch-modules.sh | 31 ++++ tests/docker-test-servers/bedework/run.sh | 43 +++++ tests/docker-test-servers/bedework/start.sh | 63 +++++--- tests/docker-test-servers/bedework/stop.sh | 4 +- tests/docker-test-servers/bedework3/README.md | 38 +++++ .../bedework3/docker-compose.yml | 14 ++ tests/docker-test-servers/bedework3/start.sh | 36 +++++ tests/docker-test-servers/bedework3/stop.sh | 12 ++ tests/fixture_helpers.py | 23 +++ tests/test_async_integration.py | 22 ++- tests/test_caldav.py | 12 +- tests/test_compatibility_hints.py | 45 ++++++ tests/test_fixture_helpers.py | 34 ++++ tests/test_servers/docker.py | 62 +++++++- tests/test_servers/registry.py | 3 +- tests/tools/convert_conf_private.py | 29 ++-- tox.ini | 6 +- 30 files changed, 915 insertions(+), 90 deletions(-) create mode 100644 tests/docker-test-servers/bedework/Dockerfile create mode 100644 tests/docker-test-servers/bedework/apacheds-setenv.sh create mode 100755 tests/docker-test-servers/bedework/build.sh create mode 100755 tests/docker-test-servers/bedework/migrate-h2.sh create mode 100755 tests/docker-test-servers/bedework/patch-modules.sh create mode 100755 tests/docker-test-servers/bedework/run.sh create mode 100644 tests/docker-test-servers/bedework3/README.md create mode 100644 tests/docker-test-servers/bedework3/docker-compose.yml create mode 100755 tests/docker-test-servers/bedework3/start.sh create mode 100755 tests/docker-test-servers/bedework3/stop.sh diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 22c9d29d..4cd44260 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -79,7 +79,7 @@ jobs: sogo_pass: testpass sogo_name: Test User sogo_fqhn: example.com - bedework: + bedework3: image: ioggstream/bedework:latest ports: - 8804:8080 @@ -301,7 +301,7 @@ jobs: echo "✗ Error: SOGo CalDAV access failed" exit 1 fi - - name: Configure Bedework + - name: Configure Bedework 3.10.3 run: | echo "Waiting for Bedework..." # Bedework/JBoss takes longer to start up @@ -330,7 +330,7 @@ jobs: BAIKAL_URL: http://localhost:8800 CYRUS_URL: http://localhost:8802 SOGO_URL: http://localhost:8803 - BEDEWORK_URL: http://localhost:8804 + BEDEWORK3_URL: http://localhost:8804 docs: runs-on: ubuntu-latest steps: diff --git a/.lycheeignore b/.lycheeignore index 1f5fe3f3..7dd8e43a 100644 --- a/.lycheeignore +++ b/.lycheeignore @@ -48,6 +48,8 @@ http://oxpedia\.org/.* http://httpd\.apache\.org/.* # GitHub URL template in sphinx conf (contains encoded braces, always 404) https://github\.com/python-caldav/caldav/blob/master/%7B.* +# Shell/Dockerfile URL templates with an unexpanded ${VAR} (encoded as $%7B) +.*\$%7B.* # Other junk that was never meant to be followed file://.*/scheme:.* diff --git a/CHANGELOG.md b/CHANGELOG.md index ce6d6ba0..5b98f47b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,15 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 ## [Unreleased] +### Changed + +* `compatibility_hints`: the `bedework` profile is renamed `bedework_3_10_3`. It was only ever measured against `ioggstream/bedework:latest`, a `quickstart-3.10.3` tree built in 2018 that can no longer even be rebuilt, while upstream Bedework is alive and released 5.0.0 in 2025 - so a profile called `bedework` was claiming far more than we have observed. `features="bedework"` now raises a `ValueError` naming the new profile rather than an `AttributeError`. (`compatibility_hints` is declared unstable for the 3.x series.) + +### Added + +* A test server for a current Bedework. `tests/docker-test-servers/bedework/` builds an image from the upstream Wildfly galleon feature pack (5.0.0), since there is no public image newer than 2018 and no single repository to build from. Three upstream bugs have to be patched at build time before the feature pack comes up at all - the pre-seeded H2 databases predate the H2 driver installed beside them, a shared module cannot load a class it needs through the war serving the request, and the bundled ApacheDS does not run on the JDK upstream prescribes. The 2018 image moves to `bedework3`. +* `compatibility_hints.bedework_5_0_0`: a profile for Bedework 5.0.0, measured 2026-09-12 with caldav-server-tester against the locally built test image as the demo user `vbede`. Nothing is inherited from `bedework_3_10_3` — 5.x creates and deletes calendars, and its text search, sync-token and principal search all behave differently. Two findings shape the rest of the profile. **Writes are asynchronous**: a read-back issued immediately after a PUT may 404 or hand back the pre-write copy, and that alone made `save-load.mutable`, `save-load.event.timezone` and `search.time-range.comp-type-optional` come out differently in two consecutive runs — the profile carries `write-delay: 3s`, without which a run measures the race rather than the server. And **a client cannot create a collection that holds tasks**: `MKCALENDAR`, extended `MKCOL` and `PROPPATCH` all answer `200 ok` for `CALDAV:supported-calendar-component-set` and then ignore it, so every collection a client creates is VEVENT-only and a VTODO or VJOURNAL PUT into one is 403. `vbede` has no usable `tasks` collection to fall back on either — the Depth:1 PROPFIND of its calendar home lists `tasks`, `Notifications` and `.pendingInbox` with a `getlastmodified` of "now" that is renewed on every listing, and all three 404 on any direct request — so the profile could not measure task support: `save-load.todo` and `search.time-range.todo` are graded `unknown`, and what was measured - an ignored component set and no tasks in an event calendar - is recorded as such. `douglm`, whose demo data ships a real `tasks` collection, can store tasks in it. The `BedeworkTestServer` in the test-server registry now uses the profile. + ### Fixed * A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index dddde0e9..a9076e58 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -307,6 +307,9 @@ class FeatureSet: "https://datatracker.ietf.org/doc/html/rfc4791#section-5.3.1", "https://datatracker.ietf.org/doc/html/rfc5689", ], + "extra_keys": { + "behaviour": "'mkcol-required' when MKCALENDAR is refused and the RFC5689 extended MKCOL has to be used instead - the library selects MKCOL for exactly this value. 'empty-207' when the server answers a successful creation with a multistatus whose DAV:response carries neither a DAV:status nor a DAV:propstat, in violation of RFC4918 section 13 (Bedework 5); purely descriptive, the library copes with it either way. 'delayed creation ...' when MKCALENDAR is accepted but the collection materialises later, with the wait in 'delay'.", + }, }, "create-calendar.auto": { "default": { "support": "unsupported" }, @@ -362,6 +365,7 @@ class FeatureSet: "description": "it's possible to save and load events to the calendar", "default": { "support": "full" } }, + "save-load.event.no-summary": {"description": "The server stores a VEVENT without a SUMMARY property. RFC 5545 section 3.6.1 makes SUMMARY optional. 'ungraceful' when the PUT is refused with an error (Bedework 5 answers 500 missingeventproperty).", "default": {"support": "full"}}, "save-load.event.recurrences": {"description": "it's possible to save and load recurring events to the calendar - events with an RRULE property set, including recurrence sets", "default": {"support": "full"}}, "save-load.event.recurrences.count": {"description": "The server will receive and store a recurring event with a count set in the RRULE", "default": {"support": "full"}}, ## This was Claude's suggestion and it works as of today, the @@ -624,7 +628,7 @@ class FeatureSet: "description": "expanding tasks" }, "search.recurrences.expanded.event": { - "description": "exanding events" + "description": "exanding events. 'quirk' with a behaviour starting 'response-per-instance' when the server returns each expanded instance in a DAV:response of its own, all under the href of the resource, in violation of RFC4918 section 14.24 (Bedework 5); the library merges them into one calendar-data" }, "search.recurrences.expanded.exception": { "description": "Server expand should work correctly also if a recurrence set with exceptions is given" @@ -1489,7 +1493,14 @@ def compare(self, observed): "calendar-order": {"support": "full"}, } -bedework = { +## Measured against the `ioggstream/bedework:latest` docker image, which is +## a quickstart-3.10.3 tree on openjdk-8 and was built 2018-11-05 - the image +## cannot even be rebuilt, as the quickstart zip its Dockerfile fetches from +## dev.bedework.org is gone. Upstream Bedework is alive and well past this: +## 5.0.0 was released 2025-09-04. Nothing here has been checked against 4.x or +## 5.x, hence the version-stamped name - a plain `bedework` would be claiming +## far more than we have measured. +bedework_3_10_3 = { ## If tests are yielding unexpected results, try to increase this: 'search-cache': {'behaviour': 'delay', 'delay': 3}, 'scheduling.auto-schedule': {'support': 'unknown'}, @@ -1539,6 +1550,138 @@ def compare(self, observed): ## save.duplicate-event is left at the default "full".) } +## Bedework 5.0.0, measured 2026-09-12 with caldav-server-tester against the +## locally built image in the caldav repo +## (tests/docker-test-servers/bedework/), demo user `vbede`. Several full runs; +## where they disagreed the difference is noted below. This is a different +## server from `bedework_3_10_3` in every way that matters - it creates and +## deletes calendars, its sync-token and text search behave differently - so +## nothing is inherited from that profile. +bedework_5_0_0 = { + ## Writes are asynchronous: a read-back issued immediately after a PUT may + ## 404 or hand back the pre-write copy. That is what separated the two + ## measurement runs - save-load.mutable came out "broken" (modification not + ## reflected after save and reload) in one and "full" in the other, and the + ## timezone probe's load() 404ed on a resource the PUT had just accepted. + ## The delay is what makes the rest of this profile reproducible. + "write-delay": {"behaviour": "delay", "delay": 3}, + + ## MKCALENDAR works, but a successful one that sets no properties is + ## answered with a 207 whose DAV:response carries neither a DAV:status nor + ## a DAV:propstat - just the href of the collection it created. RFC4918 + ## section 13 requires one or the other, so there is nothing in the answer + ## that says the creation succeeded; the collection is nevertheless there, + ## and the same request over raw HTTP answers 201. Recorded so the + ## deviation is written down somewhere; nothing in the library keys off it. + ## (A caldav older than this measurement read that body as a failure and + ## fell back to the extended MKCOL, which is why an early run of the tester + ## reported this as 'mkcol-required'.) + "create-calendar": {"support": "quirk", "behaviour": "empty-207"}, + ## Spelled out so the parent's "quirk" does not bleed down into them. + "create-calendar.auto": {"support": "unsupported"}, + "create-calendar.set-displayname": {"support": "full"}, + "create-calendar.stable-url": {"support": "full"}, + ## Not RFC properties; Bedework stores the Apple colour but not the order. + "calendar-color": {"support": "full"}, + "calendar-color.hex": {"support": "full"}, + "calendar-order": {"support": "unsupported"}, + + ## Bedework collections are typed, by default they can hold only + ## VEVENT, anything else is 403. Bedework does not support + ## creating task lists or journal lists through the CalDAV + ## protocol. Both MKCALENDAR and extended MKCOL answer "200 ok" + ## for CALDAV:supported-calendar-component-set and then ignore it. + ## A PROPPATCH of the same property afterwards is a 403. Details, + ## ref https://github.com/Bedework/bedework/issues/5#issuecomment-5652203366 + "create-calendar.with-supported-component-types": { + "support": "unsupported", + "behaviour": "the restriction is accepted with a 200 ok propstat and then ignored; the collection is VEVENT-only", + }, + ## So whether Bedework stores tasks is unknown rather than unsupported: the + ## server has a task collection type, a CalDAV client just cannot make one. + ## What was measured is that a VTODO does not go into an event calendar; + ## together with the ignored component set that leaves a client nowhere to + ## put a task. The children inherit "unknown". The same goes for journals. + "save-load.todo": {"support": "unknown"}, + "save-load.todo.mixed-calendar": {"support": "unsupported"}, + "save-load.journal": {"support": "unknown"}, + "save-load.journal.mixed-calendar": {"support": "unsupported"}, + ## Not measured either: the tester had no tasks to search for. A Bedework + ## user with a working `tasks` collection may well see this work. + "search.time-range.todo": {"support": "unknown"}, + + "save-load.event.recurrences.exception": {"support": "unsupported"}, + "save-load.mutable.attendee-partstat": {"support": "unsupported"}, + ## Seen 2026-09-13 in testChangeAttendeeStatusWithEmailGiven, which only + ## started running once save-load.mutable.attendee-partstat came out full. + "save-load.event.no-summary": { + "support": "ungraceful", + "behaviour": "a VEVENT without SUMMARY is refused with 500 missingeventproperty, though RFC 5545 section 3.6.1 makes SUMMARY optional", + }, + ## Expansion itself is right, but every instance comes back in a + ## DAV:response of its own under the same href, which RFC4918 section + ## 14.24 forbids. Until the library merged them, all but the last + ## instance of a resource were silently dropped (2026-09-13). + "search.recurrences.expanded.event": { + "support": "quirk", + "behaviour": "response-per-instance: each expanded instance comes in a DAV:response of its own, all under the same href, in violation of RFC 4918 section 14.24", + }, + ## Spelled out so the "quirk" above does not bleed into them via the parent. + "search.recurrences.expanded.exception": {"support": "full"}, + "search.recurrences.expanded.todo": {"support": "unknown"}, + ## Unchanged from 3.10.3, and still the open question in the tester's + ## docs/TODO.md. + "save-load.icalendar.related-to": { + "support": "broken", + "behaviour": "first RELATED-TO line preserved but subsequent RELATED-TO lines are stripped", + }, + "save.duplicate-uid.cross-calendar": { + "support": "ungraceful", + "behaviour": "Server error: ETagMismatchError", + }, + + "non-existing-raises-not-found.collection": { + "support": "unsupported", + "behaviour": "a non-existing calendar raises ReportError instead of NotFoundError", + }, + "principal-search": {"support": "ungraceful"}, + "principal-search.by-name.self": {"support": "ungraceful"}, + "principal-search.list-all": {"support": "ungraceful"}, + + ## Works for CATEGORIES and CLASS, not for DTEND; the children are spelled + ## out so the parent's "fragile" does not bleed down into them. + "search.is-not-defined": {"support": "fragile"}, + "search.is-not-defined.category": {"support": "full"}, + "search.is-not-defined.class": {"support": "full"}, + "search.is-not-defined.dtend": {"support": "unsupported"}, + ## Better than this feature's "unsupported" default: a time-range query + ## with no comp-type filter does return the objects in range. + "search.time-range.comp-type-optional": {"support": "full"}, + ## No text-match matches anything on a text property: a match on SUMMARY + ## comes back empty for i;octet, i;ascii-casemap and i;unicode-casemap + ## alike, on the full property value as well as on a substring. The parent + ## has to carry that verdict - with only the three children below set it + ## resolved to its default "full", which claimed a text search Bedework + ## cannot do and silently disarmed the deliberate + ## skip_unless_support("search.text") that keeps testEditSingleRecurrence + ## off this server. Enumerated properties are a different story and are + ## measured separately: a CLASS match does work, but only under + ## i;ascii-casemap, which is what search.text.case-sensitive records. + "search.text": {"support": "unsupported"}, + "search.text.case-sensitive": {"support": "unsupported"}, + "search.text.case-insensitive": {"support": "unsupported"}, + "search.text.category": {"support": "unsupported"}, + "search.time-range.alarm": {"support": "unsupported"}, + + ## The sync-token probe aborted on an ETagMismatchError (412) from its own + ## setup in every run, the configured write-delay included, so nothing + ## about sync-collection has been measured. + "sync-token": {"support": "unknown"}, + ## One account is configured, so the cross-user half of scheduling is + ## untested; the server advertises scheduling and the mailboxes are there. + "scheduling.auto-schedule": {"support": "unknown"}, +} + baikal = { ## version 0.10.1 # Baikal (sabre/dav) delivers iTIP notifications to the attendee inbox AND auto-schedules # into their calendar. diff --git a/caldav/config.py b/caldav/config.py index 07fc585a..8ff0ac1b 100644 --- a/caldav/config.py +++ b/caldav/config.py @@ -164,6 +164,42 @@ def replacer(match: re.Match) -> str: return value +## Profiles that have been renamed. The names are user-facing - they turn up +## as ``features:`` or ``base:`` in a caller's own config - so a rename has to +## explain itself rather than surface as an AttributeError from getattr. +RENAMED_FEATURE_PROFILES = { + "bedework": ( + "bedework_3_10_3", + "the profile was only ever measured against a quickstart-3.10.3 " + "docker image from 2018; use bedework_5_0_0 for a current Bedework", + ), +} + + +def _lookup_feature_profile(name): + """Look up a named profile in compatibility_hints. + + Accepts the name either bare ("synology") or module-qualified + ("compatibility_hints.synology"). + """ + import caldav.compatibility_hints + + if name.startswith("compatibility_hints."): + name = name[len("compatibility_hints.") :] + if name in RENAMED_FEATURE_PROFILES: + new_name, reason = RENAMED_FEATURE_PROFILES[name] + raise ValueError( + f"The compatibility profile '{name}' has been renamed to " + f"'{new_name}' - {reason}. Update your configuration." + ) + try: + return copy.deepcopy(getattr(caldav.compatibility_hints, name)) + except AttributeError: + raise ValueError( + f"No compatibility profile named '{name}' in caldav.compatibility_hints" + ) from None + + def resolve_features(features): """Resolve a features specification into a dict suitable for FeatureSet. @@ -175,21 +211,14 @@ def resolve_features(features): e.g. {"base": "synology", "search.is-not-defined": {"support": "fragile"}} - dict without "base": used as-is """ - import caldav.compatibility_hints - if features is None: return None if isinstance(features, str): - feature_name = features - if feature_name.startswith("compatibility_hints."): - feature_name = feature_name[len("compatibility_hints.") :] - return copy.deepcopy(getattr(caldav.compatibility_hints, feature_name)) + return _lookup_feature_profile(features) if isinstance(features, dict) and "base" in features: base_name = features["base"] if isinstance(base_name, str): - if base_name.startswith("compatibility_hints."): - base_name = base_name[len("compatibility_hints.") :] - base_features = copy.deepcopy(getattr(caldav.compatibility_hints, base_name)) + base_features = _lookup_feature_profile(base_name) for key, value in features.items(): if key != "base": base_features[key] = value diff --git a/tests/README.md b/tests/README.md index a3c307b5..dbd62457 100644 --- a/tests/README.md +++ b/tests/README.md @@ -144,7 +144,8 @@ The `docker-test-servers/` directory contains Docker configurations for: - **Nextcloud** - Full-featured cloud platform - **Cyrus** - Enterprise mail/calendaring server - **SOGo** - Groupware server -- **Bedework** - Enterprise calendar server +- **Bedework** - Enterprise calendar server (5.x, locally built image) +- **Bedework3** - the 2018 3.10.3 image, kept for its compatibility profile - **DAViCal** - CalDAV server See [docker-test-servers/README.md](docker-test-servers/README.md) for details. diff --git a/tests/caldav_test_servers.yaml.example b/tests/caldav_test_servers.yaml.example index 3beab28b..24dddc6f 100644 --- a/tests/caldav_test_servers.yaml.example +++ b/tests/caldav_test_servers.yaml.example @@ -85,13 +85,15 @@ test-servers: username: ${SOGO_USERNAME:-testuser} password: ${SOGO_PASSWORD:-testpassword} - bedework: + # Bedework 3.10.3 - the 2018 ioggstream image, kept for the historical + # compatibility profile. A current Bedework is `bedework` below. + bedework3: type: docker enabled: false - host: ${BEDEWORK_HOST:-localhost} - port: ${BEDEWORK_PORT:-8804} - username: ${BEDEWORK_USERNAME:-admin} - password: ${BEDEWORK_PASSWORD:-bedework} + host: ${BEDEWORK3_HOST:-localhost} + port: ${BEDEWORK3_PORT:-8804} + username: ${BEDEWORK3_USERNAME:-vbede} + password: ${BEDEWORK3_PASSWORD:-bedework} davical: type: docker @@ -136,6 +138,15 @@ test-servers: username: ${STALWART_USERNAME:-testuser@example.org} password: ${STALWART_PASSWORD:-testcaldav} + # Bedework 5 requires a locally built Docker image — run build.sh first. + bedework: + type: docker + enabled: false + host: ${BEDEWORK_HOST:-localhost} + port: ${BEDEWORK_PORT:-8811} + username: ${BEDEWORK_USERNAME:-vbede} + password: ${BEDEWORK_PASSWORD:-bedework} + # OX App Suite requires a locally built Docker image — run build.sh first. ox: type: docker diff --git a/tests/docker-test-servers/bedework/Dockerfile b/tests/docker-test-servers/bedework/Dockerfile new file mode 100644 index 00000000..8c00819d --- /dev/null +++ b/tests/docker-test-servers/bedework/Dockerfile @@ -0,0 +1,76 @@ +# Bedework 5 CalDAV test server. +# +# There is no usable public image for a current Bedework, and the upstream +# source is spread over ~20 git repositories that are assembled into a Wildfly +# galleon feature pack. Rather than building that from master we install the +# released feature pack the way upstream tells deployers to: +# https://bedework.github.io/bedework/#featurepack-install +# +# Override BW_VERSION to test another release, or BW_LAYERS to install a +# smaller subset (bw-democaluser-h2 is calendaring only, no public events). + +# ApacheDS - the demo system's LDAP server - cannot run on JDK 21; see +# apacheds-setenv.sh. Everything else does, so carry a second runtime rather +# than downgrade the whole image. +FROM docker.io/library/eclipse-temurin:17-jre AS jre17 + +FROM docker.io/library/eclipse-temurin:21-jdk + +ARG GALLEON_VERSION=6.1.0.Final +ARG BW_VERSION=5.0.0 +ARG BW_LAYERS=bw-demoall-h2,web-console + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install -y --no-install-recommends curl unzip procps && \ + rm -rf /var/lib/apt/lists/* + +RUN useradd --create-home --shell /bin/bash bedework + +USER bedework +WORKDIR /home/bedework + +# Galleon resolves ~950 artifacts from Maven Central into ~/.m2 on the way to +# building the wildfly tree; drop the cache and the installer again in the same +# layer so they never reach the image. +RUN curl -sSLo galleon.zip \ + "https://github.com/wildfly/galleon/releases/download/${GALLEON_VERSION}/galleon-${GALLEON_VERSION}.zip" && \ + unzip -q galleon.zip && \ + "./galleon-${GALLEON_VERSION}/bin/galleon.sh" install \ + "org.bedework.deploy:bw-wf-feature-pack:${BW_VERSION}" \ + --dir=/home/bedework/wildfly --verbose --layers="${BW_LAYERS}" && \ + rm -rf galleon.zip "galleon-${GALLEON_VERSION}" /home/bedework/.m2 + +# The feature pack ships H2 1.4.x demo databases alongside an H2 2.x driver +# that cannot read them; see migrate-h2.sh. +ARG H2_LEGACY_VERSION=1.4.197 +COPY --chown=bedework:bedework migrate-h2.sh /tmp/migrate-h2.sh +RUN /tmp/migrate-h2.sh "${H2_LEGACY_VERSION}" && rm -f /tmp/migrate-h2.sh + +# Upstream ships /cal on a module that cannot load a class the shared calendar +# service needs, and the resulting failure poisons /ucaldav too; see +# patch-modules.sh. +COPY --chown=bedework:bedework patch-modules.sh /tmp/patch-modules.sh +RUN /tmp/patch-modules.sh && rm -f /tmp/patch-modules.sh + +# OpenSearch's disk watermarks are about protecting a production cluster from +# the host filling up. Here they only mean that a busy developer machine turns +# every write into "index has read-only-allow-delete block" and every PUT into +# a 500 - the test data is a few hundred kB. +RUN printf '\n# Disposable test server: do not police the host filesystem.\ncluster.routing.allocation.disk.threshold_enabled: false\n' \ + >> /home/bedework/wildfly/opensearch/config/opensearch.yml + +COPY --from=jre17 /opt/java/openjdk /opt/java/jdk17 +COPY --chown=bedework:bedework apacheds-setenv.sh /home/bedework/wildfly/apacheds/bin/setenv.sh + +ENV JBOSS_HOME=/home/bedework/wildfly +# bwstartwildfly.sh passes this straight to standalone.sh; without it wildfly +# binds to loopback and the published port is dead. +ENV JBOSS_BIND="-b 0.0.0.0" + +COPY --chown=bedework:bedework run.sh /home/bedework/run.sh + +EXPOSE 8080 8081 9990 + +CMD ["/home/bedework/run.sh"] diff --git a/tests/docker-test-servers/bedework/README.md b/tests/docker-test-servers/bedework/README.md index f9de89c3..41777326 100644 --- a/tests/docker-test-servers/bedework/README.md +++ b/tests/docker-test-servers/bedework/README.md @@ -1,28 +1,109 @@ -# Bedework CalDAV Server Test Configuration +# Bedework 5 CalDAV Test Server -## Overview +[Bedework](https://bedework.github.io/bedework/) is an enterprise calendar +system running on Wildfly. This directory builds an image for the current +release; `../bedework3/` runs the 2018 3.10.3 image that used to be the only +one we had. -Bedework is an enterprise calendar system built on JBoss. The Docker image used for testing is `ioggstream/bedework:latest`. +## Why we build it ourselves -## Default Configuration +There is no public Bedework image newer than `ioggstream/bedework` (2018), and +there is no single repository to build from either: upstream is spread over +~20 git repos assembled into a Wildfly *galleon feature pack*. What upstream +tells deployers to install is that feature pack, so that is what the Dockerfile +does — `org.bedework.deploy:bw-wf-feature-pack`, layers `bw-demoall-h2` and +`web-console`, on JDK 21. See +. -The Bedework Docker image comes pre-configured and requires no additional setup files: +Galleon resolves ~950 artifacts from Maven Central; the build takes ~15 minutes +and the finished image is around 1.1 GB. -- **Default User**: `vbede` -- **Default Password**: `bedework` -- **CalDAV Endpoint**: `http://localhost:8804/ucaldav/user/vbede/` -- **Web Interface**: `http://localhost:8804/bedework/` +## Build -## Startup +```bash +./build.sh +``` -Bedework runs on JBoss and takes longer to start than other test servers (60-120 seconds). +To try another release or a smaller install: -## Calendars +```bash +./build.sh --build-arg BW_VERSION=5.1.0 --build-arg BW_LAYERS=bw-democaluser-h2,web-console +``` -The default user comes with two calendars: -- `calendar` - Main calendar for events -- `polls` - Bedework-specific polling calendar +## Start -## No Configuration Files Needed +```bash +./start.sh +``` -Unlike other test servers (SOGo, Baikal), Bedework doesn't require pre-seeded configuration files. The Docker image is ready to use as-is. +- CalDAV: `http://localhost:8811/ucaldav/user/vbede/` +- User: `vbede` / `bedework` (all demo accounts share that password) +- Web client: `http://localhost:8811/cal/`, Wildfly console: port 9990 + +The container runs four processes — apacheds (LDAP), h2, opensearch and +wildfly — and `run.sh` starts them in that order. + +## Stop + +```bash +./stop.sh +``` + +## Run tests + +```bash +cd ../../.. +pytest tests/test_caldav.py -k Bedework -v +``` + +Note that `-k Bedework` also matches the 3.10.3 server class; use +`-k "Bedework and not Bedework3"` to run only against this one. + +## Three upstream bugs are patched at build time + +Feature pack 5.0.0 does not come up on its own. Each fix is a separate script +with the details in its header: + +- **`migrate-h2.sh`** — the pre-seeded demo databases are H2 1.4.x files + (`format:1`) but the installed driver is H2 2.2.224, which refuses to open + them. Each database is dumped with a contemporary H2 and reloaded with the + new one. +- **`patch-modules.sh`** — the shared calendar service loads a dumprestore + class through the thread context classloader, and the module behind `/cal` + does not have it. Because the failure happens in a static initialiser in a + *shared* module, whichever request arrives first decides whether the whole + server works; the fix adds the missing module dependency. +- **`apacheds-setenv.sh`** — the bundled ApacheDS cannot run on JDK 21 (it + builds a self-signed certificate via `sun.security.x509`, which needs + `--add-exports` plus a method JDK 18 removed). It gets a JRE 17 carried in + the image; without this every authenticated request returns 500. + +`docker-compose.yml` also keeps the opensearch data directory on a named +volume: on the container's overlay filesystem OpenSearch dies at startup with +`AlreadyClosedException: Underlying file changed by an external force`. + +## Compatibility profile + +`caldav/compatibility_hints.py` has `bedework_5_0_0`, measured 2026-09-12 with +caldav-server-tester against this image as user `vbede`. Nothing is inherited +from `bedework_3_10_3` — that is a different server. + +Two things are worth knowing before reading a measurement against it: + +- **Writes are asynchronous.** A read-back issued immediately after a PUT may + 404 or hand back the pre-write copy, which made `save-load.mutable`, + `save-load.event.timezone` and `search.time-range.comp-type-optional` come + out differently in two consecutive runs. The profile carries + `write-delay: 3s`; without it a run measures the race rather than the server. +- **A client cannot create a collection that holds tasks.** MKCALENDAR and + extended MKCOL both answer `200 ok` for + `CALDAV:supported-calendar-component-set` and then ignore it, and a + PROPPATCH of the property afterwards is 403, so every collection a client + creates is VEVENT-only and a VTODO PUT into it is 403. + `vbede` has no usable `tasks` collection either: the Depth:1 PROPFIND of the + calendar home lists `tasks`, `Notifications` and `.pendingInbox` with a + `getlastmodified` of "now" that is renewed on every listing, and all three + 404 on any direct request. `douglm`, whose demo data ships a real one, can + store tasks in it. So the profile grades task and journal storage `unknown` + rather than unsupported: the tester had nowhere to put one. The analysis is + in https://github.com/Bedework/bedework/issues/5#issuecomment-5652203366 diff --git a/tests/docker-test-servers/bedework/apacheds-setenv.sh b/tests/docker-test-servers/bedework/apacheds-setenv.sh new file mode 100644 index 00000000..ed5ac478 --- /dev/null +++ b/tests/docker-test-servers/bedework/apacheds-setenv.sh @@ -0,0 +1,13 @@ +# Sourced by wildfly/apacheds/bin/apacheds.sh (its documented hook for local +# customisation). Installed by the Dockerfile. +# +# The ApacheDS bundled with the feature pack does not run on JDK 21, even +# though that is the JDK upstream tells you to install. On startup it builds a +# temporary self-signed certificate through sun.security.x509, which needs +# both an --add-exports to reach at all and a method - X509CertInfo.set(String, +# Object) - that JDK 18 removed. So run just this one process on a JRE 17 that +# the image carries alongside the JDK 21 everything else uses. + +JAVA_HOME=/opt/java/jdk17 +JAVA_OPTS="$JAVA_OPTS --add-exports java.base/sun.security.x509=ALL-UNNAMED" +JAVA_OPTS="$JAVA_OPTS --add-exports java.base/sun.security.util=ALL-UNNAMED" diff --git a/tests/docker-test-servers/bedework/build.sh b/tests/docker-test-servers/bedework/build.sh new file mode 100755 index 00000000..ccb7ee77 --- /dev/null +++ b/tests/docker-test-servers/bedework/build.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# Build the Bedework 5 Docker image for CalDAV testing. +# +# This must be run manually before starting the test server. Galleon pulls +# roughly a gigabyte of Maven artifacts, so the build takes a while. +# +# Usage: ./build.sh [--build-arg BW_VERSION=5.0.0] ... + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +echo "Building bedework-caldav-test image (this will take several minutes)..." +docker build -t bedework-caldav-test "$@" . + +echo "" +echo "Build complete. You can now run ./start.sh" diff --git a/tests/docker-test-servers/bedework/docker-compose.yml b/tests/docker-test-servers/bedework/docker-compose.yml index 51308e11..a750bac0 100644 --- a/tests/docker-test-servers/bedework/docker-compose.yml +++ b/tests/docker-test-servers/bedework/docker-compose.yml @@ -1,14 +1,27 @@ -version: '3.8' - services: bedework: - image: ioggstream/bedework:latest - container_name: bedework-test + image: bedework-caldav-test + # Not "bedework-test": that is what the 3.10.3 compose file called its + # container before ../bedework3/ was split out of this directory, so a + # checkout from before the split - another clone, an older branch - would + # silently destroy this container and put 3.10.3 up in its place. + container_name: bedework5-test ports: - - "8804:8080" + - "8811:8080" + volumes: + # OpenSearch dies at startup if its node.lock sits on the container's + # overlay filesystem ("Underlying file changed by an external force"). + # The directory only holds runtime state, and stop.sh removes the volume, + # so every run still starts from an empty index. + - opensearch-data:/home/bedework/wildfly/opensearch/data healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/bedework/"] + test: ["CMD", "curl", "-fs", "-o", "/dev/null", "-X", "PROPFIND", + "-H", "Depth: 0", "-u", "vbede:bedework", + "http://localhost:8080/ucaldav/user/vbede/"] interval: 10s timeout: 5s - retries: 15 - start_period: 120s + retries: 30 + start_period: 180s + +volumes: + opensearch-data: diff --git a/tests/docker-test-servers/bedework/migrate-h2.sh b/tests/docker-test-servers/bedework/migrate-h2.sh new file mode 100755 index 00000000..88ba9903 --- /dev/null +++ b/tests/docker-test-servers/bedework/migrate-h2.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# Convert the quickstart H2 databases to the format the shipped driver reads. +# +# The 5.0.0 feature pack pre-seeds its demo databases from bw-quickstart, which +# last released in 2022 and wrote them with H2 1.4.x (MVStore "format:1"). The +# same feature pack installs the H2 2.2.224 driver, which refuses to open them: +# +# The write format 1 is smaller than the supported format 3 [2.2.224/5] +# +# So dump each database with a contemporary H2 and reload it with the driver +# the server actually uses. The dumps need exactly one fixup: H2 1.4.x writes +# unbounded columns as VARCHAR(2147483647), above H2 2.x's precision limit. +# +# Runs at image build time; see Dockerfile. + +set -eu + +H2_LEGACY_VERSION="${1:?usage: migrate-h2.sh }" + +h2new=$(ls /home/bedework/wildfly/modules/system/layers/base/com/h2database/h2/main/h2-*.jar) +h2old=/tmp/h2-legacy.jar +work=/tmp/h2mig + +curl -sSLo "$h2old" \ + "https://repo1.maven.org/maven2/com/h2database/h2/${H2_LEGACY_VERSION}/h2-${H2_LEGACY_VERSION}.jar" + +mkdir -p "$work" +cd /home/bedework/wildfly/standalone/data/bedework/h2 + +for f in *.mv.db; do + db="${f%.mv.db}" + echo "migrating $db" + java -cp "$h2old" org.h2.tools.Script \ + -url "jdbc:h2:$PWD/$db" -user sa -password sa -script "$work/$db.sql" + sed -i 's/VARCHAR(2147483647)/VARCHAR(1000000000)/g' "$work/$db.sql" + rm -f "$db.mv.db" "$db.trace.db" + java -cp "$h2new" org.h2.tools.RunScript \ + -url "jdbc:h2:$PWD/$db" -user sa -password sa -script "$work/$db.sql" +done + +rm -f ./*.trace.db +rm -rf "$work" "$h2old" diff --git a/tests/docker-test-servers/bedework/patch-modules.sh b/tests/docker-test-servers/bedework/patch-modules.sh new file mode 100755 index 00000000..95c75de9 --- /dev/null +++ b/tests/docker-test-servers/bedework/patch-modules.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# Work around an upstream packaging bug in bw-wf-feature-pack 5.0.0. +# +# CalSvcFactoryDefault lives in the shared module org.bedework.calendar. +# common.api.rw and loads org.bedework.dumprestore.BwDumpRestore reflectively +# through the thread context classloader - i.e. through whichever war is +# serving the request. The dumprestore module is only listed as a dependency +# of org.bedework.calendar.rw-war, so a request to a war built on the +# read-only module (bw-webclient-cal, at /cal) throws ClassNotFoundException. +# +# That would just break /cal, except the failure is a static initialiser in a +# *shared* module: once it has failed, every later caller - /ucaldav included - +# gets NoClassDefFoundError and 500s until wildfly is restarted. Whichever +# request lands first decides whether the server works at all. +# +# So add the dependency to the read-only module too. +# +# Runs at image build time; see Dockerfile. + +set -eu + +modules=/home/bedework/wildfly/modules/system/layers/base/org/bedework/calendar +ro_war="$modules/ro-war/main/module.xml" + +grep -q 'name="org.bedework.calendar.dumprestore"' "$ro_war" && exit 0 + +sed -i 's||\n |' \ + "$ro_war" + +grep -q 'name="org.bedework.calendar.dumprestore"' "$ro_war" +echo "added dumprestore dependency to org.bedework.calendar.ro-war" diff --git a/tests/docker-test-servers/bedework/run.sh b/tests/docker-test-servers/bedework/run.sh new file mode 100755 index 00000000..5b04423a --- /dev/null +++ b/tests/docker-test-servers/bedework/run.sh @@ -0,0 +1,43 @@ +#!/bin/bash +# Container entrypoint: bring up the four processes a Bedework demo needs +# (apacheds, h2, opensearch, wildfly) and stay attached to wildfly. +# +# bwstartall.sh does the same thing, but it gives us no way to wait for +# opensearch before wildfly starts indexing, and it backgrounds wildfly. + +set -e + +export JAVA_HOME="${JAVA_HOME:-/opt/java/openjdk}" +cd "$JBOSS_HOME" + +echo "=== starting apacheds ===" +./bin/bwdirstart.sh + +echo "=== waiting for apacheds ===" +for i in $(seq 1 60); do + if (echo > /dev/tcp/127.0.0.1/10389) 2>/dev/null; then + echo "apacheds is up" + break + fi + [ "$i" -eq 60 ] && { echo "apacheds did not come up" >&2; exit 1; } + sleep 2 +done + +echo "=== starting h2 ===" +./bin/bwstarth2.sh + +echo "=== starting opensearch ===" +./bin/bwstartoschqs.sh + +echo "=== waiting for opensearch ===" +for i in $(seq 1 60); do + if curl -sf http://localhost:9200/_cluster/health >/dev/null; then + echo "opensearch is up" + break + fi + [ "$i" -eq 60 ] && { echo "opensearch did not come up" >&2; exit 1; } + sleep 5 +done + +echo "=== starting wildfly ===" +exec ./bin/bwstartwildfly.sh diff --git a/tests/docker-test-servers/bedework/start.sh b/tests/docker-test-servers/bedework/start.sh index 3cd6d941..6baabcf7 100755 --- a/tests/docker-test-servers/bedework/start.sh +++ b/tests/docker-test-servers/bedework/start.sh @@ -1,36 +1,59 @@ #!/bin/bash -# Start script for Bedework CalDAV test server +# Start the Bedework 5 CalDAV test server. +# +# The Docker image must be built first: +# ./build.sh +# +# The container starts four processes (apacheds, h2, opensearch, wildfly) and +# builds its opensearch indexes on every start, which takes a few minutes. +# +# Usage: ./start.sh set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$SCRIPT_DIR" -echo "Starting Bedework CalDAV server..." +if ! docker image inspect bedework-caldav-test >/dev/null 2>&1; then + echo "ERROR: Docker image 'bedework-caldav-test' not found." + echo "Please build it first with: ./build.sh" + exit 1 +fi + +echo "Starting Bedework 5 container (startup takes a few minutes)..." docker-compose up -d -echo "" -echo "Waiting for Bedework to initialize (this may take up to 2 minutes)..." -timeout=120 -elapsed=0 -while [ $elapsed -lt $timeout ]; do - if curl -f http://localhost:8804/bedework/ >/dev/null 2>&1; then - echo "✓ Bedework is ready!" +echo "Waiting for Bedework to finish deploying..." +for i in $(seq 1 90); do + if curl -sf -o /dev/null -X PROPFIND -H "Depth: 0" \ + -u vbede:bedework "http://localhost:8811/ucaldav/user/vbede/"; then echo "" - echo "CalDAV endpoint: http://localhost:8804/ucaldav/user/vbede/" - echo "Username: vbede" - echo "Password: bedework" + echo "Bedework is ready." + break + fi + if ! docker ps -q -f name=bedework5-test | grep -q .; then + echo "ERROR: Bedework container stopped unexpectedly." + docker-compose logs --tail=40 bedework + exit 1 + fi + if [ "$i" -eq 90 ]; then echo "" - echo "To stop Bedework: ./stop.sh" - echo "To view logs: docker-compose logs -f bedework" - exit 0 + echo "Timeout waiting for Bedework to deploy." + docker-compose logs --tail=40 bedework + exit 1 fi - sleep 5 - elapsed=$((elapsed + 5)) echo -n "." + sleep 5 done echo "" -echo "✗ Bedework did not start within ${timeout}s" -echo "Check logs with: docker-compose logs bedework" -exit 1 +echo "Bedework 5 is running on http://localhost:8811/" +echo " CalDAV: http://localhost:8811/ucaldav/user/vbede/" +echo " User: vbede / bedework" +echo "" +echo "Run tests from project root:" +echo " cd ../../.." +echo " pytest tests/test_caldav.py -k Bedework -v" +echo "" +echo "To stop: ./stop.sh" +echo "To view logs: docker-compose logs -f bedework" diff --git a/tests/docker-test-servers/bedework/stop.sh b/tests/docker-test-servers/bedework/stop.sh index 61f8cd9d..b91d981f 100755 --- a/tests/docker-test-servers/bedework/stop.sh +++ b/tests/docker-test-servers/bedework/stop.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Stop script for Bedework test server +# Stop Bedework 5 test server set -e @@ -9,4 +9,4 @@ cd "$SCRIPT_DIR" echo "Stopping Bedework and removing volumes..." docker-compose down -v -echo "✓ Bedework stopped and volumes removed" +echo "Bedework stopped." diff --git a/tests/docker-test-servers/bedework3/README.md b/tests/docker-test-servers/bedework3/README.md new file mode 100644 index 00000000..5aafef61 --- /dev/null +++ b/tests/docker-test-servers/bedework3/README.md @@ -0,0 +1,38 @@ +# Bedework 3.10.3 CalDAV Server Test Configuration + +## Overview + +Bedework is an enterprise calendar system built on JBoss. This directory runs +the `ioggstream/bedework:latest` Docker image. + +**That image is ancient.** It is a `quickstart-3.10.3` tree on openjdk-8, built +2018-11-05, and it cannot be rebuilt - the quickstart zip its Dockerfile fetches +from `dev.bedework.org` is gone. The compatibility profile in +`caldav/compatibility_hints.py` is therefore named `bedework_3_10_3`: it says +nothing about what a current Bedework does. + +For a current Bedework, see `../bedework/`, which builds an image from the +upstream galleon feature pack. + +## Default Configuration + +The Bedework Docker image comes pre-configured and requires no additional setup files: + +- **Default User**: `vbede` +- **Default Password**: `bedework` +- **CalDAV Endpoint**: `http://localhost:8804/ucaldav/user/vbede/` +- **Web Interface**: `http://localhost:8804/bedework/` + +## Startup + +Bedework runs on JBoss and takes longer to start than other test servers (60-120 seconds). + +## Calendars + +The default user comes with two calendars: +- `calendar` - Main calendar for events +- `polls` - Bedework-specific polling calendar + +## No Configuration Files Needed + +Unlike other test servers (SOGo, Baikal), Bedework doesn't require pre-seeded configuration files. The Docker image is ready to use as-is. diff --git a/tests/docker-test-servers/bedework3/docker-compose.yml b/tests/docker-test-servers/bedework3/docker-compose.yml new file mode 100644 index 00000000..b423c0da --- /dev/null +++ b/tests/docker-test-servers/bedework3/docker-compose.yml @@ -0,0 +1,14 @@ +version: '3.8' + +services: + bedework3: + image: ioggstream/bedework:latest + container_name: bedework3-test + ports: + - "8804:8080" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8080/bedework/"] + interval: 10s + timeout: 5s + retries: 15 + start_period: 120s diff --git a/tests/docker-test-servers/bedework3/start.sh b/tests/docker-test-servers/bedework3/start.sh new file mode 100755 index 00000000..f96b69f6 --- /dev/null +++ b/tests/docker-test-servers/bedework3/start.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# Start script for the Bedework 3.10.3 CalDAV test server + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +echo "Starting Bedework 3.10.3 CalDAV server..." +docker-compose up -d + +echo "" +echo "Waiting for Bedework to initialize (this may take up to 2 minutes)..." +timeout=120 +elapsed=0 +while [ $elapsed -lt $timeout ]; do + if curl -f http://localhost:8804/bedework/ >/dev/null 2>&1; then + echo "✓ Bedework is ready!" + echo "" + echo "CalDAV endpoint: http://localhost:8804/ucaldav/user/vbede/" + echo "Username: vbede" + echo "Password: bedework" + echo "" + echo "To stop Bedework: ./stop.sh" + echo "To view logs: docker-compose logs -f bedework3" + exit 0 + fi + sleep 5 + elapsed=$((elapsed + 5)) + echo -n "." +done + +echo "" +echo "✗ Bedework did not start within ${timeout}s" +echo "Check logs with: docker-compose logs bedework3" +exit 1 diff --git a/tests/docker-test-servers/bedework3/stop.sh b/tests/docker-test-servers/bedework3/stop.sh new file mode 100755 index 00000000..34f75027 --- /dev/null +++ b/tests/docker-test-servers/bedework3/stop.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# Stop script for the Bedework 3.10.3 test server + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +echo "Stopping Bedework 3.10.3 and removing volumes..." +docker-compose down -v + +echo "✓ Bedework 3.10.3 stopped and volumes removed" diff --git a/tests/fixture_helpers.py b/tests/fixture_helpers.py index ad7cd10d..b55804c0 100644 --- a/tests/fixture_helpers.py +++ b/tests/fixture_helpers.py @@ -241,6 +241,29 @@ def _supports(client: Any, feature: str) -> bool: return features.is_supported(feature) if features else True +def component_set_unobtainable(client: Any, comp_set: list[str] | None) -> str | None: + """Why a calendar restricted to ``comp_set`` cannot be had, or ``None``. + + Asking for a VTODO-only (or VJOURNAL-only) calendar is how the fixtures get + somewhere to put tasks on servers that will not mix them with events. When + the server also ignores the requested component set (Bedework 5: every + calendar a client creates is VEVENT-only), there is nowhere to put them, + and the caller should skip rather than fail on the PUT. + """ + if not comp_set or "VEVENT" in comp_set: + return None + if _supports(client, "create-calendar.with-supported-component-types"): + return None + for component in comp_set: + mixed = f"save-load.{component[1:].lower()}.mixed-calendar" + if not _supports(client, mixed): + return ( + f"server ignores supported-calendar-component-set and {component} " + "cannot share a calendar with events" + ) + return None + + async def atry_principal(client: Any) -> Any: """Discover the principal, or ``None`` if the server won't tell us. diff --git a/tests/test_async_integration.py b/tests/test_async_integration.py index 489c0544..51fc37f2 100644 --- a/tests/test_async_integration.py +++ b/tests/test_async_integration.py @@ -288,7 +288,16 @@ async def _afixture_calendar( own name, docstring and cal_id while the create/wipe/teardown logic lives in exactly one place - see fixture_helpers.afix_calendar. """ - from .fixture_helpers import afix_calendar, arelease_calendar, atry_principal + from .fixture_helpers import ( + afix_calendar, + arelease_calendar, + atry_principal, + component_set_unobtainable, + ) + + reason = component_set_unobtainable(async_client, supported_calendar_component_set) + if reason: + pytest.skip(reason) principal = await atry_principal(async_client) calendar, created = await afix_calendar( @@ -325,7 +334,15 @@ async def async_task_list(self, async_client: Any) -> Any: calendar is used. The calendar is reused across tests via a stable cal_id rather than being deleted and recreated, avoiding trashbin accumulation on servers like Nextcloud. + + Skips when the server cannot store a task at all, so that every test + taking this fixture skips rather than failing on the PUT: Bedework 5 hands + out a calendar happily (it accepts the component set with a "200 ok" + propstat and then ignores it, ref + create-calendar.with-supported-component-types) and answers the VTODO PUT + with a 403. """ + self.skip_unless_support("save-load.todo") ## Servers that can't hold VEVENTs and VTODOs in the same calendar ## (e.g. Zimbra, OX) need a component-restricted one. component_set = None if self.is_supported("save-load.todo.mixed-calendar") else ["VTODO"] @@ -2036,8 +2053,11 @@ async def test_change_attendee_status_with_email_given( ## direct PUT (403 Forbidden) and require iTIP scheduling instead. self.skip_unless_support("save-load.mutable.attendee-partstat") c = async_calendar + ## A SUMMARY, since some servers refuse an event without one + ## (save-load.event.no-summary) and this test is about PARTSTAT. event = await c.add_event( uid="test1", + summary="attendee status test", dtstart=datetime(2015, 10, 10, 8, 7, 6), dtend=datetime(2015, 10, 10, 9, 7, 6), ical_fragment="ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=TENTATIVE:MAILTO:testuser@example.com", diff --git a/tests/test_caldav.py b/tests/test_caldav.py index cb8d30e4..8c13eced 100644 --- a/tests/test_caldav.py +++ b/tests/test_caldav.py @@ -1563,7 +1563,13 @@ def _fixCalendar_(self, **kwargs): Delegates core create-or-find logic to fixture_helpers.get_or_create_test_calendar, handling test-infrastructure concerns (caching, cleanup, cal_id defaults) here. """ - from .fixture_helpers import get_or_create_test_calendar + from .fixture_helpers import component_set_unobtainable, get_or_create_test_calendar + + reason = component_set_unobtainable( + self.caldav, kwargs.get("supported_calendar_component_set") + ) + if reason: + pytest.skip(reason) if not self.is_supported("create-calendar"): if not self._default_calendar: @@ -2045,8 +2051,11 @@ def testChangeAttendeeStatusWithEmailGiven(self): self.skip_unless_support("save-load.mutable.attendee-partstat") c = self._fixCalendar() + ## A SUMMARY, since some servers refuse an event without one + ## (save-load.event.no-summary) and this test is about PARTSTAT. event = c.add_event( uid="test1", + summary="attendee status test", dtstart=datetime(2015, 10, 10, 8, 7, 6), dtend=datetime(2015, 10, 10, 9, 7, 6), ical_fragment="ATTENDEE;ROLE=OPT-PARTICIPANT;PARTSTAT=TENTATIVE:MAILTO:testuser@example.com", @@ -2288,6 +2297,7 @@ def testObjectByUID(self): """ It should be possible to save a task and retrieve it by uid """ + self.skip_unless_support("save-load.todo") c = self._fixCalendar(supported_calendar_component_set=["VTODO"]) c.add_todo(summary="Some test task with a well-known uid", uid="well_known_1") foo = c.get_object_by_uid("well_known_1") diff --git a/tests/test_compatibility_hints.py b/tests/test_compatibility_hints.py index 9bac24c0..3219ea4e 100644 --- a/tests/test_compatibility_hints.py +++ b/tests/test_compatibility_hints.py @@ -605,6 +605,51 @@ def test_base_with_prefix(self) -> None: assert result["sync-token"] != "fragile" +class TestRenamedProfiles: + """Renamed hint profiles must fail with an explanation, not AttributeError. + + The profile names are user-facing - they can appear as ``features:`` or + ``base:`` in a caller's own config - so renaming one has to say what it was + renamed to and why, rather than blowing up inside ``getattr``. + """ + + def test_bedework_profile_was_version_stamped(self) -> None: + import caldav.compatibility_hints as ch + + assert hasattr(ch, "bedework_3_10_3") + assert not hasattr(ch, "bedework") + + def test_bare_renamed_name_explains_itself(self) -> None: + with pytest.raises(ValueError) as exc_info: + _resolve_features("bedework") + message = str(exc_info.value) + assert "bedework_3_10_3" in message + assert "bedework_5_0_0" in message + assert "3.10.3" in message + + def test_prefixed_renamed_name_explains_itself(self) -> None: + with pytest.raises(ValueError) as exc_info: + _resolve_features("compatibility_hints.bedework") + assert "bedework_3_10_3" in str(exc_info.value) + + def test_renamed_name_as_base_explains_itself(self) -> None: + with pytest.raises(ValueError) as exc_info: + _resolve_features({"base": "bedework", "sync-token": "full"}) + assert "bedework_3_10_3" in str(exc_info.value) + + def test_new_name_resolves(self) -> None: + import caldav.compatibility_hints as ch + + result = _resolve_features("bedework_3_10_3") + assert result == ch.bedework_3_10_3 + assert result is not ch.bedework_3_10_3 + + def test_unknown_profile_names_the_offender(self) -> None: + with pytest.raises(ValueError) as exc_info: + _resolve_features("no_such_server") + assert "no_such_server" in str(exc_info.value) + + class TestFeatureSetCompare: """Test FeatureSet.compare(): declared (expected) vs observed feature sets.""" diff --git a/tests/test_fixture_helpers.py b/tests/test_fixture_helpers.py index 3796b7ee..8658c87a 100644 --- a/tests/test_fixture_helpers.py +++ b/tests/test_fixture_helpers.py @@ -14,6 +14,7 @@ import pytest +from caldav import compatibility_hints from caldav.compatibility_hints import FeatureSet from caldav.lib import error @@ -21,6 +22,7 @@ _get_or_create_impl, afix_calendar, arelease_calendar, + component_set_unobtainable, ) @@ -297,3 +299,35 @@ async def test_afix_calendar_drops_name_for_component_restricted_calendar() -> N assert principal.make_calendar_calls == [ {"cal_id": "testcal-tasks", "supported_calendar_component_set": ["VTODO"]} ] + + +IGNORED = {"create-calendar.with-supported-component-types": False} + + +@pytest.mark.parametrize( + ("hints", "comp_set", "unobtainable"), + [ + ({}, ["VTODO"], False), + ({}, None, False), + ## The restriction is ignored, but tasks go into an event calendar anyway. + (IGNORED, ["VTODO"], False), + ## Zimbra: no mixing, but a VTODO-only calendar is honoured. + ({"save-load.todo.mixed-calendar": False}, ["VTODO"], False), + ## Bedework 5: neither, so there is nowhere to put a task. + (IGNORED | {"save-load.todo.mixed-calendar": False}, ["VTODO"], True), + (IGNORED | {"save-load.todo.mixed-calendar": False}, ["VJOURNAL"], False), + (IGNORED | {"save-load.journal.mixed-calendar": False}, ["VJOURNAL"], True), + (IGNORED | {"save-load.todo.mixed-calendar": False}, ["VEVENT", "VTODO"], False), + ], +) +def test_component_set_unobtainable( + hints: dict, comp_set: list[str] | None, unobtainable: bool +) -> None: + """A restricted calendar is out of reach only when the restriction is ignored + *and* the component cannot share a calendar with events.""" + assert bool(component_set_unobtainable(FakeClient(hints), comp_set)) is unobtainable + + +def test_bedework_5_has_nowhere_to_put_a_task() -> None: + client = FakeClient(compatibility_hints.bedework_5_0_0) + assert component_set_unobtainable(client, ["VTODO"]) diff --git a/tests/test_servers/docker.py b/tests/test_servers/docker.py index 8780abfb..bd8db86f 100644 --- a/tests/test_servers/docker.py +++ b/tests/test_servers/docker.py @@ -2,7 +2,8 @@ Docker-based test server implementations. This module provides test server implementations for servers that run -in Docker containers: Baikal, Nextcloud, Cyrus, SOGo, Bedework, DAViCal, Davis, CCS, Zimbra, and Stalwart. +in Docker containers: Baikal, Nextcloud, Cyrus, SOGo, Bedework (5.x and 3.10.3), +DAViCal, Davis, CCS, Zimbra, Stalwart and OX. """ import os @@ -207,11 +208,58 @@ def is_accessible(self) -> bool: return False +class Bedework3TestServer(DockerTestServer): + """ + Bedework 3.10.3 calendar server in Docker. + + This is the ancient `ioggstream/bedework` image from 2018; see + tests/docker-test-servers/bedework3/README.md. For a current Bedework + use :class:`BedeworkTestServer`. + """ + + name = "Bedework3" + + def __init__(self, config: dict[str, Any] | None = None) -> None: + config = config or {} + config.setdefault("host", os.environ.get("BEDEWORK3_HOST", "localhost")) + config.setdefault("port", int(os.environ.get("BEDEWORK3_PORT", "8804"))) + config.setdefault("username", os.environ.get("BEDEWORK3_USERNAME", "vbede")) + config.setdefault("password", os.environ.get("BEDEWORK3_PASSWORD", "bedework")) + # Set up Bedework-specific compatibility hints + if "features" not in config: + config["features"] = compatibility_hints.bedework_3_10_3.copy() + super().__init__(config) + + def _default_port(self) -> int: + return 8804 + + @property + def url(self) -> str: + return f"http://{self.host}:{self.port}/ucaldav/user/{self.username}" + + def is_accessible(self) -> bool: + """Check if Bedework is accessible using PROPFIND.""" + try: + response = requests.request( + "PROPFIND", + f"http://{self.host}:{self.port}/ucaldav/", + timeout=DEFAULT_HTTP_TIMEOUT, + ) + return response.status_code in (200, 207, 401, 403, 404) + except Exception: + return False + + class BedeworkTestServer(DockerTestServer): """ - Bedework calendar server in Docker. + Bedework 5 calendar server in Docker. + + Built locally from the upstream galleon feature pack - see + tests/docker-test-servers/bedework/README.md - so there is no image to + pull and no CI job; ./build.sh has to be run by hand first. - Bedework is an enterprise-class open-source calendar system. + Measured 2026-09-12 into compatibility_hints.bedework_5_0_0; nothing is + inherited from the 3.10.3 profile, which describes a different server. """ name = "Bedework" @@ -219,16 +267,15 @@ class BedeworkTestServer(DockerTestServer): def __init__(self, config: dict[str, Any] | None = None) -> None: config = config or {} config.setdefault("host", os.environ.get("BEDEWORK_HOST", "localhost")) - config.setdefault("port", int(os.environ.get("BEDEWORK_PORT", "8804"))) + config.setdefault("port", int(os.environ.get("BEDEWORK_PORT", "8811"))) config.setdefault("username", os.environ.get("BEDEWORK_USERNAME", "vbede")) config.setdefault("password", os.environ.get("BEDEWORK_PASSWORD", "bedework")) - # Set up Bedework-specific compatibility hints if "features" not in config: - config["features"] = compatibility_hints.bedework.copy() + config["features"] = compatibility_hints.bedework_5_0_0.copy() super().__init__(config) def _default_port(self) -> int: - return 8804 + return 8811 @property def url(self) -> str: @@ -537,6 +584,7 @@ def is_accessible(self) -> bool: register_server_class("nextcloud", NextcloudTestServer) register_server_class("cyrus", CyrusTestServer) register_server_class("sogo", SOGoTestServer) +register_server_class("bedework3", Bedework3TestServer) register_server_class("bedework", BedeworkTestServer) register_server_class("davical", DavicalTestServer) register_server_class("davis", DavisTestServer) diff --git a/tests/test_servers/registry.py b/tests/test_servers/registry.py index dccf331c..3485ca96 100644 --- a/tests/test_servers/registry.py +++ b/tests/test_servers/registry.py @@ -235,8 +235,7 @@ def load_from_config(self, config: dict) -> None: if server_class is None: warnings.warn( f"Server '{name}': unknown type '{server_type}'. " - f"Valid types: embedded, docker, external, radicale, xandikos, " - f"baikal, nextcloud, cyrus, sogo, bedework. " + f"Valid types: {', '.join(sorted(_SERVER_CLASSES))}. " f"Server will be skipped.", UserWarning, stacklevel=2, diff --git a/tests/tools/convert_conf_private.py b/tests/tools/convert_conf_private.py index d95694a2..a1c2f5cd 100755 --- a/tests/tools/convert_conf_private.py +++ b/tests/tools/convert_conf_private.py @@ -61,6 +61,12 @@ def load_conf_private(path: Path) -> dict[str, Any]: return module +#: Legacy conf_private attribute prefixes whose test-server key was renamed. +#: `test_bedework` in a pre-3.0 conf_private.py always meant the 3.10.3 +#: docker image, which is now called `bedework3`. +SERVER_KEY_RENAMES = {"bedework": "bedework3"} + + def convert_to_yaml_config(conf_private: Any) -> dict[str, Any]: """Convert conf_private module to new YAML config format.""" result: dict[str, Any] = {"test-servers": {}} @@ -101,7 +107,9 @@ def convert_to_yaml_config(conf_private: Any) -> dict[str, Any]: servers[key] = config - # Handle boolean enable/disable switches + # Handle boolean enable/disable switches. These are the legacy + # conf_private attribute prefixes; SERVER_KEY_RENAMES maps the ones whose + # test-server key has since changed. server_names = [ "radicale", "xandikos", @@ -113,8 +121,9 @@ def convert_to_yaml_config(conf_private: Any) -> dict[str, Any]: "davical", ] - for server_name in server_names: - test_attr = f"test_{server_name}" + for legacy_name in server_names: + server_name = SERVER_KEY_RENAMES.get(legacy_name, legacy_name) + test_attr = f"test_{legacy_name}" if hasattr(conf_private, test_attr): if server_name not in servers: # Determine type based on server name @@ -126,9 +135,10 @@ def convert_to_yaml_config(conf_private: Any) -> dict[str, Any]: servers[server_name]["enabled"] = getattr(conf_private, test_attr) # Handle host/port overrides - for server_name in server_names: - host_attr = f"{server_name}_host" - port_attr = f"{server_name}_port" + for legacy_name in server_names: + server_name = SERVER_KEY_RENAMES.get(legacy_name, legacy_name) + host_attr = f"{legacy_name}_host" + port_attr = f"{legacy_name}_port" if hasattr(conf_private, host_attr): if server_name not in servers: @@ -141,9 +151,10 @@ def convert_to_yaml_config(conf_private: Any) -> dict[str, Any]: servers[server_name]["port"] = getattr(conf_private, port_attr) # Handle username/password for known servers - for server_name in server_names: - user_attr = f"{server_name}_username" - pass_attr = f"{server_name}_password" + for legacy_name in server_names: + server_name = SERVER_KEY_RENAMES.get(legacy_name, legacy_name) + user_attr = f"{legacy_name}_username" + pass_attr = f"{legacy_name}_password" if hasattr(conf_private, user_attr): if server_name not in servers: diff --git a/tox.ini b/tox.ini index 703cd349..f9178c66 100644 --- a/tox.ini +++ b/tox.ini @@ -20,9 +20,9 @@ passenv = SOGO_URL SOGO_USERNAME SOGO_PASSWORD - BEDEWORK_URL - BEDEWORK_USERNAME - BEDEWORK_PASSWORD + BEDEWORK3_URL + BEDEWORK3_USERNAME + BEDEWORK3_PASSWORD commands = coverage run -m pytest [testenv:docs] From 4a48013a8c736a18a9d838a29f6d8f407bcecef7 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Sat, 12 Sep 2026 08:32:55 +0200 Subject: [PATCH 06/18] fix: accept an all-ok multistatus on MKCALENDAR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RFC 4791 has MKCALENDAR answer 201 Created and reserves the multistatus for reporting what could not be done, so we insisted on 201. Bedework 5 answers 207 instead: every propstat 200 ok when the request carries properties, and a single DAV:response holding nothing but the href when it does not, which RFC 4918 section 13 forbids. Either way the calendar was created, yet make_calendar() raised MkcalendarError. A 207 is now accepted when nothing in it reports a failure; a multistatus with no DAV:response at all, or with a non-2xx status, still raises. Prompt: look into the new bedework test server. It should be running. It even seems impossible to createa (sic) a calendar? Followup-Prompt: fix everything [= the MKCALENDAR failure and a container-name collision that let an older checkout destroy the Bedework 5 container; the latter went into the Bedework 5 feature commit] Followup-Prompt: New bedework5 container should be running and working now. Please probe it and populate the new bedework profile in compatibility_hints.py Followup-Prompt: It should be recorded as "quirk" and 'empty-207' as behaviour. caldav library should handle it gracefully. Comments in compatibility_hints.py referring to the quirk as being in vaiolation (sic) of RFC 4918 §13 [answering what all_statuses_ok() should do with the status-less 207 the probe had turned up] Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- CHANGELOG.md | 1 + caldav/collection.py | 32 +++++++++- caldav/davobject.py | 22 +++++-- caldav/response.py | 37 ++++++++++++ tests/test_async_davclient.py | 43 +++++++++++++ tests/test_caldav_unit.py | 110 ++++++++++++++++++++++++++++++++++ 6 files changed, 237 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b98f47b..6972d8b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 ### Fixed +* Creating a calendar no longer fails on a server answering `MKCALENDAR`/`MKCOL` with a `207 Multi-Status` that reports nothing but success. RFC 4791 has the server answer `201 Created` and reserves the multistatus for reporting what could not be done, but Bedework 5 answers 207 as soon as the request carries properties - with every propstat `200 ok` and the calendar created and named. Insisting on 201 raised `MkcalendarError` for a calendar that was in fact there. The same applies to a `DAV:response` carrying neither a `DAV:status` nor a `DAV:propstat` — RFC 4918 §13 requires one or the other, and Bedework answers a property-less `MKCALENDAR` with nothing but the href of the collection it created — since nothing in such a body says the creation failed. A multistatus carrying a non-2xx status still raises, and one with no `DAV:response` at all is still not a success. * A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. ## [3.3.0] - 2026-09-03 diff --git a/caldav/collection.py b/caldav/collection.py index 5c77586f..e57d8747 100644 --- a/caldav/collection.py +++ b/caldav/collection.py @@ -48,12 +48,34 @@ from .davobject import DAVObject from .elements import cdav, dav from .lib import error, vcal +from .lib.error import errmsg from .lib.python_utilities import to_wire from .lib.url import URL, normalise_path, requote_path _CC = TypeVar("_CC", bound="CalendarObjectResource") log = logging.getLogger("caldav") +## RFC 4791 §5.3.1 (MKCALENDAR) and RFC 5689 §3 (extended MKCOL) have the +## server answer 201 Created when the collection was made and every property +## set, and reserve the multistatus for reporting what went wrong. Bedework 5 +## nevertheless answers 207 as soon as the request carries properties, listing +## each of them as 200 ok. Both are accepted here; _assert_created() then +## sorts a 207 that spells out a success from one reporting a failure. +_CREATED_STATUSES = (201, 207) + + +def _assert_created(response, method: str) -> None: + """Raise unless the server really did create the collection. + + A 207 counts as success only when every status in it is a 2xx - a + multistatus reporting that a property could not be set (or that the + collection could not be made) is the failure the RFCs use it for, and must + raise the same way any other unexpected answer does. + """ + if response.status == 201 or response.all_statuses_ok(): + return + raise error.exception_by_method[method](errmsg(response)) + # --------------------------------------------------------------------------- # Helpers for extracting calendar / principal info from PROPFIND results. @@ -930,7 +952,10 @@ def _create( if self.is_async_client: return self._async_create(path, mkcol, method, name, display_name, stable_url) - self._query(root=mkcol, query_method=method, url=path, expected_return_value=201) + response = self._query( + root=mkcol, query_method=method, url=path, expected_return_value=_CREATED_STATUSES + ) + _assert_created(response, method) # COMPATIBILITY ISSUE # name should already be set, but we've seen caldav servers failing @@ -1030,7 +1055,10 @@ def _adopt_relocated_url(self, name, relocated: list) -> None: async def _async_create(self, path, mkcol, method, name, display_name, stable_url) -> None: """Async implementation of _create (call via _create, not directly).""" - await self._query(root=mkcol, query_method=method, url=path, expected_return_value=201) + response = await self._query( + root=mkcol, query_method=method, url=path, expected_return_value=_CREATED_STATUSES + ) + _assert_created(response, method) # COMPATIBILITY ISSUE - try to set display name explicitly if display_name: diff --git a/caldav/davobject.py b/caldav/davobject.py index 7e99969f..08e810d9 100644 --- a/caldav/davobject.py +++ b/caldav/davobject.py @@ -28,6 +28,20 @@ log = logging.getLogger("caldav") +def _unexpected_status(status: int, expected: "int | Sequence[int] | None") -> bool: + """Whether ``status`` is outside what the caller asked for. + + ``expected_return_value`` is usually a single status code, but a caller + that accepts more than one (a collection creation may be answered either + with a ``201`` or with a multistatus) may hand in a sequence. + """ + if expected is None: + return False + if isinstance(expected, int): + return status != expected + return status not in expected + + """ This file contains one class, the DAVObject which is the base class for Calendar, Principal, CalendarObjectResource (Event) and many @@ -268,9 +282,7 @@ def _query( ret = getattr(self.client, query_method)(url, body, depth) if ret.status == 404: raise error.NotFoundError(errmsg(ret)) - if ( - expected_return_value is not None and ret.status != expected_return_value - ) or ret.status >= 400: + if _unexpected_status(ret.status, expected_return_value) or ret.status >= 400: ## COMPATIBILITY HACK - see https://github.com/python-caldav/caldav/issues/309 ## TODO: server quirks! body = to_wire(body) @@ -295,9 +307,7 @@ async def _async_query( ret = await getattr(self.client, query_method)(url, body, depth) if ret.status == 404: raise error.NotFoundError(errmsg(ret)) - if ( - expected_return_value is not None and ret.status != expected_return_value - ) or ret.status >= 400: + if _unexpected_status(ret.status, expected_return_value) or ret.status >= 400: ## COMPATIBILITY HACK - see https://github.com/python-caldav/caldav/issues/309 body = to_wire(body) if ret.status == 500 and b"D:getetag" not in body and b" bool: return False return True + def all_statuses_ok(self) -> bool: + """True if the multistatus reports success and nothing but success. + + Every status in it - the response-level ones and the ones nested + inside ```` alike - has to be a 2xx. Used to tell a + multistatus that merely spells out a success apart from one reporting + a failure: a server answering a collection creation with 207 (Bedework + 5 does, whenever the request carries properties) has created the + collection only if no status in the body says otherwise. + + There has to be at least one ``DAV:response``, but a response carrying + no status at all does not make the answer a failure. RFC 4918 section + 13 requires every response to carry either a ``DAV:status`` or at least + one ``DAV:propstat``, and Bedework 5 answers a property-less + MKCALENDAR with a response holding nothing but the href of the + collection it just created - reading that as a failure raised + ``MkcalendarError`` for a calendar that was there. + + This deliberately does not go through ``validate_status()``: a status + we do not accept is an answer here, not a parse error. + """ + if self.tree is None: + return False + responses = [r for r in self._strip_to_multistatus() if r.tag == dav.Response.tag] + if not responses: + return False + for response in responses: + for status in response.iter(dav.Status.tag): + ## _status_to_code() falls back to 200 for anything it cannot + ## parse, which would turn a garbled status into a success + parts = (status.text or "").split() + if len(parts) < 2 or not parts[1].isdigit(): + return False + if not 200 <= int(parts[1]) < 300: + return False + return True + def _find_objects_and_props(self) -> dict[str, dict[str, _Element]]: """Internal implementation of find_objects_and_props without deprecation warning.""" self.objects: dict[str, dict[str, _Element]] = {} diff --git a/tests/test_async_davclient.py b/tests/test_async_davclient.py index 95ee65c1..28e5a46a 100644 --- a/tests/test_async_davclient.py +++ b/tests/test_async_davclient.py @@ -1334,3 +1334,46 @@ def test_the_flavour_names_the_library_that_was_imported(self) -> None: assert _HTTPX_FLAVOUR in (None, *_ASYNC_HTTPX_CANDIDATES) assert _USE_HTTPX == (_HTTPX_FLAVOUR is not None) + + +class TestAsyncMkcalendarMultistatus: + """Async twin of ``TestMkcalendarMultistatus`` in test_caldav_unit.py: + a MKCALENDAR answered with an all-success 207 Multi-Status (Bedework 5) + created the calendar; one reporting a failing propstat did not.""" + + URL = "https://caldav.example.com/dav/user/mycal/" + + def _multistatus(self, status: str) -> bytes: + return ( + '' + "/dav/user/mycal" + "" + f"{status}" + "" + ).encode() + + async def _save_calendar(self, status_code: int, content: bytes): + from caldav import Calendar, CalendarSet + + client = AsyncDAVClient(url="https://caldav.example.com/dav/") + client.session.request = AsyncMock( + return_value=create_mock_response( + content=content, + status_code=status_code, + reason="Multi-Status", + headers={"Content-Type": "text/xml"}, + ) + ) + calendar_set = CalendarSet(client, url="https://caldav.example.com/dav/user/") + calendar = Calendar(client, parent=calendar_set, name="My Calendar", id="mycal") + return await calendar.save() + + @pytest.mark.asyncio + async def test_all_ok_multistatus_is_a_created_calendar(self) -> None: + calendar = await self._save_calendar(207, self._multistatus("HTTP/1.1 200 ok")) + assert str(calendar.url) == self.URL + + @pytest.mark.asyncio + async def test_failing_propstat_still_raises(self) -> None: + with pytest.raises(error.MkcalendarError): + await self._save_calendar(207, self._multistatus("HTTP/1.1 403 Forbidden")) diff --git a/tests/test_caldav_unit.py b/tests/test_caldav_unit.py index c552898d..60b6c5aa 100755 --- a/tests/test_caldav_unit.py +++ b/tests/test_caldav_unit.py @@ -4316,3 +4316,113 @@ def test_two_wraps_yield_identical_data(self) -> None: second.icalendar_instance = ievent assert first.data == second.data + + +class TestMkcalendarMultistatus: + """A ``207 Multi-Status`` where every status is a success means the + calendar was created. + + RFC 4791 section 5.3.1 has MKCALENDAR answer ``201 Created`` on success and + reserves the multistatus for the case where the collection could not be + created or a property could not be set. Bedework 5 answers 207 + unconditionally as soon as the request carries properties -- every propstat + ``200 ok``, the collection created, the display name set. Insisting on 201 + turned that into a ``MkcalendarError`` for a calendar that was in fact + there. A multistatus that reports a real failure must still raise. + """ + + URL = "http://cal.example.com/dav/user/mycal/" + + def _multistatus(self, status: str) -> str: + return ( + '\n' + " \n" + " /dav/user/mycal\n" + " \n" + " \n" + f" {status}\n" + " \n" + " \n" + "\n" + ) + + def _save_calendar(self, mocked, status_code: int, content: str) -> Calendar: + mocked().status_code = status_code + mocked().reason = "Multi-Status" + mocked().headers = {"Content-Type": "text/xml"} + mocked().content = content + client = DAVClient(url="http://cal.example.com/dav/") + calendar_set = CalendarSet(client, url="http://cal.example.com/dav/user/") + calendar = Calendar(client, parent=calendar_set, name="My Calendar", id="mycal") + return calendar.save() + + @mock.patch("caldav.davclient.requests.Session.request") + def test_all_ok_multistatus_is_a_created_calendar(self, mocked) -> None: + calendar = self._save_calendar(mocked, 207, self._multistatus("HTTP/1.1 200 ok")) + assert str(calendar.url) == self.URL + + @mock.patch("caldav.davclient.requests.Session.request") + def test_201_is_still_accepted(self, mocked) -> None: + calendar = self._save_calendar(mocked, 201, "") + assert str(calendar.url) == self.URL + + @mock.patch("caldav.davclient.requests.Session.request") + def test_failing_propstat_still_raises(self, mocked) -> None: + with pytest.raises(error.MkcalendarError): + self._save_calendar(mocked, 207, self._multistatus("HTTP/1.1 403 Forbidden")) + + @mock.patch("caldav.davclient.requests.Session.request") + def test_unexpected_status_still_raises(self, mocked) -> None: + """Only 201 and an all-success 207 mean "created"; the 200 some + servers might answer with is not a status we have ever accepted.""" + with pytest.raises(error.MkcalendarError): + self._save_calendar(mocked, 200, "") + + def _statusless_multistatus(self) -> str: + """Bedework 5's answer to a MKCALENDAR that sets no properties. + + RFC 4918 section 13 requires a ``DAV:response`` to carry either a + ``DAV:status`` or at least one ``DAV:propstat``; this one carries + neither, just the href of the collection it created. + """ + return ( + '\n' + " \n" + " /dav/user/mycal\n" + " \n" + "\n" + ) + + @mock.patch("caldav.davclient.requests.Session.request") + def test_statusless_response_is_a_created_calendar(self, mocked) -> None: + """Nothing in the body says the creation failed, and the collection is + there afterwards - verified against Bedework 5.0.0, where the same + request through raw HTTP answers 201.""" + calendar = self._save_calendar(mocked, 207, self._statusless_multistatus()) + assert str(calendar.url) == self.URL + + @mock.patch("caldav.davclient.requests.Session.request") + def test_empty_multistatus_still_raises(self, mocked) -> None: + """A multistatus with no response at all reports nothing about any + collection, so it cannot be read as a success.""" + with pytest.raises(error.MkcalendarError): + self._save_calendar(mocked, 207, '\n') + + @mock.patch("caldav.davclient.requests.Session.request") + def test_statusless_response_beside_a_failing_one_still_raises(self, mocked) -> None: + content = ( + '\n' + " \n" + " /dav/user/mycal\n" + " \n" + " \n" + " /dav/user/mycal\n" + " \n" + " \n" + " HTTP/1.1 403 Forbidden\n" + " \n" + " \n" + "\n" + ) + with pytest.raises(error.MkcalendarError): + self._save_calendar(mocked, 207, content) From 3c52d9c2596a80f33d983f461c47ba00c08d79c5 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Sun, 13 Sep 2026 07:46:59 +0200 Subject: [PATCH 07/18] refactor: drop hints their parent already implies A profile entry graded the same as an explicitly graded parent adds nothing, since the children of such a feature resolve to its grade. Twenty such entries across bedework_3_10_3, bedework_5_0_0, baikal, davical, purelymail, ox and infomaniak are gone, and every feature resolves exactly as before in every profile. Zimbra's delete-calendar.free-namespace stays: its comment records it on purpose. Prompt: Now the compatibility_hints explicitly lists `save-load.todo.*` as unsupported. It should suffice to set `save-load.todo` to `unknown` or `unsupported`, the children will automatically inheritate (sic) the parent. Look through the whole compatibility matrix and see if there are other profiles that can be slimmed down by removing such redundancy. Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/compatibility_hints.py | 35 +++++++---------------------------- 1 file changed, 7 insertions(+), 28 deletions(-) diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index a9076e58..75abc57e 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -1503,7 +1503,6 @@ def compare(self, observed): bedework_3_10_3 = { ## If tests are yielding unexpected results, try to increase this: 'search-cache': {'behaviour': 'delay', 'delay': 3}, - 'scheduling.auto-schedule': {'support': 'unknown'}, 'scheduling.calendar-user-address-set': {'support': 'full'}, 'scheduling.freebusy-query': {'support': 'full'}, 'scheduling.mailbox': {'support': 'full'}, @@ -1583,7 +1582,6 @@ def compare(self, observed): "create-calendar.stable-url": {"support": "full"}, ## Not RFC properties; Bedework stores the Apple colour but not the order. "calendar-color": {"support": "full"}, - "calendar-color.hex": {"support": "full"}, "calendar-order": {"support": "unsupported"}, ## Bedework collections are typed, by default they can hold only @@ -1645,8 +1643,6 @@ def compare(self, observed): "behaviour": "a non-existing calendar raises ReportError instead of NotFoundError", }, "principal-search": {"support": "ungraceful"}, - "principal-search.by-name.self": {"support": "ungraceful"}, - "principal-search.list-all": {"support": "ungraceful"}, ## Works for CATEGORIES and CLASS, not for DTEND; the children are spelled ## out so the parent's "fragile" does not bleed down into them. @@ -1659,18 +1655,14 @@ def compare(self, observed): "search.time-range.comp-type-optional": {"support": "full"}, ## No text-match matches anything on a text property: a match on SUMMARY ## comes back empty for i;octet, i;ascii-casemap and i;unicode-casemap - ## alike, on the full property value as well as on a substring. The parent - ## has to carry that verdict - with only the three children below set it - ## resolved to its default "full", which claimed a text search Bedework - ## cannot do and silently disarmed the deliberate + ## alike, on the full property value as well as on a substring. The verdict + ## goes on the parent, which its children inherit - with only the children + ## set it resolved to its default "full", which claimed a text search + ## Bedework cannot do and silently disarmed the deliberate ## skip_unless_support("search.text") that keeps testEditSingleRecurrence - ## off this server. Enumerated properties are a different story and are - ## measured separately: a CLASS match does work, but only under - ## i;ascii-casemap, which is what search.text.case-sensitive records. + ## off this server. (A CLASS match does work, but only under + ## i;ascii-casemap.) "search.text": {"support": "unsupported"}, - "search.text.case-sensitive": {"support": "unsupported"}, - "search.text.case-insensitive": {"support": "unsupported"}, - "search.text.category": {"support": "unsupported"}, "search.time-range.alarm": {"support": "unsupported"}, ## The sync-token probe aborted on an ETagMismatchError (412) from its own @@ -1696,7 +1688,6 @@ def compare(self, observed): 'save-load.journal.mixed-calendar': {'support': 'unsupported'}, 'principal-search': {'support': 'ungraceful'}, 'principal-search.by-name.self': {'support': 'unsupported'}, - 'principal-search.list-all': {'support': 'ungraceful'}, #'sync-token.delete': {'support': 'unsupported'}, ## Perhaps on some older servers? ## extra properties not specified in RFC4791/RFC5545 "calendar-color": {"support": "full"}, @@ -1773,7 +1764,6 @@ def compare(self, observed): "search.time-range.alarm": { "support": "unsupported" }, 'sync-token': {'support': 'fragile'}, 'principal-search': {'support': 'unsupported'}, - 'principal-search.list-all': {'support': 'unsupported'}, ## DAViCal skips VTODOs without DTSTART in date-range searches. 'search.time-range.todo.no-dtstart': {'support': 'unsupported'}, "old_flags": [ @@ -2156,8 +2146,6 @@ def compare(self, observed): ## was: ungraceful - observed unsupported 2026-02 (for .old-dates) 'search.time-range.todo': {'support': 'fragile'}, 'principal-search': {'support': 'ungraceful'}, - 'principal-search.by-name.self': {'support': 'ungraceful'}, - 'principal-search.list-all': {'support': 'ungraceful'}, 'auto-connect.url': { 'basepath': '/webdav/', 'domain': 'purelymail.com', @@ -2284,7 +2272,6 @@ def compare(self, observed): ## datetime-event server-side expansion, and infinite scope remain unsupported. ## (event and exception expansion are left at the default "full".) 'search.recurrences.includes-implicit.todo': {'support': 'unsupported'}, - 'search.recurrences.includes-implicit.todo.pending': {'support': 'unsupported'}, 'search.recurrences.includes-implicit.infinite-scope': {'support': 'unsupported'}, 'search.recurrences.expanded.event': {'support': 'unsupported'}, 'search.recurrences.expanded.todo': {'support': 'unsupported'}, @@ -2302,17 +2289,12 @@ def compare(self, observed): ## search still returns the categorised event; a no_class search still ## returns the CONFIDENTIAL event). Same "filter ignored" behaviour as ## search.comp-type above - silently ignored, hence unsupported. + ## (DTEND included.) 'search.is-not-defined': {'support': 'unsupported'}, - 'search.is-not-defined.category': {'support': 'unsupported'}, - 'search.is-not-defined.class': {'support': 'unsupported'}, - ## is-not-defined for DTEND is not supported - 'search.is-not-defined.dtend': {'support': 'unsupported'}, ## Freebusy queries are not supported (returns 400) 'freebusy-query': {'support': 'ungraceful'}, ## Principal search not supported 'principal-search': {'support': 'unsupported'}, - 'principal-search.by-name.self': {'support': 'unsupported'}, - 'principal-search.list-all': {'support': 'unsupported'}, ## Cross-calendar duplicate UID test fails (AuthorizationError creating second calendar) 'save.duplicate-uid.cross-calendar': {'support': 'ungraceful'}, 'save-load.icalendar.related-to': {'support': 'broken'}, @@ -2358,7 +2340,6 @@ def compare(self, observed): ## returned the stale value, an artifact of the asynchronous writes above. ## Set explicitly to 'full' since the feature default is the weaker 'fragile'. 'calendar-color': {'support': 'full'}, - 'calendar-color.hex': {'support': 'full'}, 'calendar-order': {'support': 'full'}, ## The CALDAV comp-filter is silently ignored: a calendar-query that requests ## one component type returns the calendar's whole contents regardless (a @@ -2375,7 +2356,6 @@ def compare(self, observed): ## VTODO recurrence searching is not supported (datetime VEVENT recurrence ## search, including server-side expand and infinite scope, works fine). 'search.recurrences.includes-implicit.todo': {'support': 'unsupported'}, - 'search.recurrences.includes-implicit.todo.pending': {'support': 'unsupported'}, 'search.recurrences.expanded.todo': {'support': 'unsupported'}, ## Scheduling is advertised and the calendar-user-address-set and scheduling ## mailbox are present, but the server never returns a Schedule-Tag (neither @@ -2385,7 +2365,6 @@ def compare(self, observed): ## Principal search is effectively unsupported (lists nothing / errors out). 'principal-search': {'support': 'ungraceful'}, 'principal-search.by-name.self': {'support': 'unsupported'}, - 'principal-search.list-all': {'support': 'ungraceful'}, ## This was added 2026-08-28. I believe the compatibility tests have passed ## before. I don't think this part of the test suite has changed. It could ## be that the behaviour has changed at the server side. 418 was originally an From a1644207c8c5672b82f057b9470f594831215844 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Sun, 13 Sep 2026 10:56:21 +0200 Subject: [PATCH 08/18] fix: decode Bedework's percent-encoded PUT ETag Bedework 5 answers a PUT with ETag: %22...%22, where a GET gives the quoted form, and refuses the encoded form in If-Match with 412, so every second save() of an object raised ETagMismatchError. A leading %22 is not legal entity-tag syntax (RFC 9110 section 8.8.3), so _update_tag_props() decodes it. That error had mis-graded three Bedework 5 features, re-measured with the fix: cross-calendar duplicate UIDs and attendee PARTSTAT edits work, and sync-token works except for deletes. The new save.etag and save-load.mutable.if-match-wildcard are recorded for both profiles. Prompt: Work with [a handover document from another session] - it's needed with some probes to detect the bedework etag problems, mitigation in the caldav library to handle it, and an issue report in the bedework repository Followup-Prompt: (output of `pytest -k 'bede and compat'` pasted: attendee-partstat and duplicate-uid.cross-calendar observed full on Bedework 5, if-match-wildcard observed unsupported on Bedework 3) Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- CHANGELOG.md | 1 + caldav/calendarobjectresource.py | 10 +++++++-- caldav/compatibility_hints.py | 36 ++++++++++++++++++++++++-------- tests/test_schedule_tag.py | 27 ++++++++++++++++++++++++ 4 files changed, 63 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6972d8b1..f76de41d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 ### Fixed +* An object saved to Bedework 5 can be saved again. Bedework percent-encodes the quotes of the `ETag` header in a PUT response (`%22...%22`, where a GET gives `"..."`) and then refuses that form in `If-Match` with `412`, so every second `save()` raised `ETagMismatchError`. A leading `%22` is not legal entity-tag syntax (RFC 9110 §8.8.3), so it is decoded wherever the header is read. Recorded as `save.etag: broken` in the profile, next to the new `save-load.mutable.if-match-wildcard` feature for its refusal of `If-Match: *`. * Creating a calendar no longer fails on a server answering `MKCALENDAR`/`MKCOL` with a `207 Multi-Status` that reports nothing but success. RFC 4791 has the server answer `201 Created` and reserves the multistatus for reporting what could not be done, but Bedework 5 answers 207 as soon as the request carries properties - with every propstat `200 ok` and the calendar created and named. Insisting on 201 raised `MkcalendarError` for a calendar that was in fact there. The same applies to a `DAV:response` carrying neither a `DAV:status` nor a `DAV:propstat` — RFC 4918 §13 requires one or the other, and Bedework answers a property-less `MKCALENDAR` with nothing but the href of the collection it created — since nothing in such a body says the creation failed. A multistatus carrying a non-2xx status still raises, and one with no `DAV:response` at all is still not a success. * A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. diff --git a/caldav/calendarobjectresource.py b/caldav/calendarobjectresource.py index ddf1a5e3..56126b1c 100644 --- a/caldav/calendarobjectresource.py +++ b/caldav/calendarobjectresource.py @@ -18,7 +18,7 @@ from collections import defaultdict from datetime import datetime, timedelta, timezone from typing import TYPE_CHECKING, Any, ClassVar, Optional -from urllib.parse import ParseResult, SplitResult, quote +from urllib.parse import ParseResult, SplitResult, quote, unquote import icalendar from dateutil.rrule import rrulestr @@ -1196,7 +1196,13 @@ def _update_tag_props(self, r) -> None: if not r.headers: return if "Etag" in r.headers: - self.props[dav.GetEtag.tag] = r.headers["Etag"] + etag = r.headers["Etag"] + ## Bedework 5 percent-encodes the quotes in a PUT response and then + ## refuses that form in If-Match. RFC 9110 has an entity-tag start + ## with '"' or 'W/"', so a leading %22 can only be that encoding. + if etag.startswith(("%22", "W/%22")): + etag = unquote(etag) + self.props[dav.GetEtag.tag] = etag if r.headers.get("Schedule-Tag"): self.props[cdav.ScheduleTag.tag] = r.headers["Schedule-Tag"] diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 75abc57e..9e849aa5 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -464,6 +464,11 @@ class FeatureSet: "description": "The If-Match precondition is optional when overwriting an existing calendar object resource: the server accepts a PUT that carries no If-Match etag (i.e. add_event()/save() on an object that was not first fetched). When 'unsupported', the server requires an If-Match etag for updates and rejects a no-If-Match overwrite with 409 Conflict (e.g. OX App Suite enforces optimistic concurrency). Such servers still support save-load.mutable via a fetch-then-save (etag-conditional) update; only the blind-overwrite path is affected.", "default": {"support": "full"}, }, + "save-load.mutable.if-match-wildcard": { + "description": "An overwrite carrying If-Match: * is accepted. RFC 9110 section 13.1.1 has '*' match any current representation, so it means 'overwrite, but only if the object exists'. When 'unsupported', the server answers 412 Precondition Failed even though the object is there (Bedework 5). The library does not send If-Match: * itself.", + "default": {"support": "full"}, + "links": ["https://datatracker.ietf.org/doc/html/rfc9110#section-13.1.1"], + }, "search": { "description": "calendar MUST support searching for objects using the REPORT method, as specified in RFC4791, section 7", "links": ["https://datatracker.ietf.org/doc/html/rfc4791#section-7"], @@ -720,6 +725,11 @@ class FeatureSet: "description": "Server rejects requests with wrong password by returning an authorization error. Some servers may not properly reject wrong passwords in certain configurations." }, "save": {}, + "save.etag": { + "description": "The ETag header of a PUT response is a valid entity-tag (RFC 9110 section 8.8.3: a quoted string, optionally prefixed W/), and a conditional PUT carrying it in If-Match is accepted. 'quirk' with behaviour 'percent-encoded' when the header comes back URL-encoded but the decoded form is accepted - Bedework 5 answers a PUT with ETag: %22...%22 while a GET gives the quoted form, and refuses the encoded form in If-Match with 412. The library decodes a leading %22, so a client is not affected by that shape; 'broken' when the decoded form is refused too.", + "default": {"support": "full"}, + "links": ["https://datatracker.ietf.org/doc/html/rfc9110#section-8.8.3"], + }, "save.duplicate-uid": {}, "save.duplicate-uid.cross-calendar": { "description": "Server allows events with the same UID to exist in different calendars and treats them as separate entities. Support can be 'full' (allowed), 'ungraceful' (rejected with error), or 'unsupported' (silently ignored or moved). Behaviour 'silently-ignored' means the duplicate is not saved but no error is thrown. Behaviour 'moved-instead-of-copied' means the event is moved from the original calendar to the new calendar (Zimbra behavior)" @@ -1544,6 +1554,8 @@ def compare(self, observed): 'save-load.icalendar.related-to': {'support': 'broken', 'behaviour': 'first RELATED-TO line is preserved but subsequent RELATED-TO lines are stripped'}, ## Bedework omits DAV:resourcetype from an allprop PROPFIND response. "propfind.allprop.resourcetype": {"support": "unsupported"}, + ## If-Match: * is 412 here as on 5.0.0; the PUT etag is not encoded yet. + "save-load.mutable.if-match-wildcard": {"support": "unsupported"}, ## (The old 'duplicates_not_allowed' flag was stale: Bedework does store a ## second event with the same content under a different UID, so ## save.duplicate-event is left at the default "full".) @@ -1609,7 +1621,6 @@ def compare(self, observed): "search.time-range.todo": {"support": "unknown"}, "save-load.event.recurrences.exception": {"support": "unsupported"}, - "save-load.mutable.attendee-partstat": {"support": "unsupported"}, ## Seen 2026-09-13 in testChangeAttendeeStatusWithEmailGiven, which only ## started running once save-load.mutable.attendee-partstat came out full. "save-load.event.no-summary": { @@ -1633,10 +1644,14 @@ def compare(self, observed): "support": "broken", "behaviour": "first RELATED-TO line preserved but subsequent RELATED-TO lines are stripped", }, - "save.duplicate-uid.cross-calendar": { - "support": "ungraceful", - "behaviour": "Server error: ETagMismatchError", - }, + ## bw-webdav WebdavNsIntf.putContent URL-encodes the header, GetMethod does + ## not, and CaldavBWIntf.putEvent compares If-Match as a raw string. The + ## library decodes it. Before it did, every second save() raised + ## ETagMismatchError, which had graded save.duplicate-uid.cross-calendar + ## "ungraceful" and save-load.mutable.attendee-partstat "unsupported"; both + ## are full. + "save.etag": {"support": "quirk", "behaviour": "percent-encoded"}, + "save-load.mutable.if-match-wildcard": {"support": "unsupported"}, "non-existing-raises-not-found.collection": { "support": "unsupported", @@ -1665,10 +1680,13 @@ def compare(self, observed): "search.text": {"support": "unsupported"}, "search.time-range.alarm": {"support": "unsupported"}, - ## The sync-token probe aborted on an ETagMismatchError (412) from its own - ## setup in every run, the configured write-delay included, so nothing - ## about sync-collection has been measured. - "sync-token": {"support": "unknown"}, + ## Until the library decoded the PUT etag, the sync-token probe aborted on an + ## ETagMismatchError from its own setup. Measured 2026-09-13 after that: + ## sync-collection works, a delete does not show up in it. + "sync-token.delete": { + "support": "unsupported", + "behaviour": "the sync-collection report after a delete listed no changes", + }, ## One account is configured, so the cross-user half of scheduling is ## untested; the server advertises scheduling and the mailboxes are there. "scheduling.auto-schedule": {"support": "unknown"}, diff --git a/tests/test_schedule_tag.py b/tests/test_schedule_tag.py index e325ac78..7193d2f3 100644 --- a/tests/test_schedule_tag.py +++ b/tests/test_schedule_tag.py @@ -249,6 +249,33 @@ def test_etag_captured_from_put_response(self, mocked): assert event.props[dav.GetEtag.tag] == '"etag-from-put"' + @pytest.mark.parametrize( + "header, expected", + [ + ("%2220260912T213103Z-218e%22", '"20260912T213103Z-218e"'), + ("W/%22weak-tag%22", 'W/"weak-tag"'), + ## Legal etag syntax is left alone, even with a %22 inside the quotes + ('"a%22b"', '"a%22b"'), + ], + ) + @mock.patch("caldav.davclient.requests.Session.request") + def test_percent_encoded_etag_from_put_is_decoded(self, mocked, header, expected): + """Bedework 5 percent-encodes the quotes of the ETag in a PUT response + (a GET gives the plain form) and then refuses its own encoded etag in + If-Match with 412. RFC 9110 has an entity-tag start with a quote or + W/ and a quote, so a leading %22 can only be that encoding. + """ + mocked.return_value = _make_put_response(201, {"Etag": header}) + + event = _make_event_with_tag(None) + event.save() + assert event.props[dav.GetEtag.tag] == expected + + mocked.return_value = _make_put_response(204, {"Etag": header}) + event.save() + sent_headers = mocked.call_args.kwargs["headers"] + assert sent_headers["if-match"] == expected + @mock.patch("caldav.davclient.requests.Session.request") def test_302_on_put_updates_url(self, mocked): """A 302 in response to a PUT must follow the Location header.""" From fa1acd4a1e33b3b68f76e6289fd990ecc3811fff Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Sun, 13 Sep 2026 11:52:38 +0200 Subject: [PATCH 09/18] feat: turn write-delay into synchronous-write A server-peculiarity can be neither supported, unsupported or observable, but "is a write observable once the server answered it with success" can. full is yes, unsupported is an asynchronous server, fragile one too fast to observe reliably. The post-write sleep stays in the delay key, now read through the new write_delay() helper. As a server-feature it is compared against what caldav-server-tester measures. write-delay shipped in 3.3.0, so it is still accepted and translated with a DeprecationWarning. Bedework 5 is declared fragile, since the probe reads its writes back at once. Prompt: write-delay in compatbiliity_hints.py (sic) is a "server-peculiarity". It does not make sense to have it "supported" or "unsupported". However, maybe we can reverse it and turn it into a "feature" - "synchronized_write" or something like that. "supported" then means that the observable state should be updated when getting "200 OK", while "unsupported" means there may be a delay between the "200 OK" and until the state is stored and observable. "fragile" would mean that write-operations are expected to be async, but that they are fast enough that it can be hard to observe it. Does it make sense? If it makes sense, please implement. [caldav-server-tester] also needs some work. Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- CHANGELOG.md | 3 +- caldav/compatibility_hints.py | 54 +++++++++++++--- tests/docker-test-servers/bedework/README.md | 5 +- tests/test_async_integration.py | 11 ++-- tests/test_caldav.py | 13 ++-- tests/test_compatibility_hints.py | 68 +++++++++++++++++++- 6 files changed, 130 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f76de41d..19edc9ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,11 +17,12 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 ### Changed * `compatibility_hints`: the `bedework` profile is renamed `bedework_3_10_3`. It was only ever measured against `ioggstream/bedework:latest`, a `quickstart-3.10.3` tree built in 2018 that can no longer even be rebuilt, while upstream Bedework is alive and released 5.0.0 in 2025 - so a profile called `bedework` was claiming far more than we have observed. `features="bedework"` now raises a `ValueError` naming the new profile rather than an `AttributeError`. (`compatibility_hints` is declared unstable for the 3.x series.) +* `compatibility_hints`: the `write-delay` server-peculiarity is turned around into the `synchronous-write` server-feature. A peculiarity can be neither supported nor unsupported, while "is a change observable once the server answered 200" can: `full` (the default) is yes, `unsupported` is an asynchronous server, `fragile` one that settles too fast to observe reliably. The sleep a client should take after every write is still the `delay` key, now read through `compatibility_hints.write_delay()`. Being a server-feature it is compared against what caldav-server-tester measures. `write-delay` in a configuration is translated with a `DeprecationWarning`. ### Added * A test server for a current Bedework. `tests/docker-test-servers/bedework/` builds an image from the upstream Wildfly galleon feature pack (5.0.0), since there is no public image newer than 2018 and no single repository to build from. Three upstream bugs have to be patched at build time before the feature pack comes up at all - the pre-seeded H2 databases predate the H2 driver installed beside them, a shared module cannot load a class it needs through the war serving the request, and the bundled ApacheDS does not run on the JDK upstream prescribes. The 2018 image moves to `bedework3`. -* `compatibility_hints.bedework_5_0_0`: a profile for Bedework 5.0.0, measured 2026-09-12 with caldav-server-tester against the locally built test image as the demo user `vbede`. Nothing is inherited from `bedework_3_10_3` — 5.x creates and deletes calendars, and its text search, sync-token and principal search all behave differently. Two findings shape the rest of the profile. **Writes are asynchronous**: a read-back issued immediately after a PUT may 404 or hand back the pre-write copy, and that alone made `save-load.mutable`, `save-load.event.timezone` and `search.time-range.comp-type-optional` come out differently in two consecutive runs — the profile carries `write-delay: 3s`, without which a run measures the race rather than the server. And **a client cannot create a collection that holds tasks**: `MKCALENDAR`, extended `MKCOL` and `PROPPATCH` all answer `200 ok` for `CALDAV:supported-calendar-component-set` and then ignore it, so every collection a client creates is VEVENT-only and a VTODO or VJOURNAL PUT into one is 403. `vbede` has no usable `tasks` collection to fall back on either — the Depth:1 PROPFIND of its calendar home lists `tasks`, `Notifications` and `.pendingInbox` with a `getlastmodified` of "now" that is renewed on every listing, and all three 404 on any direct request — so the profile could not measure task support: `save-load.todo` and `search.time-range.todo` are graded `unknown`, and what was measured - an ignored component set and no tasks in an event calendar - is recorded as such. `douglm`, whose demo data ships a real `tasks` collection, can store tasks in it. The `BedeworkTestServer` in the test-server registry now uses the profile. +* `compatibility_hints.bedework_5_0_0`: a profile for Bedework 5.0.0, measured 2026-09-12 with caldav-server-tester against the locally built test image as the demo user `vbede`. Nothing is inherited from `bedework_3_10_3` — 5.x creates and deletes calendars, and its text search, sync-token and principal search all behave differently. Two findings shape the rest of the profile. **Writes are asynchronous**: a read-back issued immediately after a PUT may 404 or hand back the pre-write copy, and that alone made `save-load.mutable`, `save-load.event.timezone` and `search.time-range.comp-type-optional` come out differently in two consecutive runs — the profile declares `synchronous-write` fragile (the dedicated probe does not catch it) with a 3s `delay`, without which a run measures the race rather than the server. And **a client cannot create a collection that holds tasks**: `MKCALENDAR`, extended `MKCOL` and `PROPPATCH` all answer `200 ok` for `CALDAV:supported-calendar-component-set` and then ignore it, so every collection a client creates is VEVENT-only and a VTODO or VJOURNAL PUT into one is 403. `vbede` has no usable `tasks` collection to fall back on either — the Depth:1 PROPFIND of its calendar home lists `tasks`, `Notifications` and `.pendingInbox` with a `getlastmodified` of "now" that is renewed on every listing, and all three 404 on any direct request — so the profile could not measure task support: `save-load.todo` and `search.time-range.todo` are graded `unknown`, and what was measured - an ignored component set and no tasks in an event calendar - is recorded as such. `douglm`, whose demo data ships a real `tasks` collection, can store tasks in it. The `BedeworkTestServer` in the test-server registry now uses the profile. ### Fixed diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 9e849aa5..25f4f271 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -287,13 +287,12 @@ class FeatureSet: "delay": "after this number of seconds, we may be reasonably sure that the search results are updated", } }, - "write-delay": { - "type": "server-peculiarity", + "synchronous-write": { "default": {"support": "full"}, - "description": "The server processes write operations (PUT/DELETE/MKCALENDAR/PROPPATCH/...) asynchronously: the request returns success before the change has fully taken effect, so an immediate read-back (of any kind, not just a search) may 404 or return stale data. A client must wait a bit after every write. This is the general, write-side counterpart of 'search-cache' (which only delays searches). 'full' (the default) means writes take effect synchronously.", + "description": "A write operation (PUT/DELETE/MKCALENDAR/PROPPATCH/...) has taken effect by the time the server answers it with success, so an immediate read-back of any kind - not just a search - observes the change. 'full' (the default) is that. 'unsupported' means the server processes writes asynchronously: there may be a delay between the success response and the change being stored and observable, so an immediate read-back may 404 or return stale data. 'fragile' means writes are asynchronous too, but settle fast enough that the delay is hard to observe - a single probe will usually read it as 'full'. Where a 'delay' is given, a client should sleep that long after every write before relying on the change (see write_delay()). This is the general, write-side counterpart of 'search-cache' (which only delays searches). Formerly the 'write-delay' server-peculiarity, still accepted in a configuration and translated.", "extra_keys": { - "behaviour": "'delay' to enable the post-write sleep", - "delay": "sleep this number of seconds after every write request before relying on the change being visible", + "delay": "sleep this number of seconds after every write request before relying on the change being visible. Ignored when the support is 'full'", + "save-load-delay": "observed by caldav-server-tester: seconds until a freshly PUT object could be read back", } }, "tests-cleanup-calendar": { @@ -814,6 +813,15 @@ def copyFeatureSet(self, feature_set, collapse=True): if feature == 'old_flags': self._old_flags = feature_set[feature] continue + if feature == 'write-delay': + warnings.warn( + "The 'write-delay' feature is deprecated - use 'synchronous-write', " + "e.g. {'support': 'unsupported', 'delay': 3} for {'behaviour': 'delay', 'delay': 3}", + DeprecationWarning, + stacklevel=3, + ) + self.copyFeatureSet({'synchronous-write': _from_write_delay(feature_set[feature])}, collapse=False) + continue try: ## called for the exception, not the return value: an unknown ## feature name is a typo in the configuration and gets a warning @@ -1574,8 +1582,11 @@ def compare(self, observed): ## measurement runs - save-load.mutable came out "broken" (modification not ## reflected after save and reload) in one and "full" in the other, and the ## timezone probe's load() 404ed on a resource the PUT had just accepted. - ## The delay is what makes the rest of this profile reproducible. - "write-delay": {"behaviour": "delay", "delay": 3}, + ## The delay is what makes the rest of this profile reproducible. Yet the + ## synchronous-write probe itself - one PUT, one direct GET - reads the + ## object back at once (save-load-delay 0, 2026-09-13), so it is too fast + ## to catch reliably: fragile, not unsupported. + "synchronous-write": {"support": "fragile", "delay": 3}, ## MKCALENDAR works, but a successful one that sets no properties is ## answered with a 207 whose DAV:response carries neither a DAV:status nor @@ -2349,7 +2360,7 @@ def compare(self, observed): ## before the change is queryable, so an immediate read-back 404s or returns ## stale data for several seconds. This is server-wide (not just searches), ## so we sleep after every write rather than only before searches. - 'write-delay': {'behaviour': 'delay', 'delay': 16}, + 'synchronous-write': {'support': 'unsupported', 'delay': 16}, ## VJOURNAL is not supported. 'save-load.journal': {'support': 'unsupported'}, ## Calendar colour/order work once the post-write delay is honoured (the @@ -2486,3 +2497,30 @@ def at_spelling_is_significant(features: Any) -> bool: encoded on the way out. """ return not at_spellings_are_aliased(features) + + +def write_delay(features: Any) -> float: + """Seconds to sleep after every write request before reading the change back. + + Read off ``synchronous-write``: its ``delay``, unless writes are declared + synchronous. An asynchronous server with no ``delay`` given yields 0 - + nobody has said how long to wait, and guessing would slow every write. + """ + if features is None: + return 0 + node = features.is_supported("synchronous-write", dict) + if node.get("support", "full") == "full": + return 0 + return node.get("delay", 0) + + +def _from_write_delay(value: Any) -> Any: + """Translate a value of the deprecated ``write-delay`` peculiarity.""" + if not isinstance(value, dict): + return value + if value.get("behaviour") != "delay": + return {"support": "full"} + new = {"support": "unsupported"} + if "delay" in value: + new["delay"] = value["delay"] + return new diff --git a/tests/docker-test-servers/bedework/README.md b/tests/docker-test-servers/bedework/README.md index 41777326..8cd3520d 100644 --- a/tests/docker-test-servers/bedework/README.md +++ b/tests/docker-test-servers/bedework/README.md @@ -93,8 +93,9 @@ Two things are worth knowing before reading a measurement against it: - **Writes are asynchronous.** A read-back issued immediately after a PUT may 404 or hand back the pre-write copy, which made `save-load.mutable`, `save-load.event.timezone` and `search.time-range.comp-type-optional` come - out differently in two consecutive runs. The profile carries - `write-delay: 3s`; without it a run measures the race rather than the server. + out differently in two consecutive runs. The profile declares + `synchronous-write` fragile - the dedicated probe does not catch it - with a + 3s `delay`; without it a run measures the race rather than the server. - **A client cannot create a collection that holds tasks.** MKCALENDAR and extended MKCOL both answer `200 ok` for `CALDAV:supported-calendar-component-set` and then ignore it, and a diff --git a/tests/test_async_integration.py b/tests/test_async_integration.py index 51fc37f2..5872926f 100644 --- a/tests/test_async_integration.py +++ b/tests/test_async_integration.py @@ -18,7 +18,7 @@ import pytest_asyncio from caldav import Event, FreeBusy, Todo -from caldav.compatibility_hints import FeatureSet +from caldav.compatibility_hints import FeatureSet, write_delay from caldav.lib import error from .test_caldav import ( @@ -59,7 +59,7 @@ async def wrapper(*args, **kwargs): return wrapper -## HTTP methods that change server state; a "write-delay" server settles each of +## HTTP methods that change server state; a server without "synchronous-write" settles each of ## these asynchronously, so we sleep AFTER every such request (the write-side ## counterpart of the search-cache delay, which only delays searches). _WRITE_HTTP_METHODS = frozenset( @@ -245,11 +245,10 @@ async def async_client(self, test_server: TestServer, monkeypatch: Any) -> Any: _async_delay_decorator(AsyncCalendar.search, t=delay), ) - ## Apply write-delay (sleep after every write) for asynchronous servers. + ## Sleep after every write for servers without synchronous writes. ## Wrapped on the client instance, so monkeypatch reverts it after the test. - write_delay_config = client.features.is_supported("write-delay", dict) - if write_delay_config.get("behaviour") == "delay": - delay = write_delay_config.get("delay", 10) + delay = write_delay(client.features) + if delay: monkeypatch.setattr( client, "request", diff --git a/tests/test_caldav.py b/tests/test_caldav.py index 8c13eced..f0a844e6 100644 --- a/tests/test_caldav.py +++ b/tests/test_caldav.py @@ -53,8 +53,9 @@ rfc6638_users = _config.get("rfc6638_users", []) from caldav import Calendar, DAVObject, Event, FreeBusy, Principal, Todo from caldav.compatibility_hints import ( - incompatibility_description, -) ## TEMP - should be removed in the future + incompatibility_description, ## TEMP - should be removed in the future + write_delay, +) from caldav.davclient import CONNKEYS, DAVClient, DAVResponse from caldav.elements import cdav, dav, ical from caldav.lib import error @@ -1320,7 +1321,7 @@ def foo(*a, **kwa): return foo -## HTTP methods that change server state. A "write-delay" server settles each of +## HTTP methods that change server state. A server without "synchronous-write" settles each of ## these asynchronously, so we sleep AFTER every such request to let the change ## become visible before the test reads it back (the general, write-side ## counterpart of the search-cache delay, which only delays searches). @@ -1414,12 +1415,12 @@ def setup_method(self): if foo.get("behaviour") == "delay": Calendar._search = Calendar.search Calendar.search = _delay_decorator(Calendar.search, t=foo["delay"]) - foo = self.is_supported("write-delay", dict) - if foo.get("behaviour") == "delay": + delay = write_delay(self.caldav.features) + if delay: ## Every write goes through the client request(); sleep after the ## write verbs so the asynchronous change has settled before read-back. ## Instance-level wrap (like rate-limit), torn down with the client. - self.caldav.request = _write_delay_decorator(self.caldav.request, t=foo["delay"]) + self.caldav.request = _write_delay_decorator(self.caldav.request, t=delay) if False and self.check_compatibility_flag("no-current-user-principal"): self.principal = Principal(client=self.caldav, url=self.server_params["principal_url"]) diff --git a/tests/test_compatibility_hints.py b/tests/test_compatibility_hints.py index 3219ea4e..79cfe5f0 100644 --- a/tests/test_compatibility_hints.py +++ b/tests/test_compatibility_hints.py @@ -11,7 +11,7 @@ import pytest -from caldav.compatibility_hints import VALID_SUPPORT_LEVELS, FeatureSet +from caldav.compatibility_hints import VALID_SUPPORT_LEVELS, FeatureSet, write_delay from caldav.config import resolve_features as _resolve_features @@ -793,3 +793,69 @@ def test_a_server_that_starts_answering_404_is_reported(self) -> None: observed.set_feature("non-existing-raises-not-found.collection", "unsupported") mismatches = {m["feature"]: m for m in declared.compare(observed)} assert "non-existing-raises-not-found.object" in mismatches + + +class TestSynchronousWrite: + """``synchronous-write``: is a change observable once the server said 200? + + Formerly the ``write-delay`` server-peculiarity, which could be neither + supported nor unsupported. Turned around it is a server-feature, and the + `delay` a client sleeps after every write is read off it by + :func:`write_delay`. + """ + + def test_writes_are_synchronous_by_default(self) -> None: + features = FeatureSet() + assert features.is_supported("synchronous-write") + assert write_delay(features) == 0 + assert write_delay(None) == 0 + + @pytest.mark.parametrize( + ("value", "expected"), + [ + ({"support": "unsupported", "delay": 16}, 16), + ## async, but fast enough to be hard to observe - still worth a nap + ({"support": "fragile", "delay": 1}, 1), + ## asynchronous, but nobody has said how long to wait + ({"support": "unsupported"}, 0), + ## a delay on a synchronous server is a contradiction; do not sleep + ({"support": "full", "delay": 16}, 0), + ], + ) + def test_write_delay(self, value: dict, expected: int) -> None: + features = FeatureSet({"synchronous-write": value}) + assert write_delay(features) == expected + + @pytest.mark.parametrize( + ("old", "new"), + [ + ({"behaviour": "delay", "delay": 3}, {"support": "unsupported", "delay": 3}), + ({"behaviour": "normal"}, {"support": "full"}), + ], + ) + def test_write_delay_is_a_deprecated_alias(self, old: dict, new: dict) -> None: + """``write-delay`` shipped in 3.3.0, so configurations carrying it keep working.""" + with pytest.warns(DeprecationWarning, match="synchronous-write"): + features = FeatureSet({"write-delay": old}) + assert features.is_supported("synchronous-write", dict) == new + assert "write-delay" not in features.dotted_feature_set_list() + + @pytest.mark.parametrize(("profile", "delay"), [("bedework_5_0_0", 3), ("infomaniak", 16)]) + def test_profiles(self, profile: str, delay: int) -> None: + features = FeatureSet(_resolve_features(profile)) + assert not features.is_supported("synchronous-write") + assert write_delay(features) == delay + + def test_is_compared(self) -> None: + """A server-feature the tester measures, unlike the peculiarity it replaces.""" + declared = FeatureSet({"synchronous-write": {"support": "unsupported", "delay": 3}}) + observed = FeatureSet() + observed.set_feature("synchronous-write", {"support": "full", "save-load-delay": 0}) + assert [m["feature"] for m in declared.compare(observed)] == ["synchronous-write"] + + def test_fragile_is_not_compared(self) -> None: + """Too fast to observe reliably - a single run reading 'full' is no news.""" + declared = FeatureSet({"synchronous-write": {"support": "fragile", "delay": 1}}) + observed = FeatureSet() + observed.set_feature("synchronous-write", {"support": "full", "save-load-delay": 0}) + assert declared.compare(observed) == [] From 6dfbd79097adcbf624ae9528f379e40d553d7323 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Sun, 13 Sep 2026 17:01:08 +0200 Subject: [PATCH 10/18] fix: merge expanded instances sharing one href Bedework 5 answers an expanded calendar-query with one DAV:response per recurrence instance, all under the href of the resource - RFC 4918 section 14.24 forbids that. The multistatus parser let each later calendar-data overwrite the earlier, so every expanded search lost all but the last occurrence. Repeated calendar-data is now merged into one VCALENDAR, skipping components already present (a VTIMEZONE is matched by its TZID). This fixes testRecurringDateSearch and testRecurringDateWithExceptionSearch, sync and async, against Bedework 5. Prompt: There is a handover document here, another agent has been working with it, but it did not do the "B and C", recurring-search failures for the bedework server Followup-Prompt: Is this actually a bug in the caldav library and not in Bedework? Caldav library should be fixed. If Bedework behaves according to the RFC, but still differently from other servers, then let the server checker probe and add a behaviour note for search.recurrences.expanded perhaps? AI-assisted: Claude Opus 5 via Claude Code Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- CHANGELOG.md | 1 + caldav/response.py | 41 ++++++++++++++++++- tests/test_caldav_unit.py | 86 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 127 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 19edc9ec..a58d3000 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 * An object saved to Bedework 5 can be saved again. Bedework percent-encodes the quotes of the `ETag` header in a PUT response (`%22...%22`, where a GET gives `"..."`) and then refuses that form in `If-Match` with `412`, so every second `save()` raised `ETagMismatchError`. A leading `%22` is not legal entity-tag syntax (RFC 9110 §8.8.3), so it is decoded wherever the header is read. Recorded as `save.etag: broken` in the profile, next to the new `save-load.mutable.if-match-wildcard` feature for its refusal of `If-Match: *`. * Creating a calendar no longer fails on a server answering `MKCALENDAR`/`MKCOL` with a `207 Multi-Status` that reports nothing but success. RFC 4791 has the server answer `201 Created` and reserves the multistatus for reporting what could not be done, but Bedework 5 answers 207 as soon as the request carries properties - with every propstat `200 ok` and the calendar created and named. Insisting on 201 raised `MkcalendarError` for a calendar that was in fact there. The same applies to a `DAV:response` carrying neither a `DAV:status` nor a `DAV:propstat` — RFC 4918 §13 requires one or the other, and Bedework answers a property-less `MKCALENDAR` with nothing but the href of the collection it created — since nothing in such a body says the creation failed. A multistatus carrying a non-2xx status still raises, and one with no `DAV:response` at all is still not a success. +* An expanded search against Bedework 5 no longer drops all but the last occurrence of a recurring event. Bedework returns each expanded instance in a `DAV:response` of its own, all under the href of the resource, which RFC 4918 §14.24 forbids; the multistatus parser let every later `calendar-data` overwrite the earlier ones. Repeated `calendar-data` for one href is now merged into a single `VCALENDAR`, the shape RFC 4791 §7.8.3 shows. Recorded as `search.recurrences.expanded.event: quirk` with behaviour `response-per-instance` in the `bedework_5_0_0` profile. * A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. ## [3.3.0] - 2026-09-03 diff --git a/caldav/response.py b/caldav/response.py index 45a138d8..2bbd6b72 100644 --- a/caldav/response.py +++ b/caldav/response.py @@ -9,6 +9,7 @@ from typing import TYPE_CHECKING, Any, cast from urllib.parse import unquote +import icalendar from lxml import etree from lxml.etree import _Element @@ -202,6 +203,33 @@ def _element_to_value(elem: _Element) -> Any: return elem +def _merge_calendar_data(earlier: str, later: str) -> str: + """Add the components of ``later`` that ``earlier`` does not already hold. + + A component is identified by its name, UID and RECURRENCE-ID, so a + response repeated verbatim does not duplicate anything. A VTIMEZONE has + no UID and is identified by its TZID; any other UID-less component by its + full content. + """ + merged = icalendar.Calendar.from_ical(earlier) + + def key(component: icalendar.Component) -> tuple: + if component.name == "VTIMEZONE": + return (component.name, str(component.get("TZID"))) + uid = component.get("UID") + if uid is None: + return (component.name, component.to_ical()) + rid = component.get("RECURRENCE-ID") + return (component.name, str(uid), rid.to_ical() if rid else None) + + seen = {key(c) for c in merged.subcomponents} + for component in icalendar.Calendar.from_ical(later).subcomponents: + if key(component) not in seen: + seen.add(key(component)) + merged.add_component(component) + return merged.to_ical().decode() + + class DAVResponse: """ Base class containing shared response parsing logic. @@ -707,7 +735,18 @@ def _find_objects_and_props(self) -> dict[str, dict[str, _Element]]: ## with multiple props or in multiple propstats; the 404-skip ## quirk is shared with the dataclass parsers via ## _collect_prop_elements (code-review §5.7). - self.objects[href].update(_collect_prop_elements(propstats)) + props = _collect_prop_elements(propstats) + + ## RFC 4918 section 14.24 forbids an href to appear twice, but + ## Bedework 5 answers an expanded calendar-query with one response + ## per recurrence instance, all under the href of the resource. + ## Merge the instances rather than let the last one overwrite them. + earlier = self.objects[href].get(cdav.CalendarData.tag) + later = props.get(cdav.CalendarData.tag) + if earlier is not None and later is not None and earlier.text and later.text: + later.text = _merge_calendar_data(earlier.text, later.text) + + self.objects[href].update(props) return self.objects diff --git a/tests/test_caldav_unit.py b/tests/test_caldav_unit.py index 60b6c5aa..87fb1912 100755 --- a/tests/test_caldav_unit.py +++ b/tests/test_caldav_unit.py @@ -3990,6 +3990,92 @@ def test_explicit_password_still_pairs_with_the_url_username(self): assert client.password == b"s3cret" +class TestRepeatedHrefCalendarData: + """RFC 4918 section 14.24 forbids an href to appear in more than one + DAV:response, but Bedework 5 answers an expanded calendar-query with one + response per recurrence instance, all under the href of the resource.""" + + @staticmethod + def _response(recurrence_id: str) -> str: + return f""" + /ucaldav/user/vbede/cal/yearly.ics + + + + + HTTP/1.1 200 ok + + +""" + + def _calendar_data(self, *recurrence_ids: str) -> str: + xml = ( + '\n' + + "".join(self._response(rid) for rid in recurrence_ids) + + "" + ) + result = MockedDAVResponse(xml).expand_simple_props(props=[cdav.CalendarData()]) + assert list(result) == ["/ucaldav/user/vbede/cal/yearly.ics"] + return result["/ucaldav/user/vbede/cal/yearly.ics"][cdav.CalendarData.tag] + + def test_instances_are_merged(self) -> None: + cal = icalendar.Calendar.from_ical(self._calendar_data("20261102", "20271102")) + recurrence_ids = [str(e["RECURRENCE-ID"].to_ical(), "ascii") for e in cal.walk("VEVENT")] + assert recurrence_ids == ["20261102", "20271102"] + + def test_repeated_instance_is_not_duplicated(self) -> None: + cal = icalendar.Calendar.from_ical(self._calendar_data("20261102", "20261102")) + assert len(cal.walk("VEVENT")) == 1 + + @staticmethod + def _with_timezone(tzid: str, uid: str) -> str: + return ( + "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//test//EN\r\n" + f"BEGIN:VTIMEZONE\r\nTZID:{tzid}\r\n" + "BEGIN:STANDARD\r\nDTSTART:19701025T030000\r\n" + "TZOFFSETFROM:+0200\r\nTZOFFSETTO:+0100\r\nEND:STANDARD\r\n" + "END:VTIMEZONE\r\n" + f"BEGIN:VEVENT\r\nUID:{uid}\r\nDTSTAMP:20260913T141843Z\r\n" + f"DTSTART;TZID={tzid}:20261102T100000\r\nSUMMARY:{uid}\r\nEND:VEVENT\r\n" + "END:VCALENDAR\r\n" + ) + + def test_distinct_timezones_are_all_kept(self) -> None: + from caldav.response import _merge_calendar_data + + merged = _merge_calendar_data( + self._with_timezone("Europe/Oslo", "a"), + self._with_timezone("America/New_York", "b"), + ) + tzids = sorted( + str(tz["TZID"]) for tz in icalendar.Calendar.from_ical(merged).walk("VTIMEZONE") + ) + assert tzids == ["America/New_York", "Europe/Oslo"] + + def test_repeated_timezone_is_not_duplicated(self) -> None: + from caldav.response import _merge_calendar_data + + merged = _merge_calendar_data( + self._with_timezone("Europe/Oslo", "a"), + self._with_timezone("Europe/Oslo", "b"), + ) + cal = icalendar.Calendar.from_ical(merged) + assert len(cal.walk("VTIMEZONE")) == 1 + assert len(cal.walk("VEVENT")) == 2 + + class TestPropstatStatusValidation: """Gate finding F7: a failing propstat status must raise, not vanish. From d0db435cf67e611443956af9c79098f00917c6d3 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Sun, 13 Sep 2026 16:28:22 +0200 Subject: [PATCH 11/18] test: minor test, doc and CHANGELOG tweaks Hand edits by Tobias, with some AI-based QA. The unreleased CHANGELOG is cut down and tightened. Tests no longer lock the write-delay amounts for Infomaniak and Bedework 5, Bedework's time-based sync-tokens are configured so the tests sleep, and the synchronous-write documentation is shortened. The Cyrus start.sh no longer runs `docker-compose down` before `up -d`, which killed a live server and wiped a concurrent test run's state. Prompt: It sounds like an asymmetry that cyrus runs down when starting it [docker-compose down in tests/docker-test-servers/cyrus/start.sh], please fix. Followup-Prompt: (comment from the review process) The next three commits can most certainly be squashed together - even if they are quite unreleated (sic), they are all minor commits. 38807ec9 also seems like a minor commit. a913b81a also seems like something that can be folded neatly into such a commit AI-assisted: Claude Opus 5 via Claude Code Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- CHANGELOG.md | 14 +++++--------- caldav/compatibility_hints.py | 9 +++++++-- tests/docker-test-servers/cyrus/start.sh | 5 ----- tests/test_compatibility_hints.py | 4 ++-- 4 files changed, 14 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a58d3000..a684ae6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,21 +14,17 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 ## [Unreleased] -### Changed +### Added -* `compatibility_hints`: the `bedework` profile is renamed `bedework_3_10_3`. It was only ever measured against `ioggstream/bedework:latest`, a `quickstart-3.10.3` tree built in 2018 that can no longer even be rebuilt, while upstream Bedework is alive and released 5.0.0 in 2025 - so a profile called `bedework` was claiming far more than we have observed. `features="bedework"` now raises a `ValueError` naming the new profile rather than an `AttributeError`. (`compatibility_hints` is declared unstable for the 3.x series.) -* `compatibility_hints`: the `write-delay` server-peculiarity is turned around into the `synchronous-write` server-feature. A peculiarity can be neither supported nor unsupported, while "is a change observable once the server answered 200" can: `full` (the default) is yes, `unsupported` is an asynchronous server, `fragile` one that settles too fast to observe reliably. The sleep a client should take after every write is still the `delay` key, now read through `compatibility_hints.write_delay()`. Being a server-feature it is compared against what caldav-server-tester measures. `write-delay` in a configuration is translated with a `DeprecationWarning`. +* Support for Bedework 5.0. The earlier Bedework tests were targeting the docker image `ioggstream/bedework:latest` which has been locked towards Bedework 3.10.3 since 2018. Now there is a script for building a bedework container in the docker test servers. The old Bedework docker image has been kept, but renamed into bedework3. Workarounds for various server quirks have been implemented. -### Added +### Changed -* A test server for a current Bedework. `tests/docker-test-servers/bedework/` builds an image from the upstream Wildfly galleon feature pack (5.0.0), since there is no public image newer than 2018 and no single repository to build from. Three upstream bugs have to be patched at build time before the feature pack comes up at all - the pre-seeded H2 databases predate the H2 driver installed beside them, a shared module cannot load a class it needs through the war serving the request, and the bundled ApacheDS does not run on the JDK upstream prescribes. The 2018 image moves to `bedework3`. -* `compatibility_hints.bedework_5_0_0`: a profile for Bedework 5.0.0, measured 2026-09-12 with caldav-server-tester against the locally built test image as the demo user `vbede`. Nothing is inherited from `bedework_3_10_3` — 5.x creates and deletes calendars, and its text search, sync-token and principal search all behave differently. Two findings shape the rest of the profile. **Writes are asynchronous**: a read-back issued immediately after a PUT may 404 or hand back the pre-write copy, and that alone made `save-load.mutable`, `save-load.event.timezone` and `search.time-range.comp-type-optional` come out differently in two consecutive runs — the profile declares `synchronous-write` fragile (the dedicated probe does not catch it) with a 3s `delay`, without which a run measures the race rather than the server. And **a client cannot create a collection that holds tasks**: `MKCALENDAR`, extended `MKCOL` and `PROPPATCH` all answer `200 ok` for `CALDAV:supported-calendar-component-set` and then ignore it, so every collection a client creates is VEVENT-only and a VTODO or VJOURNAL PUT into one is 403. `vbede` has no usable `tasks` collection to fall back on either — the Depth:1 PROPFIND of its calendar home lists `tasks`, `Notifications` and `.pendingInbox` with a `getlastmodified` of "now" that is renewed on every listing, and all three 404 on any direct request — so the profile could not measure task support: `save-load.todo` and `search.time-range.todo` are graded `unknown`, and what was measured - an ignored component set and no tasks in an event calendar - is recorded as such. `douglm`, whose demo data ships a real `tasks` collection, can store tasks in it. The `BedeworkTestServer` in the test-server registry now uses the profile. +* **Breaking:** the `bedework` compatibility profile is renamed `bedework_3_10_3`, and `bedework_5_0_0` is added. `features: bedework` or `base: bedework` in a config now raises a `ValueError` naming both. +* `compatibility_hints`: the `write-delay` server-peculiarity is turned around into the `synchronous-write` server-feature. The delay (relevant for tests and the caldav server tester) should still be hand-configured. ### Fixed -* An object saved to Bedework 5 can be saved again. Bedework percent-encodes the quotes of the `ETag` header in a PUT response (`%22...%22`, where a GET gives `"..."`) and then refuses that form in `If-Match` with `412`, so every second `save()` raised `ETagMismatchError`. A leading `%22` is not legal entity-tag syntax (RFC 9110 §8.8.3), so it is decoded wherever the header is read. Recorded as `save.etag: broken` in the profile, next to the new `save-load.mutable.if-match-wildcard` feature for its refusal of `If-Match: *`. -* Creating a calendar no longer fails on a server answering `MKCALENDAR`/`MKCOL` with a `207 Multi-Status` that reports nothing but success. RFC 4791 has the server answer `201 Created` and reserves the multistatus for reporting what could not be done, but Bedework 5 answers 207 as soon as the request carries properties - with every propstat `200 ok` and the calendar created and named. Insisting on 201 raised `MkcalendarError` for a calendar that was in fact there. The same applies to a `DAV:response` carrying neither a `DAV:status` nor a `DAV:propstat` — RFC 4918 §13 requires one or the other, and Bedework answers a property-less `MKCALENDAR` with nothing but the href of the collection it created — since nothing in such a body says the creation failed. A multistatus carrying a non-2xx status still raises, and one with no `DAV:response` at all is still not a success. -* An expanded search against Bedework 5 no longer drops all but the last occurrence of a recurring event. Bedework returns each expanded instance in a `DAV:response` of its own, all under the href of the resource, which RFC 4918 §14.24 forbids; the multistatus parser let every later `calendar-data` overwrite the earlier ones. Repeated `calendar-data` for one href is now merged into a single `VCALENDAR`, the shape RFC 4791 §7.8.3 shows. Recorded as `search.recurrences.expanded.event: quirk` with behaviour `response-per-instance` in the `bedework_5_0_0` profile. * A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. ## [3.3.0] - 2026-09-03 diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 25f4f271..3f65996f 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -289,7 +289,7 @@ class FeatureSet: }, "synchronous-write": { "default": {"support": "full"}, - "description": "A write operation (PUT/DELETE/MKCALENDAR/PROPPATCH/...) has taken effect by the time the server answers it with success, so an immediate read-back of any kind - not just a search - observes the change. 'full' (the default) is that. 'unsupported' means the server processes writes asynchronously: there may be a delay between the success response and the change being stored and observable, so an immediate read-back may 404 or return stale data. 'fragile' means writes are asynchronous too, but settle fast enough that the delay is hard to observe - a single probe will usually read it as 'full'. Where a 'delay' is given, a client should sleep that long after every write before relying on the change (see write_delay()). This is the general, write-side counterpart of 'search-cache' (which only delays searches). Formerly the 'write-delay' server-peculiarity, still accepted in a configuration and translated.", + "description": "A write operation is complete and immediately observable when receiving a 2XX-response from the server. 'unsupported' means the server processes writes asynchronously: the change request is received and may be queued up. For a fast server it may not be possible to reliably observe that it's asynchronous, 'fragile' can be used if the probe is non-deterministic or if a server know to be async is observed to be sync. A delay can be given, and the test code will sleep with the configured delay after each write operation", "extra_keys": { "delay": "sleep this number of seconds after every write request before relying on the change being visible. Ignored when the support is 'full'", "save-load-delay": "observed by caldav-server-tester: seconds until a freshly PUT object could be read back", @@ -1693,7 +1693,12 @@ def compare(self, observed): ## Until the library decoded the PUT etag, the sync-token probe aborted on an ## ETagMismatchError from its own setup. Measured 2026-09-13 after that: - ## sync-collection works, a delete does not show up in it. + ## the token is a timestamp of second precision (data:,20260913T125156Z-1d6a), + ## and a change made in the same second as the token was handed out is not + ## reported (0/5 with no pause, 3/3 after 1.5s). The tester graded it plain + ## "full" because its own modification happens to land a second later. A + ## delete is never reported, pause or not (0/11). + "sync-token": {"support": "full", "behaviour": "time-based"}, "sync-token.delete": { "support": "unsupported", "behaviour": "the sync-collection report after a delete listed no changes", diff --git a/tests/docker-test-servers/cyrus/start.sh b/tests/docker-test-servers/cyrus/start.sh index 30872471..8bcaf0ee 100755 --- a/tests/docker-test-servers/cyrus/start.sh +++ b/tests/docker-test-servers/cyrus/start.sh @@ -8,11 +8,6 @@ set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$SCRIPT_DIR" -# Clean up any existing container to ensure fresh state -# (No volumes used, so each start creates fresh users) -echo "Cleaning up previous Cyrus instance..." -docker-compose down 2>/dev/null || true - echo "Starting Cyrus IMAP CalDAV server..." docker-compose up -d diff --git a/tests/test_compatibility_hints.py b/tests/test_compatibility_hints.py index 79cfe5f0..247e1915 100644 --- a/tests/test_compatibility_hints.py +++ b/tests/test_compatibility_hints.py @@ -840,11 +840,11 @@ def test_write_delay_is_a_deprecated_alias(self, old: dict, new: dict) -> None: assert features.is_supported("synchronous-write", dict) == new assert "write-delay" not in features.dotted_feature_set_list() - @pytest.mark.parametrize(("profile", "delay"), [("bedework_5_0_0", 3), ("infomaniak", 16)]) + @pytest.mark.parametrize(("profile", "delay"), [("bedework_5_0_0", 1), ("infomaniak", 3)]) def test_profiles(self, profile: str, delay: int) -> None: features = FeatureSet(_resolve_features(profile)) assert not features.is_supported("synchronous-write") - assert write_delay(features) == delay + assert write_delay(features) >= delay def test_is_compared(self) -> None: """A server-feature the tester measures, unlike the peculiarity it replaces.""" From 11d05221b3c6d4f9ae399c8c1920cc54e358dfa7 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 08:49:59 +0200 Subject: [PATCH 12/18] test: put first compat mismatch on summary line pytest's short summary shows only the first line of an assertion message, which for testCheckCompatibility was a bare "compatibility mismatches:". It now carries the count and the first mismatch. Prompt: The output from the compatibility test looks like this: (nine "FAILED ...::testCheckCompatibility - AssertionError: compatibility mismatches:" lines pasted) The annoying part here is that the error is terminated after the colon. Now I have to dig through the (huge!) test logs to see exactly what kind of compatibility failure it is and it's hard to get an overview. Can the test be fixed so (at least the first)e (sic) mismatch is shown in the overview? (be aware that some other agent may be working with caldav-server-tester at the moment) AI-assisted: Claude Opus 5 via Claude Code Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- tests/test_caldav.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/test_caldav.py b/tests/test_caldav.py index f0a844e6..e6ec0b16 100644 --- a/tests/test_caldav.py +++ b/tests/test_caldav.py @@ -1693,9 +1693,15 @@ def testCheckCompatibility(self, request) -> None: fe = self.caldav.features mismatches = fe.compare(fo) - assert not mismatches, "compatibility mismatches:\n" + "\n".join( - f" {m['feature']}: declared {m['expected']!r}, observed {m['observed']!r}" + ## pytest's short test summary shows only the first line of the + ## message, so the first mismatch has to go on that line + lines = [ + f"{m['feature']}: declared {m['expected']!r}, observed {m['observed']!r}" for m in mismatches + ] + assert not mismatches, ( + f"{len(mismatches)} compatibility mismatch(es): {lines[0]}\n" + + "\n".join(f" {line}" for line in lines) ) def testSupport(self): From d12e875e78de4d4a74293ca0a37b4a12f90277d4 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 09:34:03 +0200 Subject: [PATCH 13/18] fix: grade If-Match: * from both sides in profiles caldav-server-tester now also PUTs If-Match: * to a missing object, which shows Bedework 3.10.3 and 5.0.0 have it backwards (412 on an existing object, 201 on a missing one) and SOGo ignores it; both are "broken". Zimbra compares "*" as a literal etag and stays "unsupported", now noting that If-None-Match: * overwrites. The feature description names the three shapes. Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/calendarobjectresource.py | 4 ++++ caldav/compatibility_hints.py | 25 +++++++++++++++++++++---- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/caldav/calendarobjectresource.py b/caldav/calendarobjectresource.py index 56126b1c..ea15e6b7 100644 --- a/caldav/calendarobjectresource.py +++ b/caldav/calendarobjectresource.py @@ -1368,6 +1368,10 @@ def save( * self """ + + # TODO: the overwrite/no-overwrite logic can be handled server-side for + # servers that adheres to the RFC: "If-Match: *" and "If-None-Match: *" + # Early return if there's no data (no-op case) if not self.is_loaded(): return self diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 3f65996f..58a1fd13 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -464,7 +464,7 @@ class FeatureSet: "default": {"support": "full"}, }, "save-load.mutable.if-match-wildcard": { - "description": "An overwrite carrying If-Match: * is accepted. RFC 9110 section 13.1.1 has '*' match any current representation, so it means 'overwrite, but only if the object exists'. When 'unsupported', the server answers 412 Precondition Failed even though the object is there (Bedework 5). The library does not send If-Match: * itself.", + "description": "An overwrite carrying If-Match: * is accepted, and a PUT carrying it to a missing object is refused. RFC 9110 section 13.1.1 has '*' match any current representation, so it means 'overwrite, but only if the object exists'. When 'unsupported', the server answers 412 Precondition Failed even though the object is there (Zimbra). When 'broken', a missing object is created: the condition is backwards (Bedework, 412 on an existing object) or ignored (SOGo). If-None-Match: * (section 13.1.2) has no feature of its own; the behaviour text notes where it overwrites or refuses to create. The library sends neither itself.", "default": {"support": "full"}, "links": ["https://datatracker.ietf.org/doc/html/rfc9110#section-13.1.1"], }, @@ -1425,6 +1425,10 @@ def compare(self, observed): #'save-load.get-by-url': {'support': 'fragile', 'behaviour': '404 most of the time - but sometimes 200. Weird, should be investigated more'}, ## Zimbra treats same-UID events across calendars as aliases of the same event 'save.duplicate-uid.cross-calendar': {'support': 'unsupported'}, + ## '*' is compared as a literal etag, so If-Match: * never holds and + ## If-None-Match: * always does. Measured 2026-09-15 against the docker + ## image and, repeatedly, an external Zimbra. + 'save-load.mutable.if-match-wildcard': {'support': 'unsupported', 'behaviour': 'If-None-Match: * overwrote an existing object'}, ## Zimbra DOES apply a display name set at creation (the name sticks, so ## set-displayname is 'full') - but it couples the display name to the ## calendar URL. MKCALENDAR lands the calendar at the requested cal_id path; @@ -1562,8 +1566,11 @@ def compare(self, observed): 'save-load.icalendar.related-to': {'support': 'broken', 'behaviour': 'first RELATED-TO line is preserved but subsequent RELATED-TO lines are stripped'}, ## Bedework omits DAV:resourcetype from an allprop PROPFIND response. "propfind.allprop.resourcetype": {"support": "unsupported"}, - ## If-Match: * is 412 here as on 5.0.0; the PUT etag is not encoded yet. - "save-load.mutable.if-match-wildcard": {"support": "unsupported"}, + ## If-Match: * is backwards here as on 5.0.0; the PUT etag is not encoded yet. + "save-load.mutable.if-match-wildcard": { + "support": "broken", + "behaviour": "If-Match: * holds backwards: refused with 412 on an existing object, and creates a missing object", + }, ## (The old 'duplicates_not_allowed' flag was stale: Bedework does store a ## second event with the same content under a different UID, so ## save.duplicate-event is left at the default "full".) @@ -1662,7 +1669,11 @@ def compare(self, observed): ## "ungraceful" and save-load.mutable.attendee-partstat "unsupported"; both ## are full. "save.etag": {"support": "quirk", "behaviour": "percent-encoded"}, - "save-load.mutable.if-match-wildcard": {"support": "unsupported"}, + ## 412 on an existing object, 201 on a missing one (measured 2026-09-15). + "save-load.mutable.if-match-wildcard": { + "support": "broken", + "behaviour": "If-Match: * holds backwards: refused with 412 on an existing object, and creates a missing object", + }, "non-existing-raises-not-found.collection": { "support": "unsupported", @@ -1884,6 +1895,12 @@ def compare(self, observed): "support": "ungraceful", "behaviour": "Search by name failed: ReportError at '501 Not Implemented - \n\n

An error occurred during object publishing

did not find the specified REPORT

\n\n', reason no reason", }, + ## Both '*' conditions are ignored: If-Match: * creates a missing object and + ## If-None-Match: * overwrites an existing one (measured 2026-09-15). + "save-load.mutable.if-match-wildcard": { + "support": "broken", + "behaviour": "If-Match: * is ignored: it created a missing object (201); If-None-Match: * overwrote an existing object", + }, # Ephemeral Docker container: wipe objects (delete-calendar fragile) 'test-calendar': {'cleanup-regime': 'wipe-calendar'}, From 8aded2a0dd2974d10103978e80d034b5c0297f55 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 10:42:03 +0200 Subject: [PATCH 14/18] fix: grade four servers the new probes caught The server tester's new probes caught four profiles out, each checked against the server source or a direct request. Xandikos and SOGo ignore the component set given at MKCALENDAR, Radicale 3.8.0 compares If-Match: * as a literal etag, and OX ignores the i;octet collation - declared working since b5e26c4c, which was wrong. Prompts: (Code-review questions and comments on work done in the caldav-server-tester. Follow-up-conversations urging it to do a proper job there. This resulted in some changes in the server compatibility matrix) Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/compatibility_hints.py | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 58a1fd13..40f50dfa 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -464,7 +464,7 @@ class FeatureSet: "default": {"support": "full"}, }, "save-load.mutable.if-match-wildcard": { - "description": "An overwrite carrying If-Match: * is accepted, and a PUT carrying it to a missing object is refused. RFC 9110 section 13.1.1 has '*' match any current representation, so it means 'overwrite, but only if the object exists'. When 'unsupported', the server answers 412 Precondition Failed even though the object is there (Zimbra). When 'broken', a missing object is created: the condition is backwards (Bedework, 412 on an existing object) or ignored (SOGo). If-None-Match: * (section 13.1.2) has no feature of its own; the behaviour text notes where it overwrites or refuses to create. The library sends neither itself.", + "description": "An overwrite carrying If-Match: * is accepted, and a PUT carrying it to a missing object is refused. RFC 9110 section 13.1.1 has '*' match any current representation, so it means 'overwrite, but only if the object exists'. When 'unsupported', the server answers 412 Precondition Failed even though the object is there (Zimbra, Radicale - both compare '*' as a literal etag). When 'broken', a missing object is created: the condition is backwards (Bedework, 412 on an existing object) or ignored (SOGo). If-None-Match: * (section 13.1.2) has no feature of its own; the behaviour text notes where it overwrites or refuses to create. The library sends neither itself.", "default": {"support": "full"}, "links": ["https://datatracker.ietf.org/doc/html/rfc9110#section-13.1.1"], }, @@ -1314,6 +1314,16 @@ def compare(self, observed): "auto-connect.url": {"domain": "localhost", "scheme": "http", "basepath": "/"}, "scheduling": {"support": "unsupported"}, + + ## Every collection reports and takes the same hardcoded component list + ## (xandikos/web.py), and the supported-calendar-component-set property has + ## no setter - yet MKCALENDAR still answers 201, though RFC 4791 section + ## 5.3.1 has it fail when a property cannot be set. Measured on 0.4.5, + ## 2026-09-15. + "create-calendar.with-supported-component-types": { + "support": "unsupported", + "behaviour": "the component set is ignored: a VTODO-only calendar advertises VEVENT, VTODO, VJOURNAL, VFREEBUSY and VAVAILABILITY, and a VEVENT can be saved to it", + }, } ## This seems to work as of version 3.5.4 of Radicale. @@ -1325,6 +1335,13 @@ def compare(self, observed): "search.time-range.comp-type-optional": {"support": "full"}, "search.is-not-defined": {"support": "full"}, "search.text.case-sensitive": {"support": "unsupported"}, + ## radicale/app/put.py compares the If-Match value with the item's etag + ## literally, so '*' never matches. If-None-Match: * is handled right. + ## Measured on 3.8.0, 2026-09-15. + "save-load.mutable.if-match-wildcard": { + "support": "unsupported", + "behaviour": "If-Match: * is compared as a literal etag: an overwrite of an existing object is refused with 412", + }, "search.recurrences.includes-implicit.todo.pending": {"support": "fragile", "behaviour": "inconsistent results between runs"}, "search.recurrences.expanded.todo": {"support": "unsupported"}, "search.recurrences.expanded.exception": {"support": "full"}, @@ -1901,6 +1918,11 @@ def compare(self, observed): "support": "broken", "behaviour": "If-Match: * is ignored: it created a missing object (201); If-None-Match: * overwrote an existing object", }, + ## Measured 2026-09-15. + "create-calendar.with-supported-component-types": { + "support": "unsupported", + "behaviour": "the component set is ignored: a VTODO-only calendar advertises VEVENT, VFREEBUSY and VTODO, and a VEVENT can be saved to it", + }, # Ephemeral Docker container: wipe objects (delete-calendar fragile) 'test-calendar': {'cleanup-regime': 'wipe-calendar'}, @@ -2312,10 +2334,19 @@ def compare(self, observed): ## "unsupported" (silently ignored), not "broken". Confirmed by direct probe ## 2026-06-09. Contrast bedework, which drops the todos (data loss = broken). 'search.comp-type': {'support': 'unsupported'}, - ## Text search (case-sensitive, case-insensitive, substring) now works in OX. + ## Text search (case-insensitive, substring) now works in OX. ## Confirmed full 2026-06-13. Category search remains unsupported. 'search.text': {'support': 'full'}, 'search.text.category': {'support': 'unsupported'}, + ## The i;octet collation is ignored. A direct probe 2026-09-15 sent + ## Simple, and "Simple" and + ## "SIMPLE" both matched "simple event ...". This probe has been flapping: + ## declared unsupported until b5e26c4c (2026-06-15) dropped it as working, + ## and observed unsupported in three runs out of three on 2026-09-15. + 'search.text.case-sensitive': { + 'support': 'unsupported', + 'behaviour': 'the i;octet collation is ignored: a case-sensitive search matches case-insensitively', + }, ## Recurrence searching: the sliding window hides far-past/far-future ## occurrences, but implicit expansion of *datetime* events and server-side ## expansion of exceptions work within the window (detectable now that the From 3e67902a93782c01463f7fb8f8b6bc1e22485c38 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 14:26:40 +0200 Subject: [PATCH 15/18] test: let async props test reuse a calendar `TestAsyncForNextcloud::test_set_calendar_properties` failed on every run after the first with "server advertises delete- and create-calendar, but no fresh calendar". Nextcloud moves deleted calendars to a trashbin (`delete-calendar.free-namespace` fragile), so `afix_calendar()` reuses the calendar from the previous run. The `created` assert from 812d6df6 now applies only where deletion frees the URL; elsewhere the test checks it got its own calendar back. The display name is put back after the rename, as the sync test does, so the rename is a real change on a reused calendar too. Run twice on Nextcloud and once on Radicale and Xandikos; a Nextcloud container still carrying the old "hooray-async" name fails once, then passes. Prompt: This found (sic) while doing checks in caldav: FAILED tests/test_async_integration.py::TestAsyncForNextcloud::test_set_calendar_properties - AssertionError: server advertises delete- and create-calendar, but no fresh calendar is it an async/sync problem? Followup-Prompt: [do the recommended fix: assert `created` and the creation-time name only when free-namespace is supported] Followup-Prompt: (review findings accepted: put the display name back so the rename is not a no-op on a reused calendar, check the reused calendar is the test's own, correct the 812d6df6 attribution, verify on a second Nextcloud run) Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- tests/test_async_integration.py | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/tests/test_async_integration.py b/tests/test_async_integration.py index 5872926f..f9befb7a 100644 --- a/tests/test_async_integration.py +++ b/tests/test_async_integration.py @@ -1917,6 +1917,7 @@ async def test_find_calendar_owner(self, async_calendar: Any, async_client: Any) async def test_set_calendar_properties(self, async_client: Any) -> None: """get_properties/set_properties round-trip for DisplayName.""" from caldav.elements import dav + from caldav.lib import error from .fixture_helpers import afix_calendar, arelease_calendar, atry_principal @@ -1942,14 +1943,18 @@ async def test_set_calendar_properties(self, async_client: Any) -> None: cal_id="pythoncaldav-async-props-test", calendar_name="AsyncYep", ) + assert c is not None, "no test calendar could be created or found" try: - ## Given the skips above (delete-calendar, create-calendar and - ## set-displayname/stable-url support) a fresh calendar must have been - ## created. If it wasn't, the server regressed on a feature it - ## advertises as supported - that is a failure, not a reason to skip, - ## which is how the pre-consolidation version of this test behaved - ## (make_calendar() simply raised). - assert created, "server advertises delete- and create-calendar, but no fresh calendar" + ## A fresh calendar is only guaranteed where deletion frees the URL. + ## On a trashbin server (Nextcloud) the calendar from the previous run + ## is reused, so only the creation cannot be checked there - but it + ## still has to be this test's own calendar, not a fallback. + if self.is_supported("delete-calendar.free-namespace"): + ## Here a reused calendar means the server regressed on a + ## feature it advertises - a failure, not a reason to skip. + assert created, "server frees the namespace on delete, but no fresh calendar" + else: + assert "pythoncaldav-async-props-test" in str(c.url) props = await c.get_properties([dav.DisplayName()]) assert "AsyncYep" == props[dav.DisplayName.tag] @@ -1957,6 +1962,13 @@ async def test_set_calendar_properties(self, async_client: Any) -> None: props = await c.get_properties([dav.DisplayName()]) assert props[dav.DisplayName.tag] == "hooray-async" finally: + ## Put the name back, as testSetCalendarProperties does, so a + ## calendar reused by the next run starts from "AsyncYep" and the + ## rename above is a real change rather than a no-op. + try: + await c.set_properties([dav.DisplayName("AsyncYep")]) + except error.PropsetError: + pass await arelease_calendar(async_client, c, created) # ==================== Group F – Regressions ==================== From 2e94cbee2c4272130b8392f89b6fe821bb5f542a Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 14:50:00 +0200 Subject: [PATCH 16/18] docs: explain Nextcloud's free-namespace mismatch The profile claimed the server-tester does not catch `delete-calendar.free-namespace`. It does, and grades the docker Nextcloud `full`, because setup_nextcloud.sh disables the trashbin; that step now warns when it fails instead of passing silently. `fragile` stays, as "varies by deployment", and `ecloud` - a default install - is graded unsupported. Re-enabling the trashbin was considered, but setup_nextcloud.sh records UNIQUE violations from soft-deleted objects on Nextcloud 33, and Nextcloud's `X-NC-CalDAV-No-Trashbin` header, which the library does not send, only covers calendar deletes. The robur profile also grades `save.etag` broken (a malformed etag), added by hand. Prompt: [add a delete-calendar.free-namespace probe to caldav-server-tester, as the Nextcloud profile TODO says it is not caught] Followup-Prompt: isn't it better to turn on the thrashbin (sic) feature again, and test the nextcloud as a true nextcloud server? At the other hand, I remember having an issue on ecloud that I frequently have to enter the web-ui and manually "empty" the thrashbin (sic) ... and we had similar problems with tests going into a bad state towards the local nextcloud, requiring a nextcloud server restart. Think a bit if there are any better solutions here, if not then go for the comment-only solution Followup-Prompt: [hand-edited comments in the code] Followup-Prompt: (comment from the review process) Check the latest commit, maybe this is fixed already [on the "Calendar deletion goes to trashbin" comment, which was not; now qualified to a trashbin-enabled install] Followup-Prompt: (review findings accepted: link the Nextcloud source for the header claim, warn when the setup cannot disable the trashbin) Followup-Prompt: (review findings accepted: link Calendar.php for the calendar-only limit, narrow the reason not to re-enable the trashbin to what was observed, move the ecloud line off the rate-limit TODO, correct three facts in the free-namespace comment) Co-Authored-By: Claude Opus 5 Reviewed-by: Tobias Brox --- caldav/compatibility_hints.py | 27 ++++++++++++++----- .../nextcloud/setup_nextcloud.sh | 3 ++- 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 40f50dfa..1935647b 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -1354,7 +1354,9 @@ def compare(self, observed): "calendar-order": {"support": "full"}, } -## Be aware that nextcloud by default have different rate limits, including how often a user is allowed to create a new calendar. This may break test runs badly. +## Nextcloud can be configured a lot. What particularly affects the compatibility +## matrix and test runs are rate limits, including how often a user is allowed +## to create a new calendar. This may break test runs badly. nextcloud = { 'auto-connect.url': { 'basepath': '/remote.php/dav', @@ -1381,12 +1383,24 @@ def compare(self, observed): ## could not be reproduced. No delay observed either, unlike Cyrus, so ## 'full' rather than the 'quirk' recorded there. 'delete-calendar': {'support': 'full'}, - 'delete-calendar.free-namespace': { ## TODO: not caught by server-tester - 'behaviour': "deleting a calendar moves it to a trashbin, thrashbin has to be manually 'emptied' from the web-ui before the namespace is freed up", + ## For a regular installation, there is a trashbin problem, + ## a deleted calendar's id is only available after things have been cleared + ## out from the trashbin - hence `delete-calendar.free-namespace` should be + ## set to unsupported. However, the docker test container in this project + ## has the trashbin disabled by setup_nextcloud.sh. The grade here is + ## "fragile" - as in "varies by deployment". There is never any fragility + ## for a specific installation. + ## A DELETE carrying 'X-NC-CalDAV-No-Trashbin: 1' skips the + ## trashbin, but only for a whole calendar, not for its objects: the + ## plugin sets a calendar-level flag, and only Calendar::delete() reads it. + ## https://github.com/nextcloud/server/blob/19acca6a7fd45b1bfb76952659809f858b6cd6de/apps/dav/lib/CalDAV/Trashbin/Plugin.php + ## https://github.com/nextcloud/server/blob/19acca6a7fd45b1bfb76952659809f858b6cd6de/apps/dav/lib/CalDAV/Calendar.php + 'delete-calendar.free-namespace': { + 'behaviour': "with the trashbin enabled (the default), deleting a calendar moves it to a trashbin, the trashbin has to be manually 'emptied' from the web-ui before the namespace is freed up", 'support': 'fragile', }, - # Calendar deletion goes to trashbin so delete-and-recreate doesn't give a - # fresh empty calendar. Wipe objects instead of deleting the calendar itself. + # On an install with the trashbin enabled, delete-and-recreate doesn't give + # a fresh empty calendar. Wipe objects instead of deleting the calendar itself. "test-calendar": {"cleanup-regime": "wipe-calendar"}, 'search.recurrences.includes-implicit.todo': {'support': 'unsupported'}, #'save-load.todo.mixed-calendar': {'support': 'unsupported'}, ## Why? It started complaining about this just recently. @@ -1400,10 +1414,10 @@ def compare(self, observed): 'scheduling.schedule-tag': False, } -## TODO: Latest - mismatch between config and test script in delete-calendar.free-namespace ... and create-calendar.set-displayname? ecloud = nextcloud | { #'search.is-not-defined': {'support': 'unsupported'}, ## observed to work at 4bc0de765a2b53e6f223e0b9ac51c653bac11fb7 (caldav) / 3cae24cf99da1702b851b5a74a9b88c8e5317dad (server checker) #'search.text.case-sensitive': {'support': 'unsupported'}, ## observed to work at 4bc0de765a2b53e6f223e0b9ac51c653bac11fb7 (caldav) / 3cae24cf99da1702b851b5a74a9b88c8e5317dad (server checker) + 'delete-calendar.free-namespace': False, ## TODO: this applies only to test runs, not to ordinary usage 'rate-limit': { 'enable': True, @@ -1995,6 +2009,7 @@ def compare(self, observed): 'test-calendar': {'cleanup-regime': 'wipe-calendar'}, "sync-token": {"support": "ungraceful"}, "get-supported-components": {"support": "unsupported"}, + "save.etag": {'support': 'broken', 'behaviour': "malformed etag 'a22ccfb2985aed13f50b4991a110d32253fd99aa'"}, } posteo = { diff --git a/tests/docker-test-servers/nextcloud/setup_nextcloud.sh b/tests/docker-test-servers/nextcloud/setup_nextcloud.sh index 46b863e6..2a64af9f 100755 --- a/tests/docker-test-servers/nextcloud/setup_nextcloud.sh +++ b/tests/docker-test-servers/nextcloud/setup_nextcloud.sh @@ -83,7 +83,8 @@ echo "Disabling CalDAV trashbin (calendar retention)..." # causing UNIQUE constraint violations when tests recreate a calendar with the same slug # (Nextcloud 33+ reuses the calendarid, keeping old soft-deleted objects, so adding # an event with the same UID fails). -occ config:app:set dav calendarRetentionObligation --value=0 || true +occ config:app:set dav calendarRetentionObligation --value=0 \ + || echo "WARNING: could not disable the CalDAV trashbin - deleted calendars will not free their namespace" >&2 # Purge any leftover soft-deleted calendars/objects from previous runs occ dav:retention:clean-up || true From 7bd6f3ca14c9b5605f2e832bb63650fa4c837247 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 23:08:29 +0200 Subject: [PATCH 17/18] docs: fill in 3.3.1 CHANGELOG, refresh roadmap The 3.3.1 section was missing three generic fixes (207 on MKCALENDAR, percent-encoded PUT ETag, expanded instances sharing one href) and the profile regrades that change is_supported() answers. The roadmap is brought in line with the issue tracker: the multiget ask of https://github.com/python-caldav/caldav/issues/487 shipped in 3.3.0, and closed issues are ticked off. Prompt: 3.3.1 goes out tomorrow with the current state of this branch. Please check with the roadmap what things have been closed in 3.3.0 and 3.3.1. Followup-Prompt: look into my comments [inline comments on a report about the above] Followup-Prompt: I've done some changes to the ROADMAP, please piggyback my changes into the next commit Followup-Prompt: Please verify and close [https://github.com/python-caldav/caldav/issues/487] Followup-Prompt: Please make it consistent [the multiget section number and priority in the roadmap summary table] Followup-Prompt: Update to "updated 2026-08-15" or "2026-08-16". (sic) [the roadmap header date; 2026-09-15 used] Followup-Prompt: Please fix [missing entries and a redundant sentence in the 3.3.1 CHANGELOG] Co-Authored-By: Claude Opus 5 AI Prompts: claude-sonnet-4-6: bcc4vsw9w toolu_01YACdKxjRB7MZhcXnW2qqTA /tmp/claude-7385/-home-tobias-caldav/3934293f-61bf-4490-a98f-eaf07b9abd94/tasks/bcc4vsw9w.output completed Background command "Open issue comment draft for approval" completed (exit code 0) --- CHANGELOG.md | 13 ++++++++++-- docs/design/FEATURE_COMPLETE_ROADMAP.md | 28 ++++++++++++++----------- 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a684ae6e..4da0680c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,11 @@ Changelogs prior to v3.0 are pruned, but are available in the v3.1 release This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html), though for pre-releases PEP 440 takes precedence. -## [Unreleased] +## [3.3.1] - 2026-09-16 + +The two main things in this release: +* Changes in the compatibility_hints.py needed for the upcoming caldav-server-tester 1.3.0 release. +* Workarounds for broken behaviour in the Bedework 5 calendaring server. ### Added @@ -21,10 +25,15 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0 ### Changed * **Breaking:** the `bedework` compatibility profile is renamed `bedework_3_10_3`, and `bedework_5_0_0` is added. `features: bedework` or `base: bedework` in a config now raises a `ValueError` naming both. -* `compatibility_hints`: the `write-delay` server-peculiarity is turned around into the `synchronous-write` server-feature. The delay (relevant for tests and the caldav server tester) should still be hand-configured. +* `compatibility_hints`: the `write-delay` server-peculiarity is turned around into the `synchronous-write` server-feature. While the caldav-server-tester does probe it, the delay (relevant for tests and the caldav server tester) should still be hand-configured. +* `compatibility_hints`: new caldav-server-tester probes caused the Xandikos, SOGo, Radicale, OX, Zimbra, Bedework and Cyrus profiles to be regraded. With `features: ` configured, `is_supported()` may give a different answer than in 3.3.0. ### Fixed +* `make_calendar()` raised `MkcalendarError` when the server answered MKCALENDAR with `207 Multi-Status` instead of `201 Created`, even though the calendar was created. A multistatus reporting no failure is now accepted (seen on Bedework 5). +* An ETag delivered percent-encoded (`%22...%22`) in a PUT response is now decoded; previously every second `save()` of an object raised `ETagMismatchError` (seen on Bedework 5). +* Expanded searches lost all but the last occurrence when the server answered with one `DAV:response` per recurrence instance, all under the same href. The instances are now merged (seen on Bedework 5). + * A bare `icalendar.Event`/`Todo`/`Journal` handed to caldav is wrapped in a `VCALENDAR` - that wrapper no longer gets a random RFC 7986 `UID` of its own (`icalendar.Calendar.new()` adds one). Servers taking the calendar-level `UID` to be the identity of the calendar object resource (i.e. Stalwart) saw a brand new UID on every save and rejected it with `412 no-uid-conflict`. ## [3.3.0] - 2026-09-03 diff --git a/docs/design/FEATURE_COMPLETE_ROADMAP.md b/docs/design/FEATURE_COMPLETE_ROADMAP.md index 66e2b514..b0808df7 100644 --- a/docs/design/FEATURE_COMPLETE_ROADMAP.md +++ b/docs/design/FEATURE_COMPLETE_ROADMAP.md @@ -1,6 +1,6 @@ # Feature-Complete CalDAV Library Roadmap -- **Created:** 2026-01-28, **updated** 2026-08-24 +- **Created:** 2026-01-28, **updated** 2026-09-15 - **Author:** AI-generated and human-edited based on RFC analysis and open issues - **Status:** Planning document for work after issue [#599](https://github.com/python-caldav/caldav/issues/599) completion @@ -198,13 +198,14 @@ The v3.2 roadmap covers basic scheduling improvements. Additional work for full ### 2.3 Multiget Optimization -- **Priority:** Medium +- **Priority:** Low (the main part is done) - **Estimated effort:** 8 hours - **Related issue:** [#487](https://github.com/python-caldav/caldav/issues/487) **Tasks:** -- [ ] Use `calendar-multiget` REPORT when server doesn't return object data in search -- [x] Batch retrieval of multiple objects — **done**: `Collection.multiget()`, `AsyncDAVClient.calendar_multiget()`, shared body builder `_build_calendar_multiget_body()`. The remaining gap is the [#487](https://github.com/python-caldav/caldav/issues/487) ask: using it *automatically* when a search response carried no object data +- [x] Use `calendar-multiget` REPORT when server doesn't return object data in search — **done in v3.3.0**: `search()` loads all unloaded results with one REPORT via `Calendar._batch_load_objects()` (and its async twin), falling back to per-object GET if the REPORT fails +- [x] Batch retrieval of multiple objects — **done**: `Collection.multiget()`, `AsyncDAVClient.calendar_multiget()`, shared body builder `_build_calendar_multiget_body()` +- [ ] caldav-server-tester probe for servers omitting object data in search responses - [ ] Configurable batch sizes --- @@ -603,7 +604,7 @@ features. - **Related issues:** [#690](https://github.com/python-caldav/caldav/issues/690), [#611](https://github.com/python-caldav/caldav/issues/611), [#696](https://github.com/python-caldav/caldav/issues/696) **Tasks:** -- [ ] Make the HTTP transport an extra, so `caldav` can be installed without +- [x] Make the HTTP transport an extra, so `caldav` can be installed without `niquests` ([#690](https://github.com/python-caldav/caldav/issues/690)) - [ ] Settle the v4.0 HTTP-library question ([#611](https://github.com/python-caldav/caldav/issues/611)) - [ ] Sync-mode support for the httpx family — async already has it ([#696](https://github.com/python-caldav/caldav/issues/696)) @@ -614,13 +615,16 @@ features. These open issues are bug reports, support questions or automated noise rather than roadmap items, and are deliberately left out: -[#71](https://github.com/python-caldav/caldav/issues/71) (`add_event` can update as well), -[#545](https://github.com/python-caldav/caldav/issues/545) (searches return full-day events of adjacent days), -[#612](https://github.com/python-caldav/caldav/issues/612) (support question), -[#624](https://github.com/python-caldav/caldav/issues/624) (GMX calendar creation), -[#678](https://github.com/python-caldav/caldav/issues/678) (`change_attendee_status()` async safety — see 7.1), -[#680](https://github.com/python-caldav/caldav/issues/680), [#681](https://github.com/python-caldav/caldav/issues/681), [#684](https://github.com/python-caldav/caldav/issues/684) (server-specific breakage reports), -[#685](https://github.com/python-caldav/caldav/issues/685) (automated link-checker report). +- [ ] [#71](https://github.com/python-caldav/caldav/issues/71) (`add_event` can update as well), +- [ ] [#545](https://github.com/python-caldav/caldav/issues/545) (searches return full-day events of adjacent days), +- [ ] [#612](https://github.com/python-caldav/caldav/issues/612) (support question), +- [ ] [#624](https://github.com/python-caldav/caldav/issues/624) (GMX calendar creation), +- [ ] [#678](https://github.com/python-caldav/caldav/issues/678) (`change_attendee_status()` async safety — see 7.1), +- [ ] server-specific breakage reports ... + - [ ] [#680](https://github.com/python-caldav/caldav/issues/680) + - [x] [#681](https://github.com/python-caldav/caldav/issues/681) + - [ ] [#684](https://github.com/python-caldav/caldav/issues/684) +- [x] [#685](https://github.com/python-caldav/caldav/issues/685) (automated link-checker report). Bugs get fixed when they get fixed; they do not need a phase. From d2cdff5fcc9a879788a66014c5133aebcbbd26e5 Mon Sep 17 00:00:00 2001 From: Tobias Brox Date: Tue, 15 Sep 2026 23:18:23 +0200 Subject: [PATCH 18/18] docs: minor tweaks * Link syntax fix, fixes https://github.com/python-caldav/caldav/issues/712 * The github-code-quality-bot flagged an except-block doing nothing without comments, copied the justification code comments from the equivalent sync code --- CONTRIBUTING.md | 2 +- tests/test_async_integration.py | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3e33f20a..fdd632c8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -72,4 +72,4 @@ Consider this procedures to be a more of a guideline than a rigid procedure. Us ## Code of Conduct -Code of Conduct has been moved to a [separate document](CODE_OF_CONDUCT] +Code of Conduct has been moved to a [separate document](CODE_OF_CONDUCT) diff --git a/tests/test_async_integration.py b/tests/test_async_integration.py index f9befb7a..84478fc0 100644 --- a/tests/test_async_integration.py +++ b/tests/test_async_integration.py @@ -1968,6 +1968,10 @@ async def test_set_calendar_properties(self, async_client: Any) -> None: try: await c.set_properties([dav.DisplayName("AsyncYep")]) except error.PropsetError: + ## Best-effort cleanup only: the assertion of interest has + ## already run above. Some servers reject setting the display + ## name (PropsetError); if so there's nothing to restore and + ## nothing actionable to do here, so swallow it silently. pass await arelease_calendar(async_client, c, created)