Skip to content

Commit 52e50c7

Browse files
committed
Merge #90's branch so the bridge check sees the ported drift fixes
#91's bridge coverage job reports the same three drift problems #90 fixes. Merging that branch here makes this PR's check meaningful now; the merge is a no-op once #90 lands on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Dn4gCNCU2jUGdmfPVCAGK
2 parents e62f5b6 + c512dc9 commit 52e50c7

8 files changed

Lines changed: 715 additions & 29 deletions

‎src/did/did_matlab_python_bridge.yaml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -939,7 +939,7 @@ classes:
939939
- name: document
940940
type: class
941941
matlab_path: "+did/document.m"
942-
matlab_last_sync_hash: "c95b4f2"
942+
matlab_last_sync_hash: "bb29861"
943943
status: ported
944944
python_path: "did/document.py"
945945
python_class: "Document"
@@ -1414,6 +1414,17 @@ classes:
14141414
Both languages now accept NAME_<i> as a numbered entry only when <i>
14151415
is a non-negative integer.
14161416
1417+
Sync hash bumped to bb29861 on 2026-10-23. That commit replaced the
1418+
constructor's inline char(datetime('now','TimeZone','UTCLeapSeconds'))
1419+
with did.fun.timestamp(), which clamps a ':60' leap second to
1420+
':59.999' (DID-matlab issue #53). Ported the same way: Document.__init__
1421+
now stamps base.datestamp with did.fun.timestamp() (see the timestamp
1422+
entry in did_matlab_python_bridge_util.yaml). The module-private
1423+
document._utc_timestamp it replaces had its own clamp, a plain
1424+
':60.' -> ':59.999' substitution that would have turned ':60.123' into
1425+
':59.999123'; it is removed rather than kept beside the port.
1426+
Coverage: tests/test_timestamp.py.
1427+
14171428
# -----------------------------------------------------------------------
14181429
# did.query
14191430
# -----------------------------------------------------------------------

‎src/did/did_matlab_python_bridge_implementations.yaml‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ classes:
2929
- name: sqlitedb
3030
type: class
3131
matlab_path: "+did/+implementations/sqlitedb.m"
32-
matlab_last_sync_hash: "8bea92c"
32+
matlab_last_sync_hash: "da83fa3"
3333
status: ported
3434
python_path: "did/implementations/sqlitedb.py"
3535
python_class: "SQLiteDB"
@@ -133,6 +133,60 @@ classes:
133133
new tests, including the acceptance case, the "one fetch shared
134134
across members" cache guarantee, and the handler-refused failure).
135135
136+
2026-10-23: synced through DID-matlab decd155 and da83fa3 (PRs #206,
137+
#207), the slow-drive SQLITE_READONLY work. MATLAB (1) sets
138+
busy_timeout=30000 in open_db, (2) retries a BUSY / READONLY failure in
139+
run_sql_noOpen with backoffs 0.1, 0.2, 0.5, 1, 2, 4 s, (3) closes and
140+
reopens the connection before each READONLY retry (a poisoned handle,
141+
SQLITE_READONLY_DBMOVED, which waiting never clears), and (4) prints
142+
filesystem diagnostics to stderr when the retries are exhausted.
143+
Ported: _open_db passes timeout=30.0 to sqlite3.connect (the same
144+
busy_timeout); do_run_sql_query retries on the same schedule, reopening
145+
first for READONLY; _is_retryable_sql_error, _is_readonly_error and
146+
_collect_readonly_diagnostics mirror the three new MATLAB helpers; the
147+
debug-gated progress messages are printed when self.debug is set.
148+
The three helpers are protected in MATLAB and so carry no member
149+
entries here. _is_retryable_sql_error and _is_readonly_error are
150+
staticmethods (MATLAB's take ~ for the object). The diagnostics run the
151+
same four probes -- db file and time, -journal/-wal/-shm siblings with
152+
sizes, the file's write permission (os.access for fileattrib's
153+
UserWrite), and a temp-file write probe of the directory -- but print
154+
the time as ISO-8601 UTC rather than MATLAB's local datestr.
155+
156+
DEVIATION -- where the retry may run. mksqlite autocommits every
157+
statement, so in MATLAB a failed statement is the whole unit of work
158+
and re-running it (even after a reopen) is safe. SQLiteDB groups each
159+
add/remove/branch operation into one sqlite3 transaction. Rolling back
160+
or reopening mid-transaction would silently discard the statements
161+
already pending, and a retried tail would then commit a partial
162+
operation. So do_run_sql_query retries only when the connection had no
163+
uncommitted work before the statement; inside a transaction the error
164+
is reported (with diagnostics) and raised at once, and the caller's
165+
existing rollback runs.
166+
167+
DEVIATION -- the commit is retried too. Because Python's writes reach
168+
the disk at COMMIT rather than at each statement, that is where a
169+
slow-drive failure lands here. Every commit now goes through
170+
_commit(), which retries a BUSY / READONLY COMMIT on the same schedule
171+
while the transaction is still open. It never reopens -- that would
172+
lose the transaction -- so a genuinely poisoned handle (DBMOVED) is not
173+
recovered at COMMIT: the error is raised and the operation rolled back.
174+
Replaying a whole operation after a reopen would cover that case, but
175+
_do_add_doc also copies ingested files, so it is not obviously
176+
idempotent; not attempted.
177+
178+
DEVIATION -- classification. Like MATLAB, BUSY vs READONLY is judged
179+
from the error message (sqlite3.Error.sqlite_errorcode would be exact
180+
but is 3.11+ only, and this package supports 3.10). Unlike MATLAB, only
181+
sqlite3.Error instances qualify, so an unrelated Python exception whose
182+
text says "read only" is never retried.
183+
184+
Not ported: MATLAB's run_sql_noOpen also prints every failing query
185+
and its bound values to stderr. That predates these commits, and
186+
Python never did it; only the new diagnostics block is printed here.
187+
Coverage: tests/test_sqlitedb_transient_errors.py (real lock and
188+
read-only-handle failures, with time.sleep replaced).
189+
136190
properties:
137191
- name: FileDir
138192
python_name: ~ # not ported
@@ -174,6 +228,10 @@ classes:
174228
Python: do_run_sql_query(query_str, params=()).
175229
Python uses parameterized queries. Synchronized 2026-03-15.
176230
231+
Carries MATLAB run_sql_noOpen's BUSY / READONLY retry (decd155,
232+
da83fa3), restricted to statements run with no transaction open;
233+
see the entry-level sync_notes, 2026-10-23.
234+
177235
- name: do_add_doc
178236
python_name: _do_add_doc
179237
out_of_sync: false
@@ -571,6 +629,10 @@ classes:
571629
combined "open_db / close_db" entry 2026-08-28 so each method
572630
resolves on its own.
573631
632+
Both set a 30 s busy timeout (decd155): MATLAB with pragma
633+
busy_timeout=30000, Python with sqlite3.connect(timeout=30.0),
634+
which is the same setting. Python's previous default was 5 s.
635+
574636
- name: close_db
575637
python_name: _close_db
576638
decision_log: >

‎src/did/did_matlab_python_bridge_util.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,41 @@ functions:
178178
Both provide interactive document graph visualization.
179179
Synchronized 2026-03-15.
180180
181+
# -----------------------------------------------------------------------
182+
# did.fun.timestamp
183+
# -----------------------------------------------------------------------
184+
- name: timestamp
185+
type: function
186+
matlab_path: "+did/+fun/timestamp.m"
187+
matlab_last_sync_hash: "bb29861"
188+
status: ported
189+
python_path: "did/fun.py"
190+
python_name: "timestamp"
191+
192+
input_arguments:
193+
- name: raw
194+
type_matlab: "(1,:) char, default char(datetime('now','TimeZone','UTCLeapSeconds'))"
195+
type_python: "str | None, default None (the current UTC time)"
196+
output_arguments:
197+
- name: ts
198+
type_matlab: "char"
199+
type_python: "str"
200+
201+
decision_log: >
202+
Ported 2026-10-23 from DID-matlab bb29861 (issue #53). Same regex,
203+
(T\d{2}:\d{2}:)60(\.\d+)? -> \g<1>59.999, so a leap second with or
204+
without a fraction and with or without a trailing 'Z' clamps exactly as
205+
in MATLAB, and nothing outside the seconds field is touched. The
206+
default value is the current UTC time as 'yyyy-mm-ddTHH:MM:SS.fffZ',
207+
the shape MATLAB's UTCLeapSeconds char renders. One difference, of
208+
fact rather than of design: Python's datetime cannot represent a leap
209+
second, so the default path never produces ':60' and the clamp only
210+
ever acts on a supplied raw string (e.g. one written by MATLAB).
211+
Python's validator (validate._parses_as_timestamp, via
212+
datetime.fromisoformat) rejects ':60' as java.time.LocalDateTime.parse
213+
does, so the clamp is needed on that path for the same reason.
214+
Coverage: tests/test_timestamp.py, mirroring MATLAB's TestTimestamp.
215+
181216
# =========================================================================
182217
# did.datastructures — Data structure utilities
183218
# =========================================================================

‎src/did/document.py‎

Lines changed: 2 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,9 @@
11
import contextlib
22
import os
3-
from datetime import datetime, timezone
43

54
from . import ido
65
from .common import PathConstants
7-
8-
9-
def _utc_timestamp():
10-
"""Return the current UTC time as an ISO-8601 millisecond string with 'Z'.
11-
12-
Ported from NDI-python (ndi.fun.timestamp) with the leap-second guard.
13-
``str(datetime.utcnow())`` previously emitted a space-separated,
14-
timezone-less string ('2026-07-20 22:36:19.611068') that diverges from the
15-
DID-matlab UTCLeapSeconds ISO-8601 output and from base.schema.json's own
16-
default ('2018-12-05T18:36:47.241Z'); a JS ``new Date()`` parses the
17-
tz-less form as LOCAL time. Emit '%Y-%m-%dT%H:%M:%S.%fZ' truncated to
18-
milliseconds, clamping the (theoretical) leap-second :60 to :59.999.
19-
"""
20-
now = datetime.now(timezone.utc)
21-
ts = now.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3]
22-
ts = ts.replace(":60.", ":59.999")
23-
return ts + "Z"
6+
from .fun import timestamp
247

258

269
class Document:
@@ -30,7 +13,7 @@ def __init__(self, document_type="base", **kwargs):
3013
else:
3114
self.document_properties = self.read_blank_definition(document_type)
3215
self.document_properties["base"]["id"] = ido.IDO.unique_id()
33-
self.document_properties["base"]["datestamp"] = _utc_timestamp()
16+
self.document_properties["base"]["datestamp"] = timestamp()
3417

3518
for key, value in kwargs.items():
3619
path = key.split(".")

‎src/did/fun.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,36 @@
1+
import re
2+
from datetime import datetime, timezone
3+
14
import networkx as nx
25

6+
# A leap-second seconds field (':60' with an optional fraction), kept with its
7+
# 'T HH:MM:' prefix so the date portion is never touched.
8+
_LEAP_SECOND = re.compile(r"(T\d{2}:\d{2}:)60(\.\d+)?")
9+
10+
11+
def timestamp(raw=None):
12+
"""
13+
A current UTC timestamp string that is safe for DID schema validation.
14+
15+
Returns the current time in the ISO-8601 form used throughout DID,
16+
``'2026-09-18T21:37:45.123Z'``. ``timestamp(raw)`` sanitizes the given
17+
string instead of the current time, which is mainly useful for testing.
18+
19+
A leap second (``':60'``, with or without a fraction) is clamped back to
20+
``':59.999'``: the schema validator parses timestamps the way MATLAB's
21+
``java.time.LocalDateTime.parse`` does, which rejects any seconds value
22+
above 59. Python's ``datetime`` never renders a leap second itself, so on
23+
the default path the clamp is a guard rather than a fix; it matters for
24+
strings that came from MATLAB's ``UTCLeapSeconds`` clock.
25+
26+
This function mimics the behavior of the Matlab `did.fun.timestamp`
27+
function (DID-matlab issue #53).
28+
"""
29+
if raw is None:
30+
now = datetime.now(timezone.utc)
31+
raw = now.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
32+
return _LEAP_SECOND.sub(r"\g<1>59.999", raw)
33+
334

435
def docs_to_graph(document_objs):
536
"""

0 commit comments

Comments
 (0)