Skip to content

Port did.fun.timestamp, leap-second datestamp clamp, and SQLite BUSY/READONLY retry - #90

Merged
stevevanhooser merged 1 commit into
mainfrom
claude/port-timestamp-leapsecond-sqlite-readonly
Oct 23, 2026
Merged

stevevanhooser merged 1 commit into
mainfrom
claude/port-timestamp-leapsecond-sqlite-readonly

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Fixes the three problems bin/check_bridge_coverage.py reports against DID-matlab main, which make the "Bridge coverage vs DID-matlab" check fail on every PR. After this change the check reports 0 problem(s).

What was ported

1. did.fun.timestamp (new; DID-matlab bb29861)

  • Added did.fun.timestamp(raw=None). It returns the current UTC time as yyyy-mm-ddTHH:MM:SS.fffZ. It clamps a :60 leap second to :59.999 using MATLAB's regex, with or without a fraction and with or without a trailing Z.
  • New bridge entry (status: ported) in did_matlab_python_bridge_util.yaml. Python's datetime can't represent a leap second, so on the default path the clamp never fires. It only changes a raw string that is passed in, such as one MATLAB wrote.

2. document (c95b4f2 → bb29861)

  • Document.__init__ now sets base.datestamp with did.fun.timestamp(), as MATLAB's constructor now does.
  • Removed the private document._utc_timestamp. Its clamp was a plain :60. → :59.999 substitution, which would have turned :60.123 into :59.999123.

3. sqlitedb (8bea92c → da83fa3; decd155 + da83fa3)

  • Busy timeout: _open_db connects with timeout=30.0, the same setting as MATLAB's pragma busy_timeout=30000. The previous Python default was 5 s.
  • Retry: do_run_sql_query retries BUSY/READONLY errors with MATLAB's backoff schedule (0.1, 0.2, 0.5, 1, 2, 4 s). For READONLY it reopens the connection before each retry, which is what clears a poisoned handle (DBMOVED).
  • Helpers: _is_retryable_sql_error, _is_readonly_error and _collect_readonly_diagnostics mirror the three new protected MATLAB helpers.
  • Diagnostics: once the retries run out, the filesystem diagnostics are printed to stderr, as in MATLAB.

Deviations (recorded in the entry's sync_notes)

  • Retry only outside a transaction. mksqlite commits every statement on its own, so in MATLAB a failed statement is the whole operation. SQLiteDB runs each add, remove or branch operation as one transaction. Rolling back or reopening partway through would drop the statements already pending, so the statement retry only runs when no transaction is open. Inside a transaction the error is reported and raised, and the caller's existing rollback runs.
  • COMMIT is retried too. In Python, writes reach the disk at COMMIT, so that is where a slow-drive failure shows up. Every commit now goes through _commit(), which retries a BUSY/READONLY commit while the transaction is still open. It never reopens, because that would lose the transaction. A truly poisoned handle at COMMIT is therefore raised rather than recovered. Replaying the whole operation was not attempted because _do_add_doc also copies ingested files, so a replay is not obviously safe.
  • Error classification. Like MATLAB, the error type is judged from the message: sqlite_errorcode would be exact but needs Python 3.11+, and this package supports 3.10. Unlike MATLAB, only sqlite3.Error instances count.
  • Not ported: MATLAB also prints every failing query to stderr. That behaviour predates these commits and Python never had it.

Tests

  • tests/test_timestamp.py mirrors MATLAB's TestTimestamp. It checks timestamps against Python's validator and that Document stamps through the helper.
  • tests/test_sqlitedb_transient_errors.py uses real sqlite3 failures: a second connection holding the lock, and a handle swapped for a read-only one. Only time.sleep is faked. It covers the 30 s timeout, error classification, BUSY retry without reopen, READONLY reopen, retries running out, no retry inside a transaction, the COMMIT retry (including through add_branch), and the diagnostics.

Local runs: black --check, ruff check (pinned 0.16.5), pytest (766 passed), and check_bridge_coverage.py against a full DID-matlab clone (0 problems).

Found, not fixed here

_outside_brackets and _strip_comment in bin/check_bridge_coverage.py read a MATLAB transpose as the start of a string, for example the ' in reshape(varargin,[],numRows)'; catch, end inside run_sql_noOpen. Everything after that line in sqlitedb.m drops out of the member checks. That includes the methods (Static, Access=public) block, whose 8 public static methods have no bridge entry. Fixing the parser turns those 8 into reported problems, so that belongs in its own PR.


Generated by Claude Code

…USY/READONLY retry

Brings the three entries bin/check_bridge_coverage.py reported against
DID-matlab main back in sync.

timestamp (new, DID-matlab bb29861): did.fun.timestamp(raw=None) returns
the current UTC time as 'yyyy-mm-ddTHH:MM:SS.fffZ' and clamps a ':60'
leap second (with or without a fraction or trailing Z) to ':59.999' with
MATLAB's regex.

document (bb29861): Document stamps base.datestamp with
did.fun.timestamp(). The private _utc_timestamp it replaces had a looser
clamp (':60.123' -> ':59.999123'), so it is removed.

sqlitedb (decd155, da83fa3): 30 s busy timeout on open; do_run_sql_query
retries BUSY/READONLY with MATLAB's backoff schedule, reopening the
connection first for READONLY (poisoned handle / DBMOVED); exhausted
retries print the filesystem diagnostics. Because SQLiteDB groups writes
into transactions where mksqlite autocommits, the statement retry runs
only with no transaction open, and COMMIT is retried (never reopened)
through a new _commit(). Deviations are recorded in the bridge entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Dn4gCNCU2jUGdmfPVCAGK
stevevanhooser pushed a commit that referenced this pull request Oct 23, 2026
#91's bridge coverage job reports the same three drift problems #90
fixes. Merging that branch here makes this PR's check meaningful now; the
merge is a no-op once #90 lands on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Dn4gCNCU2jUGdmfPVCAGK
@stevevanhooser
stevevanhooser merged commit aa7a34b into main Oct 23, 2026
7 checks passed
@stevevanhooser
stevevanhooser deleted the claude/port-timestamp-leapsecond-sqlite-readonly branch October 23, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants