[stevevanhooser]

Why

In NDI's lightsheet viewer, adding a downloaded tile to the file cache took 30–60 s, while the download itself took 3–7 s. The time went into this cache.

Changes (src/did/file.py)

1. Thread-safe lock (Python only). get_cache() returns one FileCache per process, and BinaryTable.has_lock was a single flag on it. While one thread held the lock, every other thread saw has_lock True, took the "already held" path, and rewrote the index without the lock. A per-table threading.Lock now serializes threads from get_lock to the outermost release_lock; the lock file still serializes processes. has_lock counts as held only for the thread that took it. reset_lock_state only clears the calling thread's own hold.
2. Binary-search lookups. is_file, add_file and touch scanned the index row by row, reopening it for every row (about 0.2 s per lookup at 20,000 files). The index is kept sorted by name, so these now use find_row(..., sorted=True). find_row's unsorted path, still used by remove_file and check, reads the column once.
3. Opt-in fast polling. checkout_lock_file takes an optional max_poll_interval. When it's given, a waiter checks again after 5 ms and backs off to at most that interval. When it's omitted, a waiter still checks once a second, as before, so long-held locks keep their cadence. BinaryTable.get_lock passes 0.1 s.
4. The cache lock expires after 20 s, as MATLAB's does. get_lock used checkout_lock_file's defaults, a one-hour expiry. A holder killed mid-update (an NDI viewer quitting while a daemon download thread was adding a tile) left a lock that every other process then waited 30 s on and gave up, for the next hour. This was the 30-s-per-tile stall seen in the viewer.

Changes 2 and 3 are also made in VH-Lab/DID-matlab#221; MATLAB already uses the 20 s expiry. Until DID-matlab#221 merges, Python is ahead, and the bridge entries' decision_logs say so. After it merges, the drift check will flag binaryTable, fileCache and checkout_lock_file, and their matlab_last_sync_hash should be bumped to the merge commit.

Testing

• New tests in tests/test_file_cache.py:
  • 24 threads adding to one cache at once: all rows land, the index stays sorted, and currentSize is right.
  • A brief lock held by another thread or another process is taken in under 0.5 s.
  • Opt-in polling hands a brief lock on in under 0.5 s; default polling still waits about 1 s.
  • The cache lock's expiry is at most 20 s.
  • is_file hits and misses are correct, and an unsorted find_row opens the index once.
• Each fix's tests fail without that fix.
• Full suite: 735 passed, 10 skipped, 1 xfailed. ruff and black are clean.
• bin/check_bridge_coverage.py against DID-matlab main reports 3 problems, all present on main before this change: +fun/timestamp.m has no entry, and document and sqlitedb have drifted. This PR adds none.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJ9PUeM7x2vLsg8FcZQfXo

[stevevanhooser]

Bridge coverage vs DID-matlab is red. All three problems it reports come from DID-matlab main moving on since DID-python main last ran CI (Sept 14), not from this PR:

• src/did/+did/+fun/timestamp.m has no bridge entry.
• document.m drifted: bb29861 (leap-second datestamp clamp).
• sqlitedb.m drifted: da83fa3 and decd155 (SQLITE_READONLY retry and reopen).

None of these touches binaryTable, fileCache or checkout_lock_file. The same check run locally against DID-matlab main without this PR reports the same 3 problems. No fix for them exists yet. Each needs its MATLAB change ported, which is separate work, so I haven't added it to this PR.

Once VH-Lab/DID-matlab#221 merges, this check will also flag binaryTable, fileCache and checkout_lock_file. Their matlab_last_sync_hash should then be bumped to the merge commit; the decision_logs already describe the change.

---

Generated by Claude Code