Repository navigation
Conversation
…aster, and note the file a crash died on Large RAWs and layered TIFFs ran the indexing helper's reading past its memory and killed it — Swift's allocation trap, "indexer: exited (5)" — faster than the watchdog's once-a-second tick could note the file. The file stayed at the queue's head, and every restart took it first and died on it again: the ARahim3#182 report has 55,000 files waiting behind one. A picture whose header says its reading would ask for more than the helper's memory limit is now marked "couldn't be read" before decoding: RAW at 192 bytes per pixel (measured 17 B/px on a real 45.44 MP Nikon Z9 NEF on macOS 26.5.1, 782 MB peak, and ≥176 in the field at an 8 GB watchdog — 192 chosen above the field floor, disclosed conservative for modern ImageIO), layered TIFFs at 16 B/px and the stb side at 8 B/px (design estimates, disclosed as unmeasured in the code). Plain TIFFs, JPEGs and PNGs keep no preflight: their scaled decode streams in tiles, measured at 70 MB for a 2,125 MP LZW TIFF and 18 MB for a 100 MP PNG. The watchdog now looks ten times a second. A helper killed on the spot writes a note beside the index from a signal handler (syscalls only, path fixed at install), and the next helper marks the file the crash was on. The Mac app and the Tauri shell treat a fatal-signal exit like a watchdog stop (restart, cap), and the 4-stops-in-10-min cap resets once the queue has moved 100 files between stops, so an index full of oversized RAWs works through them instead of freezing. Which file a crash was on is the hard half, and the pre-push review of this branch pointed out three places where the first version got it wrong: * A note with no path used to mean "mark the queue's head", which it is not: media runs in batches of eight and the whole batch stays pending till the batch ends, so a trap on the third image failed an innocent first one and left the poison queued (7 innocents per poison, with 4 as the restart cap — the queue froze before the bad file reached the head). The note is therefore still pathless (the crash beats `Indexer.nowReading`), and **the app names the file for the next helper**: both apps already see every `file` event, so `Engine` / the shell keep the last one reported and pass it as `DIGUP_STOPPED_FILE` on the restart; the head-mark stays as the fallback for a spawn with no name (a terminal run, a crash before any file event). * A fatal-signal exit was classified from the exit code alone, which is inert on Linux: `ExitStatus::code()` is None for a child killed by a signal, so the shell saw 0 and restarted nothing. The `ended` event now carries the signal (`ExitStatusExt::signal()`, Unix-only; Windows has no signals and carries the class as the SEH exit code — breakpoint, access violation, stack overflow, fast fail, illegal instruction, divide-by-zero, all matched), the Mac list gained SIGBUS (10), and the classification is a small function with unit tests in both spellings. * The preflight itself could trap: the layered-TIFF sizes come from 4-byte LONG tags, and a header saying 0xFFFFFFFF overflowed the parser's own `Int` sum — a trap inside the guard built to stop traps. Sizes above 1<<24 are garbage and get no estimate; the sums are `UInt64` and saturating on both platforms, and the budget comparisons no longer force `Int(budget)` (an absurd `--memory-limit` trapped in the helper's own setup too, via `<< 20`). Measured on this box (Apple M1, 8 GB, macOS 26.5.1): the Z9 NEF is marked in 0.17 s under the 4 GB default limit ("needed an estimated 8.1 GB ... a RAW picture of 45 MP") and decodes in 5.9 s at a 16 GB limit; a queue of NEF+HEIC+JPEG ends 2 done 1 failed with exit 0; a helper killed with SIGTRAP mid-run is followed by a helper that marks the file and finishes the rest (19 done, pending 0). Named marking, end to end against the real helper on a sandbox index of three queued text files with the newest (the queue's head) a.txt: the note alone marks a.txt (the fallback, `state='failed'`, event `stopped` naming it); the note plus `DIGUP_STOPPED_FILE=c.txt` marks **c.txt** and leaves a.txt pending — the innocent head is not failed and the queue moves past the poison. Suite: 138 green (base 125, +13, all of them the new `PreflightTests` suite: 4 layered-TIFF parser tests including the garbage-dimensions case, 3 preflight-through-ImageIO tests, 2 camera-file tests gated on `DIGUP_TEST_RAW`, 3 stopped-file note tests, 1 wording test). `cargo test` 17 green (base 13, +2 crash-class tests, +2 `reap` tests on real children: SIGTRAP gives `(None, Some(5))`, `exit 75` gives `(Some(75), None)`), and `cargo check --tests` is clean. Disclosed, as written in the code: the layered-TIFF and stb bytes-per-pixel coefficients are design estimates, not field measurements; the Windows `CrashNotes` net covers only the SEH classes the C runtime translates there (SEGV/FPE/ILL), so a Windows crash can leave no note — the fallback then marks nothing and the cap bounds the restarts, and the Windows-side blocks (the SEH codes, the `fatal_signal` no-op) compile only for Windows targets, so this box's runs do not exercise them; installing plain signal handlers trades Swift's own crash print for the note (macOS keeps its `.ips` via the re-raise, Linux loses the print, and chaining Swift's handler through `sigaction` is the natural next step); the preflight tests share the helper's global budget (`ImageExtractor.decodeMemoryBudget`) and run serialized, coupled to one another by design; and the GUI paths — the Mac app restarting a helper after a real crash, the shell app on Windows — are not exercised on this box, which runs the helpers and the unit tests, not the Tauri app.
Ocean572
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #182.
Problem
Large RAW images and layered TIFF files could exceed the indexing helper's memory limit during decoding, triggering a Swift allocation trap and terminating the helper.
The shell reported the failure as:
indexer: exited (5)The existing watchdog checked the helper once per second. This was not frequent enough to reliably identify the file being processed when the crash occurred.
The resulting failure could permanently block the indexing queue:
The original report described approximately 55,000 files waiting behind a single problematic file.
Changes
This PR introduces three related improvements:
1. Preflight Oversized Images Before Decoding
The indexer now estimates image memory requirements from image metadata before attempting to decode the image.
If the estimated memory requirement exceeds the helper's configured memory limit, the file is marked as couldn't be read without attempting the expensive decode operation.
Different image formats use different estimated memory coefficients:
These estimates allow the indexer to reject potentially oversized images before they trigger an allocation failure.
Formats That Do Not Require Preflight
Plain TIFF, JPEG, and PNG files are not subject to this new preflight rejection.
Their decoding paths already support scaled, tiled processing, which avoids allocating the entire decoded image in memory.
Measurements from this machine demonstrate the difference:
Rejecting these formats based solely on their total pixel count would unnecessarily exclude images that can be processed efficiently through tiled decoding.
2. Increase Watchdog Sampling Frequency
The watchdog now checks the indexing helper 10 times per second, instead of once per second.
This provides more responsive detection of helper failures and improves the ability to track indexing progress and identify problematic files.
3. Record Crashes and Identify the Problematic File
The indexing helper now writes a
StoppedFilecrash note alongside the index when it is terminated by a fatal signal.The note is written from a signal handler using system calls only, with the destination path established during installation.
When the helper restarts, the application uses the recorded information to identify and mark the file associated with the crash.
Both the macOS application and the Windows/Linux shell now treat fatal-signal exits similarly to watchdog-triggered stops, allowing the existing restart and restart-cap mechanisms to handle them.
Restart Protection
The existing restart protection limits repeated stops to four within a ten-minute period.
However, this could still prevent an index containing many oversized images from making progress.
The restart counter now resets after the queue successfully moves forward by 100 files between stops.
This allows the indexer to work through multiple problematic files without permanently exhausting its restart allowance.
Important Implementation Details
Pre-push code review identified three significant defects in the initial implementation. All three have been addressed.
1. Identify the File Actually Being Processed, Not the Queue Head
The initial implementation attempted to identify the problematic file using the queue's current head.
This was incorrect because media indexing processes files in batches of eight. The entire batch remains pending until processing completes.
For example, if the third image in a batch crashes the helper, the first image is not necessarily responsible.
Marking the queue head would incorrectly fail an innocent file while leaving the actual problematic file pending.
With seven innocent files ahead of a problematic image, repeated restarts could exhaust the four-stop restart allowance before the problematic file ever reached the queue head.
Corrected Approach
The crash note remains pathless because the helper can crash before
Indexer.nowReadingis updated.Instead, both applications use information they already receive through the existing
fileevents:DIGUP_STOPPED_FILEwhen restarting the helper.The queue-head marking behavior remains as a fallback when no filename is available, such as:
fileevent is reported.DIGUP_STOPPED_FILE.This avoids incorrectly failing an unrelated file when the actual crash location is known.
2. Detect Fatal Signals Instead of Relying on Exit Codes
The initial implementation classified fatal crashes using
ExitStatus::code()alone.This does not work reliably on Unix systems.
When a child process is terminated by a signal, its exit code is
None. Consequently, the shell could interpret the result incorrectly and fail to restart the helper.Unix Handling
The
endedevent now includes the terminating signal through:ExitStatusExt::signal()This is implemented for Unix targets.
The macOS fatal-signal classification also now includes
SIGBUS(signal 10), which was previously missing.Windows Handling
Windows does not use Unix-style termination signals.
Instead, the shell identifies supported fatal Structured Exception Handling (SEH) exit codes.
The classification covers:
Both Unix signal classification and Windows exit-code classification are implemented as small, independently testable functions.
3. Prevent the Preflight Parser from Crashing
Review also identified a potential overflow in the layered-TIFF dimension parser.
Layered-TIFF dimensions are obtained from 4-byte LONG tags. A malformed header containing a value such as
0xFFFFFFFFcould overflow the parser's ownIntarithmetic.This created a dangerous situation where the memory guard designed to prevent crashes could itself trigger an allocation or arithmetic trap.
The parser has been hardened:
1 << 24are treated as invalid and excluded from memory estimation.UInt64with saturating arithmetic on both platforms.Int.The last change also prevents an unusually large
--memory-limitvalue from causing an overflow during the helper's own initialization through the<< 20conversion.Performance and End-to-End Verification
Measured on the development machine:
Hardware and environment: Apple M1, 8 GB RAM, macOS 26.5.1.
RAW Image Preflight
Tested using a 45.44-megapixel Nikon Z9 NEF file.
Under the default 4 GB limit, the file is rejected before decoding with the following message:
needed an estimated 8.1 GB ... a RAW picture of 45 MPThis demonstrates that the preflight can identify oversized RAW files without triggering the expensive decode operation.
Mixed Image Queue
Tested with a queue containing NEF, HEIC, and JPEG files.
The indexer completed with:
Crash Recovery
A helper was deliberately terminated with
SIGTRAPduring indexing.The subsequent helper restart successfully identified and marked the problematic file, then continued processing the remaining queue.
Final result:
Stopped-File Identification
Tested end-to-end against the real helper using a sandbox index containing three queued text files:
a.txt,b.txt, andc.txtThe queue head was
a.txt.Two scenarios were tested:
Scenario A: Crash note without a named file
The helper received the stopped-file note without
DIGUP_STOPPED_FILE.Result:
a.txtwas marked as failed using the queue-head fallback.state='failed'.stoppedevent correctly identifieda.txt.Scenario B: Crash note with
DIGUP_STOPPED_FILE=c.txtThe helper received both the crash note and the explicit filename.
Result:
c.txtwas marked as failed.a.txtremained pending.This verifies that explicit crash identification takes precedence over the queue-head fallback.
Testing
Swift Tests
138 tests passed.
The original baseline was 125 tests, with 13 new tests added in the
PreflightTestssuite:DIGUP_TEST_RAW.Rust Tests
17 tests passed.
The original baseline was 13 tests, with four new tests:
reaptests using real child processes:SIGTRAPproduces(None, Some(5)).exit 75produces(Some(75), None).Additionally:
cargo check --testscompleted successfully.Scope and Limitations
The following platform-specific details and limitations are important for acceptance testing.
Memory Estimation Accuracy
The memory coefficients are estimates rather than universal measurements:
Windows Crash Notes
The Windows
CrashNotesimplementation covers the SEH exception classes that the C runtime translates into the supported fatal signal categories (SEGV/FPE/ILL).Other Windows crash types may not produce a crash note.
In those cases, the fallback may not identify or mark the problematic file. The restart cap still limits repeated crashes.
The Windows-specific signal plumbing and SEH exit-code handling compile only for Windows targets and have not been verified on this development machine.
Signal Handler Behavior
Installing plain signal handlers replaces Swift's default crash-printing behavior.
The current implementation re-raises the signal so macOS can still generate its
.ipscrash report. However, Linux loses Swift's normal backtrace output.Chaining Swift's existing signal handler through
sigactionis identified in the code as a potential follow-up improvement.GUI Acceptance Testing
The development machine runs the indexing helpers and unit tests but does not exercise the Tauri GUI application.
The following still require acceptance testing:
Platform-Specific Test Coverage
The two camera-file tests require a real NEF file on the machine running the suite and remain gated behind
DIGUP_TEST_RAW.The three ImageIO-path tests compile out on Linux, where the two stb-side preflight tests run instead.
The new
PreflightTestssuite was introduced with this branch, so all 13 tests in that suite are new.