diff --git a/.agents/skills/project-journal-compatibility/SKILL.md b/.agents/skills/project-journal-compatibility/SKILL.md index a57594de..ab3a49c1 100644 --- a/.agents/skills/project-journal-compatibility/SKILL.md +++ b/.agents/skills/project-journal-compatibility/SKILL.md @@ -25,7 +25,53 @@ Do not use this skill for: ## Mandatory Knowledge -- Baseline compatibility target is `systemd/systemd` tag `v260.1`. +- SDK journal-format compatibility targets `systemd/systemd` tag `v260.1`. +- The portable file-backed journalctl contract targets tag `v262` + (8cc40e0c5e9234bf45084751ac53b1fbfe70b492). Run the pinned full-feature + oracle; host versions and feature-limited builds cannot certify this target. + Query-matrix qualification requires `--journalctl PATH`; every helper must use + that selected oracle, including version reporting and empty-result cases. +- Native PCRE2-8 >=10.42 is CLI-private. Enable CGO only for Go CLI builds + and tests; retain CGO-disabled SDK checks. Declare the C compiler, pkg-config + and PCRE2 development prerequisites in command-inclusive coverage/CI. +- File-backed CLIs preserve native Unix path bytes through selection, facade + opening and discovery. File patterns use platform globbing with a temporary + thread-local environment locale; literal paths bypass expansion. Windows + uses Unicode paths and brace/wildcard selection. Keep these native calls in + CLI modules and retain pure SDK checks. Test replacement-named decoys and + locale-sensitive patterns against the pinned oracle, with portable Unicode + controls in native OS CI. +- Color parity includes policy/environment selection, priority and audit + styles, PCRE2 first-match offsets, multiline preparation, verbose fields, + JSON primitives, separators and table headers. Compare ANSI bytes as well + as visible content; normalize JSON key order without discarding its styles. + Retain buffered match offsets without a second regex evaluation and keep + color-only work off the plain-output path. +- Parser recognition is insufficient. Compare encounter order and terminal + actions, raw required values, empty-versus-absent options, final-action scope, + timestamps/DST/TZif state, binary MESSAGE behavior, status and complete output. + Check calendar rendering independently from bound parsing: POSIX TZ, custom + TZDIR, leap records, transition-free footers and malformed-rule fallback must + agree in entry output, headers and tables. Omitted DST dates must exercise + the configured posixrules file, absent-file defaults and transition flags; + provider defaults alone do not establish native parity. Preserve raw epoch widths and + lazily resolve rendering zones once, without per-row file reads or zone allocation. + Do not let intentional follow termination hide a child failure or a blocked + stdout pipe. Retain native exit status and drain large backlogs to files. +- Use native OS jobs for commands with native dependencies. A CGO-disabled + cross-build of the SDK does not establish native CLI portability. +- Audit complete CLI selector components after a verified mismatch. Unit show + groups and invocation-discovery groups differ; test naming/glob boundaries, + raw field bytes, current-user conversion, ID precedence, global bounds, + directional limits, secondary-ID offsets and frozen follow selection. + Compare supplied oracle binaries explicitly, including live command builders. + Exercise platform fallback matchers on the primary test host. FreeBSD unit + brackets need the shared CLI compatibility matcher: native fnmatch lacks + named classes and collating elements. Validate malformed brackets as well + as positive matches, multibyte-to-byte retry, and native OS execution. Keep + ordinary patterns on native fnmatch without compatibility allocations. + Use indexed + eligibility and metadata getters; do not expand unrelated message payloads. - The test scope is SDK conformance plus file-backed journalctl behavior. - Do not implement daemon-only journalctl commands such as daemon sync, flush, rotate, or relinquish-var operations. - journalctl already treats repeated matches for the same field as OR alternatives and different fields as AND. @@ -126,11 +172,70 @@ Do not use this skill for: reindex/invalidation impact in the active SOW, product scope, and release notes. If not bumped, record why old cache entries remain semantically safe. - Avoid unnecessary DATA decompression, repeated `FIELD=value` parsing, allocation, sorting, hashing, or syscalls in reader/writer hot paths. Treat these as regressions unless tests and benchmark evidence prove they are required. +- Relocated baseline/candidate source copies must include the tests' referenced + fixture and helper files at their original relative paths. Check those + dependencies before running the suite and verify copied bytes. Select Cargo + packages from their manifest names, not directory names or dependency aliases. - Explorer column catalogs must be built from FIELD indexes. The Rust `ExplorerQuery::debug_collect_column_fields_by_row_traversal` switch is a debug-only discrepancy tool; production reports, benchmarks, or compatibility claims with that switch enabled are invalid and must be treated as an explorer defect. +- Indexed aggregation changes must force temporary DATA reuse during nested + posting and ENTRY reads. From `go/`, run + `go test -count=1 ./journal -run '^TestExplorerIndexedWindow'` for both layouts, + all DATA compression modes, + mmap/auto eviction and ReadAt scratch reuse. Keep possible mmap faults in child + processes. Compare full results and indexed counters, including empty/binary + values, singleton DATA, unset/repeated values, time filters, bucket integer + boundaries and partial read-error counters. A small fixture or + a successful zero-row query does not establish the default-window lifetime. + Benchmark matching and unmatched low/high-cardinality queries with + `BenchmarkExplorerIndexedValueOwnership`; record allocations separately from + elapsed time. Use `BenchmarkExplorerIndexedCompressedValueOwnership` with an + explicit `JOURNAL_SDK_EXPLORER_COMPRESSED_CORPUS` for prebuilt codec controls; + check indexed/traversal outputs, actual decompression and update totals before + timing. Freeze the same complete test helper for baseline and candidate, and + exclude fixture construction and reader teardown from timing. An incorrect or + crashing baseline cannot certify a speedup. +- Inspect CLI-helper timer boundaries before interpreting profiles or batching. + The Go reader helper opens/closes a fresh reader and checksums/sorts results + on every `--loops` iteration; its final ExplorerStats describe the last loop. + Pilot one iteration before selecting a batch count for each workload. Keep + CPU-profile post-timer activity and heap-profile forced GC separate from + query-cost claims; sampled cumulative allocation is not peak live memory. +- For Go ENTRY-slot header acquisition, run `TestEntrySlotHeader` controls on + both backends, default/tiny/non-power-of-two windows, boundary headers, + partial errors, closed readers, row pins, eviction and live array growth. + Measure `BenchmarkReaderOpen`, `BenchmarkReaderEntryHeader` and + `BenchmarkReaderAllocation` with the same complete header benchmark helper + in both builds. Record actual allocation classes with + `TestReaderAllocationShape`; raw sizeof does not establish heap cost, and + 64-bit layout preservation does not establish 32-bit layout preservation. + Exclude candidate-only direct-helper correctness tests from both timing + binaries, document that exclusion and run those tests separately. Preserve + the baseline validator's policy when isolating acquisition changes from + other paused work, and verify matching pure-build flags and source hashes. + For uncertain small-open timings, compare the same immutable five-entry + corpus using `JOURNAL_SDK_READER_OPEN_SMALL_CORPUS` and retain a generated-file + control with those same binaries. Preserve earlier results: a changed helper + can alter test linking even when production source is byte-identical. +- For Go native-file opener changes, run `TestNativeReaderFile` uncached on + amd64 and 386. Verify read-only/blocking/close-on-exec flags, typed path + errors, symlink identity, descriptor closure and sparse reads above 2 GiB. + Preserve interrupted-open retry, native malformed-file cleanup and existing + `.zst` source/staging behavior; build non-Linux delegation targets. The direct + Linux opener bypasses Go's internal test-cache input hook, so use `-count=1` + for fixture-dependent tests and retain explicit immutable fixture hashes. + Exclude candidate-only opener contract tests from both timing builds, as for + direct header tests. Primitive syscall savings do not establish full-reader + performance; repeat small, large and higher-level reader/query controls. +- For cached constructor statistics, run `TestReaderAccessorInitial` for + empty/nonempty files and initial mmap failure. Verify explicit counters, + selected/requested modes and fallback reason against a fresh statistics + query. Windows zero-byte raw files fail explicit mmap construction and Auto + falls back to ReadAt; preserve that platform behavior in test expectations. + Exclude candidate-only statistics tests from both timing builds. - Smoke tests are not sufficient evidence for production compatibility. SOW validation must record exact stock systemd version, commands/helpers, stress duration, entry counts, reader counts, and failure criteria. - Common compression-library dependencies are allowed after dependency review. Journal parsing/writing must not depend on systemd/libjournal; CGO and linking to system journal libraries remain disallowed unless the user explicitly changes those separate constraints. - Every external-agent prompt must include the canonical repository-boundary block verbatim from `AGENTS.md` or `.agents/skills/project-agent-orchestration/SKILL.md`. @@ -197,12 +302,117 @@ Do not use this skill for: optional-argument edge cases where bare and explicit-empty forms differ, such as `--boot` versus `--boot=`, bare `--lines` versus `--lines=`, and bare `--case-sensitive` versus `--case-sensitive=`. +- CLI integer controls must exercise native base prefixes/octal, leading and + trailing whitespace, signed 32-bit descriptor limits and explicit ID-offset + signs. Preserve raw Unix match operands separately from raw grep patterns. +- Short-output controls must include multiline and distinct-suffix messages, + width/line/byte thresholds and repeated MESSAGE values; grep/cat select the + first value while short output selects the last. Prove bounded reads stop + before further matching, boot observation precedes grep, and follow-tail + selection/resume state includes nonmatching native rows. +- Check short MESSAGE preparation before printability/truncation: tabs, + completed and malformed CSI/OSC sequences, CR boundaries and show-all raw + controls. JSON duplicate controls must include a binary first occurrence; + a byte array must not become the repeated-values collection. +- Cursor controls must compare final native position as well as selected + output: grep misses, bound crossings, zero counts and after-terminal seeks. + Verify follow cursor-file contents after a skipped suffix or no matches, + rather than accepting file existence alone. Size parsing needs compound + suffix, fraction, unit-order/case and overflow controls, including terminal + help encounter order. Preserve raw source-option presence on Unix. - For verifier changes, run `tests/interoperability/run_verify_matrix.py` and require stock `journalctl --verify --file` plus Rust and Go verification paths to agree on positive regular, zstd/xz/lz4 DATA-compressed, compact, compact plus DATA-compressed, and sealed files, and on negative object type, object size, payload hash, hash-chain, entry-array, header-counter, seqnum, monotonic, and TAG/FSS corruption classes. +- For ordinary traversal or compressed-reader changes, build both conformance + adapters and run `tests/interoperability/run_reader_hardening.py` against an + explicitly selected file-backed stock oracle. Use a new `.local/` output + directory. The paired corpus covers cycles, missing required posting links, + acyclic compatibility controls, facade/directory callers, complete Zstandard + streams and whole-file staging. `--only chains --only streams` does not + validate public crash recovery. Strict verification is a separate gate; + ordinary stock corruption handling can differ from the SDK's completeness + errors, so record the actual expected contract for each fixture. +- For recovery changes, test fresh exact queries as well as queries after a + full unfiltered drain. Earlier traversal can hide a missing-pointer bug by + activating recovery first. Include sparse postings, same-size unpublished + tails, partial arrays, forward/reverse/time positioning, directory refresh, + and strict capture/append rejection. Release DATA guards before cold recovery + reads. Do not replay controlled Explorer callbacks or sampling state to retry + a failed indexed operation. +- Reader recovery must not mutate source journals. Separate static damaged-file + extraction, actual writer publication, and synthetic fault injection in test + evidence. Restoring previously committed bytes is not evidence of journald + publication or a required repair lifecycle. Before adding state or repeated + scans for a restore/retry scenario, establish its supported upstream trigger. +- Recovery EOF controls must separate repeated same-direction/count polls from + reversal, explicit seeks and actual append count changes. Check bounded read + work on an unchanged damaged file and verify source bytes remain unchanged. + Cover raw as well as filtered readers; Go explicit Refresh is a separate path. +- Count-bounded ENTRY-array traversal must reject the first required revisit; + a lazy unbounded-cycle guard alone can miss it before the count completes. + Cover short cycles, descending-prefix/ascending revisits, public Rust Node + iteration and writer tail selection before mutation. Preserve acyclic + backward links and ignore reader links beyond the declared population. + Prefix checks may inspect only already visited arrays, not future headers. +- Requalification must use one stable observed recovery view across every + Boolean branch. Test a failure after an earlier branch was clipped; do not + install a partially qualified expression. Synthetic ENTRY/DATA restoration + can exercise rollback but does not establish a production repair contract. + Pending builders and installed expressions + must retain retry state and known DATA identities, including empty lists. + A failed backward preparation must retain captured populations rather than + reread DATA counts after a later append. Check public expression composition + where exposed. Clearing matches must rewind file/directory/facade readers + without an explicit seek; test this after multiple sparse matches. +- Filter position controls must cover installing at a resolved row, reversal + after EOF and early AND/OR exhaustion, duplicate realtime bounds, and failed + selected ENTRY loads followed by getters and transient I/O retry. Test both SDK and + facade APIs, regular/compact layouts and Go access backends. Directory + controls must include sequential file transitions, partial merge prefetch, + initial unset errors and pending realtime seeks; preserve candidate order + through complete retry/EOF traversal. Explicit filtered realtime search checks + visited posting slots; separately prove ordinary traversal still encounters + damaged skipped slots. Use probe counts plus isolated release comparisons to + establish search work and elapsed cost respectively. + Test recovery first discovered while sequential traversal crosses files, + followed by a later merge-probe error. Internal merge restarts + must reuse the inner routine; one public movement owns rollback of its + selected child without discarding other prefetched candidates. + Install and replace filters at a committed directory row, including group + operators. Preserve pending seeks after an initial failed load. Prove native + sequential discovery with an unvisited later-file error: nearer rows must + remain readable before that file is reached. Raw/filtered mode changes must + reuse proven sequential positions as well as filter replacements. + Audit strict index consumers for suppressed new traversal errors and cached + partial results. Exercise requested facet/source posting chains, positive + counts with missing pointers and actual cache omission with a healthy sibling; + retain documented value tolerance and propagate required traversal failures. +- Exercise recovery through public default constructors, not only a private + enablement switch. Cover file/directory/facade and whole-file compressed input, + normal EOF plus observed status, unchanged source bytes and declared counters, + and strict verification/capture/append rejection. Test live publication with + actual writers separately from synthetic restoration. A zero status is not a + full-index integrity result. Preserve errors for unknown secondary posting + tails and malformed compressed streams. + Run `TestRecoveryEqualCountFacetWithoutPostings` and measure + `BenchmarkRecoveryUnmatchedFacet` when changing Go facet shortcuts: incomplete + unused array capacity can leave visible and declared counts equal while + suppressing postings. Keep the large orphan value out of unmatched key copies; + record allocation evidence rather than inferring it from correct results. +- Before reader performance acceptance, freeze matching helper sources against + baseline and candidate SDKs. Keep immutable fixtures and complete semantic + counters. `tests/benchmarks/run_reader_regression_comparison.py` alternates + order and retains per-process output, measured runs and comparisons even when + a later case fails. An unsuccessful baseline is not a valid timing reference. +- Select representative performance controls from the actual changed work. + Careful code review and high-level tests can suffice for unchanged hot work + or cold-only checks. Use targeted timing/profiling for credible added costs + and observed regressions; do not mechanically expand every API, layout and + backend combination or build a large framework for small changes. Retain + adverse evidence, and distinguish uncertainty from established equality. - Separate reader support for existing historical files from writer feature milestones. - Record excluded upstream tests with a reason and extract file-level behavior where practical. @@ -226,6 +436,16 @@ Do not use this skill for: 7. Run live one-writer/multiple-reader tests for every affected writer and reader. 8. Record benchmark or profiling evidence when the SOW includes performance claims. +## Vacuum Ordering Validation + +- Exercise valid cross-history clock rollback, reversed creation order, ties + and unsequenced archives. Native sorting can have contradictory comparisons; + the approved deterministic history-head merge is an explicit parity exception. + Compare ordinary compatible histories with the pinned oracle and ambiguous + histories with the documented policy, including actual retained files. +- Retain count, size and age controls and single/many-history cost evidence. + Do not catch Rust sort panics or blindly substitute another sorting algorithm. + ## Validation Checklist Before claiming production-grade compatibility: diff --git a/.agents/skills/project-release/SKILL.md b/.agents/skills/project-release/SKILL.md index fc5816bd..15c442ba 100644 --- a/.agents/skills/project-release/SKILL.md +++ b/.agents/skills/project-release/SKILL.md @@ -65,7 +65,9 @@ actions already authorized. Loading this skill does not start a release. The source can precede the workflow's merge commit. Ordinary PRs/pushes do not publish, and later master changes do not change the selected source. 3. Require successful source checks and both full language suites at the - source-declared compiler minimums, with CGO disabled for Go. + source-declared compiler minimums, with CGO disabled for the Go SDK and + a separate CGO-enabled Go CLI test. Command-inclusive suites require PCRE2-8 + development files and pkg-config; SDK publication does not depend on PCRE2. 4. Require all eight Rust crates to pass in dependency order: **common → registry → core → host → log-writer → index → engine → public SDK**. For each new version, CI dry-runs before obtaining temporary publishing diff --git a/.agents/sow/SOW-status.md b/.agents/sow/SOW-status.md index 3d52a1af..63ec8597 100644 --- a/.agents/sow/SOW-status.md +++ b/.agents/sow/SOW-status.md @@ -1,13 +1,31 @@ # SOW Status -Last updated: 2026-10-11 +Last updated: 2026-10-26 ## Current -- None. +- SOW-0045 - Go Reader Alignment Optimization: regression reopened and paused. + Unchanged HEAD reproduces skipped appends after healthy empty/EOF refresh, + including unique-triggered refresh. Separate positioning work remains queued; + a pre-existing directory realtime-seek/getter gap and unfiltered time-probe + error suppression are also recorded for that analysis. SOW-0108 is complete; + SDK-then-CLI execution remains unchanged. + +- SOW-0071 - Runtime Purity And Optional Platform Services: regression + reopened and paused. The static guard misses standard Go process-launch + spellings despite its original acceptance promise. No SDK runtime launch + is demonstrated; scanner design and implementation remain unapproved. + Tracking adds no prerequisite or change to the 0108 -> 0159 -> 0121 order. ## Pending +- SOW-0161 - File Indexer Source Posting Membership: open source-analysis + lead, not activated. Investigate inherited source-map target membership and + captured-tail handling before selecting any behavior/performance change. +- SOW-0160 - Go XZ Reader Dictionary Allocation: open, not activated. Tracks + unchanged cumulative dictionary allocation measured during SOW-0108, separate + from required regression prevention. No decoder redesign is authorized and + the approved current execution order is unchanged. - SOW-0156 - Fixed-Header Consistency Audit: separate follow-up agreed in PR #9. Define invariants, audit both SDKs, then implement/test missing rules. Not activated by the current array-count repair. @@ -40,9 +58,10 @@ Last updated: 2026-10-11 on 2026-06-21. Still valid as Rust quality debt, but live-path work is high-risk and not mandatory before integration/release work. - SOW-0122 - Journalctl Performance Followups: open. Tracks non-blocking - SOW-0121 performance follow-up for indexed `--list-invocations`, portable + SOW-0121 broader performance follow-up for large invocation inputs, portable `--follow` scalability, and large-fixture performance evidence before any - large-archive performance claim. + large-archive performance claim. Mandatory no-regression proof for the new + SDK/journalctl repairs belongs directly to SOW-0159/SOW-0121. - SOW-0123 - Event-Time Lane Fresh-Journal Realtime Clock Seed: open. Records the Rust fresh-lazy realtime floor gap and Go parity/documentation gap for event-time-ordered journals. Project-manager diagnosis and implementation @@ -55,6 +74,45 @@ Last updated: 2026-10-11 ## Recently Closed Or Completed +- SOW-0121 - File-Backed Journalctl Full Parity And Ship Decision: completed. + Directional cursor integration and PR12 fixes pass three internal reviews, + 3,514 native comparisons and146 portable controls. Pushed-head native CI is + pending; historical qualification follows. + The vacuum regression is repaired under approved deterministic history-head + ordering. Both languages pass independent 10,000-set models; native CI passes + Linux/macOS/Windows/FreeBSD with 130 portable controls. Existing 78 utility + oracle observations pass. Final internal review passes the repair; lifecycle + impact review and normal final-commit CI complete publication. Upstream + Codacy configuration retry and coverage upload fail remotely; native tests, + coverage generation and CodeQL pass. These CI limitations stay explicit. + Broader scalability remains SOW-0122; PR #12 is draft, not merged/released. + +- SOW-0159 - Rust/Go SDK Hardening And Reader Compatibility: completed. + SDK directional cursor APIs pass three independent reviews; CLI integration is + completed in SOW-0121. Latest SDK validation is in its closure record. + Both SDKs provide default intact-prefix recovery/status, complete streams, + native filtering and approved position/time/error retry behavior. Go floor + amd64/386/race checks pass 1,995/1,996/1,995 events, excluding only the unchanged + live-stock stress test with a retained isolated pass and earlier deadline + failure. Rust floor workspace/allocative checks pass, including 200 SDK tests. + Final tools pass 552 shared observations and 63/34/42 v262 verifier/directory/ + mixed checks; 35 documentation examples and 16 pages pass. Performance is + accepted under representative evidence and careful review, with uncertain + intervals and adverse history retained. The fresh whole-SOW review has one + positive verdict, five coverage gaps and no verified unresolved blocker. + The complete-and-clean closing audit passes; signed commit 25accee keeps + implementation and completed lifecycle together. Journalctl SOW-0121 later reopened for CLI review findings; its vacuum repair remains accepted. +- SOW-0108 - Cross-Language Reader Window Accessor Architecture: completed + again with the approved Go indexed aggregation lifetime/header/opener repair. + Full/race/386/integrated/floor and cross-platform checks pass. The current + 118-case timing matrix records 79 improvements, 39 inconclusive results and + no regression flag; inconclusive results do not establish equivalence. All + 1,176 measured allocation pairs avoid increases. External review is + adjudicated with no verified blocker; Mimo's P2 is disproven and Qwen/Kimi + coverage gaps remain explicit. Independent findings are owned by paused + SOW-0045/SOW-0071 and pending SOW-0160. Signed prerequisite commit `4c49205` precedes completed SDK + SOW-0159 and reopened CLI SOW-0121. + - SOW-0158 - v0.10.0 Publication: completed. Release run 38123573886 passed both language suites and published/verified all eight Rust `0.10.0` crates. Annotated `v0.10.0` and `go/v0.10.0` peel to `288efe015c45`; fresh exact @@ -131,11 +189,6 @@ Last updated: 2026-10-11 `4619776b74311338b1689b2ab1fb0936b7746e49`; both release tags resolve to that commit, all eight Rust crates are published and non-yanked, and clean Go `1.26.2` and Rust `1.91.0` registry-consumer verification passes. -- SOW-0121 - File-Backed Journalctl Full Parity And Ship Decision: completed - again after repairing its mixed-directory boot-row oracle. The harness now - strips only the stock table header, requires a complete stock oracle, and - compares every normalized boot row; the full-feature mixed matrix passed - 42/42 and tagged-v260.1 directory/query validation passed. - SOW-0136 - Rust Directory Harness Explicit Identity: closed without implementation. Its evidence is merged into the user-approved regression reopening of originating SOW-0115. @@ -406,14 +459,6 @@ Last updated: 2026-10-11 O(1) indexed row collection. Local validation, four-peer high-row comparator evidence, `git diff --check`, and six production-grade reviewer votes passed. Remaining reviewer-discovered edge parity items are tracked by SOW-0109. -- SOW-0108 - Cross-Language Reader Window Accessor Architecture: completed - after regression repair. Rust and Go public file-path verification APIs now - use bounded reader-backed byte sources instead of materializing whole journal - files; Go verifier source slices return owned buffers to avoid mmap/window - aliasing hazards. Focused Rust/Go tiny-window sealed verifier tests, - `go test ./...`, affected Rust package tests, the shared verify matrix, - wiki docs validation, `git diff --check`, and SOW audit passed. All six - reviewers ultimately voted `PRODUCTION GRADE` after the reopened repair. - SOW-0104 - Python Explorer And Netdata Parity To Rust: completed. The Python SDK now carries the full Rust feature surface: Explorer (filters, facets, histogram, FTS, Traversal/Index/Compare strategies, @@ -647,14 +692,9 @@ Last updated: 2026-10-11 and repo-local QEMU FreeBSD validation passed for Rust, Go, Python, and Node.js; FreeBSD no-stock single-file and directory matrices passed; files generated on macOS, Windows, and FreeBSD passed Linux stock - `journalctl --verify --file`; SOW-0071 and SOW-0072 blockers are completed. -- SOW-0071 - Runtime Purity And Optional Platform Services: completed. Core - reader/writer paths in Rust, Go, Node.js, and Python no longer host-probe, - execute subprocesses, or acquire writer locks implicitly; identity discovery - and writer locks are optional helpers; legacy Rust `jf` host identity helpers - were removed; runtime-purity scans cover core, facade, Python I/O helper, and - legacy `jf` runtime files; Linux/macOS/Windows validation and three - whole-SOW reviewer rounds passed. + `journalctl --verify --file`; SOW-0071/SOW-0072 runtime dependencies were + resolved for that validation. SOW-0071 now separately tracks a paused + guard-coverage regression, with no demonstrated SDK runtime launch. - SOW-0067 - Go Cross Platform Portability: completed. Go SDK portability implementation, whole-SOW reviews, Linux/Windows tests, FreeBSD/macOS compile checks, and parent native macOS/Windows generated-file validation passed. @@ -704,16 +744,6 @@ Last updated: 2026-10-11 rows/s single-file and 628k rows/s open-files. Go tests, mixed-directory, cross-language, live regular/compact matrices, read-only reviewers, and audit passed. -- SOW-0045 - Go Reader Alignment Optimization: completed. Go reader now has - mmap-backed Unix access by default, live/snapshot bounds, byte-preserving RAW - field APIs, current-entry payload visitor/enumerator APIs, libsystemd facade - DATA fast paths, non-overlapping directory fast-path coverage, and shared - reader benchmark integration. Compact 100k-row benchmark medians measured Go - single-file `sdk-payloads` live/mmap at 1.07M rows/s and `facade-data` at - 1.09M rows/s versus stock systemd DATA enumeration at 565k rows/s; Go - open-files `sdk-payloads` measured 697k rows/s versus stock systemd open-files - DATA at 532k rows/s. Rust remains faster; the cursor-seek systemd divergence - discovered during review was closed by SOW-0055. - SOW-0054 - Node.js Reader And Writer Rust Port: completed. Node.js now carries the finalized reader/writer compatibility slice where practical under the no-native-runtime policy: byte-preserving RAW field access, active-file diff --git a/.agents/sow/audit.sh b/.agents/sow/audit.sh index 1b2a0630..40b7967b 100755 --- a/.agents/sow/audit.sh +++ b/.agents/sow/audit.sh @@ -145,6 +145,8 @@ scan_sensitive_file() { # The canonical public Git SSH remote is not a personal email address. my $email_line = $line; $email_line =~ s{(? SOW-0159 -> SOW-0121 execution chain without a dependency finding +or a user scheduling decision. Original completion evidence remains below. + ## Requirements ### Purpose @@ -426,3 +435,107 @@ implementation. ## Regression Log None yet. + +## Regression - 2026-10-15 + +### What Broke And Evidence + +Healthy Go iteration can skip the first append after an explicit Refresh or a +unique-field query that refreshes the entry list. Two states reproduce it: +open an empty valid journal, append one row, refresh and Step; or consume a +one-row journal through EOF, append one row, refresh and Step. Both Step calls +return false/nil despite the newly loaded row. QueryUnique triggers the same +transition. A private diagnostic against unchanged HEAD `dafcdac4b971` reproduces +all four combinations; retain `sow-0045-refresh-position-probe.log` and its +synthetic test source under ignored task scratch. + +At HEAD, `go/journal/reader.go:606-609` leaves the empty constructor index at +zero, successful refresh restores the old numeric index at `:561-562`, and +Next increments it before selecting a row. At populated forward EOF the old +index equals the old row count; refresh then makes that index refer to the +first append, which the next increment skips. Automatic Step-driven tail +refresh has a separate adjustment and does not establish that explicit refresh +works. `go/journal/reader_unique.go` and directory unique-cache refresh call +the same refresh path. Commit `6a48ccc5` introduced that preservation behavior +in this SOW; the original execution/validation above claimed live refresh and +live/snapshot parity. + +### Why Earlier Validation Missed It + +Existing live tests begin with a populated reader and use automatic tail +refresh. Explicit-refresh retained-row tests refresh while positioned on a row +and do not subsequently step from EOF. The new SOW-0108 header test initially +assumed empty-open positioning; its intended header/array-growth coverage now +explicitly SeekHead before appending. That adjustment does not repair or claim +acceptance of this separate positioning defect. + +### Repair Plan And Validation + +Keep this queued until scheduled. Before code, specify position preservation +for empty/head/unset, a current row, forward EOF, explicit tail, reverse +movement, repeated/no-op refresh and unique-triggered refresh. A blanket +old-length-minus-one adjustment is not an established complete repair: it can +change reverse movement or expose a current row before a successful Step. +Cover public Reader, facade and directory callers, regular/compact files, +mmap/ReadAt, tiny/default windows, retained payloads, failed-refresh rollback +and snapshot exclusion. Require appended rows exactly once and preserve native +index paths and measured performance. + +Rust is not a confirmed passing oracle for this transition. Source analysis +suggests a possible healthy-tail cached-population issue; this is an unexecuted +working theory requiring a separate paired runtime check before any Rust +finding or scope decision. SOW-0159 already handles its distinct empty-file and +recovery contracts; do not silently broaden that SOW to this healthy-position +state machine. + +### Artifact Impact And Follow-Up Mapping + +The originating SOW is reopened here rather than replaced by a new SOW. +Canonical/root status indexes record its paused state, and SOW-0159 and +SOW-0108 cross-reference this finding. No positioning runtime code is changed. +Synthetic fixtures contain no real journal data. During repair, update the +product live-reader contract, caller guidance and compatibility skill if the +resolved behavior requires it; preserve existing compiler/purity constraints. +Validation, review disposition and a complete-and-clean audit remain required +before this regression can close. This is tracked work outside the active +header/lifetime repair; no completion is claimed. + +### Related Queued Directory Seek-State Finding - 2026-10-16 + +Read-only SOW-0159 impact analysis finds an independent pre-existing getter-state +gap: `go/journal/directory_reader.go:739` clears realtime/candidate state during +SeekRealtimeUsec but retains the selected file index. After a successful Step, +SeekRealtimeUsec followed by GetEntry/GetCursor can therefore expose the old +row before movement. Rust's `rust/src/journal/src/directory.rs:201-208` clears +its index and its getters report NoEntry until movement succeeds. The Go +omission exists in commit `fcca32fe56f4`; SOW-0027 claimed broad facade and seek +support but did not specify getter state between seek and movement. + +Track this alongside the queued healthy-position analysis here; it is not a +consequence of native filtering and adds no SDK-first prerequisite. Before any +runtime change, reproduce both public APIs and facades, determine the affected +seek/refresh state combinations and obtain any unresolved behavior decision. + +### Additional Unfiltered Time-Search Evidence - 2026-10-17 + +SOW-0159's directory error-position controls find another inherited unfiltered +Go seek gap. `reader.go:firstRealtimeIndexAtOrAfter` and +`lastRealtimeIndexAtOrBefore` use a `sort.Search` predicate that treats an ENTRY +read error as a comparison result and then returns a nil error. A backward +merged-directory direction change can consequently skip the damaged candidate +without reporting its ENTRY-read failure. The same source pattern exists in +the frozen SOW-0159 baseline. This is separate from retaining the previous row +after an error that is actually returned. + +Track propagation and retry of errors from unfiltered time probes here with +the existing seek/refresh scope. The approved SOW-0159 binary-search change is +for explicit filtered realtime positioning, and its ordinary unfiltered time +policy remains unchanged. Its Go directory position regression matrix excludes +the unfiltered backward merged case for that specific reason; filtered cases +and sequential unfiltered file transitions remain covered. Before implementing +this item, add a direct file and directory SDK/facade reproducer, compare Rust +and the frozen baseline, resolve any remaining behavior choice, and measure +the existing/new binary-search paths. This SOW remains paused while SOW-0159 +is the sole implementation SOW. +No new position contract or implementation is approved by this tracking note. +SOW-0159 retains its distinct reverse-filter and selected-entry error decisions. diff --git a/.agents/sow/done/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md b/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md similarity index 87% rename from .agents/sow/done/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md rename to .agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md index 14cb3c6a..a0e5df53 100644 --- a/.agents/sow/done/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md +++ b/.agents/sow/current/SOW-0071-20260530-runtime-purity-and-optional-platform-services.md @@ -2,11 +2,16 @@ ## Status -Status: completed +Status: paused Sub-state: implementation, local/macOS/Windows validation, three whole-SOW review rounds, final cleanup, and final local validation are complete. +Regression reopened 2026-10-16: the static core-runtime guard misses standard +Go process-launch API variants. Tracking is paused; no SDK runtime impurity is +proved and no scanner implementation is authorized by this finding. Preserve +the active SOW-0108, then SOW-0159, then SOW-0121 execution order. + ## Requirements ### Purpose @@ -706,3 +711,84 @@ Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative. + +## Regression - 2026-10-16 + +### What Broke And Evidence + +The completed outcome says the static guard rejects subprocess APIs in core +runtime files (Acceptance Criteria: original lines 108-109; Validation Plan: +273-274; implemented scan: 493-496). `tests/runtime_purity/test_core_runtime_purity.py:100` recognizes +`exec.Command(` but not `exec.CommandContext(`. The same pattern collection +also misses `os.StartProcess(` and `syscall.ForkExec(`. In-memory matching of the +actual pattern list confirms the existing positive control and all three +misses. No example process was executed and no core source was mutated. + +The issue was found during SOW-0108 external review. It predates that repair. +The reviewed runtime contains no newly introduced process-launch path; the +reviewed `exec.CommandContext` use is an allowed test-binary child for mmap +failure isolation. This is a guard-coverage regression, not evidence that the +current SDK launches an external program. + +### Why Previous Validation Missed It + +The guard checks current source against selected literal patterns. Passing that +scan does not show that equivalent standard APIs would be detected. The source +list is also explicit, so its intended module coverage needs an audit before +claiming a complete core-runtime boundary. + +### Regression Pre-Implementation Gate + +Status: needs-user-decision; paused tracking only. + +- Problem/root cause: literal API patterns cover one Go spelling without + proving equivalent standard process-launch variants or module coverage. +- Evidence reviewed: the scanner pattern and source lists, this SOW's original + validation promises, AGENTS.md runtime layers and the SOW-0108 read-only + external finding. Private static evidence remains with the originating repair. +- Affected contracts: enforcement of the pure core SDK boundary in both Go and + Rust; tests and optional helpers retain their documented exceptions. No SDK + API, journal bytes or runtime behavior changes are proposed here. +- Existing patterns: reuse the scanner's explicit core/import checks and + representative synthetic snippets; inspect both language surfaces before + choosing the enforcement mechanism. +- Risk/blast radius: an incomplete gate can miss an unwanted core dependency; + an overbroad replacement can falsely reject tests, CLI code or optional + services. Keep runtime source and performance unchanged. +- Sensitive-data handling: use synthetic snippets and relative source pointers + only; no identities, host inputs, credentials or live journals are needed. +- Implementation plan: audit actual core modules and language call forms, + present the bounded enforcement options, record the user's choice, then + implement the approved scanner/test changes as this SOW alone. +- Validation plan: use positive and negative mutation/snippet controls for + supported process-launch variants and the allowed test/helper boundaries; + run the existing purity suite and prove module inventory coverage. Record + lexical-analysis limitations rather than promising detection of arbitrary + language indirection. +- Artifact impact: review AGENTS.md and the compatibility skill for workflow + consequences; update specs only if the enforcement/public contract changes; + consumer API docs need no change for a scanner-only repair. No output skills + exist. Maintain this SOW and both status indexes when reopened, activated or + closed. +- Open decisions: the coverage/enforcement approach requires investigation and + user design approval on activation. Current authorization is tracking only; + no implementation or priority change is authorized. + +### Current Validation And Artifact Mapping + +- Static evidence: the actual regex list flags the `exec.Command` control and + misses all three standard alternatives; no executable fixture was launched. +- Same-failure search: the explicit source list and both-language process-call + patterns are implicated. Full module/alias analysis is required on activation, + not claimed complete by this tracking record. +- Runtime validation: not run for this record because no runtime or scanner + implementation changes. A passing current scan would not disprove the gap. +- Review: the original external finding is non-blocking for SOW-0108 and is + retained there. External-review authorization for this SOW must be decided + separately; authorization for SOW-0108/0159/0121 does not extend here. +- Artifacts: original narrative remains first, this dated regression is appended, + and current/paused state plus both indexes will reflect reopening. No code, + public spec or consumer-document change accompanies tracking. +- Follow-up: all guard repair work remains owned here; no new SOW number or + execution dependency is created. Scope/design approval remains required + before implementation. diff --git a/.agents/sow/done/SOW-0108-20260614-cross-language-reader-window-accessor.md b/.agents/sow/done/SOW-0108-20260614-cross-language-reader-window-accessor.md index e5570ae7..483fcf33 100644 --- a/.agents/sow/done/SOW-0108-20260614-cross-language-reader-window-accessor.md +++ b/.agents/sow/done/SOW-0108-20260614-cross-language-reader-window-accessor.md @@ -38,6 +38,29 @@ mmap aliasing blocker, round 3 found one process-only ledger blocker, and round 4 resolved the final process gate. All six reviewers ultimately voted `PRODUCTION GRADE` for the repaired regression surface. +Regression reopened 2026-10-15: valid-input Go indexed Explorer reads retain an +evictable DATA value across nested posting reads. Both frozen baseline facet +and histogram commands crash. The user approved repairing this +prerequisite first, preserving current SDK work and measuring allocation costs. +SOW-0159 is paused; no concurrent implementation is authorized. +Earlier local placements pass forced-access and cross-language large-fixture +checks but flag a 386 small-open slowdown. The approved Linux native-reader +opener and cached constructor statistics pass full, race, 386, integrated, +compiler-floor and cross-platform checks. The current 118-case timing matrix +finishes with 79 improvements, 39 inconclusive results and no regression flags. +Inconclusive results do not establish equivalence. Earlier small-open timing +flags remain recorded; controlled same-executable and four-binary crossover +repeats show improvements. The timing cause is not established. Private storage +variants remain excluded. Local validation and two independent internal final +reviews pass. External review is adjudicated with no verified blocker: GLM, +Minimax and Deepseek return PRODUCTION GRADE; Mimo returns NEEDS CHANGES with +its sole P2 disproven by source and 18 executed checks. Qwen fails twice with +HTTP 429 and Kimi times out twice without a verdict; both coverage gaps remain +explicit. The approved hybrid header reads retain one fixed 16-byte reader +buffer, fitting native slices and cached readAt fallback. This original SOW +closes with the scoped repair and lifecycle in one signed commit after final +audit and independent exact-revision impact checks. SDK work resumes afterward. + ## Requirements ### Purpose @@ -3934,3 +3957,2805 @@ Disposition: Per-scalar verifier allocation and per-DATA-object scratch allocation remain accepted verifier behavior because verification is an integrity-check path, not a query hot path. + + +## Regression - 2026-10-15 + +### What Broke + +Go indexed facet counting retains a temporary DATA value while nested posting +reads can evict and unmap its window. Converting that value to an owned string +after traversal can crash. Indexed histogram counting retains the same value +across both posting and ENTRY reads. The original lifetime guarantee remains +required; no relaxation is proposed. + +### Evidence + +- A frozen executable built from SDK commit `dafcdac4b971` crashes on a valid + regular 100,000-row, 32-field, approximately 176 MiB synthetic journal using + default rolling mmap and an indexed filter/facet query. An independent direct + invocation reproduced the fault in `runtime.slicebytetostring` from + `indexedCountFacetGroup` after `visitDataEntryOffsets`. +- `go/journal/explorer.go`: `indexedCountFacetGroup` and + `indexedCountHistogram` retain `splitRawPayload`'s borrowed value across + accessor reads. `reader_access.go` temporary slices are evictable; + `reader_access_mmap_unix.go` unmaps evicted windows. +- Commit `a8119457` changed these readers from stable whole-file access to + temporary rolling slices. This SOW explicitly included Explorer's internal + lifetime preservation. The defect therefore reopens this original SOW, + rather than becoming a new SDK-hardening SOW. +- Both facet and histogram failures are reproduced with default mmap settings. + Smaller-window and zero-row baseline controls can silently return wrong keys + even when they do not crash; compare complete outputs before using a baseline + for any timing claim. + +### Why Previous Validation Missed It + +Existing successful tests did not force eviction of a retained facet/histogram +DATA value during nested posting reads. Closed-file validity, small fixtures +and ordinary row-lifetime tests do not establish this temporary callback +lifetime. The prior accepted outcome is contradicted for these callers. + +### Repair Plan And Scope Decision + +The user approved prerequisite repair on 2026-10-15, the recommended +long-term-best option. Repair and validate this original SOW first, preserving +SOW-0159's working changes and measuring allocations, then resume SDK and CLI +work sequentially. The existing no-performance-regression constraint remains. + +The smallest candidate owns a facet/histogram string key before nested reads. +Histogram can reuse one key per DATA object. This candidate can add allocations +for unmatched facet values, so the user's no-performance-regression constraint +still applies. A larger count-before-payload reorder is an alternative only if +measurement rejects the ownership candidate; it changes more error/accounting +paths and requires its own concrete analysis before implementation. + +Do not use current-row pins across an entire FIELD walk: they persist until row +advance and can force unrelated reads through scratch or exceed row budgets. +No public API, memory-envelope or compiler-floor change is proposed. + +### Pre-Implementation Gate For This Regression + +Status: ready for the owned-key candidate and forced-eviction validation; +performance and review remain acceptance gates. + +- Problem / root-cause model: facet and histogram callbacks retain an unowned + value from a temporary accessor slice while nested posting/ENTRY reads may + recycle positioned-read storage or unmap its mmap window. A later string + conversion dereferences invalid storage. +- Evidence reviewed: frozen valid-input baseline facet crash at SDK commit + `dafcdac4b971`; originating commit `a8119457`; both callback bodies in + `go/journal/explorer.go`; accessor eviction/lifetime rules; existing Explorer + parity, reader-window and borrowed-payload tests. Rust already owns the + corresponding value before posting callbacks; no Rust runtime repair is needed. +- Affected contracts and surfaces: Go indexed facets and histograms, both + compact/regular formats, mmap/ReadAt and compressed/uncompressed DATA. Public + results, unset counts, stats, error propagation and memory bounds must remain + equivalent. SDK-hardening and CLI behavior remain separately owned. +- Existing patterns to reuse: owned Go string map keys, existing native posting + traversal and candidate sets, traversal-strategy parity oracle, current + reader-window stats, retained matched-helper binaries and synthetic corpora. +- Risk and blast radius: a copy before nested reads fixes lifetime but can + allocate for unmatched DATA. Histogram can reuse the owned key instead of + copying once per matching row. Measure both dense and sparse/high-cardinality + workloads; do not accept an allocation/timing regression or treat a crashing + baseline as a faster implementation. +- Sensitive data handling plan: synthetic identities and fields only. Keep + fixtures, crash logs and raw measurements under ignored `.local/`; durable + evidence uses public commit IDs, relative source paths and aggregate results. +- Implementation plan: preserve SOW-0159 source changes. Own each facet or + histogram key while the payload is still valid, then reuse it after nested + access. Add forced-eviction and scratch-reuse coverage plus matching/unmatched + allocation benchmarks. If measurement rejects this candidate, analyze the + count-before-payload alternative before changing implementation strategy. +- Validation plan: reproduce the baseline histogram failure; exercise both + layouts/backends and compressed DATA in child processes where an mmap fault + would terminate the test runner. Compare indexed/traversal and Rust results, + stats and unset behavior. Run the Go Explorer/window suites, real 176 MiB + fixture, paired valid-control performance/allocation checks, same-failure + search, authorized whole-regression review or explicit waiver, and clean audit. +- Artifact impact plan: preserve existing lifetime specs; add regression test + procedures to the runtime compatibility skill, update docs only if usage or + guarantees change, and update this SOW plus both status indexes. Stage only + the repair's exact source hunks and artifacts; do not commit paused SDK work. +- Open decisions: scheduling and lifetime contracts are approved. The updated + numbered header-allocation design choice below supersedes the earlier + direct-slice proposal, which fails the cached-boundary performance constraint. + No header strategy has been transferred into the prerequisite runtime. + External review authorization or waiver is required after local validation. Performance + failure requires a concrete new candidate, never a silently relaxed gate. + +### Validation Plan + +- Force mmap eviction and positioned-read scratch reuse in facet and histogram + paths, including exact filters, matching/unmatched values and both layouts. +- Compare complete indexed results with the traversal strategy and the Rust + sibling; preserve unset accounting, statistics and bounded memory. +- Measure allocations and timings for sparse/high-cardinality unmatched facets + as well as matching values. A crash or incorrect baseline cannot certify a + speedup. Do not weaken SOW-0159's performance gate. +- Keep the existing SOW-0159 working changes uncommitted and separately owned. + Any prerequisite commit must contain only this regression's repair/artifacts; + validation must disclose when both working surfaces are present. +- Obtain the required regression review disposition, update affected artifacts, + run the complete-and-clean SOW audit and close this repair before resuming + dependent implementation. + +### Artifact And Follow-Up Mapping + +The existing product/accessor lifetime specs already require the correct +behavior; update their evidence if repair changes the contract. Update runtime +compatibility guidance for forced-eviction indexed-query checks. Consumer docs +need a change only if public usage or guarantees change. Canonical/root status +indexes record this reopened, active regression. The original narrative and +prior verification-repair history remain intact above. + +### Current Validation And Outcome + +Source/lifetime investigation and both baseline crash reproductions are complete. +Prerequisite repair is authorized and active; SOW-0159 is paused. The previous +`hybrid-slots-1` placement passes 303 focused native/integrated and 32-bit +controls, the standalone full Go suite (1,115 leaves), 311 race cases and all +16 large-fixture language/strategy checks, but reproduces a 386 regular-small +open slowdown. Its initial/repeat timing evidence remains retained. Earlier +`hybrid-packed-1` keeps the same hybrid algorithm and packs two private booleans: +measured sizes/classes are amd64 624/640 and 386 524/576. Its five pure builds +and allocation-shape controls pass; full functional/performance revalidation +was not completed because its small-open timing flag persists. The user +subsequently approved the Linux native-reader opener; `native-open-1` passes +new amd64 descriptor controls and shared-file mmap timings, but its longer +original-layout 386 repeat flags a slowdown. Later controlled same-executable +and four-binary crossover repeats show improvements; the cause of the earlier +discrepancy remains unproven. The current native-open-stats-1 candidate passes +full/focused/race, 386, integrated, compiler-floor and non-Linux build checks. +Its complete performance matrix is running. Private storage variants did not +establish a cure and remain excluded from the root. +Earlier candidates' failed compact timings or increased allocation costs remain +recorded below. Full functional/performance acceptance of the current candidate +remains open; no earlier placement's results substitute for its complete gate. +No review approval or completed outcome is claimed. Current SDK work remains +preserved and separately owned. + +### Repair Execution - 2026-10-15 + +- Reproduced both facet and histogram fatal faults with the frozen pre-change + SDK on the valid 176 MiB corpus, default 32 MiB mmap windows and 100 returned + rows. Zero-row and smaller-window controls can succeed; they do not establish + lifetime safety and cannot replace the failing production settings. +- Added child-isolated indexed-versus-traversal checks across both layouts, + uncompressed/Zstandard/XZ/LZ4 DATA and ReadAt/mmap/auto access. Before the fix, + all twelve uncompressed facet/histogram cases fail with wrong results; the + owned decompressed-payload controls pass. The test forces eviction or scratch + reuse and checks rows, counts, histograms and memory bounds. +- Retained a pre-repair test executable and source snapshot with all paused + SOW-0159 changes present, then implemented only early owned keys in facets and + histograms. This isolates the prerequisite repair for before/after benchmarks. + Matching/unmatched low/high-cardinality allocation benchmarks are included; + no performance or review acceptance is claimed yet. + +### Ownership Candidate Measurement And Revision - 2026-10-15 + +The early-copy candidate passes all 48 lifetime cases but is rejected by the +performance gate. Ten alternating measured pairs after two warmups, pinned to +one CPU, show sparse high-cardinality facets 54.0% slower and histograms 53.8% +slower. Facet allocations rise from 1,922 to 2,930 per operation and histogram +allocations from 1,935 to 2,933; approximately 645 KiB of unnecessary key copies +are added per operation. Dense histograms improve, but that does not compensate +for the filtered-query regressions. Raw results and immutable executable hashes +are retained under `.local/`; no acceptance is inferred from the passing tests. + +The next candidate keeps payload validation before posting traversal and owns a +key only when a matching result needs it. This is an implementation refinement +within the approved lifetime repair; it changes no public contract: + +- Facets acquire their key at the first matching posting. The inline posting + callback precedes any nested read, so its original value remains valid. + Compressed values are owned decoder results. For a later uncompressed posting, + save the value's file offset/length before traversal, reacquire that byte range + and immediately copy it. Reuse the key thereafter. +- Histograms acquire their key at the first actual bucket update, after ENTRY + validation and time checks. Reacquire an uncompressed value by its saved + offset/length because the ENTRY read can invalidate even the inline value. + Compressed values need no second decode. A separate readiness flag preserves + valid empty values and prevents repeated copies. +- Unmatched DATA and values with no bucket update allocate no key. Each matching + uncompressed DATA requires at most one additional temporary accessor read; + measure that cost against the valid, non-evicting baseline. Keep existing + counters, unset tracking, payload-error precedence and posting error propagation. +- Do not reorder postings ahead of payload validation or add row pins, buffers, + duplicate decompression or a new accessor API. The paused SDK chain guards + must not become a dependency of this prerequisite repair. + +Validation adds empty/binary values, time-excluded histogram values and malformed +posting/ENTRY errors to the forced-access coverage. Rebuild both benchmark +executables together if their shared test helper changes. Re-run the paired +allocation/timing checks, the default-window reproductions and the standalone +repair without SOW-0159 before claiming local completion. + +The first lazy candidate passes all 78 expanded child cases, including complete +indexed statistics and actual compressed-DATA activity. Its facet allocations +match the baseline exactly. Histogram allocations decrease in every measured +case and dense histogram elapsed time improves by 35.6% at low cardinality and +5.5% at high cardinality. However, facets remain 2.6%-5.8% slower across the +four matched cases, so this candidate also fails acceptance. The added +first-match branch executes for every matching posting; moving ownership outside +that callback is the next measured refinement, not a relaxed performance gate. + +For the next facet candidate, preserve the original posting callback exactly. +When every posting is a candidate, own the required key before traversal. For +filtered candidates, count first and acquire a key only if the count is nonzero; +reacquire uncompressed raw bytes only when there is more than the inline +posting. A single-inline traversal performs no accessor read; compressed values +already own their bytes. This preserves the same validation/error order and +zero unmatched-key allocations while removing the added per-posting branch. +Retain the histogram candidate, which already waits until an actual update. +Qualify the all-postings shortcut once using existing population metadata: +the visible ENTRY count must equal the declared header count. A recovered +prefix can suppress all of a DATA object's postings even in an all-candidates +query; those views retain count-first ownership without depending on paused +SDK recovery fields. Empty values need neither a reread nor an allocation. + +The count-first candidate passes all 78 cases. Ten matched pairs show six +improvements (both high-cardinality facets and all histograms); the two +low-cardinality facets are inconclusive, not accepted. Their allocations match +the baseline. Assembly inspection establishes that the inner posting callback +has identical instructions and captures, with the same 40-byte frame. The outer +DATA callback grows from 248 to 288 stack bytes and computes/spills a raw value +offset even when it will never be used. The next local refinement computes that +offset only inside the reacquisition branch, using the validated uncompressed +DATA suffix (`object size - value length`), and postpones the compression flag +until after posting traversal. It changes no reads or result ordering. + +Standalone HEAD-plus-repair validation initially ran before its read-only test +input links were ready: all Go packages except the journal package passed, and +that package reported only a missing FSPRG-vector input. The input exists in the +main repository; this is a private validation-workspace setup failure. Correct +that setup before rerunning the failed test and the final complete suite. Do not +report the initial run as passing. + +### Independent Large-Fixture And Standalone Evidence - 2026-10-15 + +- A private source archive from HEAD contains only the two Explorer repair + functions and the new regression tests; it contains none of SOW-0159's runtime + changes. The failed FSPRG input test passes after the input-link setup is + corrected. Final full-suite validation is still required after the last change. +- Its standalone Go reader helper passes the original default-window faulting + query on the 176 MiB regular fixture and the 128 MiB compact fixture. Both + indexed and traversal strategies agree with the frozen, unchanged Rust SDK: + 16 language/layout/strategy/operation observations, each with 100 returned + rows, matching row-byte/checksum and facet/histogram checksums, and 6,250 + aggregation updates. The original default-window failure is therefore fixed + independently of paused SDK work. +- The non-crashing frozen Go controls also produce wrong value checksums. + Neither a successful exit nor the same aggregate total establishes their + correctness. Use the small, non-evicting matched benchmark fixtures as valid + baseline performance controls; do not count the crashing or incorrect large + runs as speed comparisons. +- The large-fixture single-run timings are correctness/reference evidence only, + not an accepted performance comparison. Rust supplies the sibling aggregation + oracle. Stock journalctl has no equivalent facet-count/histogram operation; + unique-field enumeration does not preserve multiplicities or time buckets. +- Source search found no additional internal instance of the same retained + temporary DATA pattern. Netdata boot discovery owns the key before nested + reads; row classification copies retained labels; snapshot and unique-value + callbacks consume/copy values before posting access. Rust indexed aggregation + already owns its value while the DATA guard is alive. No Rust runtime change + is required to restore the existing paired lifetime contract. + +### Final Facet Callback Refinement - 2026-10-15 + +Twenty alternating 500 ms pairs after the deferred-offset change still leave +low-cardinality facet timings inconclusive: dense median ratio 1.00013 with +interval 0.99373-1.01167; sparse ratio 0.98358 with interval 0.96892-1.01319. +Keep that gate open. Assembly confirms the original callback tests the same +all-candidates flag for every posting. Select the all-candidates or filtered +callback once per DATA instead: both retain the original reference/count/map +update order, while the filtered callback captures only the candidate map and +the dense callback needs no membership test. Hoist these inputs outside FIELD +traversal to avoid copying the complete candidate structure into closures. + +This uses the same native posting visitor, maps, error propagation, public API +and ownership rules; it introduces no capacity guesses or additional indexes. +The selection uses candidate membership independently of the complete-population +ownership shortcut, so recovered views retain their existing clipping behavior. +Check escape analysis and allocations, all regression cases, standalone full +tests and the matched performance matrix before acceptance. + +### Current Standalone Gate And Proposed Allocation Scope - 2026-10-15 + +- The final callback-specialized ownership candidate passes the complete + pristine-HEAD-plus-repair Go suite with `CGO_ENABLED=0`: all five packages with + tests pass; the journal package completes in 85.584 seconds. Tests, benchmarks + and source archives remain repository-local. The earlier input-setup failure + is corrected and its focused FSPRG retry also passes. +- The isolated before/after performance matrix has eight completed cases. + Dense low/high-cardinality histograms improve (ratios 0.66786 and 0.95036). + The other six cases are inconclusive; no performance acceptance is claimed. + Facet allocation counts/bytes are unchanged in all four cases. Histogram key + reuse reduces allocations/bytes in every case. Combined-SDK candidate timings + cannot substitute for this isolated prerequisite evidence. +- Allocation profiling finds ENTRY-array header checks responsible for much of + the remaining cost. Original `validEntryObjectOffset` creates a 16-byte buffer + per posting; the paused SDK implementation already uses a temporary accessor + slice and parses it immediately. Its wider checks/recovery rules remain owned + by SOW-0159. The buffer-read optimization alone could be separated without + changing those original validation rules. +- A numbered scope decision is pending: include that existing buffer-read + optimization in this prerequisite and validate reader opening plus aggregation, + or retain facet/histogram-only scope and continue profiling. The recommended + long-term-best option keeps the performance gate and removes unnecessary + allocation using an existing immediate-parse pattern. No scope expansion or + additional header-reader implementation is authorized until the user answers. +- Independent race validation of the already approved ownership repair passes + in 90.209 seconds. External review remains unauthorized; request its disposition after + local validation and the performance gate are resolved. + +### Audit Fixture False Positives - 2026-10-15 + +The expanded changed-file audit exits 2 for two address-pattern matches in +`go/adapter/main.go` and `rust/src/adapter/main.rs`. Inspection confirms both are +the same deliberately malformed, all-zero journal archive filename, already +present at HEAD, used to test filename error classification. They contain no +personal address or private data. Do not alter the compatibility fixtures or +mask genuine addresses to make the audit pass. + +The active SOW's required audit-repair gate owns a minimal recognition fix: +remove only that exact double-quoted all-zero filename token from the email +candidate text before matching, as the audit already does for the canonical +public Git SSH remote. Actual address tokens on the same line must still be +reported. Validate the exact Go/Rust fixture shapes, adjacent addresses, +nonzero near-misses and suffix near-misses, then rerun the expanded audit. +This changes no SDK or CLI behavior and is independent of the pending header +allocation scope decision. Record the resulting clean verdict before closing. + +Seven isolated pattern controls pass: exact Go/Rust filename fixtures are +excluded, while a synthetic address, an adjacent address and nonzero/suffix/ +unquoted near-misses remain detected. Shell syntax validation passes. The +expanded changed-file audit then exits zero with the full verdict +`SOW initialization complete and clean`; the initial audit failure is resolved. + +The standalone Explorer and reader-window/borrow race selection also passes +in 90.209 seconds. These checks use the ownership-only repair; the proposed +ENTRY-header allocation change remains unimplemented in this prerequisite +while its scope decision is pending. + +A separate pristine-HEAD run of the new regression tests fails 20 of 78 parent +subcases: ten uncompressed lifetime cases and ten empty/binary value cases; +compressed and malformed-index controls pass. The repaired standalone suite +passes all 78. An initial rough log count included the nested child failure +report twice; the retained structured summary counts parent subcases only. +ReadAt facets happen to pass in the original copy-based header implementation; +the paused SDK's temporary-header path exposes scratch reuse there too. The +repair covers both without depending on that paused change. + +### Additional Profiling While The Scope Decision Is Pending + +Callgrind collection is limited to `indexedCountFacetGroup` and its callees, +using retained pristine before/after executables, one operation per process, +two warmup pairs and twenty alternating measured pairs on one CPU. This is an +instruction metric, not native elapsed-time evidence. Low-cardinality dense and +sparse median ratios are 0.99879 and 0.99860; their intervals still cross one +(0.99438-1.00032 and 0.99444-1.00357). It does not close the performance gate. +Raw profiles, commands, individual runs and executable hashes are retained. + +The proposed direct header read is lifetime-safe because parsing copies scalar +fields immediately and posting-array buffers already own their bytes. Source +analysis identifies an access-pattern distinction to validate if approved: +cross-window headers currently assemble cached chunks, whereas a contiguous +temporary read can use scratch ReadAt. Warm boundary performance, tiny windows, +short/wrong/zero headers and borrowed-row pin pressure therefore join the +reader-open/scan validation plan. No original structural checks, recovery rules +or public API changes belong in this proposed allocation-only transfer. + +### In-Progress Implementation Review + +An independent read-only source review finds no new lifetime, count, error or +unset-accounting defect in the two-function repair. It confirms raw suffix +arithmetic, descriptor-only length reads after eviction, the singleton/no-reader- +access path, owned compression output, empty-key readiness and unchanged +membership semantics. The same-failure search finds no additional internal +borrowed DATA use spanning nested reads. This is ordinary implementation review, +not the required authorized whole-SOW acceptance review. + +The review qualifies one future recovery interaction: equal visible/declared +ENTRY counts do not prove recovery is inactive when only unused ENTRY_ARRAY +capacity is missing. A recovered view can suppress all postings of a DATA even +with equal global counts. The prerequisite's ordinary walker always visits its +positive inline posting, so no recovery dependency belongs here. SOW-0159 must +handle that interaction when it resumes; the actual current SOW file records +the required optimization qualification and validation. No unmatched-allocation +claim for every recovered view is accepted by this prerequisite evidence. + +The time-range value fixture is strengthened so an additional DATA value occurs +only outside the requested interval. This tests a value with no contributing +posting, in addition to empty/binary values. Benchmark fixture/functions are +unchanged; retained matched benchmark binaries remain the measured artifacts. +All twelve strengthened shape cases pass in the isolated repair (0.142 seconds). + +### Larger Valid-Control Comparison - 2026-10-15 + +While the header-allocation scope decision is pending, validation continues +within the approved two-function repair. Build matched reader helpers from +pristine HEAD and HEAD plus that repair, with the same helper source and CGO +disabled. Compare the immutable 100,000-row regular/compact corpora using an +explicit 256 MiB window, which holds each fixture without eviction. This makes +the original baseline a valid control; its default-window crash/wrong output +cannot serve that purpose. + +The retained driver checks every result against the unchanged Rust helper, +compares complete Go indexed counters between phases, and requires matching +returned-row and aggregation checksums. Cover low/high-cardinality facets and +histograms, filtered/unfiltered, with two warmup pairs plus ten alternating +measured pairs on one CPU. Retain every command, stdout/stderr, GNU time/RSS +record and fixture/executable hash. This supplements the small-fixture results; +it does not certify default-window baseline performance or erase inconclusive +cases. The pending header optimization remains outside the prerequisite. + +All sixteen large controls finish with matching Rust semantics and complete Go +indexed counters. Seven timings regress, three improve and six are inconclusive. +The regressions include regular low-cardinality dense facets (ratio 1.0698), +regular high-cardinality sparse facets (1.0358), regular high-cardinality dense +and sparse histograms (1.0995 and 1.0357), compact low-cardinality dense facets +(1.0232), compact high-cardinality sparse facets (1.0313), and compact +high-cardinality dense histograms (1.0311). Each listed interval excludes one. +Retained commands, paired observations and intervals are in the private +`sow-0108-large-valid-control-performance` workspace. The candidate fails the +performance gate; small-fixture improvements do not override these results. + +### Counter And Singleton Refinement Plan - 2026-10-15 + +The singleton histogram path currently rereads each contributing raw value +after its ENTRY read. That adds an accessor operation for every unique DATA, +where no reuse can repay the extra read. Keep the existing owned-string +approach, but acquire a singleton key before ENTRY access when candidate +membership and query/bucket bounds prove that it must contribute. Hoist the +proof once per histogram: no realtime query bounds, nonempty buckets, normalized +first-bucket width, and the maximum possible unsigned timestamp quotient fits +in `int`. The last condition preserves the existing bucket-index behavior for +extreme timestamps and 32-bit builds. A singleton posting invokes its callback +before any accessor read, so the raw value is still valid there. Filtered-out +postings, time-bounded queries and uncertain bucket ranges retain lazy ownership. +No key allocation is added to valid no-update paths; invalid ENTRY errors can +occur after the same required key was acquired. Compute the raw suffix offset +only when the fallback actually rereads the value. + +Batch facet reference statistics after each DATA posting walk. Dense callbacks +already count every delivered posting; filtered callbacks keep a separate local +visited count. Add the actual count before checking the posting-walk error, so +partial error statistics and later reread-error precedence remain unchanged. +There are no progress callbacks inside these indexed walks and progress +snapshots copy statistics by value. This removes a per-posting result write +without changing native traversal, membership, maps, allocations or public APIs. + +These are implementation refinements within the approved two-function repair; +the header allocation transfer remains pending. Recheck forced-access/error +cases and compiler captures, then measure both small allocation controls and +the larger valid-control matrix using fresh immutable candidate executables. + +The singleton/scalar-capture candidate passes all 126 forced-access cases then +measured. The small allocation matrix has two improvements, one regression +(filtered high-cardinality facets, ratio 1.0400, interval 1.0051-1.0643), and five +inconclusive cases. Facet allocations are unchanged; all histogram allocation +counts decrease. The large valid-control matrix has one improvement, eight +regressions and seven inconclusive cases. Performance acceptance remains open. +The fresh private workspaces are `sow-0108-singleton-scalar-small-performance` +and `sow-0108-large-singleton-scalar-performance`; no unsuccessful result is +discarded or averaged against an unrelated improvement. + +Native CPU profiles of 150 complete regular low-cardinality facet queries +retain matching complete results. Baseline elapsed time is 7.536 seconds and +candidate time is 7.870 seconds. Posting-header validation remains a major cost; +the sampled increase is concentrated in unchanged row-scan/header-access +callees, so the profile does not prove that one new ownership branch explains +the whole slowdown. These single profiles are diagnostic, not paired acceptance +evidence. Compiler analysis confirms all aggregation callbacks stay off the +heap and the histogram captures only the DATA size scalar, not a whole header. + +The next narrow refinement reduces callback setup and captures. Select the +facet posting callback once per facet, outside DATA traversal, and reuse local +count/visited scalars reset for each DATA. Indexed walkers are synchronous and +non-reentrant, so this preserves the same counts and error flush. In histograms, +capture only the candidate all flag and offset map rather than copying the +56-byte candidate structure into both callbacks. These changes do not add +reads, allocations, caches or new public behavior. New direct-helper tests +cover partial facet counters and singleton bucket integer boundaries, zero +width and empty buckets; validate these along with the prior forced-access +matrix before accepting another measurement. + +The hoisted-callback candidate passes all 142 current focused cases, including +the twelve explicit singleton bucket-boundary cases and four direct-helper +partial-counter cases. Those direct tests retain the private result on error, +since the public Explorer API correctly discards an unsuccessful result. No +partial-result public contract is added. Compiler analysis confirms the reused +facet closures do not escape and histogram callbacks capture the candidate map +and flag rather than the full candidate structure. Large valid-control and +small allocation comparisons remain required for this candidate. + +The completed hoisted-callback small matrix has three improvements and five +inconclusive cases, with no measured regression in that matrix. Facet +allocations remain unchanged. Low-cardinality dense histogram allocations fall +from 3,694 to 1,912 per operation; high-cardinality dense histograms fall from +4,464 to 3,702. Sparse histograms also allocate less (1,973 to 1,948 and 2,728 +to 2,718). These counts apply to the matched 2,048-row fixture only. + +The large valid-control matrix still fails the gate: four regressions, two +improvements and ten inconclusive cases. Regressions are regular filtered +low-cardinality facets (ratio 1.02251, interval 1.00672-1.05379), regular dense +high-cardinality histograms (1.17639, 1.01987-1.36940), regular filtered +high-cardinality histograms (1.04482, 1.02804-1.05610), and compact dense +high-cardinality histograms (1.03099, 1.01082-1.09616). The wider interval in one +case is retained, not converted into a performance claim. Every paired run +still matches the unchanged Rust helper and complete Go indexed statistics. +Evidence workspaces are `sow-0108-hoisted-small-performance` and +`sow-0108-large-hoisted-performance`. + +The latest standalone helper also passes the original default-32-MiB-window +large-file query: all sixteen language/layout/strategy/operation checks agree +on complete aggregation and 100 returned rows. Fresh 32-bit Go execution passes +the twelve singleton bucket-boundary cases (0.162 seconds), including the +platform-dependent integer conversion fallback. This is correctness evidence; +32-bit performance has not been measured. + +A second read-only implementation inspection confirms that hoisted visitors +are synchronous and never retained; serialized Reader use remains required. +The existing accessor has no cheap validity probe for a temporary value. +Diagnostic counter snapshots scan windows and do not guarantee invalidation +detection after a failed scratch read. Do not replace ownership with a guessed +counter or backend-specific pointer check. The pending allocation-only header +scope decision remains separate; no transfer has been made. + +The latest pristine-HEAD-plus-repair full Go suite passes with CGO disabled and +four Go execution threads; all five packages containing tests pass and the +journal package completes in 100.298 seconds. The expanded changed-file audit +also returns the complete-and-clean verdict, and `git diff --check` is clean. +The new candidate's race selection is still running. Performance acceptance, +the pending scope choice and authorized review/waiver remain unresolved; this +SOW is still in progress and neither requested SDK/CLI SOW has resumed. + +### Histogram Callback Reuse Plan - 2026-10-15 + +The remaining large high-cardinality histogram failures exercise a new posting +closure for each DATA object. Compiler output shows that closure stores the +value/key state plus candidate/reader/result metadata repeatedly. Reuse one +synchronous posting visitor for the histogram query, as the latest facet path +already does. Reset value, compression, singleton eligibility, DATA end, key and +key-readiness before every valid DATA walk. Preserve all membership checks, +ENTRY access, timestamp/bucket checks and error propagation in their existing +order. Clear the borrowed/decoded value immediately after the posting walk, +before either returning its error or loading the next DATA; otherwise the +hoisted slice could unnecessarily retain a previous decompression buffer while +the next one is allocated. The map key remains owned. + +This changes callback setup only, stays inside the two approved functions, and +introduces no new accessor interface, buffer policy, public API or native read. +Keep earlier candidate binaries/results immutable. Validate the complete +forced-access matrix and compiler escape behavior, then compare both timing +matrices again. The separate header-allocation transfer remains unapproved. + +Clear the shared owned key after each walk as well as the value slice. This +avoids retaining an unused duplicate string while the next DATA is decoded; +initialization and the previous-walk clear establish the empty key before the +next valid DATA. No additional allocation or clearing pass is needed. + +Measurement correction: earlier small comparisons reused an unchanged +benchmark fixture/function against a baseline binary with the older test +catalog. The added singleton/boundary/error tests can change linked code +placement even though they are not timed. Retain those runs as exploratory +observations; do not use them as the final matched timing gate. Fresh HEAD and +HEAD-plus-two-functions source archives now contain the exact same complete +142-case test file, verified by hash, and build both timed binaries with the +same flags. Diagnostic escape-analysis compilation is separate from the timed +executables. Large helper controls are also rebuilt with identical flags and +the same helper source. No prior successful timing gate is being withdrawn: +all earlier candidates already had an open or failed performance gate. + +The preceding hoisted-facet candidate's race selection completes successfully +in 229.020 seconds. The new reused-histogram candidate separately passes all +142 focused cases. Its diagnostic compilation confirms both reused visitors +and shared value/key state remain on the stack; there is no new closure heap +allocation. Final timing/whole-suite evidence must name this new executable; +the preceding candidate's full-suite/race results do not certify the refinement. + +The reused-histogram large single-query comparison retains six regressions, +one improvement and nine inconclusive cases, with all semantic/statistical +completeness checks passing. In particular, compact low-cardinality facets +change substantially even though their source path is unchanged by the latest +histogram-only refinement. That observation does not establish a cause. Before +making further code changes, run the same paired harness with the exact same +baseline executable in both phases. This qualifies measurement variability +without pretending an identical implementation has improved or regressed. +Retain the original candidate results; the performance gate stays open. + +The identical-binary control completes with all sixteen intervals including one +and all results/counters matching. It finds no phase bias in that run. This +does not invalidate the candidate's regressions or establish equivalence of a +changed implementation. The fully matched small allocation comparison is now +running; no additional code refinement or header-scope transfer has begun. + +The fully matched eight-case small comparison completes: two improvements, +five inconclusive cases and one regression. Filtered high-cardinality +histograms regress (ratio 1.06029, interval 1.01727-1.13308), despite allocating +less. All facet allocation counts/bytes remain the same; all histogram counts +decrease by the same amounts as the earlier observations. The new reuse +candidate therefore also fails the timing gate. Its matched source/binary +hashes, commands, per-process output and allocation samples are retained in +`sow-0108-reused-clear-small-performance`. + +The scope question is presented again with the completed evidence: transfer +only the existing immediate header-buffer read optimization from paused +SOW-0159 and validate reader-opening/boundary behavior, or keep the two-function +scope and continue its performance investigation. Recommend the former as +long-term-best; the allocation profile identifies existing per-posting header +allocations as a material cost. No answer has been received, so no header +transfer or public Reader behavior change is implemented in this prerequisite. + +The current reused-histogram standalone helper passes all sixteen original +default-window Go/Rust index/traversal checks. Its 32-bit singleton-boundary +selection also passes (8.568 seconds). The expanded changed-file audit reports +complete and clean, and whitespace validation is clean. All validation jobs +started for these checks have completed. The final chosen implementation still +needs its full-suite/race confirmation after the performance/scope choice is +resolved; the earlier whole-suite/race runs remain named candidate evidence. +Before acceptance, also qualify representative compressed-DATA aggregation +timings/allocation behavior: codec correctness alone does not establish the +performance of the changed shared visitor/key path. Keep that work in this +active regression gate rather than treating it as a separate optional follow-up. + +### Filtered Histogram Wrapper And Codec Validation - 2026-10-15 + +The preceding goal turn made implementation and validation progress; the +header-allocation scope question remains unanswered. Continue only the approved +facet/histogram repair and its validation. Assembly from the frozen helpers +shows the original histogram posting callback has 122 instructions, 549 code +bytes and a 264-byte frame. The reused posting callback has 213 instructions, +986 bytes and a 296-byte frame. Its DATA callback shrinks from 280 to 200 stack +bytes, but rejected postings still enter the larger matched-work frame. + +Select a filtered wrapper once per histogram query. Dense queries invoke the +existing matched body directly, with its all-candidates branch recording the +reference statistic. Filtered queries use a small wrapper that records the +same statistic, checks the candidate map and calls the matched body only for +accepted offsets. The matched body's false all-candidates flag prevents a +duplicate increment. Keep ownership, ENTRY/time/bucket processing, error order +and post-walk value/key clearing unchanged. An independent read-only inspection +confirms the source-level count/lifetime model; actual frame reduction, escape +behavior and benefit must be measured. No extra journal read or unmatched key +allocation is allowed. + +For compressed performance controls, render the existing synthetic JSON writer +from `tests/interoperability/go_fixture_writer.py` into a fresh private builder +and compile it once against HEAD. Preserve the current benchmark's 64 DATA/16 +FIELD buckets, compression threshold 32 and publication cadence zero. Generate +immutable 2,048-row regular/compact fixtures for Zstandard, XZ and LZ4, with +low/high-cardinality mixed values and a separate true-singleton shape. Reuse +the current VALUE/SELECT patterns and synthetic identities; no host data enters +the fixtures. Check native codec flags and successful query decompression. + +Add an optional prebuilt-corpus benchmark to the ownership test helper. It +reads explicitly selected private fixture paths, skips when no corpus is +provided, and excludes fixture creation/encoding from the timer. Before timing, +compare complete index/traversal aggregation plus expected update totals: +mixed shapes 2,197 dense/34 sparse, true singletons 2,048/32. Copy the identical +complete helper into HEAD and repaired archives and build with matching flags. +Retain fixture/source/binary hashes, warmups, alternating measured pairs, +allocation counts/bytes and every result. Cover both layouts/codecs with dense +low-cardinality and sparse high/singleton queries for facets and histograms. +This is validation infrastructure within the active regression, not resumed +SDK or CLI implementation; the performance gate remains unchanged. + +All eighteen immutable codec fixtures pass native structural inspection. Their +8 MiB reserved files fit the default 32 MiB accessor window. The baseline's +36 one-operation preflight cases agree with traversal output and expected +update totals, and report actual decompression; those observations are +correctness/setup checks rather than accepted timing samples. + +The generated filtered wrapper has 32 instructions, 110 bytes and a 32-byte +frame. It contains one candidate map lookup and an indirect matched-body call, +with no ENTRY/string/bucket work. The matched body shrinks to 188 instructions, +847 bytes and a 288-byte frame, and dense traversal uses it directly. Compiler +escape inspection shows the wrapper/matched/DATA closures do not escape. +Conditional write barriers through captured pointer state remain visible in +assembly; stack placement alone does not prove their CPU cost is zero. + +Add four direct-helper histogram error cases (both layouts and all/filtered +candidate sets) to verify that a malformed matched ENTRY preserves the exact +one-reference partial count. The focused matrix is now 146 cases. Freeze the +updated complete helper identically in both archives before using either for +timing; earlier 142-case binaries remain immutable diagnostic evidence. + +The filtered-wrapper candidate passes all 146 focused cases, including the new +partial histogram counters. All 36 codec preflight cases also pass complete +index/traversal output checks, expected update totals, actual decompression and +zero-eviction/temporary-copy controls. Those one-operation preflights are not a +timing gate. The next comparison first measures the previously regressing raw +filtered high-cardinality histogram using identically frozen complete helpers. + +That matched ten-pair comparison reports a 1.01927 median ratio with a +1.01179-1.03797 interval. Allocations decrease from 2,728/63,656 bytes to +2,718/57,384 bytes, but the time regression remains. The wrapper refinement is +therefore not accepted. Preserve its executable/results and investigate the +captured DATA value state before spending time on the wider timing matrix. + +Refine that state within the histogram function: retain the shared slice only +for independently decoded data or a qualified inline posting. Other raw values +need only a numeric offset/length until an accepted bucket actually needs the +key. Reacquire raw bytes into a callback-local slice and convert immediately; +do not store that temporary pointer back through the captured slice. Clear +shared slices only after a path that assigned them, and clear the key only when +it was made ready. Initial nil/empty state and these post-walk clears establish +the next DATA's state, including error returns. Empty values still become the +empty key. Time-filtered singleton and multi-posting reads still reacquire only +after their unchanged ENTRY/time/bucket checks. This reduces captured pointer +stores without adding an accessor API, changing reads or eagerly allocating +keys for unmatched DATA. Re-run the focused lifetime/error/boundary matrix and +measure; source reasoning does not establish the timing outcome. + +The numeric-value refinement passes all 146 focused cases. Independent +read-only source analysis agrees with its empty/singleton/time-bound/error +state model. It also confirms that the existing accessor has no general borrow +generation token: LRU touches and post-copy counters cannot prove an old slice +survived. Inspecting concrete window internals would bypass the accessor +abstraction, so no such shortcut is added. The separate header scope choice +remains unanswered. A matched narrow timing run is now evaluating the changed +pointer stores before broader timing validation. + +The narrow numeric-state comparison is inconclusive (median 1.01312, interval +0.98649-1.03247), with the same reduced allocation counts. It does not establish +performance acceptance. The complete eight-case raw matrix is running. + +An independent read-only validation-infrastructure check finds two limits to +correct before final measurement: deferred reader close is currently included +once per benchmark calibration, and corpus hashes are compared only at run +start/end rather than against the generation manifest. Stop benchmark timers +before deferred close and require the recorded generation hashes as well. +Earlier runs remain useful diagnostic evidence with the close cost included; +none is accepted as the final timing gate. Full indexed counters are compared +for untimed preflight calls across phases/calibration; timed iterations check +the exact update total. The 146 focused cases independently compare full +indexed statistics and outputs. Do not describe timed-loop validation as a +comparison of every counter on every iteration. + +The eight-case numeric-state raw comparison completes with three improvements, +four inconclusive cases and one regression. The remaining filtered +high-cardinality histogram has median ratio 1.00804 and interval +1.00160-1.07206. Facet allocation counts remain unchanged; histogram counts are +lower. Retain these results in `sow-0108-numeric-value-small-performance` with +their close-in-timer limitation. They do not close the performance gate. The +codec comparator also rejects truncated calibration output rather than +claiming comparison of statistics that Go's benchmark logger omitted. + +The numeric-state standalone helper passes all sixteen default-window +Go/Rust index/traversal checks on the original large regular/compact corpus, +with exact row and aggregation checksums and 6,250 updates. Retain its +manifest and outputs in `sow-0108-numeric-value-large-validation`. These are +correctness checks; their incidental process timings are not a performance +comparison. Fresh before/candidate test executables contain the same updated +whole helper with timers stopped before reader close. The first compressed +timing selection covers singleton histograms for Zstandard/LZ4 and both layouts; +it is a four-case diagnostic selection, not the complete 36-case codec gate. + +The four-case compressed selection completes with two inconclusive Zstandard +cases and two LZ4 singleton histogram regressions (regular median 1.00891, +interval 1.00570-1.01763; compact median 1.02683, interval +1.00591-1.03948). Allocations and complete logged preflight counters agree. +These inline-only postings do not exercise the proposed posting-array header +optimization, so that scope change alone cannot establish the full gate. + +Measure a simpler histogram callback lifetime: reuse the prior per-DATA +visitor structure, but keep its payload slice immutable and reacquire raw bytes +only into a callback-local slice. Earlier versions reassigned the captured +payload, forcing pointer indirection even for already owned decoded data. +Without that reassignment, Go may capture the slice by value in the DATA-local +visitor; its key/readiness lifetime ends with that DATA callback, so query-wide +slice/key assignment and clear operations disappear. Preserve the qualified +inline ownership proof, candidate checks, ENTRY/time/bucket order, one key per +used DATA, all counters and error handling. Raw slices are dereferenced only +before nested reads or after immediate reacquisition; decoded slices remain +owned. This is a source refinement within the approved histogram function, +not an accessor/header expansion. Freeze it separately, inspect capture/escape +behavior and remeasure the failing raw/codec selections after focused checks. + +The immutable DATA-local candidate passes all 146 focused cases. Compiler +diagnostics confirm `value` is captured by value (24 bytes, not reassigned), +both closures remain nonescaping and ownership strings enter only result maps. +Its DATA callback has no write-barrier checks, versus two in the former shared +state callback; the posting callback retains two for key assignments. Frame and +callback setup costs still need timing evidence, so this assembly observation +is not an acceptance claim. Its complete helper includes the corrected timer +boundary and is frozen identically into HEAD and candidate source archives. + +Its first raw high-cardinality filtered histogram comparison is inconclusive: +median ratio 1.02848, interval 0.91412-1.15887, with about ten-percent variation +in both phases. Allocations retain the 2,728-to-2,718 decrease. This noisy run +cannot establish either a performance gain or equivalence. Measure the +compressed singleton selection next; preserve this run rather than treating +the compiler result as a timing win. + +The immutable-local four-case compressed selection completes with all four +intervals including one and unchanged allocation counts. Median ratios range +from 0.99476 to 1.00755; the earlier LZ4 regressions are not reproduced by this +candidate. These remain inconclusive measurements, not established equivalence +or the full codec gate. Raw measurement noise and the existing scope choice +remain unresolved. Results are retained under +`sow-0108-immutable-local-singleton-codec-performance`. + +A longer one-second-per-calibration run establishes a remaining regular LZ4 +singleton regression: median 1.01996, interval 1.01176-1.03918, despite unchanged +allocations. Do not interpret the earlier shorter inconclusive run as parity. + +For compressed singleton DATA only, preserve the original direct posting +visitor: the decoded payload is independently owned, and exactly one posting +cannot benefit from key reuse. Select that visitor before initializing any +raw-value/key-reuse state. Keep the native posting walker, candidate check, +ENTRY/time/bucket order, direct value conversion, partial counters and common +classification footer. Other DATA uses the immutable-local repair unchanged. +This avoids ownership bookkeeping where ownership already exists and reuse has +no work to save. It adds a guarded implementation path inside the same approved +histogram function, without a public contract or reader-access change. Extend +the direct malformed-ENTRY counter check to LZ4 singleton DATA and both layouts/ +candidate sets; existing all-codec lifetime/singleton tests cover successful +behavior. Freeze matching helpers and measure the failing cases again. + +The longer compact LZ4 run also regresses (median 1.02089, interval +1.00751-1.03923), confirming the need to measure the specialized decoded path. +The eight direct malformed-ENTRY/counter controls pass against unchanged HEAD, +including actual compressed singleton flags, so the extended checks do not +encode only the repair's behavior. The full focused matrix now contains 150 +cases. Generated code for the decoded singleton posting callback is 549 bytes +with a 264-byte frame and zero write-barrier checks, matching the original +callback's size/frame; the general raw/reuse path remains separate. Its DATA +dispatcher is larger, so equal inner callback size is not a timing proof. + +The decoded-singleton candidate passes all 150 focused cases. A one-second +calibration comparison is running for the same two LZ4 singleton histogram +cases that established the previous candidate's regression. The scope remains +the two Explorer functions; the header transfer is still unanswered and has +not been implemented. Whole-suite/race, full timing matrices, audit and review +remain final-candidate gates, rather than implied by prior candidate runs. + +The decoded-singleton one-second comparison retains a regular LZ4 regression +(median 1.00521, interval 1.00340-1.02729); compact is inconclusive (median +1.00961, interval 0.99558-1.03941). The inner callback matches the original size, +but DATA dispatch and its shared footer still retain the compression flag +across the native walk to branch on it again. The candidate is not accepted. + +Move the decompression counter update into each successful dispatch branch. +For the decoded-singleton branch it is unconditional, because that branch +already proves compression. The general branch retains its conditional update. +Check the walk error before either update, then execute the common classified/ +loaded counters. No callback or fallible operation occurs between these three +counter writes, so their final and partial-error values are unchanged. This +removes a redundant known-true footer test and may shorten compression-flag +liveness on the fast branch. Preserve the existing two posting bodies and +native walker; no new reader scope or ownership rule is introduced. Confirm +with the current successful/error tests and matched timings rather than +assuming the branch removal is measurable. + +The decoded-footer candidate passes all 150 focused cases, twenty selected +32-bit boundary/partial-counter controls and all sixteen original default-window +large Go/Rust/index/traversal checks. Its two-case long LZ4 timing comparison is +running. No full timing or whole-SOW acceptance is claimed. + +Both aggregation callbacks currently call `splitRawPayload` and then compare +its field to an already known FIELD-chain name. `reader_unique.go` already uses +a direct known-prefix/boundary check. Investigate reusing that pattern within +these two functions to avoid a repeated delimiter scan and non-inlined split +call. Exact equivalence additionally requires checking that the requested field +contains no `=`; otherwise prefix matching could accept a name the existing +first-delimiter split rejects. Empty/binary names, missing/wrong delimiters and +mismatched FIELD DATA payloads must retain their current behavior. Keep the +native chain walk and its errors, even for an unmatchable field name. This is +an implementation investigation inside the approved scope; do not weaken +field validation to obtain a timing gain. + +The byte-level equivalence is established: the old comparison succeeds exactly +when the first delimiter is at the requested field's length and all preceding +bytes equal that field. Therefore the field has no delimiter itself. Conversely, +a delimiter-free field, equal prefix and delimiter at that boundary guarantee +the same first delimiter and value suffix. This includes zero-length and binary +fields. Compute the field-delimiter flag once per field, keep the check inside +the DATA callback and reuse the existing prefix/boundary pattern there. + +Add 24 direct-helper controls across both layouts and facet/histogram helpers: +wrong name, wrong boundary, early delimiter, empty parsed name, a requested FIELD +name containing `=`, and that unmatchable name pointing at a wrong-type object. +Use one FIELD hash bucket and same-length synthetic byte mutations, updating the +FIELD hash when renaming it; do not change file bounds or posting arrays. The +native lookup must actually find the mutated FIELD before testing the result. +All ordinary mismatches remain one unset update with no classified DATA; +wrong-type FIELD chains retain their error. The latter also proves that an +unmatchable name does not bypass native chain errors. Public Explorer rejects +delimiter-containing query names, so use the existing internal helpers for +that compatibility control. Independent read-only source inspection confirms +the fixture/hash strategy. Freeze the expanded identical helper in both phases +and run the new controls against unchanged HEAD as well as the repair. + +The prefix candidate passes all 174 focused cases. All 24 newly added malformed +FIELD/DATA controls also pass against unchanged HEAD, including lookup of the +mutated FIELD and preservation of wrong-type chain errors. Its source archives +freeze the same complete helper in both phases. The earlier decoded-footer +long LZ4 comparison completes with both cases inconclusive and unchanged +allocations; regular median 1.00067 (interval 0.99198-1.09818), compact 0.98774 +(0.94295-1.00076). Those noisy results do not establish acceptance. The current +prefix candidate still needs its own timing and larger-file qualification. + +The current prefix candidate also passes 44 selected 32-bit boundary/error/ +malformed-field controls and all sixteen original large-file Go/Rust strategy +checks. Both status indexes now name the 174-case focused matrix. The product +spec distinguishes the shared ownership guarantee from Go's contributing-key/ +reuse implementation: unchanged Rust already owns a temporary value before +posting traversal, so no new claim that Rust delays that copy is introduced. +The matched eight-case raw timing/allocation matrix is running; SDK and CLI +implementation remain paused behind this prerequisite. + +The current prefix candidate's eight-case matched raw comparison completes with +four improvements and four inconclusive cases, without a measured regression. +All facet allocation counts are unchanged. Histogram counts decrease by +1,782/25/762/10 for low-dense/low-sparse/high-dense/high-sparse respectively; +their byte totals also decrease. The formerly failing high-cardinality filtered +histogram is now inconclusive with median 0.98104 and interval +0.96607-1.01426. Low-cardinality filtered facets are also inconclusive (median +1.00997, interval 0.98605-1.03134), so no whole-gate acceptance is inferred from +the absence of a significant regression in this matrix. Raw commands, warmups, +all samples, allocation arrays and immutable hashes are retained in +`sow-0108-field-prefix-small-performance`. The full 36-case prebuilt codec +comparison is now running, followed by the matched large timing controls. + +Contributor benchmark guidance now documents the focused test command, raw +allocation cases and optional prebuilt codec-corpus contract. It names the +exact expected shape/update totals and keeps fixture generation, validation, +opening and close outside measurement. No consumer SDK API/usage change is +introduced by this repair; the caller-facing wiki remains owned by paused +SOW-0159. The runtime compatibility skill records these validation procedures. + +The full codec run finds a regular LZ4 singleton facet regression (median +1.01751, interval 1.00823-1.03757), with unchanged allocation counts. Other +cases continue to run against immutable executables; preserve the whole result. +The filtered facet path resets `visited` and flushes it to result statistics +for every DATA, even though a singleton has only one posting. Accumulate this +reference count for the FIELD walk and flush once immediately after that walk, +before propagating its error. Reset only the per-value matching `count` for +each DATA. Dense traversal retains its current count/flush path. + +No fallible operation or external callback occurs between FIELD-walk return, +the flush and error propagation. The final successful and partial-error +reference totals are the same modular sum as per-DATA flushes, including errors +during posting reads or later value reacquisition. No result key, native read, +membership test or other counter changes. This removes redundant filtered +singleton bookkeeping inside the already approved facet function. Extend the +partial facet counter control with one successful preceding DATA object to +prove prior references/updates survive a later partial posting error, while +retaining the original no-prefix controls. Complete the current immutable +codec run before starting builds or timing the next candidate. + +Also fold qualified raw singletons into the direct singleton histogram visitor. +The existing guard becomes `nEntries == 1 && (compressed || ownInlineValue)`. +After the unchanged reference/membership checks, own a raw callback-local key +before ENTRY access; decoded keys are owned only after successful time/bucket +checks. A singleton cannot reuse a key, and a local key needs no captured +key/readiness pointer. The native and paused recovery walkers invoke its inline +callback zero or one times with no prior accessor I/O. Keep the native walker. +General DATA retains lazy local reacquisition/key reuse; its old inline branch +is unreachable after this dispatch and is removed. Empty values remain the +empty key. The shared fast branch conditionally records decompression after +successful traversal because it now covers both storage forms. + +Independent read-only source analysis confirms these lifetime/error/counter +conditions. Existing raw/codec singleton, time-bound, extreme bucket, malformed +ENTRY and 32-bit controls exercise the dispatch and fallback. This refinement +addresses the earlier raw high-cardinality histogram overhead without adding a +reader API or changing the bucket proof. Compressed timing must still be +remeasured: the merged callback gains compression branches even while raw +singleton bookkeeping decreases. Preserve the previous 36-case run as named +candidate evidence; do not mix its timing samples with the new executable. + +The prefix candidate's complete 36-case codec comparison finishes: eleven +improvements, twenty-four inconclusive cases and the one regular LZ4 singleton +facet regression already recorded. Every preflight result/counter and expected +update total matches, and allocation counts do not increase (histogram reuse +reduces them where applicable). Its manifest verifies binary and fixture hashes +through the completed run. Keep this failed candidate's whole matrix under +`sow-0108-field-prefix-codec-performance`. + +After that run ends, freeze the filtered FIELD-flush and merged singleton +candidate with the same complete 178-case helper in HEAD and candidate archives. +Build timing executables with identical flags, separately from any diagnostic +compiler build. The next validation includes the new partial-counter controls +against HEAD, the full focused matrix and the previously regressing cases. + +The new successful-prefix facet control initially fails on unchanged HEAD: +the fixture assumed FIELD DATA insertion order. `writer_objects.go` prepends +new DATA to the FIELD chain, so the damaged DATA was visited first and its +one-reference/zero-update result was correct. Correct the synthetic fixture by +appending its successful unique DATA last, select that row for the filtered +case, and assert the native FIELD head before applying the posting mutation. +Keep the original expected two-reference/one-update result for that now-proven +walk order. This is a test-fixture correction, not a reader behavior change; +retain the failed logs and rerun the corrected controls against HEAD first. + +The corrected 40 baseline partial-error/malformed-field controls pass, including +the successful-prefix facet cases and all zero classification/decompression +counters after failed singleton ENTRY reads. The FIELD-flush/singleton candidate +passes all 178 focused cases, 52 selected 32-bit controls and all sixteen +original large-file cross-language/strategy checks. The matched helper files +and source manifests are frozen under `sow-0108-field-flush-stats-178-*`. +The CLI validation helper uses the same Explorer source as this archive; +subsequent changes only corrected/strengthened tests. Fresh paired timing starts +with the sixteen valid large-file controls, then the raw and codec matrices. +Performance acceptance, complete-suite/race confirmation and review remain +open; earlier passing tests or inconclusive timing intervals do not close them. + +The FIELD-flush/singleton candidate's sixteen-case valid large comparison +finishes with eight regressions, seven inconclusive cases and one improvement. +Regular dense low facets/histograms and dense/filtered high facets/histograms +include roughly 7-12% median slowdowns; compact dense low facets and filtered +high histograms also regress. All Go counters, row/aggregation checksums and +Rust-oracle results remain exact. This candidate fails the gate. Preserve all +samples and hashes in `sow-0108-large-field-flush-performance`. + +Provenance checks confirm both CLI executables use Go 1.27.1, the same compiler/ +architecture/CGO/build settings, identical dependency versions and the same +benchmark-helper source. Comparing every production Go file and module metadata +in their frozen archives finds only `journal/explorer.go` changed. Do not dismiss +the large regressions as a helper mismatch. Profile the exact non-evicting +large-case settings next to distinguish aggregation cost from surrounding work; +the earlier profiles used a different candidate and cannot explain this one. + +The exact 256 MiB regular dense low-cardinality facet diagnostic repeats 200 +complete helper loops: baseline 9.994624748 seconds, candidate 10.841001244 +seconds, with identical results and final statistics. CPU samples attribute +0.16 seconds to unchanged `parseEntryHeader` in baseline and 1.26 seconds in +candidate; posting validation decreases from 3.77 to 3.40 cumulative seconds. +This locates an unexplained cost outside the repaired aggregation functions; +it does not establish its mechanism or excuse the failed timing gate. Inspect +the compiled parser and run unchanged-path controls before further source edits. + +Clarify the helper timer: each `--loops` iteration opens a fresh reader, builds +the query, executes Explore, computes checksums and sorted summaries, and closes +the reader. `read_seconds` covers that whole pipeline. Final ExplorerStats are +the last loop's logical counts, not a work-cost measurement or an aggregate of +all loops. CPU profiling continues through post-timer output/heap-profile GC; +heap `alloc_space` is cumulative sampled allocation and `inuse_space` after +explicit GC/close is not peak query lifetime. Retain these profiles as +diagnostics only. Paired unchanged row-only Explorer and sequential SDK controls +will use the same executables, fixtures, explicit non-evicting window and timer, +without calling either repaired aggregation function. Any code-layout or GC +explanation remains a hypothesis until supported by those controls and assembly. + +The first unchanged-path control, regular row-only Explorer, is reproducibly +slower (median ratio 1.14858, 95% interval 1.11392-1.16770). It requests neither +facets nor a histogram and calls neither changed aggregation function. An +independent read-only comparison checks eleven row/helper functions: normalized +instructions and stack-frame sizes match. The parser moves by 512 code bytes, +preserving its cache-line alignment; profile excess is attributed mainly to two +adjacent stack stores. Thus added instructions or a larger checked frame do not +explain this row-only result. Broader code-address/cache interactions remain +unproven. Preserve the measured regression rather than classifying it as noise. + +Hardware-counter collection is unavailable under the existing process +permissions. The read-only `perf stat` availability check fails; no permission, +service, package or host configuration change is attempted. Continue using the +available Go profiles, assembly and matched diagnostic controls. The complete +four-case unchanged-path control is still running; this first case is not a +complete diagnostic outcome or performance acceptance. + +Next bounded source refinement: use one reusable facet posting visitor for both +dense and filtered candidates. Increment one FIELD-local visited total on every +callback, apply the original all-candidates-or-map-membership test, then update +the DATA-local count and distinct-row set. Flush the visited total once after +FIELD traversal, before propagating any error, for both candidate forms. Remove +the duplicate dense visitor setup and per-DATA dense statistics flush. The +existing native traversal, key ownership, result maps and all other counters +remain unchanged. No caller callback or fallible work observes the internal +intermediate reference count; final and partial-error modular totals are exact. +Existing dense/filtered successful-prefix/error controls exercise the change. + +This simplifies the approved two-function implementation and removes duplicated +setup/bookkeeping. It restores the baseline's per-posting all-candidate branch; +actual speed and text size must be measured rather than inferred. It is not +address padding or an asserted fix for the unexplained unchanged-path cost. +Keep the current frozen diagnostic running without changing its executables. +Source editing and separate-CPU local validation may proceed while it runs; +retain those later diagnostic samples with this concurrency qualification, and +do not use them as final acceptance. New acceptance timing waits until all such +jobs finish. No header allocation or reader contract transfer is authorized. + +The single-visitor candidate passes all 178 focused cases and the complete +standalone Go suite (`CGO_ENABLED=0`, journal package 83.410 seconds). Its facet +parent is 64 compiled bytes smaller and its duplicate dense visitor is removed; +this is a source simplification, not established elapsed-time improvement. +Selected 32-bit controls pass; race validation is running on separate CPUs. + +A one-loop Callgrind row-scan diagnostic on baseline and FIELD-flush binaries +returns identical outputs/counters. Simulated instruction totals differ by +less than 0.05%; modeled instruction-cache misses and branch misses do not +increase in candidate. This model does not explain the native slowdown and does +not emulate every processor address/prediction effect. Keep its raw traces and +do not infer native timing equivalence. Next, a short three-binary row-only +diagnostic on another CPU will compare baseline, FIELD-flush and single-visitor +placements. Concurrent work and the different CPU make it exploratory only; +final paired timing still waits for all jobs to finish. + +The single-visitor candidate's race selection passes (216.406 seconds), its +52 selected 32-bit controls pass, and all sixteen original large-fixture +Go/Rust indexed/traversal checks pass with the default 32 MiB window. The new +correctness manifest also checks both fixture and binary hashes. Its first +CPU-4 launch was refused before any query because that CPU was outside the +caller's inherited affinity; explicit process pinning makes the intended +separate-CPU validation pass. Preserve that failed preflight log, not as a +reader failure. These checks do not certify performance. + +The exploratory three-binary row-only probe has medians 0.59414 seconds +(baseline), 0.69535 (FIELD-flush) and 0.61083 (single visitor) for twenty loops. +Five observations per binary, another CPU and concurrent validation make these +diagnostic rather than equivalence evidence. The smaller candidate substantially +reduces the old unexplained cost in this probe; no mechanism is claimed. + +The optional sequential control was overbatched: each process expands every +field over twenty complete scans and takes approximately 45-50 seconds. A +separate user question requests permission to interrupt only that private +diagnostic driver and verified descendants, retain completed evidence and use +shorter current-candidate checks. Until answered, do not stop those processes. +Otherwise let the run finish. Its first row-only case was measured before +concurrent validation; later cases carry the concurrency qualification already +recorded. Benchmark batching/timer lessons are added to the runtime compatibility +skill and benchmark README. No product/API contract changes result. + +The current focused suite also passes in the combined working tree with paused +SOW-0159 changes present (22.745 seconds). The expanded changed-file audit again +reports `SOW initialization complete and clean`, and whitespace validation is +clean. An isolated code/artifact review patch applies cleanly to the unchanged +index under `git apply --check --cached`; nothing is staged. It contains only +the two Explorer functions, their complete new helper, the lifetime spec, +compatibility/benchmark instructions and the narrow audit correction. Paused SDK +runtime changes are excluded. Eventual lifecycle staging must also preserve the +paused dependency SOW records and coherent status indexes. + +An ordinary independent source review finds no new lifetime, partial-error, +counter-wrap or recovery-walker issue in the single-visitor diff. Dense calls +skip map lookup; all emitted callbacks contribute to FIELD-level references; +the total is flushed before propagating errors. No caller callback can observe +intermediate statistics. Existing prefix/error controls cover the change. This +is implementation analysis, not authorized final acceptance review. + +The previously failing regular LZ4 singleton facet is faster in a CPU-29 +exploratory comparison (ratio 0.96341, interval 0.95712-0.96922), with exactly +2,161 allocations and 1,473,112 bytes per operation in both phases and identical +full counters. Concurrent diagnostic work prevents treating this as final timing +acceptance. Hold the source fixed for the fresh complete matrices. Any flagged +case needs independent reproduction using the same executables before another +source refinement; retain all samples rather than replacing an unfavorable run. + +The remaining top-level timer/dispatch comparison also finds identical +normalized bodies and frames in all three binaries, including `main.main`, +`readBenchOnce`, `runBenchLoops`, query configuration and result processing. +Apparent disassembly differences resolve to identical constant bytes and +relocated symbol references. No propagated row-only frame growth or additional +stack-growth check is established. The cause of unchanged-path timing differences is +still unresolved; source-cost and native timing evidence remain separate. + +The current candidate is frozen for acceptance measurement. Prepare a shorter +four-case unchanged-path harness with ten row-only loops and one full SDK scan +per process, matching complete counts/counters and preserving binary/fixture +hashes. It supports stopping normally between processes and a total run budget. +After the old diagnostic and all other jobs finish, run that control plus the +sixteen large, eight raw and thirty-six codec comparisons sequentially. Do not +substitute the concurrent exploratory measurements for those final matrices. + +The final build-settings check catches an acquisition-command mismatch before +acceptance measurement: the initial single-visitor CLI used default CGO enabled, +whereas the retained baseline explicitly disabled it. Both exploratory test +binaries used the same default, but retain all of those runs as exploratory. +Rebuild the candidate CLI and both matched test executables explicitly with +`CGO_ENABLED=0`; preserve the initial executables under their original names. +The new pure CLI matches baseline's Go 1.27.1/compiler/architecture/dependency/ +VCS settings. The parser and facet symbol positions/sizes are unchanged by this +rebuild, but the final gate must name the rebuilt executables. Their focused +178-case suite and all sixteen original large-fixture checks pass again. The +earlier full Go suite already used CGO disabled on these same source files; +the race run intentionally uses its required instrumentation environment. + +The old 96-process unchanged-path diagnostic finishes normally with a complete +hash manifest; no process was interrupted and the related approval question is +obsolete. FIELD-flush row-only queries regress on both layouts (regular ratio +1.14858; compact 1.03819), while full sequential scans are inconclusive +(1.01686 regular; 0.99907 compact). These remain named diagnostic evidence for +that superseded candidate, with the concurrency qualification above. + +All other validation jobs have finished. Start the current pure single-visitor +candidate's final four-case unchanged-path comparison on CPU 28, followed by +large/raw/codec matrices sequentially. Both core CLI build settings and complete +test-helper provenance are matched; record outcomes before deciding whether +another source change is justified. No performance acceptance is claimed yet. + +The current pure candidate's four unchanged-path controls complete with exact +results/counters and clean hashes. Regular row-only queries regress (ratio +1.02838, interval 1.01657-1.04083), as do compact row-only queries (1.02087, +1.00597-1.02863). Full sequential reads are inconclusive on regular (1.00674, +0.99556-1.02617) and compact (0.99830, 0.98792-1.02198). Preserve these results +in `sow-0108-single-facet-final-unchanged-controls`. + +An independent same-binary repeat increases row-only batches from ten to forty +fresh-reader loops, keeping two warmup and ten alternating measured pairs. +Regular remains slower by its median but is inconclusive (1.02379, +0.98648-1.03495). Compact independently reproduces the slowdown (1.01916, +1.00681-1.02606). Retain both complete runs; the favorable regular interval +does not erase the initial result, and compact fails the current gate. Neither +query calls a repaired aggregation function. Continue the frozen candidate's +large/raw/codec accounting before deciding the next implementation/scope step; +do not claim unchanged-path equivalence or tune code addresses as a substitute +for a concrete improvement. + +The sixteen-case large comparison completes with two flagged regressions, two +improvements and twelve inconclusive cases. All checksums, complete Go counters, +Rust-oracle results and hashes match. The flagged cases are regular filtered +low-cardinality histograms (1.04016, interval 1.00158-1.04438) and compact dense +low-cardinality facets (1.02268, 1.00126-1.04295). Preserve the full matrix under +`sow-0108-single-facet-final-large-performance`; independently repeat those cases +with the same executables and averages of multiple complete fresh processes per +phase. Retain every process and block mean, not only favorable samples. + +The raw eight-case allocation/timing matrix is running before the codec matrix. +No source change is made in response to a single flagged small difference. +The current performance gate remains failed/open because compact row-only +slowdown independently reproduced. Complete the approved allocation accounting +and remaining frozen-source measurements before requesting the next scope +decision. The separate header-allocation proposal still has no user answer. + +The eight raw cases finish with four improvements, three inconclusive results +and one flagged low-cardinality filtered facet (ratio 1.00940, interval +1.00033-1.01989). All facet allocation counts and bytes exactly match baseline. +Histogram allocations decrease by 1,782/25/762/10 for low dense/low filtered/ +high dense/high filtered respectively; corresponding byte reductions are +1,121,536/15,872/468,736/6,272. The complete hash manifest and all samples are +retained in `sow-0108-single-facet-final-raw-performance`. Independently repeat +the single flagged raw case with the same executables. The full thirty-six-case +codec matrix now runs sequentially, using the rebuilt matched pure executables. + +The complete codec matrix finishes: six improvements, twenty-nine inconclusive +cases and one flagged regular XZ high-cardinality facet (ratio 1.04097, interval +1.00574-1.07138). Full preflight statistics, update/decompression totals, fixture +and executable hashes match. Detailed allocation medians include small positive +differences in seven Zstandard/XZ cases (0.5-24 bytes and up to 0.5 allocations); +retain and investigate them rather than asserting allocation equivalence. Retain the +complete run in `sow-0108-single-facet-final-codec-performance` and independently +repeat its flagged case with longer measured batches using these same binaries. +The fixed-candidate large repeats start first, using ten complete processes per +phase and retaining both individual measurements and paired block means. + +The larger-process large-case repeats preserve all semantics and hashes. +Compact dense low-cardinality facets independently regress (ratio 1.02975, +interval 1.02253-1.03826). Regular filtered histograms are inconclusive with +substantial variation (1.00196, 0.99696-1.18830); neither this wide interval nor +its near-one median establishes equivalence. The raw filtered low facet's +independent one-second repeat is inconclusive (1.00230, 0.97834-1.01583), with +allocation counts/bytes unchanged. Retain all initial and repeat measurements. + +Repeat the seven compressed cases with positive allocation medians using +one-second batches and fixed binaries; this includes the XZ timing flag. In +parallel, prepare only an isolated HEAD-based test helper for the existing +header-read primitives. It compares the allocating readAt check and the proposed +immediate readSlice/parse check within/crossing warmed windows on both backends. +Its purpose is to quantify proposal tradeoffs before a scope decision, not to +transfer the header optimization into the prerequisite. No product runtime +source, recovery behavior or approved two-function implementation changes. +Compile/run that analysis only after the current performance job finishes. + +All seven longer codec repeats finish with matching median allocation counts, +overlapping byte ranges and inconclusive timing intervals. The initial XZ time +flag is not independently reproduced (regular high facet ratio 0.99903, +interval 0.92040-1.03192); retain both runs. Byte medians differ by -4 to +1 bytes +against baseline totals from about 20 MB to 17 GB per query. This does not +establish exact timing equivalence or justify erasing the independently +reproduced compact row/facet regressions. Unchanged XZ dictionary allocation is +an opportunity outside the prerequisite repair; no decoder/pooling work is +implemented or accepted into this scope. + +### Updated Header-Allocation Proposal And Decision - 2026-10-15 + +The original direct-slice-only recommendation is withdrawn. Isolated existing- +primitive measurements show a cached crossing header needs one new positioned +file read under `readSlice`, while the existing `readAt` copies two cached +chunks. In the four-strategy comparison, mmap cached header medians are: + +| Strategy | Within window | Crossing window | Bytes / allocations per check | +|---|---:|---:|---:| +| Existing allocating readAt | 38.54 ns | 58.06 ns | 16 / 1 | +| Direct slice | 17.71 ns | 740.90 ns | 0 / 0 | +| Reusable owned buffer | 23.33 ns | 42.05 ns | 0 / 0 | +| Hybrid slices/buffer | 19.07 ns | 42.87 ns | 0 / 0 | + +The hybrid uses direct slices for fitting headers in power-of-two windows and +owned-buffer readAt otherwise. Non-power-of-two windows use the buffer path; +both layouts of the public journal format still share the same 16-byte object +header. The analysis checks 64 correctness cases on mmap/ReadAt, window sizes +64/63/8/default, zero/flagged-zero/wrong-type/short/overflow headers and retained +rows. All pass. Thirty-two microbenchmark cases with three measurements each +finish, asserting actual temporary-read and cache-work counts. Hybrid and buffer +paths allocate zero per check and introduce no temporary reads in warmed cases. +The runtime source in both analysis archives is byte-identical to HEAD; only +private test helpers differ. This is proposal evidence, not full-reader or +prerequisite acceptance. Reader construction/size-class costs remain to be +measured if a design is approved. + +Updated user decision 1 was resolved on 2026-10-15: the user answered `1a`, +selecting option A below. Preserve B/C as the alternatives considered. + +- A, recommended long-term-best: include hybrid native slices plus one fixed + 16-byte owned buffer per reader. Preserve cached boundary reads and measure + the complete affected reader surface. +- B, surgical: reuse the owned buffer for every header. Simpler acquisition, + preserving the existing read method; common cached headers are slower than A + in the isolated comparison, while both improve on the original. +- C: retain the two-function scope and continue its unresolved timing work. + +No answer to an earlier proposal is assumed to approve this changed design. +The scope decision is reserved to the user by `AGENTS.md:142`; no runtime +implementation of A or B starts before the corresponding choice is recorded. + +### Pre-Implementation Gate For The Proposed Header Scope + +Status: analysis complete; user approved option A on 2026-10-15. + +- Problem / root-cause model: the two-function lifetime repair is correct but + independently reproduces slower compact row/facet queries. Unchanged row + functions have matching instructions/frames; no causal layout mechanism is + established. Separately, the existing native ENTRY-offset validator allocates + a 16-byte header buffer per check. Removing this measured allocation is a + concrete reader-work reduction; it is not claimed to explain the slowdown. +- Evidence reviewed: HEAD `reader.go:680`, escape/CPU/allocation profiles, + accessor cached-window and temporary-copy paths, complete fixed-candidate + timing matrices and repeats, and the isolated primitive source/hash manifests. +- Affected contracts and surfaces: Go ordinary open, live ENTRY-array reload, + reverse posting arrays, Explorer filters/facets/histograms, directory/facade + readers and boot-array fallback. Preserve current validation, zero-header and + error behavior. Transfer no SOW-0159 recovery/bounds changes. Rust already uses + its native header-view path; no new public feature or parity exception applies. +- Existing patterns to reuse: native `readSlice`, existing cached `readAt`, + immediate scalar parsing, normalized default window size and private Reader + state. The measured placement refinement below supersedes the initial tail + placement: reuse the verified unused cursor/position slots for the buffer. +- Risk and blast radius: one fixed 16-byte buffer may change Reader allocation + class. Validate empty/small/large construction as well as hot checks. Preserve + bounds/error classes, cached-crossing behavior, retained rows and closed + readers; no header borrow may span another accessor operation. +- Sensitive data handling plan: synthetic header bytes, fields and identities + only; raw fixtures/profiles stay under ignored `.local/`. Durable evidence + contains only relative paths, public source commits and aggregate measurements. +- Implementation plan: after choice A or B, add only the private acquisition + helper and fixed buffer, replace the allocating acquisition in the original + validator, and retain its remaining body unchanged. A uses fitting native + slices for power-of-two windows and cached readAt fallback; B always uses + readAt. Preserve all unrelated paused SDK work. C authorizes no header change. +- Validation plan: turn the approved strategy's primitive controls into normal + tests; exercise both backends/layouts, tiny/default/non-power-of-two windows, + pinned rows, partial errors, closed readers and live refresh. Re-run full Go, + race, selected 32-bit and cross-language large-file checks. Freeze matched + helpers/build flags for opening, unchanged-path and raw/codec timing/allocation + gates. Require review disposition and a complete-and-clean audit before close. +- Artifact impact plan: maintain the lifetime spec, compatibility skill, + benchmark procedures, regression evidence and both status indexes. Internal + acquisition adds no consumer API or usage change; document constant memory + and operational details where measurements require it. Keep SOW-0159 paused + and track its direct-slice boundary issue there without implementing it yet. +- Open decisions: final technical review authorization or explicit waiver + after local validation. The user selected A; no performance gate is relaxed. + +### Independent Proposal Analysis And Current Checkpoint - 2026-10-15 + +Read-only internal analysis found no correctness defect in the private header +prototype. Production constructors normalize options once, pass those same +options to the accessor and store them in Reader. There is one production +Reader literal in HEAD and one in the paused SDK implementation; no production +clone, options mutation or accessor replacement invalidates the window-size +pairing. Backend fallback preserves the normalized window size. The hybrid +arithmetic admits only fitting headers in power-of-two windows; short and +non-power-of-two windows use cached readAt. The independent owned buffer is +parsed synchronously into scalars, and failed reads return before parsing. + +The evidence remains deliberately limited: primitive timing reuses warmed +windows and excludes Reader construction, cold mapping, eviction pressure and +Reader size-class effects. Methods run in grouped order with three samples; +small differences between hybrid and buffer-only strategies require paired +whole-reader validation after authorization. Correctness strategies also share +one Reader sequentially. These observations do not establish production +performance or satisfy the final technical review gate. + +All owned build and measurement jobs have finished normally; no process was +stopped. No header production implementation, staging or commit has occurred. +SOW-0159 and SOW-0121 remain paused. The expanded changed-artifact audit reports +`SOW initialization complete and clean` in +`sow-0108-decision-checkpoint-audit.log`; whitespace checks pass. User choice +A/B/C and, after successful +local validation, the final review disposition remain required before close. + +### Approved Hybrid Header Implementation - 2026-10-15 + +The user selected `1a` after receiving the crash context, measured compact +row/facet slowdowns, allocation evidence, options, memory/layout risks and +whole-reader acceptance conditions. Implement hybrid acquisition in the shared +Go ENTRY-slot validator: use a native slice only when the 16-byte header fits +inside a power-of-two cache window, and otherwise use cached readAt into one +private fixed 16-byte Reader buffer. Keep the buffer at the cold tail of Reader +state and parse immediately without exposing a borrowed header. Preserve the +validator's original validation/error behavior in the isolated SOW-0108 build; +preserve paused SOW-0159 validation when applying the acquisition helper in the +working tree. Rust's existing native header path needs no equivalent code change. + +This approval expands the two-function Explorer repair only by the private +Reader acquisition helper/buffer and the required tests, benchmarks and artifact +updates. Measure opening costs, actual Reader allocation class, retained memory, +cached/crossing/eviction work, unchanged queries and raw/compressed indexed +aggregation with matched helpers and pure builds. The isolated microbenchmarks +do not establish that this change resolves the compact slowdown. Keep SOW-0159 +and SOW-0121 paused until this complete repair passes local validation, review +disposition and the complete-and-clean audit. + +### Hybrid Implementation Validation And Allocation Measurement - 2026-10-15 + +The approved hybrid helper is implemented with the original candidate's buffer +at Reader's tail. Native slices parse immediately; boundary/non-power-of-two +headers use cached readAt. The frozen prerequisite transfers only the two +Explorer functions, the helper/buffer and header acquisition into HEAD. Paused +SOW-0159 bounds/shape/recovery rules remain absent from that standalone build. +Both timing binaries include identical complete Explorer and header benchmark +helpers; the direct-helper correctness-only file is excluded from both timing +builds and included in separate correctness builds. Runtime hashes and matched +benchmark hashes are identical between the two correctness test revisions. + +- Full standalone pure Go suite passes: journal 98.896 seconds; 1,091 leaf + cases across all passing packages, including the initial 100 header cases + and 178 Explorer lifetime/error/boundary cases. +- Expanded focused validation passes 303 leaves on native and 386 builds: + 124 header cases, 178 Explorer cases and the allocation-shape diagnostic. +- Race selection passes 287 leaves in 216.556 seconds; the added 24 raw-header + vectors also pass a separate race run. Working-tree header checks pass all + 124 cases with the paused SDK validation still present. +- All sixteen large-file Go/Rust indexed/traversal checks pass with matching + results, immutable fixtures and default 32 MiB windows. The first harness + launch rejected its requested CPU before reading files; retain that failed + CPU-availability guard log and the successful explicitly pinned retry. +- The fixed 16-byte tail field changes standalone Reader size from 632 to 648 + bytes, and Go 1.27.1 linux/amd64 heap class from 640 to 704 bytes. A retained + 4,096-object allocation diagnostic confirms both classes. Empty-reader opens + allocate 64 additional bytes; five-row opens save five header allocations + and net 16 bytes. These are measured fixed costs, not only raw sizeof. +- Ten paired allocation-only measurements flag a slowdown: 86.790 to 91.825 ns, + ratio 1.05805, 95% interval 1.03400-1.08427. Continue the complete retained + opening/header matrix before refining placement; no performance acceptance + is claimed. Investigate whether verified unused Reader fields can hold the + same approved buffer without changing the hybrid algorithm or adding memory. + +The initial live header test exposed a separate healthy-positioning defect: +opening an empty reader, appending then explicitly refreshing skips the first +row unless the caller first SeekHead. A diagnostic against unchanged HEAD +confirms the same failure after populated EOF and unique-triggered refresh. +Original SOW-0045 is reopened/paused with all four reproductions and the full +position/caller validation plan. The header test explicitly positions at head; +no positioning repair is mixed into this SOW. This valid separate finding is +tracked, not silently discarded or claimed fixed. + +### Buffer Placement Refinement Within Option A - 2026-10-15 + +Keep the approved hybrid algorithm and fixed Reader-owned buffer. The initial +tail placement increases Go's allocation class and allocation-only latency, so +reuse the two private fields `cursor uint64` and `position int` instead. Complete +HEAD/current source searches find no reads, writes or literal initializers for +either Reader field. All Reader literals are keyed; no reflection lookup, +unsafe cast, field-offset constant, serialization or assembly dependency was +found. Other cursor references belong to distinct Explorer or CLI types. + +On gc/amd64 the unused fields occupy bytes 400-415. A fixed [16]byte buffer in +that span preserves direction at offset 416, every surviving later field and +the existing GC pointer layout; removing the new tail buffer restores size +632. On gc/386 the old span is 12 bytes, so replacement shifts subsequent +fields by four bytes and predicts size 524 -> 528. Both are predicted to fit +the checked runtime's 576-byte allocation class; measure rather than claim +that prediction as a result. Private field names/count change in both builds. + +This is a storage-placement refinement of the selected option A, with the same +header routes, ownership, validation and public behavior. No additional reader +algorithm or unrelated cleanup is introduced. Record this plan before code; +then re-run functional/race/386 checks, retained allocation diagnostics and the +complete matched performance gates. Retain the first tail-buffer executable, +source manifest and full matrix as rejected/diagnostic evidence. No performance +acceptance follows from a static layout argument. + +The original tail-buffer matrix finishes all 27 cases: seventeen improvements, +nine inconclusive results and the allocation-only regression above. All four +large opens improve; empty-reader latency is inconclusive while retaining the +measured 64-byte allocation increase. Preserve all cases and the complete +manifest in `sow-0108-hybrid-1-header-construction-performance`; do not accept +the tail placement by averaging its unfavorable allocation case away. + +The revised native allocation diagnostic measures size 632 and heap class 640 +for both baseline and candidate. The 386 diagnostic measures size 524 -> 528 +but heap class 576 for both. Each uses 4,096 escaping Reader allocations. These +results confirm the predicted class preservation; full revised latency/query +comparisons remain required. No additional public API or format change occurs. + +### Revised Placement Validation Checkpoint - 2026-10-15 + +The frozen `hybrid-slots-1` runtime differs from HEAD only in the two Explorer +functions and the approved Reader acquisition/storage changes. All six native/ +386 test and CLI artifacts report CGO_ENABLED=0; source, helper and executable +hashes are retained in `sow-0108-hybrid-slots-1-validation-build-manifest.json`. +The separately built race executable uses CGO only for race instrumentation. + +- The full standalone Go suite passes serially, 1,115 leaves and journal + package 106.386 seconds. The first run overlapped heavy validation and failed + the stock live-reader stress harness because its writer exceeded 45 seconds. + Preserve that failure. Unchanged baseline passes the same live case alone in + 22.65 seconds; the complete candidate suite then passes without other private + validation jobs. An initial direct-binary diagnostic used the wrong working + directory and skipped its C oracle build; that skipped run is not evidence. +- Race selection passes all 311 leaves in 261.348 seconds. Native/integrated + and 386 focused controls each pass 303 leaves: 124 header cases, 178 Explorer + cases and one allocation-shape diagnostic. The 386 run takes 96.911 seconds; + the working-tree integrated run takes 88.704 seconds. +- Default 32 MiB-window validation completes all sixteen large Go/Rust indexed + and traversal commands with exact results/counters and clean fixture/binary + hashes. Keep the successful new-placement manifest separate from prior + candidates; no stale executable is used for this result. +- Expanded audit reports complete-and-clean before performance; whitespace + checks pass. The isolated code/artifact review patch checks cleanly against + the index without staging it; paused SDK runtime changes remain excluded. + +All correctness jobs have finished before final timings start. Run constructor/ +header, unchanged-query, large aggregation, raw and codec matrices sequentially +with fixed pure builds on CPU 28. Preserve every case, allocation observation, +semantic counter and initial/repeat result. Large-open controls already remove +100,000 allocations/1.6 MB per open; this does not establish the complete query +performance gate. Final technical review remains unauthorized and unwaived. + +### Revised Placement Paired Performance Checkpoint - 2026-10-15 + +The frozen `hybrid-slots-1` pair completes four matrices sequentially on CPU +28 with two warmup pairs and ten alternating measured pairs per case. Runtime, +helper, executable and fixture hashes remain fixed. The retained driver source +manifest records the exact measurement programs. The 95% paired bootstrap +intervals describe observed timing uncertainty; an inconclusive interval is +not evidence of equivalence and no slowdown tolerance is introduced. + +- Constructor/header, 27 cases: eighteen timing improvements, nine + inconclusive, no flagged slowdown. Every one of the 324 paired allocation + observations, including warmups, has non-positive candidate byte/allocation + deltas. Reader allocation remains 640 bytes/one allocation; its paired time + ratio is 0.99015, interval 0.98139-1.03736, inconclusive. +- Each 100,000-entry open removes exactly 100,000 allocations and approximately + 1.6 MB. Regular mmap time is 21.75 -> 19.11 ms, ratio 0.8835, interval + 0.8593-0.9230; compact mmap 17.26 -> 14.77 ms, ratio 0.8716, interval + 0.8430-0.8847; regular ReadAt 68.70 -> 66.02 ms, ratio 0.9592, interval + 0.8801-0.9835; compact ReadAt 46.41 -> 40.28 ms, ratio 0.8906, interval + 0.8333-0.9803. All four are improvements. Empty-open allocation counts and + bytes are unchanged; five-entry opens save five allocations/80 bytes. +- Every direct header case saves one allocation/16 bytes. Cached fitting + checks have paired time ratios about 0.466-0.473; cached crossing checks + 0.7452 mmap and 0.7434 ReadAt. Eviction still allocates cache metadata: + candidate mmap 130 bytes/one allocation and ReadAt 205 bytes/two allocations. + The spec explicitly distinguishes reusable header storage from cache-window + allocation rather than implying that all reader operations allocate zero. +- Unchanged row/entry controls, four cases: compact rows improve, ratio + 0.91640, interval 0.91027-0.96146. Regular rows are inconclusive at 0.96822, + interval 0.91723-1.00102; regular entry iteration 0.98578, + 0.94822-1.01318; compact entry iteration 0.97715, 0.95089-1.01229. + Exact result/checksum/work-counter preflights pass. This new compact-row + result does not erase the rejected two-function candidate's reproduced + slowdown. +- Large aggregation, sixteen cases: twelve improvements, four inconclusive, + no flagged slowdown; Go/Rust output and work counters agree exactly. The + prior problematic compact low-cardinality dense facet is now inconclusive: + ratio 0.93848, interval 0.92203-1.03772. Other inconclusive dense cases are + regular high-cardinality facet, compact high-cardinality facet and compact + high-cardinality histogram. The latter has ratio 1.00525, interval + 0.99507-1.06816, so it needs a retained longer-batch confirmation before + accepting that path. All comparisons use the valid 256 MiB control window; + separate default-window correctness remains the sixteen passing commands. +- Raw aggregation, eight cases: all improve, with paired ratios 0.60258 to + 0.95338. Every case lowers both bytes and allocation counts; low-cardinality + dense histogram falls from 3,694 to 130 allocations and 1,259,536 to 109,488 + bytes per query. High-cardinality dense facet falls from 3,699 to 2,937 + allocations and 879,288 to 867,096 bytes. Raw timing logs check errors and + nonzero updates; they do not retain complete maps or indexed counters. + Semantic/counter equality comes from focused correctness and large-matrix + validation rather than from those raw timing observations alone. + +Evidence directories are +`sow-0108-hybrid-slots-1-header-construction-performance`, +`sow-0108-hybrid-slots-1-unchanged-controls`, +`sow-0108-hybrid-slots-1-large-performance` and +`sow-0108-hybrid-slots-1-raw-performance`, each retaining its complete manifest, +per-process observations and comparisons under ignored task scratch. + +The 36-case compressed matrix is running. After it finishes, measure selected +386 constructor/header cases against the frozen 386 baseline, because the +storage replacement shifts later fields by four bytes on that architecture +despite preserving its measured 576-byte allocation class. Repeat the four +inconclusive dense large aggregations with longer timed batches and the same +executables. Preserve all initial/repeated evidence; investigate any reproduced +slowdown instead of averaging it into other cases. No source optimization, +scope change, final performance acceptance or review approval follows from +this checkpoint. + +The unchanged XZ dictionary-allocation opportunity is now explicitly tracked +by pending SOW-0160. No decoder work or execution dependency is added here. +This settles the earlier outside-scope note without promising untracked work. + +Complete the sibling performance reference with the frozen unchanged Rust +helper and this Go candidate on the four regular/compact low-cardinality dense +facet/histogram cases. Both timers include reader open, query/result processing +and close. Use the same fixture, explicit 256 MiB window, output checks and +alternating paired procedure; interpret cross-language differences as reference +costs, not Go before/after regressions. Stock journalctl has no equivalent +facet-count/time-bucket operation, as recorded above. This additional reference +does not replace the full Go baseline/candidate performance gate. + +### Current Artifact Maintenance And Follow-Up Ledger - 2026-10-15 + +- AGENTS.md: unchanged; the existing lifetime, one-SOW, review and pure-runtime + rules already govern this repair. No workflow or product-scope exception is + added beyond the user's recorded header-acquisition approval. +- Runtime project skills: the compatibility skill now requires forced DATA + invalidation, error/counter/bucket controls, matched helper and timer + boundaries, constructor/header checks and measured allocation classes. +- Specs: `product-scope.md` records owned internal aggregation keys and + synchronous header views/reusable fixed storage. These restore internal + lifetime guarantees without adding an API. Cache metadata allocation remains + explicitly allowed by the existing bounded accessor contract. +- End-user/operator docs: `tests/benchmarks/README.md` documents the focused + lifetime/header/codec controls, immutable fixtures, timer boundaries and + allocation diagnostics. SDK usage, CLI syntax and published API docs do not + change in this prerequisite; their paused SDK edits are not included here. +- End-user/operator skills: no output/reference skills exist or are affected. +- SOW lifecycle: keep this SOW in-progress/current until the complete gate is + satisfied. Preserve paused SOW-0159 and SOW-0121. Original SOW-0045 is reopened + and paused for the independently reproduced explicit-refresh positioning + regression; it does not reorder this chain. Pending SOW-0160 tracks the + unchanged XZ allocation opportunity without authorizing implementation. +- Status indexes: canonical and root indexes reflect the active prerequisite, + the paused work and those concrete tracked findings. +- Sensitive data: all new tests use synthetic journal values/identities. Raw + fixtures, profiles, build/measurement manifests and diagnostic executables + remain in ignored repository-local scratch. Durable evidence contains source + pointers, public commit identity and aggregate observations only. +- Review disposition: the internal source/data investigations are ordinary + implementation analysis, not external acceptance. External review remains + unauthorized and unwaived; request that decision after all local validation. +- Audit: the expanded pre-performance audit was complete-and-clean. Repeat it + after this final evidence/tracking work; no close is claimed from an earlier + audit or an exit code without its full verdict. + +Lessons retained in the compatibility skill and benchmark procedures: an +incorrect default-window baseline cannot certify a speedup; per-payload header +allocation and Reader heap-class growth must be measured separately; a fitting +native slice and a crossing temporary slice can have different file-read costs; +and incomplete timing intervals cannot be reported as equivalence. Deferred +recovery interaction checks remain explicitly in SOW-0159. The positioning and +XZ opportunities now have their own concrete lifecycle destinations above. + +### Revised Placement Complete Codec Matrix - 2026-10-15 + +All 36 compressed cases finish: seventeen improvements, nineteen inconclusive +and no timing slowdown flags. Complete-stream/output/indexed preflights and +fixture/executable checks pass. All 360 measured baseline/candidate allocation +pairs lower both bytes and allocation counts; retain the separate warmup data +as well. The complete matrix is +`sow-0108-hybrid-slots-1-codec-performance`; unchanged decoder costs are not +presented as introduced by the repair. + +Three cases warrant longer one-second batches using the same frozen pair: + +| Case | Paired time ratio | 95% interval | Reason | +|---|---:|---:|---| +| regular/xz/low/facet | 0.96926 | 0.90170-1.27119 | Widely varying initial timings | +| regular/xz/high/facet | 1.00816 | 0.96065-1.07747 | Positive initial point estimate | +| compact/lz4/high/facet | 1.02479 | 0.76851-1.25093 | Positive estimate and wide interval | + +This is confirmation of measured uncertainty, not acceptance of a slowdown or +a request to redesign a decoder. Preserve initial and repeated results rather +than selecting only favorable observations. The targeted 386 constructor/header +matrix runs first; longer large-case confirmations and the Rust reference remain +in the sequential measurement plan. Final performance assessment and technical +review disposition remain open. + +### Targeted 386 Initial Flag - 2026-10-15 + +The selected 386 matrix flags regular five-entry mmap open in its initial +ten paired measurements: median 24,136.5 -> 28,073.5 ns, paired ratio 1.16724, +95% interval 1.15516-1.37576. Bytes fall from 1,648 to 1,568 and allocations +from 22 to 17, so allocation reduction alone cannot establish acceptance. +The retained 386 Reader diagnostic still measures the same 576-byte heap +class. No root cause is established for this timing result. + +Finish all thirteen selected cases, then independently repeat every flagged +case using one-second timed batches and the same frozen binaries. Preserve the +initial result and repeat rather than substituting larger-file gains. Do not +change storage, header routes or the approved design in response to a single +initial flag. A reproduced slowdown must be resolved before closing this repair. + +The complete selected 386 matrix finishes thirteen cases: eight improvements, +four inconclusive and the single regular-small-open flag above. Its retained +manifest is complete. The one-second repeat now runs first, before the queued +large/codec/sibling comparisons; no source or executable changes are made. + +Independent read-only data analysis of the completed compressed matrix checks +all 864 process outputs, 2,020 preflight counter records and fixture/executable +hashes. All 432 pairs reduce both bytes and allocations, including all 72 +warmup pairs; the 360 measured pairs alone populate the comparison arrays. +This confirms measurement bookkeeping, not timing equivalence or final +acceptance. Initial intervals and selected repeats remain recorded above. + +### Reproduced 386 Flag And Large Confirmations - 2026-10-15 + +The same regular-small-open 386 case independently regresses in one-second +batches: 23,719.5 -> 26,824.5 ns, ratio 1.12920, 95% interval +1.07498-1.15086. Bytes remain 1,648 -> 1,568 and allocations 22 -> 17. Retain +both initial and repeated observations in +`sow-0108-hybrid-slots-1-386-small-open-repeat`; this is a failed performance +gate, not an acceptable tradeoff for faster large journals. Do not claim +readiness or run external acceptance review while this failure is unresolved. +Investigate the frozen binaries, matched benchmark setup and 386 layout before +changing implementation or requesting an irreducible design decision. + +Longer large-case confirmations complete with ten fresh processes per phase +per paired block, immutable binaries/fixtures and exact semantics/counters: + +| Case | Paired time ratio | 95% interval | Assessment | +|---|---:|---:|---| +| regular/high/facet/dense | 0.95082 | 0.91040-0.96213 | Improvement | +| compact/low/facet/dense | 0.96712 | 0.92380-0.99642 | Improvement | +| compact/high/facet/dense | 1.00735 | 0.99724-1.03804 | Inconclusive | +| compact/high/histogram/dense | 1.01645 | 0.96837-1.06635 | Inconclusive | + +All records remain in `sow-0108-hybrid-slots-1-large-dense-repeat`. The prior +problematic compact low facet now improves in this longer run. Neither that +improvement nor the two remaining uncertain large cases can cancel the confirmed +386 failure. The queued three-case codec confirmation runs next, unchanged. + +The three one-second codec confirmations complete with unchanged counters, +fixtures and executable hashes. All are inconclusive: regular XZ low facet +ratio 0.98470, interval 0.96383-1.02891; regular XZ high facet 1.00500, +0.97499-1.07477; compact LZ4 high facet 0.98725, 0.97381-1.01790. None confirms +a slowdown, and none proves equivalence. Retain the complete original matrix +and `sow-0108-hybrid-slots-1-codec-repeat` together. + +Read-only 386 source analysis establishes that Reader options, accessor and +file-size fields precede the four-byte shift. Accessor/backend/window objects +are unchanged separate allocations; both fitting header paths reach the same +window lookup. The changed Explorer functions do not run during ordinary open. +Warmed 386 fitting-header checks improve about 82 -> 50 ns, which does not +explain the positive complete-open delta. Later-field/GC/link placement remains +an unproven hypothesis. Process CPU totals include untimed setup/calibration; +their larger system-time delta is only a diagnostic lead. + +After the bounded Rust reference finishes, profile the existing 386 binaries +with an equal fixed number of small opens, then inspect syscall counts/timing +and existing generated-fixture/backend controls. Profiles are diagnostic and +cannot replace uninstrumented paired acceptance. No production source change +is authorized by an unproven attribution; record any necessary helper extension +before implementing it, using identical helpers for both builds. + +### Frozen 386 Diagnostic Evidence - 2026-10-15 + +Equal-count CPU profiles complete 150,000 regular-small opens per binary: +baseline 24,244 ns and candidate 24,295 ns. Profiling changes execution and +these two runs are not paired acceptance. Top samples remain dominated by +syscalls; they do not identify a causal added reader operation. Separate +3,000-iteration syscall diagnostics report exactly matching principal counts: +3,014 openat/close, 3,003 fstat64, 3,023 mmap2, 3,005 munmap, 12,034 fcntl and +3,008 epoll_ctl, including setup/calibration. Traced elapsed time is not an +uninstrumented speed comparison. No additional window mapping or file read is +shown by these counts. + +The complete diagnostic manifest, profiles, syscall tables and opening +disassembly remain in `sow-0108-hybrid-slots-1-386-small-open-diagnostics`. +Neither diagnostic establishes why the independently paired one-second case +regresses. Test that calibration/iteration-count lead next with ten alternating +uninstrumented pairs of exactly 150,000 opens using the original frozen +binaries. Preserve every earlier result. No production or helper source changes +have been made for this investigation; performance acceptance remains failed. + +The bounded sibling reference also completes. At the same 256 MiB window and +with matching 100-row output/100,000 updates, Go candidate/Rust baseline paired +ratios are 0.79062 regular facets, 0.79604 regular histogram, 0.76150 compact +facets and 0.77443 compact histogram, each interval below one. This is only the +four measured low-cardinality dense cases, not a broad cross-language speed +claim. Go process peak RSS is about 4.4-5.3 MiB higher in this reference; retain +both phases' RSS observations. The fixed Go baseline/candidate matrices, not +these sibling ratios, decide regression acceptance. Evidence is retained in +`sow-0108-hybrid-slots-1-sibling-reference`. + +### Shared Small-Fixture Measurement Plan - 2026-10-15 + +Uninstrumented fixed-count confirmation is inconclusive: paired ratio 1.09857, +95% interval 0.99806-1.15678, phase medians 23,772 -> 26,155 ns. This retains +the positive estimate and does not clear the initial/repeated slowdown. + +The current small benchmark creates a deterministic-content file per process +and calibration, but does not hash/freeze the actual inode/path across phases. +Add an optional `JOURNAL_SDK_READER_OPEN_SMALL_CORPUS` benchmark-only input, +with the same regular/compact path shape and five-entry preflight as the +generated path. Keep generation as the default. Prepare one synthetic regular +and compact file using the same writer options/rows, verify bytes and counts, +then hash both before and after every comparison. + +Rebuild both 386 timing binaries with the identical extended helper and +byte-identical original runtime sources. Compare generated versus immutable +small fixtures with those same rebuilt binaries, and mmap versus ReadAt with +the shared corpus. Retain old binaries/results: new test linking can itself +change code placement, so disappearance in both fixture modes would not prove +that fixture generation caused the original flag. No SDK runtime, approved +header routing or Reader storage changes belong to this measurement refinement. + +### Shared-Fixture Outcome And Private Storage Refinement - 2026-10-15 + +The extended-helper builds have byte-identical runtime source to the original +pair. Their generated regular-small mmap control is inconclusive at ratio +0.99363, interval 0.97127-1.00925. All four shared five-entry cases are also +inconclusive: regular mmap 1.00737 [0.98035,1.08793], regular ReadAt 0.99699 +[0.98264,1.01835], compact mmap 0.99790 [0.88568,1.03366], compact ReadAt +1.00365 [0.98141,1.02437]. Fixture hashes/count preflights pass. These new test +binaries do not establish why the older pair independently slowed; retain +both build contexts and all results. No performance acceptance follows. + +Refine only private storage within approved option A: move the existing +`entryDataActive bool` beside `currentHeaderValid bool`, consuming padding, +and remove its old padded slot. Keep the fixed 16-byte owned header buffer, +hybrid routes and all scalar/error policy unchanged. Source analysis predicts +standalone Reader size 528 -> 524 on 386, restoring original size, and +632 -> 624 on amd64; measure these predictions and actual heap classes. +This eliminates known unnecessary struct growth. It is not claimed to explain +the earlier timing result or to guarantee a latency improvement. + +Evidence/risk gate before code: all five boolean uses are named reads/writes; +Reader literals are keyed; no Reader unsafe cast, field-address/offset, +reflection, serialization or assembly dependency was found. Reader remains +non-concurrent. Earlier fields and the header algorithm remain unchanged; +unique-state field offsets and GC metadata move. Preserve every unrelated +paused SDK field/change. This is a field-placement refinement of A, not a +new API, cache algorithm, architecture exception or accepted slowdown. + +Freeze new before/candidate/correctness sources and matched helpers. First +verify native/386 sizes/classes and the previously problematic generated/shared +small-open cases, retaining an old-helper comparison as an independent link +context. If those warrant continuing, complete focused/full/race/386 and +default-window large-file correctness, then the complete affected performance +matrices using the new fixed pair. Record every unfavorable observation; +do not infer performance acceptance from packing or allocation counts alone. +All data remains synthetic under ignored scratch. Existing lifetime/header +specs and controls apply; record measured layout/evidence and update the review +bundle/status indexes without transferring paused SDK implementation. + +### Packed Storage Build And Layout Measurements - 2026-10-15 + +Five frozen pure builds complete: before/candidate amd64 and 386 with the +extended matched helper, plus a 386 candidate with the original matched helper. +All report CGO_ENABLED=0. Runtime manifests verify the only change from the +previous candidate is the private boolean move in `reader.go`; the Explorer +repair and hybrid acquisition algorithm are byte-identical. + +Retained 4,096-object diagnostics measure baseline/candidate sizes/classes: + +| Architecture | Reader size before -> packed | Allocation class before -> packed | +|---|---:|---:| +| amd64 | 632 -> 624 bytes | 640 -> 640 bytes | +| 386 | 524 -> 524 bytes | 576 -> 576 bytes | + +The old-helper 386 packed build independently measures 524/576 as well. These +results confirm the source layout calculation, not a timing fix. Evidence is +`sow-0108-hybrid-packed-1-build-manifest.json` and its per-build shape logs. +First run the formerly problematic regular-small mmap case against the original +frozen baseline with identical original benchmark helpers. Preserve that link +context separately from the extended-helper/shared-file controls. Full +functional and performance revalidation of this packed candidate remains open. +The earlier review patch targets the previous candidate and is superseded; +regenerate it from the new frozen source only after local gates are assessed. + +### Packed Old-Helper Flag And Unchanged-SDK Control - 2026-10-15 + +The packed candidate still flags regular-small mmap open against the original +old-helper baseline: 23,411 -> 26,639 ns, paired ratio 1.14553, interval +1.12112-1.15893. Allocations remain 22 -> 17 and bytes 1,648 -> 1,568. +Restoring 386 sizeof did not clear this old-link-context timing gap. Retain +`sow-0108-hybrid-packed-old-helper-1-small-open`; stop broad acceptance reruns +until this path is understood. Do not claim a causal fix from boolean packing. + +The newer before build also has an observed approximately 26 us small-open +cost even though every SDK runtime source hash matches the original before +build. Test that lead directly: compare the original before executable with +the extended-helper before executable, using alternating one-second pairs, +no profiling and no shared-corpus override. These have byte-identical SDK +source and different benchmark-helper/build contexts. This negative control +can establish whether the timing gap requires an SDK change; it is not a +replacement baseline/candidate acceptance comparison. Keep all prior evidence +and do not infer fixture or linker causality without that discriminating result. + +The unchanged-SDK negative control completes: original-before 23,196 ns versus +extended-helper-before 23,285 ns, ratio 0.99803, interval 0.96564-1.02146, +inconclusive, with exactly identical 1,648 bytes/22 allocations. This does not +reproduce the old SDK candidate's gap or establish its cause. Build metadata +matches across original before/candidate, extended before and packed old-helper +candidate, including compiler/runtime settings; no flag mismatch is found. +Retain `sow-0108-386-small-open-unchanged-sdk-control` without changing the +earlier failure disposition. Next measure the packed candidate against its own +matched baseline using the immutable small corpus in both layouts/backends. + +### Same-Process Header Strategy Diagnostic Plan - 2026-10-15 + +The packed shared regular-small mmap case initially flags ratio 1.03865, +interval 1.01015-1.08464; regular ReadAt is inconclusive at 0.98728, +0.96302-1.03191. Finish this matrix before further jobs. Neither packing nor +changing test linking clears the full uncertainty. + +Prepare a private analysis-only benchmark in a separate source archive. Open +the existing five-entry fixture through the normal catalog constructor, wrap +its accessor, then invoke the unchanged ENTRY-array loader and normal close. +For the five known ENTRY-header offsets only, the wrapper selects native +tempSlice, cached readAt into a reusable owned 16-byte buffer, or the original +allocating 16-byte readAt. Other reads delegate unchanged. Every strategy uses +the same wrapper, constructor, fixture, count checks and close path; assert +that exactly five headers were intercepted. Rotate strategy order across +processes and include mmap/ReadAt controls. + +This exercises existing primitives without changing SDK runtime code, reader +options or the approved product strategy. The wrapper and constructor split +add common overhead and alter call/link context; it is diagnostic evidence, +not a public API implementation or an acceptance speed comparison. Keep it +out of all final timing binaries and committed regression tests. No unsafe +memory access, native dependency, live host journal or non-synthetic data is +needed. Preserve full outputs, fixture hashes and algorithm/counter preflights. + +The packed shared-file matrix finishes one regular mmap timing flag and three +inconclusive cases; all save five allocations/80 bytes. Compact mmap ratio is +1.00203, interval 0.88213-1.08504; compact ReadAt 0.99800, +0.97044-1.01871. Preserve the complete matrix and existing failure disposition. + +The same-process strategy diagnostic completes fourteen processes, including +twelve measured processes with balanced rotating orders. Native/allocating +paired ratio is 0.94950 [0.90079,0.99107]; owned-buffer/allocating is 0.98108 +[0.90499,0.99205]. Both save the expected five allocations/80 bytes. Every +strategy intercepts exactly five ENTRY headers; inputs/hashes remain fixed. +These observations support the approved primitive choice but do not explain +the separately linked full-open flag or satisfy acceptance. The generic +comparison helper retained median key names containing `seconds`; their actual +input/output numeric unit in this diagnostic is ns/op, as recorded in each +benchmark row. Ratios and intervals are dimensionless. + +Next use a separate private executable to compare full construction with the +same metadata-only constructor followed immediately by the unchanged ENTRY +loader. No accessor wrapper or alternative header strategy is used. Both paths +must return five identical offsets, metadata and close outcomes. This isolates +the constructor call/frame context using one linked program. Source inspection +shows the original metadata constructor does nothing after optional ENTRY +loading except return the Reader. The split is diagnostic only: no SDK source +or constructor contract changes until it provides evidence for a concrete +implementation refinement. Retain both layouts/heap classes and every prior +failure; benchmark strategy order remains rotated rather than grouped. + +The same-process constructor split is inconclusive: ratio 0.99930, interval +0.98739-1.01298, with identical allocation counts/bytes and valid five-row +results. Reject it as an implementation change; it provides no demonstrated +benefit and does not explain the flagged complete-open comparisons. + +### Read-Only File-Open Scope Lead - 2026-10-15 + +No new file-opener runtime implementation is authorized. Source investigation +identifies a concrete separate work reduction to measure before requesting any +scope decision: Unix `openReaderFile` calls `os.Open`; installed Go's Unix +`newFile` attempts poller setup for regular files and restores blocking mode +after registration fails. The frozen Linux traces record four fcntl calls and +one failed epoll_ctl per reader open. `os.NewFile` given an already blocking +descriptor follows the non-pollable path. This is independent of, and not +claimed as the cause of, the header candidate's timing gap. + +Changing the shared opener would also affect whole-file compressed input, +writer/log metadata and retention inventory, so it is not a routine header +placement change. Analyze a separate native-journal-read helper that leaves +those callers and non-Linux behavior intact. Preserve Go/Rust public behavior, +pure runtime, compiler floors, close-on-exec, interrupted opens, typed path +errors and descriptor cleanup. User approval is required before any runtime +scope expansion. + +Measure only existing standard-library primitives in an isolated Linux +benchmark module: os.Open versus blocking syscall.Open with O_CLOEXEC followed +by os.NewFile, each with identical stat/close checks. Use synthetic shared +files, both local amd64/386 builds, matched single-program rotating order and +retained syscall/allocation observations. This estimates the available work +reduction for a concrete proposal; it does not implement an SDK opener or +establish full-reader performance. Read-only platform/contract analysis runs +in parallel; no installs, services or live journals are involved. + +### File-Open Primitive Evidence And New Scope Decision - 2026-10-15 + +The isolated standard-library module completes both pure-Go architectures, +two warmups and twelve measured processes per architecture, alternating the +two strategy orders on CPU 28. Both strategies open the same immutable +five-row synthetic file, stat it, obtain its descriptor and close it. Neither +uses or changes an SDK opener. + +| Architecture | os.Open median | Blocking open/NewFile median | Paired time ratio and interval | Bytes / allocations | +|---|---:|---:|---:|---:| +| amd64 | 6,154.5 ns | 4,795.5 ns | 0.77795 [0.77368,0.78505] | 440 / 4 both | +| 386 | 7,835.5 ns | 5,627.5 ns | 0.71409 [0.70542,0.73667] | 344 / 4 both | + +Fixed-count 3,000-open traces confirm the work reduction: fcntl counts drop +12,006 -> 3,006 and epoll_ctl 3,001 -> 1, with identical open/close/stat counts. +The extra six fcntl calls and single epoll_ctl are process setup. Thus the +primitive removes four calls per open, including the failed regular-file poller +registration. Full source, fixture/executable hashes, per-process outputs, +allocation observations and traces are retained in +`sow-0108-file-open-primitives` under the task's ignored scratch directory. + +This is not a demonstrated cure for the complete-reader regression. The current +packed candidate still has a regular-small mmap paired time flag of 1.03865 +[1.01015,1.08464] with the shared fixture, and 1.14553 [1.12112,1.15893] in the +original matched-helper control. Both save five allocations/80 bytes. No +allocation saving compensates for a failed timing gate. + +Read-only platform and contract checks establish: + +- Existing `openReaderFile` in `go/journal/file_open_unix.go:15` is shared by + native reads, `.zst` sources, log metadata and root retention. Do not change + that shared helper for the narrow proposal. +- A separate helper called only by the native branch in + `go/journal/reader.go:305-307` confines the change. Other platforms can + delegate to their existing helper; compressed sources continue at line 310. +- Linux standard `syscall.Open` supplies large-file handling on 386. Preserve + read-only access, atomic close-on-exec, normal symlink following, interrupted + open retries, typed `os.PathError` values and descriptor ownership/cleanup. + No dependency, compiler-floor or CGO change is needed. +- Public native readers require sized random access and expose neither file + deadlines nor a concurrent-close guarantee. Explicit paths currently have + no regular-file rejection. Do not add such a rejection or a stat/reopen race. + Keep compressed sequential-source scheduling unchanged. +- Direct descriptor opening bypasses Go's internal `testlog.Open` hook. + Fixture-dependent validation must run uncached and retain input hashes; + importing internal hooks or using linkname is not an acceptable workaround. +- Go/Rust parity concerns observable behavior and guarantees, not identical OS + calls. Rust needs no matching runtime change for this internal Linux cost. + +New user decision 1 remains open; the earlier `1a` approved hybrid header +acquisition only and does not authorize this additional scope: + +- A, recommended long-term-best: add a Linux-only native-reader opener with + the preserved contract above, retaining the hybrid algorithm. Benefit: + removes independently measured work on both architectures without new + allocation. Cost/risk: new platform-specific code and descriptor-contract + coverage; full-reader improvement is not established by primitive results. +- B, surgical: retain the existing opener and restore the original allocating + header acquisition on 386 while retaining the approved hybrid on amd64. + Benefit: limits the change to header acquisition and preserves the original + 386 header costs. Cost/risk: gives up five small-open or 100,000 large-open + allocation savings on 386 and still requires the complete performance gate; + the independently linked timing cause remains unproven. + +Recommend A because it eliminates verified unnecessary work rather than adding +an architecture-specific allocation fallback. No runtime opener or 386 routing +change starts before the user chooses. Either option preserves the full +correctness, allocation and per-path timing gates and the approved sequential +SOW-0108 -> SOW-0159 -> SOW-0121 execution order. + +The explicit numbered scope question has been submitted to the user and is +pending; no dependent implementation starts meanwhile. The expanded changed- +artifact audit passes with `SOW initialization complete and clean` in +`sow-0108-file-open-proposal-audit.log`; `git diff --check` is clean. Independent +allocation-result inspection confirms exact equality in all 56 primitive +observations, including warmups. All owned measurement processes completed +normally. No files are staged or committed and the two requested SOWs remain +paused behind this active prerequisite. + +### Approved Native-Reader Opener And Pre-Implementation Gate - 2026-10-15 + +The user answered `A` to the additional opener question. This authorizes the +recommended long-term-best Linux native-journal-only helper, while retaining +the approved hybrid header algorithm and the full performance gate. The +original shared opener, compressed-source path and other platforms retain +their existing behavior. No CGO, dependency or compiler-floor exception is +introduced. SOW-0108 remains the sole active implementation. + +- Problem / root-cause model: small 386 opens still fail timing acceptance. + Independently, standard Linux regular-file opening performs four avoidable + poller-setup syscalls. Removing that measured work is approved; it is not + claimed to explain the header timing difference. +- Evidence reviewed: primitive module and all 56 allocation observations, + syscall traces, existing platform openers and their callers, local Go 1.26.2 + and 1.27.1 descriptor construction and open semantics, Reader ownership and + public concurrency/deadline contracts, current packed candidate matrices. +- Affected contracts and surfaces: native Go journal opening on Linux, + including mmap/ReadAt, regular/compact, facade/directory/query and live reads. + Preserve read-only access, atomic close-on-exec, 386 large-file support, + symlinks, EINTR retry, typed path errors, descriptor cleanup and malformed- + file outcomes. No explicit native regular-file rejection is added. +- Existing patterns to reuse: standard syscall.Open plus os.NewFile; existing + platform opener delegation, Reader constructor error cleanup and immutable + fixture/paired benchmark infrastructure. The descriptor stays private. +- Risk and blast radius: blocking descriptors skip poller setup. Restrict the + new helper to native sized random-access files; retain compressed sequential + source opening. Direct open bypasses Go test-cache input observation, so run + fixture-dependent validation uncached and retain fixture hashes. No runtime + hook, host probing or new API is needed. +- Sensitive data handling plan: synthetic paths, identities and journal fields + only; temporary files, builds, logs and traces stay in ignored repository + scratch. Durable evidence uses relative paths and aggregate measurements. +- Implementation plan: add a Linux helper that opens read-only with O_CLOEXEC, + retries EINTR, wraps open failures as os.PathError, transfers ownership to + os.File and closes the raw descriptor if wrapping unexpectedly fails. Add a + non-Linux delegator. Call it only in openJournalFile's non-.zst branch. +- Validation plan: focused read-only/blocking/CLOEXEC, typed-error, symlink, + sparse >2 GiB and integration controls; retain current compressed-input, + malformed, cleanup and live tests. Freeze new HEAD-based baseline/candidate + archives with only approved repair changes and identical timing helpers. + First repeat small-open controls, then full/focused/race/386/large functional + checks and affected performance matrices without overlapping private jobs. + Verify syscall reduction through the real Reader and build non-Linux targets. +- Artifact impact plan: update compatibility spec, runtime validation guidance, + benchmark procedures, this SOW and both status indexes. Consumer APIs and + usage do not change; explain any operational/testing implications in the + appropriate existing artifacts. Preserve all paused SDK implementation hunks. +- Open decisions: this opener design is approved. External review authorization + or explicit waiver remains necessary after the complete local gate; no other + product choice is needed to implement this design. + +### Native Opener Execution And Storage Refinement - 2026-10-15 + +The Linux helper and non-Linux delegator are implemented. Only the native +openJournalFile branch changes; the shared opener and compressed sources do +not. Root amd64 descriptor checks pass eleven leaves, including sparse >2 GiB +reads. Five frozen pure builds and their allocation-class diagnostics pass. +Timing excludes both candidate-only direct-header and opener contract tests. + +The first complete-reader comparison on immutable shared small files records +two mmap improvements and two inconclusive ReadAt cases, all saving five +allocations/80 bytes. Regular mmap ratio is 0.82745 [0.75815,0.92560]; compact +mmap 0.85482 [0.84128,0.91294]. The original matched-helper setup is initially +inconclusive at 1.04544 [0.99676,1.05503]. Its longer sixteen-pair repeat +confirms a remaining slowdown: 1.03227 [1.01946,1.04638], medians 23,915.5 -> +24,739.5 ns. Preserve both complete results. Opener savings do not clear this +failure and full functional validation has not started for this candidate. + +Reflection-only layout diagnostics show the existing fixed buffer replaces +twelve unused bytes on 386 with sixteen bytes. Although packing preserves the +524-byte Reader and 576-byte allocation class, direction moves 372 -> 376, +entryOffsets 376 -> 380, entryIndex 388 -> 392 and currentHeader 396 -> 400. +No cache or compiler cause is established by these offsets. + +Next test a private storage-placement refinement within the approved design: +remove the fixed buffer from the old cursor/position location, keep direction +there, and place the same sixteen-byte buffer after entryDataIndex, using the +space recovered by the existing boolean packing. This changes no algorithm, +public API, bounds/error policy or amount of buffer state. It may keep the +ENTRY-offset slice together and align currentHeader on 386; measure rather +than assuming a performance benefit. Freeze new source/build/hash manifests, +check both architecture sizes/classes, and repeat the original and shared +small-open comparisons before transferring a winning placement to the root. +Retain every earlier candidate/failure. This is an internal placement choice, +not an additional opener or architecture-specific header-routing design. + +The late-buffer original-helper repeat also flags a slowdown: ratio 1.02144 +[1.00831,1.03184], with unchanged allocation savings. Reject field placement +as an established cure; do not transfer the private late layout to the root. + +### Internal Constructor Work Refinement - 2026-10-15 + +Source inspection identifies duplicate accessor statistics recomputation. +`newRollingReaderAccessor` initializes every statistics field; a successful +`probeInitialWindow` updates counters/windows and refreshes the snapshot again. +Each successful return in `newReaderAccessor` then calls `stats()`, recomputing +that already-fresh snapshot. No intervening operation can change it, and all +production callers discard this constructor return value. + +Within the approved accessor/performance repair, replace only those three +returns with the cached `statsValue`. Preserve the existing three-result +internal signature and every consumer-visible statistic, backend selection, +fallback reason, error and ownership rule. This is routine removal of duplicate +work, with no additional product/design choice, state, API or dependency. It is +not claimed as the cause of the separately linked timing difference. + +Validate initial returned counters against explicit empty/nonempty mmap/ReadAt +expectations, a fresh statistics query and Auto's initial-mmap-failure fallback. +Use synthetic files and the existing constructor/backend patterns. Exclude the +new correctness-only tests from both timing builds, retaining identical complete +benchmark helpers and the original matched-helper comparison. Freeze this +three-line change separately from paused SDK accessor changes. Retain original +Reader placement; check native and 386 allocation classes and repeat the failed +open comparison before full validation. Existing artifact, purity, data handling +and per-path performance/review gates continue to apply unchanged. + +The cached-statistics candidate's sixteen-pair original-helper comparison still +flags ratio 1.02597 [1.01353,1.04283]. Seven new initial-statistics controls pass; +this duplicate-work removal is not established as a timing cure. No full +functional acceptance is claimed for this candidate. + +Real-Reader tracing verifies 3,003 opens with identical open/close/stat/map/unmap +counts. Candidate fcntl calls fall 12,018 -> 3,009 and epoll_ctl 3,004 -> 1: +four calls saved per open. A controlled openat EINTR injection is followed by +a successful retry and passing one-iteration Reader benchmark. The first trace +checker expected adjacent EINTR/INJECTED markers; strace inserts its errno +explanation between them. Preserve that checker failure and raw output; the +corrected checker validates the retained successful commands and input hashes. +Hardware-counter profiling is unavailable in the installed perf invocation; +retain its error without changing tools, permissions or host settings. + +### Private Lazy Buffer Representation Plan - 2026-10-15 + +Test a representation refinement of the same approved single fixed-size owned +buffer: store a pointer to a sixteen-byte array and allocate that array once +only if a header needs cached readAt fallback. Fitting native headers keep the +existing zero-allocation path. This retains one buffer per Reader, constant +storage, immediate scalar consumption and all acquisition/error/bounds rules. +It adds no public option or architecture-specific routing. The first fallback +allocates one sixteen-byte object, matching the original validator's allocation +for that read; subsequent fallbacks reuse it. Measure cold fallback costs as +well as warmed checks and whole construction; do not assume a timing win. + +The motivation is concrete layout pressure: replacing twelve unused bytes with +an inline sixteen-byte array displaces 386 fields. A pointer is four bytes on +386 and eight on amd64. Keep existing boolean packing, measure rather than +assume the resulting size/allocation class, and retain every failed inline +placement. Review Go lifetime/escape behavior: Reader owns the array, borrowed +header slices never survive another accessor operation, and array allocation +is skipped for every fitting-header path. No unsafe conversion or shared global +scratch is permitted. Freeze this private prototype independently, preserving +the native opener, three cached-statistics returns and matched helper sources. +Repeat the original-layout opening failure before transferring it to the root. + +The lazy representation builds but misses the allocation-class objective: +amd64 size/class is 616/640 and 386 is 512/576. Raw size alone does not determine +class; retain the measured result rather than assuming 512-byte allocation. +It introduces a first-fallback allocation without demonstrating the intended +class reduction. Do not transfer it to the root or claim performance success. + +### Bounded Layout-Size Storage Prototype - 2026-10-15 + +Source evidence provides a smaller representation change while keeping the +approved inline array: the three private cached layout sizes are uint64, but +`configureLayout` assigns only constants 4, 8, 16, 64 and 72 from +`go/journal/format.go:16-23`. Repository search finds only those six constant +assignments and the three uint64-returning getters. No parsed file offset or +size is assigned to these caches and no outside-package caller can access them. + +Prototype uint8 storage for those three caches, with explicit widening in +their existing uint64 getters. Constant assignments remain compile-time range +checked, so a future larger layout constant cannot silently truncate. Preserve +the inline sixteen-byte buffer, existing boolean packing, native opener and +cached-statistics returns. This changes only private storage and load widths; +format widths, offset arithmetic, APIs, validation and acquisition algorithms +stay unchanged. The same historical-format, compact/regular, cached-header and +retained-row gates apply. Measure actual classes on both architectures and +repeat the original matched-helper failure before transferring the prototype. +No new consumer decision is involved in representing proven bounded internal +constants; no unsafe conversion, dependency, additional state or fixture-helper +change is allowed. Retain all unsuccessful placement/representation evidence. + +The narrower cache prototype achieves a 504-byte / 512-byte-class 386 Reader, +but its sixteen-pair original-helper repeat still flags a slowdown: ratio +1.04889 [1.01636,1.05987], medians 24,517.5 -> 25,528 ns, 1,696 -> 1,552 B/op +and 22 -> 17 allocations/op. Do not transfer it to the root. The root retains +the approved inline buffer, original cache widths, native opener and cached +constructor statistics. Preserve all private candidate sources and results. + +### Bounded Measurement-Context Diagnosis - 2026-10-15 + +Two read-only source/harness investigations find no extra entry-array traversal +or timer/parser error explaining the remaining small-open regression. The +original helper validates exactly five ENTRY headers in both builds. Current +runtime work removes those five allocations, one duplicate statistics refresh +and four descriptor syscalls. These reductions do not establish end-to-end +timing acceptance. Earlier 386 disassembly shows an extra helper frame, but no +causal timing attribution is established. + +The shared and original-helper measurements change both executable/helper +link context and fixture/setup context. The current complete benchmark helper +can select an immutable small corpus; otherwise it creates and closes a Writer +before resetting timing counters. That reset does not reset process allocator, +GC or kernel cache history. Their timing contribution remains a hypothesis. + +Run a bounded discriminator without runtime changes or rebuilds: use the same +frozen native-open-stats-1 386 baseline/candidate new-helper binaries in both +generated and immutable shared-file contexts. Alternate context order and +baseline/candidate order; use two warmups plus sixteen measured pairs per +context at 1.5 seconds. Retain every process result, binary/fixture hashes and +separate within-context comparisons. Do not average contexts, erase earlier +failures, or treat an inconclusive result as equivalent performance. + +The fixed-work private narrow-cache diagnostic completed four measured pairs +at 200,000 opens each, retaining wall/user/kernel CPU, page faults and allocation +counts in `sow-0108-native-open-sized-1-fixed-work-diagnostic`. Timings vary +substantially across individual pairs. Whole-process counters include untimed +fixture generation and cleanup; they cannot attribute the timed API difference +to a constructor stage. This diagnostic is not performance acceptance. + +The same-executable discriminator completes with improvements in both contexts: +generated ratio 0.90502 [0.89126,0.92952], medians 26,721.5 -> 23,959 ns; +shared ratio 0.89099 [0.87765,0.90295], medians 23,559 -> 21,032 ns. Both remove +five allocations / 80 bytes. Retain all 72 process observations and checked +hashes in `sow-0108-native-open-stats-1-context-diagnostic`. Fixture generation +alone does not reproduce the timing reversal in this executable pair. The +original-helper failure is not superseded or excused by this result. + +Proceed with the already-planned full, focused, race, 386, integrated, +compiler-floor and cross-platform functional validation of the frozen root +candidate, serially without benchmark overlap. This completes necessary +correctness evidence while the bounded read-only machine-code investigation +compares the separately linked helpers. Performance acceptance and external +review remain open; no SOW advances on this diagnostic result. + +Next, prepare identical private constructor-stage benchmark helpers in new +copies of the original-helper frozen sources. Keep all runtime source and +compiler flags unchanged. After serial functional validation completes, measure +header-only open/close, accessor creation/close on an already-open descriptor, +ENTRY-array loading on an existing cached accessor and raw header parsing. +The array stage clears only entryOffsets before each load to retain its normal +allocation behavior. Validate counts before timing and retain allocations, +binary/source hashes and exact stage boundaries. These diagnostic timings do +not replace the original complete-open control or the full performance gate. + +### Test-Cache Invocation Consequence - 2026-10-15 + +The approved direct Linux opener bypasses Go's internal fixture-input tracking. +The documented uncached validation rule must therefore also reach the existing +repository test entrypoints. Source search finds two shell test invocations: +`.github/workflows/release.yml:119` and +`tests/coverage/run_go_coverage.sh:19`. Both currently omit `-count=1`. + +Add that flag to those two existing commands, preserving test selection, +coverage instrumentation, compiler floors, build caches and release behavior. +This is a necessary validation consequence of the approved opener, not an +additional SDK or release design choice. Validate shell syntax and workflow +parsing/command preservation; the complete functional run already executes the +same uncached Go command successfully. Do not install tools, run a release or +repeat the full suite merely to validate the invocation-only edit. Include the +two files in the scoped review patch and artifact ledger. + +### Current Candidate Functional And Crossover Evidence - 2026-10-15 + +`sow-0108-native-open-stats-1-functional/manifest.json` records all eleven +serial steps passing without benchmark overlap: + +| Gate | Evidence | +| --- | --- | +| Uncached complete pure-Go suite | 1,133 leaves; 106.475 s | +| Focused race instrumentation | 329 leaves; 220.688 s | +| 386 focused pure-Go suite | 321 leaves; 31.020 s | +| Preserved SDK workspace integration | 321 leaves; 24.238 s | +| Go 1.26.2 floor, amd64 and 386 | 143 leaves each; 15.221 / 14.840 s | +| Non-Linux compilation | Windows, Darwin and FreeBSD amd64, CGO disabled | +| Complete-reader helpers | Frozen before/candidate pure-Go executables built | + +The CI/coverage invocation edits pass `bash -n` and workflow parsing; the Test +Go step retains its source checkout directory and exact test selection. + +The four-binary crossover completes two warmups and sixteen measured pairs per +helper family, using the original private temporary directory, identical CPU, +benchtime and inherited process environment. Original-helper ratio is 0.90752 +[0.89207,0.91455], medians 26,599 -> 23,954 ns. New-helper ratio is 0.89669 +[0.89254,0.90511], medians 26,733.5 -> 24,116 ns. Both report 1,648 -> 1,568 B/op +and 22 -> 17 allocations/op. Hashes of all four executables are unchanged. +Retain all 72 observations in `sow-0108-native-open-stats-1-helper-crossover`. + +Earlier original-helper runs report 1,696 -> 1,616 B/op. That absolute-byte +discrepancy is unexplained, so the crossover does not demonstrate an identical +allocation context or identify the cause of earlier timing flags. It does +establish that the old executable pair does not consistently reproduce them +under the controlled crossover. No failed evidence is discarded. + +Actual four-binary disassembly finds the same SDK instruction shapes and +function addresses between helper families. The timed benchmark closures have +the same instructions and move by 256 bytes, preserving 64-byte alignment. +Relocated data references and fixture/setup state remain possible influences, +not established causes. The prepared constructor-stage helper is retained as +a diagnostic option; no additional runtime optimization is justified by this +investigation alone. + +Run the already-planned current-candidate matrix serially: default-window +large-file Go/Rust/index/traversal correctness; native and 386 construction and +header controls; all eight raw-value and 36 compressed-value cases; sixteen +large aggregation cases with a valid non-evicting baseline; four unchanged +complete-reader controls. Retain allocation bytes/counts separately from +timing and stop at any failed command or flagged per-case regression. Earlier +full matrices remain algorithm evidence, not substitute current-build results. + +### Correctness-Test Windows Empty-File Expectation - 2026-10-15 + +Read-only implementation inspection identifies a portability defect in the new +initial-statistics test. The existing Windows backend creates a file-mapping +handle in its constructor (`go/journal/reader_access_mmap_windows.go:35-48`, +`:145-153`). A zero-byte raw file therefore fails explicit Mmap construction; +Auto legitimately selects ReadAt with a fallback reason. The new test assumed +successful Mmap and no fallback for that file on every platform. Cross- +compilation checks syntax but cannot validate this runtime expectation. + +Keep all backend/runtime behavior unchanged. Correct only the test: assert +explicit Mmap failure for a Windows zero-byte raw file; assert Auto's ReadAt +selection and nonempty reason there; retain success, exact initial counters, +fresh-snapshot equality and expected mode for all other cases. This follows +existing failure/fallback contracts and needs no new product choice. Freeze a +new correctness-only source snapshot preserving the prior snapshot and timing +executables. After timing completes, run the focused statistics controls on +native/386/compiler-floor targets and cross-compile Windows. No production or +matched timing source changes, so ongoing measurements remain applicable. + +Microsoft's [file-mapping documentation](https://learn.microsoft.com/en-us/windows/win32/memory/creating-a-file-mapping-object) +confirms that mapping a zero-length file with zero maximum size fails with +ERROR_FILE_INVALID. The platform expectation follows that existing behavior; +Windows runtime execution has not been performed. The same inspection also +finds the compatibility skill's Explorer test example missing `-count=1`; +align that command with the already-approved uncached validation policy. +No additional runtime defect is verified by this ordinary implementation +inspection; it does not replace the whole-SOW review gate. + +### Current Construction And Raw-Aggregation Matrix - 2026-10-15 + +Both complete construction/header matrices finish without a flagged case: +amd64 has 19 improvements and eight inconclusive results; 386 also has 19 +improvements and eight inconclusive results. Each retains 27 cases, ten paired +measurements after two warmups, complete allocation metrics and checked input +hashes. Inconclusive timing is not classified as equivalent performance. +Current 100,000-row mmap opening ratios are 0.8886 regular / 0.8688 compact on +amd64 and 0.8823 / 0.8547 on 386. Measured Reader allocation classes remain +640 bytes on amd64 and 576 bytes on 386; private smaller-cache experiments are +not part of this candidate. + +All eight raw aggregation cases improve, with paired median ratios ranging +0.6048 to 0.9532. Every case also lowers median allocation bytes and counts. +For the unfiltered low-cardinality histogram, allocation bytes fall +1,259,536 -> 109,488 and allocation count 3,694 -> 130 per query. This includes +the approved ownership repair and header acquisition together; it does not +attribute the complete gain solely to descriptor opening. + +Evidence directories are `sow-0108-native-open-stats-1-header-performance`, +`sow-0108-native-open-stats-1-386-header-performance` and +`sow-0108-native-open-stats-1-raw-performance`. The compressed and large-reader +matrices are still running; no final performance or review acceptance is +claimed from these completed subsets. + +### Complete Current-Candidate Performance Evidence - 2026-10-15 + +The serial `sow-0108-native-open-stats-1-performance-pipeline` completes all +seven stages. All six executable hashes and all driver hashes match their +starting manifests. A separate result check verifies 2,832 paired-process +records, two warmup pairs and ten measured pairs per case, plus sixteen Rust +oracle records for large aggregation. Every current-build process succeeds. + +| Matrix | Cases | Improvement | Inconclusive | Regression flag | +| --- | ---: | ---: | ---: | ---: | +| amd64 construction and headers | 27 | 19 | 8 | 0 | +| 386 construction and headers | 27 | 19 | 8 | 0 | +| Raw indexed aggregation | 8 | 8 | 0 | 0 | +| Compressed indexed aggregation | 36 | 17 | 19 | 0 | +| Large indexed aggregation | 16 | 15 | 1 | 0 | +| Complete-reader controls | 4 | 1 | 3 | 0 | +| Total | 118 | 79 | 39 | 0 | + +All 36 compressed cases lower median allocation bytes and allocation counts, +including the nineteen timing-inconclusive cases. Together with the eight raw +cases, all 44 measured aggregation allocation comparisons improve. Compressed +preflights preserve exact indexed counters, outputs, stream checks and all +eighteen immutable fixture hashes. The large matrix preserves Go/Rust output +and full Go counter agreement. Its before/candidate timing uses the recorded +256 MiB non-evicting window because the unchanged default-window baseline +crashes; it is not default-window before/after timing evidence. Sixteen separate +default-window large-file correctness commands pass for the repaired Go path +and Rust indexed/traversal references. + +The complete-reader controls retain twenty loops / two million rows per SDK +entry process. Their final paired median candidate/baseline ratios are: + +| Case | Ratio | Paired bootstrap 95% interval | Assessment | +| --- | ---: | --- | --- | +| Regular Explorer query | 0.96280 | [0.91827, 1.00127] | Inconclusive | +| Regular SDK entry | 1.00019 | [0.98393, 1.02891] | Inconclusive | +| Compact Explorer query | 0.95180 | [0.92072, 0.96964] | Improvement | +| Compact SDK entry | 0.99606 | [0.99402, 1.00752] | Inconclusive | + +No tolerance or cross-case average is applied. These measurements do not prove +equivalence for any inconclusive case and do not erase earlier flags. The +unexplained historical timing and 48-byte allocation-context discrepancies +remain explicit limitations for whole-SOW review. The current matrix has no +reproduced flagged case requiring another runtime candidate. Raw observations, +allocation metrics, commands and checked manifests remain under ignored local +evidence; `sow-0108-native-open-stats-1-performance-verification.json` records +the independent count/hash/allocation checks. + +### Correctness-Test Correction Validation - 2026-10-15 + +The platform-aware initial-statistics correction passes seven leaf tests each +on current Go amd64/386, Go 1.26.2 amd64/386 and the preserved integrated SDK +workspace. Windows amd64 cross-compilation also passes. No Windows runtime +execution is claimed. These six steps are recorded in +`sow-0108-native-open-stats-1-test-fix-validation/manifest.json`. + +The corrected correctness-only v2 snapshot changes one test file; every runtime +source hash remains identical to the previously validated and timed candidate. +The prior full 1,133-leaf suite, focused race and other functional results remain +runtime evidence; the five focused runs above validate the final test edit. +The ordinary read-only implementation inspection found this test expectation +and one missing uncached-test flag in the compatibility skill. Both are fixed. +That inspection is not the required independent whole-SOW acceptance review. + +### Diagnostic And Opportunity Disposition - 2026-10-15 + +- The prepared constructor-stage helper was not built or run. Further stage + measurements are rejected as required repair work: the controlled crossover + and complete current-candidate matrix do not reproduce the old flag, and + another isolated stage experiment would not establish its historical cause. + Retain the helper and all unsuccessful results without a delivery promise. +- Read-only accessor analysis finds that `touchWindow` recomputes unchanged + statistics on an ordinary cache hit unless it newly pins a window. The clock + and last-used updates still affect eviction. No correctness failure or timing + significance is established. A new shared-accessor optimization is rejected + from this repair and its required follow-up scope: it is unnecessary to the + approved ownership/header/opener outcome and expands the blast radius to all + cache hits. No runtime change or performance claim follows from this finding. + Existing public `AccessStats()` tests refresh the snapshot and would mask + stale cached fields; this limits what those tests could prove for such a + separate optimization. +- Recovery interaction checks remain part of paused SOW-0159; CLI v262/PCRE2 + work remains in paused SOW-0121. The independently reproduced explicit-refresh + positioning regression remains in reopened SOW-0045. Unchanged XZ dictionary + allocation remains tracked by pending SOW-0160. None is silently accepted or + implemented in this prerequisite. + +### Final Local Artifact Gate Checkpoint - 2026-10-15 + +AGENTS.md and the runtime-purity architecture remain unchanged. The compatibility +skill, product-scope spec and benchmark guide now cover the approved Linux +opener, fixed header storage, ownership checks, measured allocation classes, +initial statistics, cross-platform boundaries and uncached test invocations. +The release workflow and coverage runner add only `-count=1` to their existing +Go test commands. Their syntax/command-preservation checks pass; no release, +installation or host configuration operation was performed. + +Published SDK usage and API contracts are unchanged by this prerequisite, so +the paused SDK wiki edits are excluded. No output/operator skill exists to +update. Tests use synthetic identities and values; raw fixtures and diagnostic +artifacts remain ignored. No new dependency, compiler minimum or external +journal-library linkage is introduced. Canonical/root status indexes must show +this SOW in-progress, local measurements complete and review disposition open. +The expanded changed-artifact audit reports `SOW initialization complete and +clean` in `sow-0108-native-open-stats-1-final-local-audit.log`, including 158 SOW +records and 99 scanned durable artifacts. `git diff --check` passes. The scoped +sixteen-file code/artifact v3 patch applies cleanly to HEAD using a separate +scratch index; the actual staging index is untouched. Ordinary same-failure +analysis covers both aggregation callers, nested posting/ENTRY reads, +constructor statistics returns, native/crossing header callers and native-file +cleanup, including the broader backend/layout/codec/error tests above. + +Two independent internal final reviews now inspect the whole repair, with +runtime and measurement emphasis respectively. Their verdicts and any required +dispositions remain open. External review is still unauthorized and unwaived; +request that decision after the local review has no unresolved required fix. +No claim of completed SOW, final performance equivalence or publication follows +from this local checkpoint. + +### Independent Internal Final Review And Disposition - 2026-10-15 + +Both read-only reviewers assess the complete approved repair and its direct +and indirect consequences. Both return PASS with no unresolved P0-P2 defect. +Their verdicts are internal technical assessments; neither authorizes external +review, waives that gate, establishes timing equivalence or completes this SOW. + +Reviewed HEAD is `dafcdac4b971d04e48768a1f831974f9252061e2`. Both reconstruct +the sixteen-file scoped v3 patch against HEAD and verify its Go files against +the frozen correctness-v2 source. The patch SHA256 is +`f9c98fbbe2e4118037ab2a3297ab911f73504ac7bf5976ddfdd38844a39d726d`. +All 89 frozen runtime-file hashes match. The preserved paused SDK changes are +excluded. Neither reviewer builds, tests, profiles, benchmarks or writes files. + +The runtime review verifies aggregation ownership/reacquisition, immediate +header consumption, the fixed-buffer lifetime, constructor statistics, +descriptor/error semantics, platform delegation and uncached test entrypoints. +It independently recounts all retained functional events: 1,133 full-suite +leaves, 329 race leaves, 321 each for 386 and integration, 143 per compiler-floor +target and seven per corrected-statistics execution target. No failure events +are found. + +The measurement review verifies all 2,848 retained process outputs: 2,832 paired +baseline/candidate records and sixteen Rust oracle records. It independently +recomputes all 118 paired medians, intervals, classifications, phase medians and +variation values. Across all 1,176 benchmark allocation pairs, including +warmups, neither bytes/op nor allocations/op increases. Large and complete- +reader CLI matrices do not expose allocation counts; RSS observations do not +establish allocation equivalence. + +One verified P2 evidence-preservation finding is resolved. The pinned drivers +imported an untracked comparison module owned by paused SOW-0159. A later edit +there could prevent reproducing calculations from the retained package alone. +The resolution preserves that module and the unchanged HEAD reader-benchmark +module byte-for-byte in `sow-0108-native-open-stats-1-calculation-provenance`, +with hashes and tracked status. The parent independently replays all 118 +recorded ratios, intervals and classifications using the captured source; all +match. The measurement reviewer +verifies the captured bytes and replay coverage. No runtime change, benchmark +rerun or transfer of paused SDK implementation scope occurs. The manifest +explicitly labels these hashes post-measurement; no historical starting hash is +asserted. + +The comparison module SHA256 is +`4b6b93f8cabf08923ed4438e26dd01ba583539ebff5642a92a6e53639ccc9b1e`; +the calculation-provenance manifest SHA256 is +`0e45946735c6368758d124e98821e6fdba7ea68b240b8588c3156e1a0d0f1703`. +This resolves reproducibility without erasing the original finding. + +Both reviews retain these limits: 39 timing-inconclusive cases are uncertain; +historical timing and 48-byte allocation-context differences remain unexplained; +large before/after timing needs the valid non-evicting baseline; and non-Linux +execution was not performed. Codec complete-counter checks occur in preflights +and calibration, while timed iterations check update totals. Raw timing helpers +also have narrower assertions than the correctness suites. No broader assertion +about every timed iteration or platform follows from their PASS verdicts. + +The separate `touchWindow` optimization is not required for this approved +outcome. Deferred implementation destinations remain SOW-0159, SOW-0121, +SOW-0045 and SOW-0160 as mapped above; no new untracked delivery promise is +created. Final record-only changes and evidence preservation do not alter the +reviewed runtime/test patch. Repeat the expanded audit and whitespace check, +pin the final handoff records, and request external-review authorization or an +explicit user waiver. The SOW remains in-progress/current until that gate and +the required combined implementation/lifecycle commit are complete. + +The post-review expanded audit reports `SOW initialization complete and clean` +in `sow-0108-native-open-stats-1-post-review-audit.log`; the whitespace check +passes. A final audit after this result-only record and an independent impact +check of the final ledger changes are captured with the handoff evidence. +External-review authorization or waiver is the remaining user gate; neither +internal PASS changes the SOW status or authorizes starting the paused SOWs. + +### External Review Authorization - 2026-10-16 + +The user selected option A: authorize read-only external review for SOW-0108 +now, then SOW-0159 and SOW-0121 after each complete local validation milestone. +Standing authorization includes further rounds after material fixes within +those three SOWs. Run the six reviewers named in the approved question through +the system-wide external-reviewers skill. Do not expand implementation scope, +relax the performance contract or activate multiple SOWs concurrently. + +The reviewed runtime/test patch and measurements remain unchanged. Record this +authorization and refresh handoff hashes before invocation; display the full +neutral prompt and review the complete regression repair with its direct and +indirect consequences. SDK and CLI work remain paused until this prerequisite +passes review, audit and the combined implementation/lifecycle commit. + +### External Whole-Repair Review And Adjudication - 2026-10-16 + +The user-authorized six-reviewer round uses the same complete approved repair +prompt and frozen sixteen-file v3 patch for every reviewer. The runtime/test +patch SHA256 remains +`f9c98fbbe2e4118037ab2a3297ab911f73504ac7bf5976ddfdd38844a39d726d`. +The full prompt SHA256 is +`c8b8f130ecee9014a036f2629918dfc455bb9c52b6feb50e2b34f0745b5e183e`. +Review inputs and each original report, invocation result and adjudication are +retained privately with the regression evidence. No SDK implementation is +accepted by this prerequisite's external review. + +- GLM: PRODUCTION GRADE, no in-scope finding. Its reconstruction checks all + sixteen scoped files, 89 frozen runtime hashes, functional records, the + 118-case calculations and allocation evidence. Existing SOW-0045/SOW-0160 + issues remain separately owned. +- Minimax: PRODUCTION GRADE with three P3 observations. Its claimed unnecessary + audit exception is disproven using the exact Perl matcher: the trailing + tilde creates a word boundary after the archive filename's `.journal` + portion, which still matches the email-like pattern. Retain the narrow + all-zero quoted filename exception. Its timing/platform observations are + existing limitations. A hypothetical cross-call retention test for a private + synchronously consumed header slice is not required by the approved + contract. References to recovery internals concern paused SOW-0159; they + are not public SDK acceptance evidence. Frozen sources match their manifest; + the root working tree also preserves deliberately excluded paused SDK edits. +- Deepseek: PRODUCTION GRADE with three P3 observations. Constructor snapshot + freshness is already proved by source and tests; no stale return is shown. + Byte offsets in mutation fixtures intentionally target the journal layout + independently of parser implementation. Dense full-value assertions reside + in retained preflights and Go/Rust comparisons rather than every timed + iteration; no failed dense behavior or broken approved gate is shown. Extra + comments, unrelated offset refactoring and duplicate testing are rejected + from required repair scope. Its comparison-module observation is resolved + by the retained source capture and complete replay described above. +- Mimo: original verdict NEEDS CHANGES with one P2, rejected as a false + positive. Frozen `go/journal/explorer.go:1215` begins the facet loop; + `var count, visited uint64` at :1223 is inside it. The counter is flushed + at :1273 before an error returns at :1274-1275. Therefore prior facets are + not added repeatedly. An independent read-only check verifies this scope + and the reproducer's runtime hashes. Eighteen executed cases cover one and + three rows, two facets and Traversal/Index/Compare on current amd64, current + 386 and minimum-compiler amd64. All return exactly two or six DATA refs. + Valid indexed queries explicitly select AllValues; the default FirstValue + query in the report is rejected by the existing contract. Healthy cases + were executed; partial-error scope is established by source and existing + interrupted-posting tests, without claiming a new multi-facet fault-injection + execution. No production correction or material code change follows. +- Qwen: no technical verdict. The invocation and its one permitted retry both + fail with HTTP 429. Preserve both failures and the resulting coverage gap; + no substitute or third attempt is made. +- Kimi: no technical verdict. The first attempt and its one permitted + continuation reach their supervised 1,800-second limits (recorded durations + 1,800.457 and 1,801.112 seconds). Neither returns a complete review block. + Preserve both attempts and the resulting coverage gap; no third attempt + or substituted verdict is made. + +The separate runtime-purity guard observation belongs to original SOW-0071: +its completed criteria explicitly promise rejection of subprocess APIs, but +its literal patterns miss standard Go variants. Static matching proves the +misses without executing any example. This is not evidence that current core +SDK code launches a process. Reopen the original SOW for paused tracking, +preserving its original narrative and the active 0108 -> 0159 -> 0121 order; +no scanner implementation, new prerequisite or additional runtime scope is +introduced. SOW-0045 and SOW-0160 remain the other separately tracked findings. + +The current matrix has 79 improvements, 39 inconclusive results and no flagged +slowdown across 118 cases. Inconclusive intervals do not establish equivalence. +All 1,176 benchmark allocation pairs avoid an increase; CLI matrices do not +expose allocation counts. Historical timing and absolute-byte discrepancies, +the non-evicting large-baseline requirement and non-Linux runtime coverage +limits remain unchanged and are not erased by reviewer verdicts. + +The master adjudication finds no unresolved blocking repair finding. An +independent read-only check also supports the received-report dispositions +and preserved verdicts. The round is complete with the two explicit model +availability gaps; those gaps are not positive verdicts. The runtime/test +patch remains unchanged. Finish the artifact/lifecycle audit and exact-revision +impact check before the required combined signed commit. + +### Final Regression Artifact And Follow-Up Gate - 2026-10-16 + +The final runtime/test patch is unchanged by review disposition and tracking +updates. The accepted repair restores the existing ownership guarantee for +Go indexed facets and histograms, uses the approved bounded hybrid header +acquisition, and retains the approved native-reader opener and constructor +snapshot optimization. Public SDK/CLI behavior changes remain outside this +prerequisite. The preceding validation sections record the exact functional, +platform, performance and review evidence and their limits. + +- Same-failure search: both aggregation callers, singleton and multi-posting + paths, nested posting/ENTRY reads, partial errors and counter flushes, + compressed/raw value ownership, synchronous header consumers, constructor + returns and descriptor cleanup are covered by source inspection and the + retained focused/large/integrated checks. Mimo's additional multi-facet + allegation is disproven independently by source and 18 executed cases. +- AGENTS.md: no workflow or responsibility change; its existing ownership, + sequential implementation, review and pure SDK contracts continue to apply. +- Runtime project skills: the scoped compatibility changes document forced + invalidation, allocation classes, matched timers/helpers, hybrid acquisition, + native opening, cached initial statistics and uncached Go checks. +- Specs: the scoped product-scope changes document the restored internal + lifetime and header-use guarantees. Paused SDK behavior/spec edits remain + excluded from this repair. +- End-user/operator docs: the benchmark guide records the actual methods and + limits. Published SDK API and CLI usage do not change in this prerequisite; + their paused wiki edits remain excluded. +- End-user/operator skills: none exist or are affected by this repair. +- Sensitive-data handling: tests use synthetic values and identities. Raw + journals, logs, profiles, binary/source manifests and reviewer reports remain + ignored; durable records contain relative pointers and sanitized aggregates. +- SOW lifecycle/status indexes: close this original reopened SOW with its + implementation in one commit. Preserve paused SOW-0159/SOW-0121 and the + 0108 -> 0159 -> 0121 sequence. Original SOW-0045 remains paused for healthy + EOF/refresh positioning; original SOW-0071 is reopened for paused guard-gap + tracking; pending SOW-0160 owns unchanged XZ dictionary allocation. None adds + an execution prerequisite or runtime implementation here. SOW-0122 keeps its + separate CLI scalability scope and references the current CLI SOW location. +- Lessons: invalid default-window results cannot certify timing; native and + crossing header access need separate controls; allocation-class effects and + instruction/syscall evidence do not establish latency causation; retain + conflicting measurements and uncertainty. The compatibility skill and + benchmark guide hold the resulting procedures. +- Follow-up scan: historical pending design/measurement notes above are + superseded by the recorded decisions and completed checks. Remaining SDK + recovery interactions are owned by SOW-0159, CLI v262/PCRE2 work by SOW-0121, + positioning by SOW-0045, guard enforcement by SOW-0071, and XZ allocation by + SOW-0160. Additional constructor-stage diagnosis, a shared touchWindow + optimization and the non-blocking duplicate/comment/refactoring suggestions + are explicitly rejected as required work for this approved outcome. No + untracked implementation promise remains. + +The post-external tracking audit is complete and clean, scanning 100 durable +artifacts with no sensitive-data patterns. Whitespace checks pass. Repeat the +full audit after this final lifecycle move, then independently check the exact +closing revision and commit the implementation/lifecycle together. Completion +is not reported to the user before that combined signed commit succeeds. + +The post-move lifecycle audit reports `SOW initialization complete and clean`, +including 100 scanned durable artifacts with no sensitive-data patterns. +Whitespace checks pass. The final closing revision is pinned for an independent +impact check before the combined signed commit. Runtime/test bytes remain the +reviewed v3 patch; preserve the paused SDK working changes throughout staging. diff --git a/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md b/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md index d06a47b2..a9403434 100644 --- a/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md +++ b/.agents/sow/done/SOW-0121-20260621-file-backed-journalctl-full-parity.md @@ -6,8 +6,10 @@ Status: completed `completed` is the successful terminal status. `done` is a directory name, not a status value. Do not use `Status: done` or `Status: complete`. -Sub-state: the 2026-07-27 mixed-directory harness regression is repaired and -validated. Rust and Go behavior did not change. +Sub-state: PR12 review regressions completed locally 2026-10-26. Both CLIs +use the approved directional SDK APIs. Three internal final scopes pass after +3,514 native comparisons and146 portable controls. Final-commit native CI is +pending publication; PR12 remains draft and no merge/release is authorized. ## Requirements @@ -2228,3 +2230,3371 @@ Validation: No Rust or Go production source, API, behavior, file format, durability, default, or compatibility contract changed. + +## Regression - 2026-10-14 File-Backed Parity And Performance Preservation + +Status: in-progress after signed SDK commit 25accee. Implementation authorized on 2026-10-15; +the user selected the complete systemd v262 CLI contract. + +### User Request And Workstream Ownership + +The user requested two SOWs, SDK and journalctl, with performance maintained or +improved and no performance harm. The project regression rule requires reusing +this originating SOW rather than creating a replacement journalctl SOW. +SOW-0159 owns SDK traversal/decoder/recovery primitives; this record owns CLI +parsing, dispatch, query controls, rendering, and exit behavior in Rust and Go. + +The user approved SDK-first execution on 2026-10-15 because the commands consume +those primitives. Only one implementation SOW may execute at a time. This CLI +SOW remains paused until the SDK deliverable passes its gates; this reopening +does not authorize concurrent implementation. + +### What Broke And Evidence + +The completed claim of full portable v260.1 parity is contradicted by synthetic +file-backed comparisons. The command sources below were unchanged between +audited SDK commits `2b33553` and `dafcdac4b971d04e48768a1f831974f9252061e2`. +Runtime probes used stock systemd 255.4; expected behavior was also checked in +exact v260.1 source. They establish the listed failures, not a fresh full +target-version runtime certification. + +| Case | Stock behavior | Observed Rust/Go failure | +| --- | --- | --- | +| `-F PRIORITY -n1` / `-n0` | Limit unique values / emit none | Rust ignores the limit; Go dispatches ordinary entry output instead of values. | +| `-N -nN`, combined action options | Preserve selected enumeration; last action wins | Go line dispatch overrides enumeration; both use fixed action priority. | +| Cursor or after-cursor plus `-n1` | Emit cursor row or its successor | Both return the final row instead. | +| Follow with `-S @0` or `--lines=all` | Emit all 1005 initial fixture rows | Both emit only the final ten. | +| `--no-tail -n1` | Override limit and emit all rows | Both emit one row. | +| Valid PCRE2 lookahead, for example `(?=.)` | Compile and match | Both current regex engines reject it. | +| Grep with no matches | Exit 1 | Both exit 0; empty grep/missing MESSAGE has an additional Go mismatch. | +| `--list-boots -o json` | JSON table data | Both emit a text table. Invocation-list JSON requires the same audit. | +| Empty/repeated output-field selection | Default/union/set semantics | Empty selection suppresses cat output; Go loses earlier selections; Rust rejects repeats. | +| Repeated flags/scalars | Stock repeat/override semantics | Rust rejects repeated `-q`, output, lines, and other setters. | +| Leading/trailing `+` | Parse failure | Both accept malformed disjunction placement. | +| Empty `--since=` or UTC-suffixed timestamp | Reject empty / accept valid UTC text | Both silently ignore empty time; both reject the valid UTC form. | +| `--system --user --unit=alpha.service` | Retain system-unit interpretation | Both apply the user-unit rewrite and lose matching rows. | + +Source evidence: + +- `go/cmd/journalctl/main.go:380`, `:1039`, `:1431`, `:1613`, `:2099`, + `:2684`, `:2736`; `go/cmd/journalctl/output.go:49`, `:180`. +- `rust/src/cmd/journalctl/main.rs:458`, `:548`, `:574`, `:1939`, `:2702`, + `:2980`, `:3097`, `:4009`; `rust/src/cmd/journalctl/output.rs:143`. +- `.local/journalctl-parity-2rx8qht8/behavior-probes.json` and + `supplemental-probes.json` contain generated diagnostic results. The counts + and cases above are the durable sanitized evidence summary. +- Both language CLI suites and all 132 parser cases per language passed during + the audit. Those successes did not cover these behavioral combinations. + +Time bounds with `-F` are not a v260.1 defect: stock and both rewrites enumerate +all stored values, including outside an empty time interval. v262 changes that +contract by rejecting scoped `-F`/`-N` combinations. Do not silently implement +time-windowed enumeration; the compatibility-target decision below is explicit. + +### Why Previous Validation Missed It + +- Parser probes generally accept any recognized-option result, even a later + validation/open error. Name recognition is not operational parity. +- Grep cases use literal patterns and normally require stock success; no-match + exit status and PCRE2-specific syntax were absent from the oracle cases. +- Follow-since tests supplied `--no-tail` or explicit small limits, masking the + default-ten history loss. Cursor/line combinations and enumeration/line + combinations were not exercised together. +- ID listings, repeated options, action order, empty field selections, and + malformed separator positions lacked complete semantic/status assertions. + +### Pre-Implementation Gate For This Regression + +Gate status: target, regex direction and execution order approved; prerequisite +SOW-0108 and SDK SOW-0159 are complete in signed commits 4c49205 and 25accee. The earlier activation gate +above records historical work; this gate controls this regression implementation. + +Problem / root-cause model: + +- Independent action booleans and shared show-line handling do not reproduce + stock action/argument order. Cursor paths retain the last selected rows rather + than stopping after the cursor-relative limit. Follow defaults erase explicit + all/since intent. Output and timestamp parsers implement incomplete stock + semantics; regex dialect and no-match status differ from PCRE2 journalctl. + +Evidence reviewed: + +- The cases/source above; current product scope and parity matrix; prior SOW + outcomes; shared parser/query/directory/verify harnesses; pending SOW-0122; + SDK findings and ownership in SOW-0159; benchmark/reporting contracts. + +Affected contracts and surfaces: + +- Rust and Go file-backed commands, parser normalization, action dispatch, + cursor/line/follow behavior, grep, text/JSON/export output, exit status, public + help/docs, shared oracle fixtures, and direct CLI performance measurement. +- SDK core repairs belong to SOW-0159 and must be consumed without duplicating + their decoder or recovery logic in command wrappers. + +Existing patterns to reuse: + +- FIELD/DATA indexed catalogs and unique visitors; native cursor/ENTRY-array + seeking; bounded row selection; shared stock-oracle query/output/follow cases; + existing binary-safe rendering and facade lifetime rules. + +Risk and blast radius: + +- High compatibility risk for scripts consuming exit codes, cursors, limits, + JSON, and repeat semantics. High performance risk from extra scans/buffers or + a slower universal regex engine. Pure-language/non-Linux constraints remain. + +Sensitive data handling plan: + +- Use synthetic repository-local files/directories only. Never query host + journals or invoke daemon operations, logger, or systemd-cat. Place scratch, + caches, profiles, cursor files, and isolated oracle builds under `.local/`. + Durable artifacts contain sanitized counts, relative paths, and public commits. + +Implementation plan: + +1. Resolve target/regex design and ordering; preserve release baseline binaries + and fixture/output identities before runtime edits. +2. Add paired stock-oracle regressions for every case family above, including + explicit expected nonzero statuses and complete output/cursor assertions. +3. Repair parser/dispatch, enumeration/line precedence, cursor-limited seeking, + and follow history while preserving indexed paths and bounded state. +4. Repair approved grep, timestamp, source-scope, and rendering semantics; + validate the complete approved option surface after the last material fix. +5. Run the performance/portability/interoperability gates, update artifacts, + obtain whole-SOW review or explicit waiver, audit, and close in one commit. + +Validation plan: + +- Run `tests/parser-parity/check_v260_manifest.py` and + `run_parser_parity.py`, then the query, directory, mixed-directory, and verify + matrices plus affected Rust/Go CLI unit/subprocess suites. +- Require an actual full-feature stock runtime for the approved target; record + version, commit, compression and sealing features. PATH systemd 255.4 is + supplementary evidence and cannot certify a v260.1 or v262 target. +- Cover text/binary/repeated fields, all affected output framings, first/last + cursor identity, chronological/reverse order, limits zero/one/all, and live + file/directory appends. Check both values and return codes without masking + reader failures or sorting away order-sensitive mismatches. +- Preserve Windows/macOS/FreeBSD support; existing toolchains and repository + caches only. Additional installations or remote execution need authorization. +- Apply the performance gate below, repeat the same-failure search, then run + `.agents/sow/audit.sh` and require its complete-and-clean verdict. + +Artifact impact plan: + +- AGENTS.md: existing performance/purity/parity rules suffice unless a baseline + or dependency-policy change is explicitly approved. +- Runtime project skills: update the compatibility workflow for added semantic + oracle and CLI performance gates when implemented. +- Specs: correct the parity matrix/product scope to the accepted shipped target + and behavior; do not advertise unimplemented outcomes during preparation. +- End-user/operator docs: maintain `docs/Journalctl-CLI.md`, help, and related + guides for approved behavior, limitations, and migration implications. +- End-user/operator skills: none exist; check again at close. +- SOW lifecycle: reopen the original record, preserve its full historical + narrative, and append this regression. SOW-0159 owns SDK repairs. SOW-0122 + remains separate broad scalability work, not an excuse to omit this gate. +- SOW-status.md: update canonical/root indexes and completion together with the + eventual implementation commit, not as a later lifecycle-only commit. + +Open-source reference evidence: + +- `systemd/systemd @ c0a5a2516d28601fb3afc1a77d7b42fcfe38fced` (`v260.1`): + `src/journal/journalctl.c:744`, `:964`, `:1072`, `:1121`; + `journalctl-misc.c:187`, `:214`; `journalctl-show.c:73`, `:601`; + `src/shared/pcre2-util.c:58`; `src/shared/logs-show.c:1452`. +- `systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492` (`v262`): + `src/journal/journalctl.c:958`; `journalctl-filter.c:26`; `NEWS`. + +Open decisions: + +1. Resolved on 2026-10-16: the user authorized read-only external review after + complete local validation, including further rounds after material fixes. + This SOW remains paused for SOW-0108 then SOW-0159; review has not run here. + +Resolved on 2026-10-15: the user selected the complete v262 CLI contract, +including a full option/surface delta audit, fixtures and migration documentation. +The user approved SDK-first sequential execution. The SDK binary-format target +remains v260.1; the newer CLI target does not change journal bytes or compiler +minimums. +The user also approved native PCRE2 in the CLIs only, including a Go CGO +exception, retaining the performance gate. Core SDK purity and the prohibition +on system journal linkage remain intact. Dependency/version and portable build +details require source review and validation during this SOW, not a new product +design fork. The user selected prerequisite SOW-0108 repair before SDK/CLI work +resumes; preserve all existing working changes. + +### Acceptance Criteria And Performance Gate + +- Implement and test every confirmed CLI case family above in both languages; + language-specific code differences do not justify different observable results. +- The user's no-performance-harm instruction is a hard acceptance condition. It + overrides generic allowances for a documented slower path in this workstream. +- Before runtime changes, retain baseline release binaries, compiler/dependency + metadata, immutable file/directory fixtures, output sinks, and baseline + variance. Alternate baseline/candidate runs with warmups and repeated samples. +- Require unchanged or improved performance per affected valid-input case. No + permissible slowdown percentage or aggregate-score offset is authorized. + Inconclusive/noisy results need investigation before claiming acceptance. +- Measure complete-command elapsed/CPU time, first-output latency, rows/bytes + per second, peak RSS, allocation/profile evidence where available, and follow + catch-up/append/idle costs. SDK loop benchmarks alone do not certify startup. +- Include indexed `-N/-F`, low/high-cardinality values, cursor `-n1`, tail/reverse + small limits, narrow time ranges, literal/approved complex grep, projected + JSON, complete scans, and follow history on small/large synthetic inputs. +- Validate counts, order, cursors, and output checksums before interpreting + timings. A ten-row broken follow result cannot be the performance standard + for a correct 1005-row backlog; use equivalent complete work and controls. +- Preserve FIELD/DATA enumeration, seek/index paths, bounded row selection, + reusable DATA, and requested-scope decompression. Do not insert complete + archive scans, full-result buffering, repeated parsing/render allocation, + full verification, or a disabled fast path to simplify the fixes. +- Add a focused repeatable process-measurement runner around shared fixtures; + no dedicated CLI benchmark currently proves these cases. If core/facade + paths change, also run the shared reader-core benchmarks and render reports + through `tests/benchmarks/report_benchmarks.py`. +- Compare equivalent stock and fastest-sibling behavior where practical; record + versions and limitations. Profile adverse/uncertain results. Reproducible + regressions block acceptance and require redesign rather than a silent waiver. + +### Implementation Review And Failure Handling + +- CLI implementation is authorized after SOW-0108 and SOW-0159. Native PCRE2 + direction and the CLI-only Go CGO exception are recorded above; validate + dependencies and performance before acceptance. Keep implementation sequential. +- Standing external review is authorized as recorded in Open decisions above. + Review the complete repaired SOW after local validation using the system-wide + reviewer skill, retaining failed invocations and their coverage gaps. +- Every reviewer remains read-only and receives the canonical repository-boundary + block above, the complete SOW scope, source versions, performance evidence, + unresolved findings, and prohibition on host-journal probing. +- Record failed correctness, performance, audit, and review gates. Do not mark + this SOW completed or submit an unreviewed parity claim while they remain open. + +### Preparation Validation And Artifact Maintenance + +- Accepted decisions recorded: two sequential workstreams, SDK first, the + complete v262 CLI target and no accepted performance regression. Regex + direction and the Go CLI CGO exception are approved; SDK recovery termination + belongs to SOW-0159 and uses normal EOF plus explicit status. +- Tests/evidence: the diagnostic comparisons and 132/132 parser results above + are existing investigation evidence; no final implementation PASS is claimed. +- Preparation validation on 2026-10-14: whitespace, gate/state/index, and exact + historical-narrative preservation checks passed. `.agents/sow/audit.sh` + reported `SOW initialization complete and clean`, with no sensitive-data + patterns in the scanned durable artifacts. Independent read-only planning + review found no blocking consistency flaw; this is not implementation approval. +- Same-failure search: parser/dispatch, show loops, regex construction, field + selection, time grammar, and ID-list rendering were inspected in both CLIs. + Finish reachability and semantic-interaction scans during implementation. +- Real-use evidence: synthetic explicit-file reads/follow only; no live host + journal, daemon operation, production service, or software install was used. + +Sensitive data gate: + +- Appended text contains sanitized counts, public source commits, and relative + paths only. No raw private data or workstation mirror paths are copied here. + +- Artifact maintenance: both SOW indexes and the SOW-0122 dependency/reference + notes are updated during preparation. Product behavior/specs/docs are not + changed before implementation. There are no output/reference skills to edit. +- Lessons: recognized options, passing parser probes, and valid closed-file + reads do not establish semantic parity or preserved end-to-end performance. +- Follow-up mapping: all current CLI defects and their performance gate belong + here; SDK causes belong to SOW-0159. Indexed invocation listing and general + polling scalability remain SOW-0122. Unrelated pending work stays separate. +- Close only after full implementation, approved target/design, paired + correctness/performance evidence, review or waiver, artifact gates, and clean + audit. Move back to `done/` and commit with the implementation together. + +### Review Authorization - 2026-10-16 + +The user selected option A, granting standing read-only external-review +authorization for SOW-0108, SOW-0159 and this CLI SOW at their complete local +validation milestones, including further rounds after material fixes. Use the +six named reviewers and the system-wide reviewer skill. Earlier unauthorized +review notes describe the prior preparation checkpoint and are superseded by +this decision. Keep complete v262 parity, the CLI-only native PCRE2 decision, +the no-performance-harm gate and sequential prerequisite execution unchanged. + +### Resumption And Detailed Implementation Gate - 2026-10-17 + +Gate status: satisfied. The SDK prerequisite is delivered in signed commit +25accee. Only this CLI SOW executes; independent SOW-0045, SOW-0071, SOW-0160 +and SOW-0161 remain paused or pending. No new product decision is outstanding. +The approved complete v262 target, CLI-only native PCRE2 and portable command +contract govern the routine implementation choices below. + +Problem and evidence: + +- Ordered upstream getopt processing cannot be represented by independent action + booleans, a global help prescan or short-argument rewriting. Required values + must remain verbatim. Setters validate in encounter order; terminal utilities + return immediately. Final grep compiles after global checks and before every + nonterminal action. The final action and scope decide enumeration conflicts. +- Cursor-relative limits, explicit all/since follow history, first raw MESSAGE + matching and cat output, repeated projections, ID-table JSON, timestamp grammar + and no-match status need paired behavioral controls rather than recognition. +- Exact authority: systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492 + (v262), especially src/journal/journalctl.c, journalctl-filter.c, + journalctl-show.c, journalctl-misc.c, src/shared/pcre2-util.c, + time-util.c, parse-util.c and cli-util.h. The isolated full-feature oracle + has PCRE2 10.42, XZ, LZ4, Zstandard and OpenSSL. The host binary is supplementary + only. SDK binary-format target v260.1 and compiler floors remain unchanged. +- Baseline release-build-8/retained preserves 11 binaries, 359 source files and + verified hashes. Its CLI binaries are the pre-CLI comparison; later SDK timing + reports are not substituted for command startup or rendering evidence. + +Affected surfaces and reused patterns: + +- Both command parsers, action selection, post-filters, timestamp conversion, + output tables/renderers and selection loops; existing parser/query/follow, + directory, mixed and verify harnesses; CLI build callers, CI, docs and specs. +- Reuse SDK FIELD/DATA enumeration, cursor and realtime seeks, bounded row + selection and binary-safe entry payloads. SDK algorithms and dependencies + are not redesigned here. Reuse current flag storage, output-mode enums, + validation helpers and fixture generators; no general parser framework. + +Implementation sequence and bounded costs: + +1. Advance the existing manifest/checker to pinned v262 OPTION macros, actions, + output modes and hidden --introspect-cli. Add semantic oracle cases before + repairs, preserving raw exit status, complete outputs and cursor identity. +2. Add one ordered CLI-private pass per language. Preserve required values, + interspersed operands, --, clusters, attached optional values, native special + boot/line lookahead, unique long prefixes and per-occurrence validation. + Use an explicit action with last-action/rotate-vacuum semantics. Reuse Go + flag storage and Rust Args/help storage without rebuilding parsers per row. +3. Implement PCRE2-8 directly in CLI-private wrappers. Compile the final raw byte + pattern once; mirror stock casing and default byte mode without JIT/UTF/UCP. + Rust owns native handles with RAII. Go reuses compiled code; one CGO call per + requested MESSAGE creates match data, matches borrowed bytes, copies scalar + results and frees match data before returning. This respects implicit CGO + argument pinning without copying subjects or retaining Go pointers. Empty + subjects use a native empty byte. Propagate matching errors and close every + handle. SDK runtime retains CGO=0 and no PCRE2 dependency. +4. Apply v262 final-action -F/-N scope rejection. Keep native unique enumeration + and value limits; fix cursor and follow limit precedence without scans or + unbounded buffering. Preserve first raw MESSAGE, binary cat bytes, projection + union/defaults, JSON ID tables and grep exit 1 with empty-success stderr. +5. Complete portable timestamp grammar once during command parsing. Use checked + integer duration/epoch arithmetic and calendar fraction rounding; preserve + local, UTC, offsets, named zones and native DST selection. Rust Jiff 0.2.38 + is a CLI-private pure-Rust zone provider (MSRV 1.70, Unlicense OR MIT), with + existing Go time zone facilities. Retain glibc-compatible inverse conversion + and narrow TZif metadata/leap handling where the provider omits required + information. No process-global TZ mutation or new Go native exception. +6. Keep native PCRE2 build requirements confined to CLIs: Go cgo build tags and + explicit CLI CGO=1 build calls; pure SDK builds/tests remain CGO=0. Maintain + Linux/Windows/macOS/FreeBSD build paths with existing toolchains and caches. + Native package installation on this server is not authorized or required by + the plan. Review ABI/version/platform details before each dependency change. + +Risk, validation and artifacts: + +- High compatibility blast radius for scripts: encounter order, empty versus + absent options, binary bytes, status codes and cursor positions. Dedicated + stock-oracle negative controls and language-local regressions verify these. + Additional per-message native-call cost needs focused complete-command + evidence. No extra archive scan, MESSAGE copy, full-result buffer, SDK graph + verification or universal per-row parser is permitted. +- Run both supported-floor CLI suites, the advanced parser manifest/parity, + complete query/follow and directory/mixed/verify matrices, binary outputs, + and available portable builds. Keep failures and exact source/tool hashes. + Use synthetic repository fixtures only; never query host journals or alter + services. Scratch/caches remain repository-local, durable evidence sanitized. +- The user's later performance decision 11 in SOW-0159 supersedes this record's + earlier mechanical measurement list. Trace actual added work; careful review + and representative high-level tests suffice for cold-only/equivalent paths. + Use existing helpers and focused elapsed/profile controls for native grep, + startup and credible costs. No large matrix/framework is required. Retain + adverse results and investigate reproducible slowdowns; an uncertain interval + alone proves neither equality nor regression. No slowdown waiver is granted. +- Update AGENTS.md and compatibility skill for the separate CLI target/native + exception; parity matrix, product spec, CLI docs/help and build guidance for + actual delivered behavior. Keep SDK target/compiler floors unchanged. No + consumer output/reference skills exist. Reconcile follow-ups and both status + indexes, complete whole-SOW review under standing authorization, run the + complete-and-clean audit, then commit implementation and completed lifecycle + together with an explicit signed commit. No push is authorized. + +Open decisions: none. Portable implementation details implement existing approved +contracts; any genuinely incompatible design fork must be presented before a +change that would require it. + +Initial semantic controls retain 116 baseline observations with 103 comparison +failures, including field-catalog ordering differences. Catalog comparisons now +compare names as sets, matching the existing unordered SDK catalog contract; +entry/cursor sequences and unique-value limits remain ordered and exact. The +first paired implementation reduces this batch to 14 failures: Rust ID-table +JSON and separator placement, plus two timestamp forms in each language. +Native PCRE2 syntax, raw-byte matching, no-match status, action order and cursor +limits pass in that intermediate batch. These are development results, not +final acceptance. Go CLI tests pass; a Rust subprocess banner test still names +the historical target and is updated to assert the approved v262 banner. + +Timestamp occurrences retain one command-local inverse-conversion context and +cached numeric bounds. Reparse/reordering would change relative-now values and +DST fold choices; execution reuses parsed values. Jiff 0.2.38 and required pure +zone packages are fetched only into the repository Cargo cache. Cargo lock +changes add CLI dependencies and disambiguate the pre-existing bitflags entry; +no existing dependency version or SDK compiler minimum is raised. Native build +calls explicitly enable CGO only for the command; SDK/test fixture builds stay +pure. Private embedded Go tzdata keeps named zones available in standalone +non-Linux commands. + +The initial existing harness run passes 300 static, 6 cursor-file, 34 portable +error, 60 utility and 18 empty-result observations. Of 174 output checks, two +expose the inherited C-locale ellipsis difference: stock uses three ASCII dots, +while both commands always use one Unicode ellipsis. Both renderers now select +the marker by character locale and account for its width. Each parser passes +its 134 recognition/interaction checks; the newly listed scope check is then +implemented rather than left skipped. + +Development timestamp fixture failures are retained: stock rejects raw journal +header times outside VALID_REALTIME (zero or >= 2^55). The fixture now uses only +valid journal times and requires stock verification before comparisons. The +valid fixture yields 164 observations with three failures (spaced Z in both +commands, lowercase IANA name acceptance in Rust); fresh/seeded Berlin/New York +folds, both spring gaps and forced seasonal abbreviations pass. Rust's provider +lookup is constrained to exact IANA spelling, and spaced Z follows the separate +zone path instead of the failed welded-Z attempt. + +The expanded follow check first exposes test setup limits, not product evidence: +initial history is individually synced before its five-second ready deadline. +The fixture writer now syncs initial history once before publishing ready, while +live appends still sync individually. Large stdout pipes then block at about +144 JSON rows before harness termination. File-backed output capture removes +that collection limit and retains actual exit codes plus intentional-stop +metadata. All 33 follow observations pass, including 1,005-row explicit-all, +since-default and no-tail backlogs and explicit zero. Earlier failed setup and +collection runs remain retained; message equality cannot mask child exit 1. + +Additional bounded controls reproduce three timezone-provider differences in +both commands: a positive TZif leap changes a calendar bound by one second; +fresh TZif names can include a future standard abbreviation absent from the +current seasons; and a zero-transition TZif ignores its conflicting footer in +glibc. The CLI-private timestamp adapter will retain validated TZif metadata +alongside the existing provider, initialize and update the abbreviation pair, +apply leap corrections to UTC/local probes, and use the first standard type +before any transition. This is the narrow metadata supplement already specified +in the implementation gate. It runs once per timestamp context, never per row, +and introduces no SDK change or additional native binding. Named zones use the +installed database when one exists, so a bundled alias absent from that database +does not silently become a valid native spelling. Portable bundled fallback +remains for platforms without an installed zoneinfo database. + +Focused release comparisons retain identical complete outputs across nine +representative commands, using the unchanged SDK and immutable pre-CLI binaries. +The first 100,000-row grep comparison is faster (Go median 0.819s to 0.511s; +Rust 0.215s to 0.206s), while tiny-command repeats expose a startup cost that +requires investigation before performance acceptance. Parent-fork high-water +RSS contaminates the initial process runner's RSS field; those numbers are not +accepted memory evidence. Dedicated hyperfine startup repeats and syscall +summaries isolate startup without expanding the benchmark framework. Static +link variants are diagnostic builds only. Independently, four Go CLI regular +expressions currently compile during every invocation although they serve only +boot descriptors or vacuum options. Move those constant compilations behind +standard sync.OnceValue helpers; this changes no matching semantics and removes +unrequested cold-path work. Validate boot/vacuum controls and compare complete +commands again. No performance waiver or linking-policy change is assumed. + +### Build Entry-Point Review And Outstanding Decisions - 2026-10-17 + +The real binary interoperability entry point builds both writers and commands +from current sources, then passes all 18 binary JSON/export/verification checks +against stock v262 and the libsystemd test adapter. Earlier semantic wrappers +supplied prepared binaries and did not validate these build entry points. +Review found missing environment forwarding in the binary/basic builders and +hard-coded Rust artifact directories despite inherited CARGO_TARGET_DIR. Those +two builders now forward environments and select the actual target directory. + +The same-pattern search found six further builders that conditionally copy a +Rust artifact, then accept an existing destination when the source is absent. +The repair uses the live runner's existing explicit source-file check followed +by unconditional copying. Resolve relative target-directory overrides against +Cargo's repository-root working directory in all affected builders, so build +and artifact selection refer to the same location. This changes test setup +only, adds no SDK/CLI runtime cost, and needs no new framework. Validate with +the existing actual-build binary matrix plus bounded builder failure controls. + +Open decisions at the 2026-10-17 checkpoint, subsequently resolved below: + +1. Authorize a separate signed CI snapshot on a new ci/journalctl-v262 branch + to run the prepared Linux/macOS/Windows/FreeBSD workflow, or retain local-only + work with native non-Linux validation outstanding. No push is yet authorized. +2. Accept the measured CLI-only fixed startup cost of native PCRE2/CGO and the + portable timestamp provider, or retain strict startup parity and approve + development of static native build profiles. Repeated tiny-command controls + show Go approximately 3.0ms to 3.8ms and Rust 1.5ms to 1.8ms; complete + 100,000-row grep improves in both. Static builds are diagnostic evidence + only. The no-regression requirement has not been waived, and dynamic builds + remain the current source default pending this decision. + +### Local Implementation And Validation Ledger - 2026-10-17 + +Implementation ownership remains one local writer. Read-only investigations +checked timestamp-provider differences, parser contracts and native build setup; +these are implementation evidence, not the authorized whole-SOW acceptance +review. No SDK runtime file changes belong to this CLI change set. + +Implemented behavior: + +- Both ordered parsers preserve required argument bytes, encounter-order + validation, immediate terminal options, repeated setters and final-action + dispatch. Empty source/cursor options preserve native presence semantics. + The manifest, its generator and source checker now target exact v262, + including hidden introspection, 72 long options, 28 short options, 16 output + modes and 21 action enum values. The generator reproduces committed JSON. +- Final -F/-N actions reject entry/time/cursor/grep scope while retaining + unrestricted indexed FIELD/DATA enumeration and unique-value limits. + Forward cursor limits begin at the cursor. Follow honors resolved cursor + history, since-only/all/no-tail backlog and explicit zero without retaining + an unbounded output buffer. An empty cursor file retains the ordinary tail. +- CLI-private PCRE2 uses raw first-MESSAGE bytes, stock automatic casing and + native compile/match failures. No-match show returns 1 with empty stderr. + Patterns compile once. Rust reuses owned match data; Go borrows subjects for + one native call that frees match data before returning, with no retained Go + pointer or subject copy. SDK builds retain CGO=0 and no PCRE2 dependency. +- ID tables support four JSON framings and numeric microsecond bounds. + Projection repetition forms a union without trimming field-name bytes. + Cat preserves first raw values, NUL/newline bytes and delayed field-name + validation. Locale-dependent text ellipses use the correct marker width. +- Timestamp parsing caches each occurrence once, preserving command-local + inverse-conversion state. Checked durations/epoch arithmetic, calendar + normalization/fraction rounding, offsets, exact named zones, folds/gaps, + seasonal abbreviation state, TZif leap metadata and POSIX local zones are + covered by the expanded native-oracle controls. No timestamp work is added + to SDK row traversal. Go boot/vacuum constant regexes initialize on demand. + +Validation uses the full-feature stock v262 executable built from +systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492, with PCRE2 10.42, +XZ/LZ4/Zstd and OpenSSL. Host systemd 255.4 is not substituted for this oracle. +Private evidence root: .local/sow-0121-v262-krauk7l5/. + +| Check | Result / retained evidence | +| --- | --- | +| Go CLI at supported Go 1.26.2 floor, with race detection | PASS; go-floor-race-3.log | +| Rust CLI at supported Rust 1.91 floor | 21 unit and 11 subprocess tests PASS; rust-floor-1.log | +| Parser recognition/interactions | 134/134 per language, no skips; parser-final-1.log | +| Additional v262 semantic observations | 160/160; v262-final-1.log | +| Timestamp semantic observations | 222/222; timestamp-final-1.log | +| Existing query/cursor-file/error/output/empty controls | 300/6/34/174/18 PASS; query-final-1.log | +| File-backed utility actions | 60/60; utilities-final-2.log | +| Follow history/cursors/live appends | 48/48; follow-4.log | +| Verifier/directory/mixed-directory observations | 63/34/42 PASS; integration-final-1/ | +| Actual builders plus cross-language binary JSON/export | 18/18 twice; binary-actual-2.log also exercises a relative target override | +| Fresh/missing Rust artifacts, relative target overrides and CLI-only CGO in all nine builders | 18/18; builder-controls-9pnv6kng/results.txt | +| Portable smoke contract on Linux | 28/28; portable-smoke-linux-1.log | +| Pure Go SDK build and runtime-purity guard controls | PASS; runtime-purity-1.log; SDK runtime diff empty | +| Executable documentation examples | 35/35; docs-validation/results.json | +| Documentation structure, Python syntax, changed workflow actionlint and diff whitespace | PASS | +| In-progress project audit | Complete-and-clean verdict; audit-in-progress-2.log | + +Failed development/collection observations remain retained. In particular, +query-final-1 reused a vacuum fixture directory and encountered FileExistsError; +the utility collection was rerun in a fresh private workspace and all 60 passed. +The first follow setup synced each historical row and exceeded its ready +deadline; batched initial sync fixed setup. Pipe-backed collection then blocked +large histories; file-backed capture now drains complete output and records the +actual child exit plus intentional-stop status. Neither incomplete output nor +child exit 1 is accepted as a passing follow observation. + +The build-entry-point same-failure search covered every interoperability CLI +builder. Missing env forwarding affected two; stale-copy fallback affected +seven including the binary builder, and all are repaired. The existing manifest +generator also retained v260.1 despite the updated JSON; its target, hidden +option, action and interaction additions now exactly regenerate v262 JSON. +These are validation-infrastructure consequences of the CLI target/native +dependency changes, without a new framework or product surface. + +Performance evidence remains provisional while decision 2 above is open. Nine +complete-command comparisons retain equivalent complete output and immutable +pre-CLI binaries. Complete scan medians are Go 0.514s to 0.493s and Rust 0.220s +to 0.219s; 100,000-row grep is Go 0.819s to 0.511s and Rust 0.215s to 0.206s. +Short-command and projected-output adverse samples remain in command-perf-1/; +the follow-up cold-start repeats and loader/syscall inspection isolate the +fixed native startup cost. Unchanged SDK/index and ordinary row-render paths +retain their algorithms; no full scan, unnecessary decompression, full-result +buffer, per-row option parser or subject copy is introduced. The initial +parent-contaminated RSS field is rejected as memory evidence. Static diagnostic +builds demonstrate an alternative but do not establish a new build policy. + +Dependency/build review: + +- Native PCRE2 is confined to command wrappers and requires 8-bit ABI >=10.42; + Rust uses size_t-compatible usize, not platform-dependent C long. PCRE2's + declared license is BSD-3-Clause. SDK manifests/runtime do not link PCRE2 or + system journal libraries. The libsystemd adapter above is a test oracle only. +- Jiff =0.2.38 and bundled timezone data are CLI-private pure Rust, declared + Unlicense OR MIT and rust-version 1.70, below the supported Rust 1.91 floor. + Lock additions do not upgrade existing dependencies. Optional defmt/bitflags + lock entries do not become SDK runtime dependencies. The saved SDK Cargo + dependency tree contains neither Jiff nor PCRE2. Go's embedded tzdata stays + command-private. Compiler minimums and SDK journal-format target are unchanged. +- CLI build callers explicitly enable CGO; coverage/release retain independent + pure SDK checks. CI declares PCRE2 development prerequisites. The prepared + native workflow covers Linux, macOS ARM64, Windows CLANG64/GNU LLVM and + FreeBSD. Local SDK cross-builds cannot certify these native CLI dependencies; + actual non-Linux runtime evidence remains open under decision 1 above. + The workflow selects the image-provided python3 on Unix and python on + Windows, avoiding reliance on an unverified macOS unversioned alias. + Changed workflows pass actionlint after that adjustment. + +Artifact maintenance at this local checkpoint: + +- AGENTS.md distinguishes the CLI v262 target and approved native exception + from pure SDK v260.1 behavior. No additional runtime-purity exception exists. +- Compatibility skill records semantic/status/binary/follow and native build + evidence rules. Release skill and RELEASING.md distinguish pure SDK checks + from command-inclusive native tests; publication/tagging policy is unchanged. +- Product scope and parity matrix record delivered CLI semantics, native build + requirements and the v262 migration. Historical filenames remain usable. +- Rust/Go README and CLI wiki explain the target, SDK/CLI dependency boundary, + scoped enumeration rejection, cursor/follow fixes, binary projection, JSON + tables and timestamp rules. Interoperability/coverage guides record build + prerequisites and accurate output/status collection. +- No end-user/operator output skills exist in this repository. No copied + output skill requires an update. Both status indexes retain this sole active + implementation SOW and the completed signed SDK dependency. +- Durable changes use synthetic identities, public commits, relative source + paths and sanitized counts. No private journal, identity, credentials or + customer material is used. Scratch/logs/caches remain repository-local. +- Lessons: option recognition does not establish semantics; validate full + cursor/backlog output and actual status; provider timezone metadata can + change native parsing; actual builder entry points need fresh artifacts; + separate SDK purity from CLI native prerequisites; trace added work and + investigate reproducible costs without a large measurement framework. +- Follow-up mapping: current target/parser/query/output/timestamp/native-build + work remains here. Indexed invocation listing and broad polling scalability + remain the real pending SOW-0122. Reopened SOW-0045/0071 and pending + SOW-0160/0161 remain separate, with no implementation mixed into this SOW. + +Still required before close: explicit disposition of the two open decisions, +native platform evidence or a user-approved qualification limit, whole-SOW +external review under standing authorization with findings resolved, final +artifact/audit gates and one signed implementation-plus-lifecycle commit. +No whole-SOW readiness, performance PASS, native platform PASS or completion +claim is made at this checkpoint. No push, release or deployment has occurred. + +### User Decisions And Normal PR Validation - 2026-10-18 + +The user agrees with the recommended bounded CLI startup allowance and asks +why normal PR CI would not cover the requested native platform checks. The +prepared workflow already runs on pull_request, so no separate CI-only branch +or special workflow dispatch is necessary. Use the normal PR's Linux, macOS, +Windows and FreeBSD jobs, with PCRE2 development prerequisites and CLI-only +CGO, while retaining independent pure SDK checks. Restrict the additional push +trigger to master, matching ordinary post-merge coverage and avoiding duplicate +native jobs for each PR-branch push. This is CI configuration within this SOW, +not a request for additional server software, services or remote infrastructure. + +The accepted performance exception is specific: measured fixed startup on the +tiny Linux command is approximately Go 3.0ms to 3.8ms and Rust 1.5ms to 1.8ms, +for native PCRE2/CGO and the portable timestamp provider. Keep default native +linking; static builds remain diagnostics rather than a new packaging policy. +This allowance does not permit SDK or sustained query-throughput regressions, +and faster grep does not offset an unrelated regression. Existing reviewed +native-index/row paths and representative complete-command evidence remain the +performance basis under the user's engineering-judgment rule. No new timing +framework or universal per-case measurement requirement is introduced. + +Both earlier product/authorization decisions are resolved. Complete the local +artifact/audit gate and authorized whole-SOW external review, then use a normal +PR for native CI. Keep this SOW open until platform results, reviewer findings +and final implementation/lifecycle gates are resolved. No merge, release or +deployment is authorized by this decision. The default docs/specs must describe +the accepted fixed CLI cost and retain the SDK/native dependency boundary. + +### Post-Decision Primary Gate - 2026-10-18 + +The complete internal implementation audit and local validation are finished +for the CLI candidate against signed SDK base 25accee. Runtime code is unchanged +since the passing supported-floor, semantic/output/timestamp/follow and actual +builder runs above. The final changes record the accepted bounded startup +allowance in this SOW, product scope and CLI docs; update both status indexes; +and use the ordinary PR trigger plus master-only post-merge push coverage. +There is no additional CI-only branch or special remote execution procedure. + +All 35 executable documentation examples pass again in the private +docs-validation-20261018 workspace, and all 16 wiki pages validate. Changed +workflows pass actionlint, changed Python sources parse, whitespace checks +pass, and the added-source identity scan is clean. The pre-review audit reports +SOW initialization complete and clean. No SDK runtime file differs from the +signed base. The candidate manifest retains all changed source files and both +dynamic CLI binaries under the private review-candidate directory referenced +by .local/sow-0121-v262-krauk7l5/current-review-candidate.txt. + +No verified in-scope implementation blocker remains in the primary audit. The +candidate is ready for the authorized independent whole-SOW static review; +native non-Linux runtime qualification will run through the normal PR checks +and remains a pre-close gate. This local checkpoint does not claim those remote +results. Runtime source is frozen while review runs; only review evidence may +be appended. Any verified material defect requires repair plus complete renewed +internal validation before another full-scope round. + +### Whole-SOW External Review - 2026-10-18 + +After the complete primary local audit, validation and post-decision artifact +updates, the full neutral prompt was displayed and all six authorized static +reviewers started in parallel in session b3ee3abf5edc47409040187cb33fbeb3. +The prompt covers this entire SOW, all tracked/untracked changes against signed +SDK base 25accee, indirect build/CI/consumer consequences, native reference +behavior, accepted fixed startup cost and the remaining normal PR platform +gate. Review is read-only: no tests, builds, writes, delegation or host-journal +access. This CLI round is distinct from both completed SDK rounds. Retain each +result/failure and independently adjudicate findings before PR submission and +closure; failed invocations supply no technical verdict. + +All six initial invocations terminate after about 31-33 seconds with empty +stdout, repeated reconnect messages and the provider's temporary high-demand +error. No reviewer produces a technical verdict. These are concrete transient +transport failures under the reviewer policy; each receives its sole permitted +retry in the same session, using the identical full prompt displayed above. +There are no successful reviewers to rerun. Runtime source remains unchanged. + +The six permitted retries also terminate with empty stdout and the same +temporary high-demand/reconnect failure after approximately 42-46 seconds. +All twelve attempts are retained; this round supplies zero technical verdicts +and six coverage gaps. No further retry is allowed in this round, and no +approval or review waiver is inferred. The technical review gate remains open. + +### Normal Draft PR And Native Validation Plan - 2026-10-18 + +Continue the already-authorized normal PR validation while keeping this SOW +in-progress. Use one ordinary feature branch, fix/journalctl-v262-parity, and +a signed provisional CLI commit so GitHub can run the prepared platform jobs. +This is a draft for review/CI, not a completed SOW, merge or release. The PR +also carries the two earlier signed prerequisite/SDK commits not yet on origin. + +After all remaining gates are resolved, fold validation/closure evidence and +the completed SOW move into the signed final CLI commit, preserving the +provisional commit as a local rollback point. Do not rewrite master or other +contributors' work. Any update of the task's own provisional PR head must +verify the exact captured remote commit and preserve the previous commit. +The final SOW lifecycle must remain coherent with its implementation; a +standalone completion-only commit is not the delivery plan. + +### First Normal PR Native Results And Windows Checkout Repair - 2026-10-18 + +Normal draft PR #12 is open with signed provisional CLI commit c3b4b79, +following signed SDK/prerequisite commits. Native workflow run 39134674439 +passes Linux and macOS ARM64, including both CLI suites and 28 portable smoke +controls. FreeBSD remains in progress at this checkpoint. Windows fails before +any build during actions/checkout: two upstream corrupted fuzz fixture names +contain colons and cannot be represented as Windows filenames. This is a +checkout setup failure, not a demonstrated CLI runtime failure. + +The complete tracked-path inventory finds exactly two Windows-invalid names, +both under fixtures/systemd/test-data/corrupted/id:*.zst. Preserve all upstream +files in the repository; use Windows-only non-cone sparse checkout to omit +those two inputs. Do not omit the full corrupted directory: the Rust CLI's +verify_corrupted_fixture test requires zstd-truncated-frame.zst there. +The Go CLI verification tests require that same retained fixture. Independent +read-only reachability review confirms that neither CLI suite nor the portable +smoke uses either colon-named input. Git sparse-checkout check-rules, evaluated +against all 867 tracked paths, excludes exactly those two inputs and retains +the other 865, including all required fixtures. Actionlint and whitespace +checks pass; actual Windows qualification remains outstanding. +Linux/macOS/FreeBSD retain their ordinary checkouts. No SDK/CLI runtime changes +or test exclusions are needed. Validate Git's actual sparse-rule selection +against every tracked path, retain the required malformed-stream fixture, +run actionlint and obtain the real Windows CI result. Capture current CI +results before pushing this correction. + +### Native Checkout And Independent Review Repairs - 2026-10-18 + +Normal PR CI at f5063ba passes Linux, macOS ARM64 and FreeBSD again. +Windows still fails before building: sparse selection is correct, but Git for +Windows validates index paths before omitting worktree files. Source evidence +is git-for-windows/git @ cc4dbf752a05efdc0e04e71fd3e8110d11bdd35c, +compat/mingw.c:4093-4099. Preserve the exact two sparse exclusions and disable +core.protectNTFS only through the checkout step's command-scoped environment. +Do not persist the setting, rename fixtures or omit other corrupted fixtures. +The pinned checkout action forwards that environment to each Git invocation; +subsequent status/log and sparse-index refresh paths do not rebuild entries. +Local rule inventory is evidence for selection, not proof of Windows success; +the ordinary PR Windows job must supply that result. + +Independent static review finds five concrete gaps in the full v262 contract. +Pinned stock and both retained dynamic candidates reproduce the first four; +the metadata zero-limit case is also reproduced separately. Private before +records are retained under the current-review-reproductions.txt pointer. + +- Go parseBoolOption trims whitespace, so invalid boolean values before help + succeed while Rust/v262 reject them. Remove only that normalization. +- Bounded forward output performs PCRE matching after the requested final row: + a valid first MESSAGE followed by a non-UTF8 MESSAGE makes -n+1 with + --grep=(*UTF). print the row then fail, while v262 succeeds. Stop callbacks + immediately after consuming the final selected row. Search and repair the + corresponding chronological-tail collectors, and avoid Rust head-zero + traversal. Preserve unlimited, cursor, render-exclusion and follow behavior. +- Both field-value actions accept invalid names, including an empty name with + -n0. Validate at field enumeration time using the existing native field-name + predicate, before the zero-limit shortcut. Preserve earlier terminal help + and keep SDK APIs unchanged. +- Rust positional matches coerce raw OS arguments into String and reject a + valid binary MESSAGE match that Go/v262 accept. Keep OsString through the + CLI, test '+' and '=' as bytes, then use the existing parse_match_bytes and + facade AddMatch. Lossy conversion is allowed only for diagnostics. Preserve + current portable path-argument rejection and generated textual filters. +- Both metadata tables turn -n0 into successful empty output, while v262 exits + one with no table. Apply no-data handling after limit selection, respecting + quiet/JSON suppression of the diagnostic. Preserve input, unit and boot + validation before that result. Check actual empty metadata selections too. + +These are required consequences of the already-approved full CLI contract, +not new product choices or SDK work. Add paired shared controls before runtime +changes, retain their red results, repair the minimal CLI paths, and rerun both +supported-floor CLI suites plus parser/query/output and portable controls. +Removing trimming and excess traversal adds no hot-path work; the native +indexes and SDK row loops remain unchanged. Retain complete-command +performance evidence rather than adding another timing framework. + +The initial primary checkpoint did not discover these gaps and is superseded +by this repair gate. Complete independent full-scope review and local repair +validation before a fresh whole-SOW external round. Provider failures from the +earlier round remain coverage gaps. The SOW remains in-progress. + +The completed independent changed-surface audit adds a finite adjacent repair +set. Source comparison and pinned-oracle probes reproduce both CLI no-full +multiline/suffix loss, quiet verification PASS diagnostics, nonnative priority +aliases/case, facility numeric bounds and native integer syntax/width. Private +output-option-before.json and integer-before.json retain the exact results. +The original tests covered a 500-byte identical-message threshold case, not +ordinary multiline/suffix shortening, and decimal parser values only. + +- Short output must process original lines before indentation, retain stock's + 90-percent prefix/suffix behavior, and apply its three-line/300-byte threshold + and UTF8 cell-width rules. Keep full/show-all bypasses and avoid extra work + in the default full-width path. Reuse existing output fixture/control tools. +- Pass effective quiet to verification reporting and suppress informational + PASS lines only. Preserve malformed-file, missing-key and invalid-key errors. +- Use one small CLI-private native-integer parser per language for the six + stock safe_ato consumers: lines, boot/invocation offsets, priority, facility + and vacuum-files. Preserve leading ASCII whitespace, base prefixes/octal, + signed 32-bit limits where applicable and unsigned 64-bit vacuum counts. + Reject trailing whitespace/underscores and nonnative priority names. Numeric + facilities extend through127. Boot-ID suffixes require an explicit sign and + the descriptor's ID form follows the pinned native parser. No SDK parsing + contract changes. +- Reproduce three-boot hit/miss/hit filtering in both directions. Observe boot + transitions before grep, preserve intervening/trailing transition events, + and keep quiet/merge/rendering and selected-row limits intact. Audit show, + tail and follow callers rather than changing SDK traversal. +- Derive grep direction from the effective pager/line limit, including the + implicit pager-end1000 count, and preserve explicit all in Go. Reuse the + existing1005-row fixture for the latter control. +- Preserve an explicitly supplied empty Go cursor-file path through action + use, rather than ignoring the requested update; earlier terminal help must + still succeed. Named empty-content cursor files remain valid. + +The last three source findings require native reproductions before repair. +These are required full-contract fixes, with no additional user design fork. +The reviewer found no SDK runtime changes or new native dependency leakage. +The verdict applies to the frozen provisional candidate, not the moving repair +work; complete validation and renewed whole-SOW review remain mandatory. + +The expanded shared suite records280 observations and96 failures on the +retained provisional binaries, covering the complete finite repair list. +Three-boot separators and pager effective-limit controls reproduce the source +findings. The empty cursor-file probe additionally shows Rust rejecting before +rows while stock prints rows then reports the requested write failure; retain +OS-string parsing for that path in Rust and explicit presence in Go. +Follow is part of the boot-observation repair: select its initial tail from +native-filtered rows before grep, and retain a separate last-visited resume +cursor so nonmatching rows do not repeat boot transitions across polls. +Keep displayed/cursor-file state separate and preserve owned Rust cursor +strings where possible to avoid a new allocation per skipped row. + +The first repaired-binary pass reduces shared failures from96 to10, all in +short output. The new controls expose two adjacent native rendering details: +v262 prints the last repeated MESSAGE in short mode (while grep/cat still use +the first), and its literal unknown-label fallback is printed without adding +those eight bytes to the indentation/width counter. Source logs-show.c:649- +733 and retained expected/actual bytes establish both. Preserve those exact +mode-specific rules, keep custom multiline fixtures free of the unrelated +repeated-MESSAGE control, and add a separate repeated-MESSAGE short case. +Return the actual label and its fallback-width adjustment together to avoid +repeating field lookups in each rendered row. No SDK change is required. + +### Repair Validation And Normal PR Update - 2026-10-18 + +The complete finite independent review batch is implemented in the CLIs. +Supported floors Go1.26.2 (race) and Rust1.91 pass their full CLI suites; +Rust retains21 unit and11 subprocess controls. Shared repaired binaries pass +286/286 v262 semantic controls, both134-case parser suites,592/592 existing +query/cursor/error/utility/output/empty controls and54/54 follow controls. +The follow additions compare native-tail-before-grep behavior and complete +short output with intervening boot events across polls. New controls retain +old-binary red evidence; primary review also checks every matching caller. + +The first new follow run times out awaiting the stock fixture writer's ready +file, before reader execution. The exact saved three-row input completes in +0.514 seconds on a fresh isolated run; the fresh complete54-case follow run +then passes. Preserve the earlier failure rather than treating it as CLI or +performance evidence. No process or service outside test children is stopped. +One repaired Rust fallback-label error is caught in the paired semantic pass +and corrected before the286-case clean result. + +No SDK runtime file differs from signed base25accee. Native DATA/FIELD/ENTRY +index use, SDK allocation behavior and decompression scope are unchanged. +Native integer checks are cold; the boot-descriptor regex is eliminated. +Grep still runs once per visited eligible row; follow retains a visited cursor +without adding a Rust string allocation per rejected row. Shortening work runs +only with no-full; the default full-width branch adds no scans or width-table +work. A targeted release command comparison will check the changed show/grep +paths using the existing fixture and runner; no new measurement framework is +introduced. The previously accepted tiny startup allowance remains the sole +intentional CLI cost exception. + +Normal PR quality findings are adjudicated individually. Deterministic +bootstrap sampling keeps its seeded generator with a local B311 annotation. +The explicit stock-oracle version invocation gets B603 on the actual call, +replacing its misplaced annotation. Five newly introduced blank-line findings +are removed. The eighth aggregate Pylint result is an unused expected local +in the already-changed follow harness; remove it without changing behavior. +These are test/tooling-only adjustments, including two comment-only SDK test +helpers, not a reopening of SDK runtime work. Raw reports remain private. + +Docs/specs now describe integer/empty/quiet rules, mode-specific repeated +MESSAGE selection, native shortening and follow boot/position behavior. The +compatibility skill records the missed test classes. No new public SDK API, +compiler minimum, cache version, host service, release or installation changes +are involved. Both status indexes remain in-progress. Native Windows and the +complete final review remain outstanding before lifecycle closure. + +### Final Short-Header Same-Pattern Audit - 2026-10-18 + +Primary source review of the changed short-prefix helpers finds and reproduces +five adjacent cases in both languages: `_EXE` incorrectly replaces the native +unknown label; an explicitly empty identifier is treated as absent; duplicate +identifiers select the first rather than last value; with-unit drops the user +unit when both units exist; and an invalid identifier is printed instead of +falling back to `_COMM`. The retained label-audit-current.txt pointer identifies +all15 before observations. Authority is systemd/systemd at the pinned v262 +commit, src/shared/logs-show.c:193-205,649-675. + +Repair the existing CLI-private header builder in both languages rather than +adding per-caller patches. Read the last raw value through borrowed field-map +views, distinguish absent/empty, apply native printable/length fallback rules, +combine the native system/user unit fields, and retain native width accounting +including its unknown/user-unit quirks. Construct Rust headers as bytes so +show-all never loses raw bytes through lossy String conversion. Keep grep/cat +first-MESSAGE behavior and all SDK code unchanged. Add paired controls for +these cases, raw show-all and the fallback width, then repeat affected CLI, +semantic/output and representative command checks. Existing header allocation +can be reduced by avoiding temporary value lists and cloned label strings. +This is part of the same approved native rendering contract and introduces +no new product choice or dependency. + +The same label audit also reproduces exclusion and opaque-option consequences: +-Tfirst wrongly suppresses a row whose last identifier is second. Rust rejects +raw-byte identifier/exclusion arguments and silently drops a non-UTF8 +projection, expanding JSON to all fields or making cat succeed. Pinned native +and Go observations are retained in exclude-before.json/opaque-before.json. +Preserve native last-identifier/C-string exclusion and raw OS arguments for +these opaque CLI values. Normalize projection names once at option construction: +retain valid UTF8 names for field lookups, retain the first invalid cat name +for delayed rendering-time errors, and keep explicit projection presence even +when every name is unmatchable. This avoids a UTF8 check for every projected +field on every returned row. Identifier filters can use existing byte AddMatch; +exclusions can compare borrowed raw bytes without lossy conversion/allocation. +No SDK API, dependency, or new user-visible syntax is introduced. + +Full-output newline ownership is checked directly against native output before +freezing. Both languages lose the indentation of a trailing blank line and +drop lines after a NUL in show-all because they truncate the whole message +as one C string. Six native/candidate observations are retained under the +newline-boundaries-current.txt pointer. Trim the renderer-owned final newline +after indentation, and apply C-string truncation separately to each original +line only on the show-all/NUL path. Ordinary full output avoids an extra scan; +Rust reuses the borrowed last-MESSAGE view. Add paired full/shortened/trailing +blank/NUL controls, and remove the exclusion visitor made unused by the +borrowed last-value repair. These are the final same-path newline consequences +of the approved per-line native rendering contract. + +### Complete Second Primary Review Repair Plan - 2026-10-18 + +The second independent whole-scope pass reports six finite classes, with no +P0/P1 finding. Its FAIL applies to the inspected candidate, not later repairs. +The scope includes parser, native regex, time, output, cursor/follow, +maintenance, consumers, dependencies, CI and artifact/performance contracts. + +- Rust's empty-source guard uses failed UTF8 conversion as an empty value. + Pinned-oracle probes reproduce nonempty raw root/image silently cleared. + Test actual OsStr emptiness; preserve namespace presence and the approved + portable-unsupported classification. Native ignores namespace with an + explicit file, so that intentional qualification must not be represented + as native rejecting the raw value. +- Both short renderers omit native tab/ANSI/CR preparation. Mirror the native + byte state machine before newline truncation and printable/length checks; + keep incomplete or unsupported controls intact, bypass show-all, and leave + cat/grep raw. Borrow ordinary messages and allocate only when a control + needs processing. The unchanged printable scan can detect those controls + without adding another scan of ordinary ASCII messages. +- Both cursor checkpoints retain the last grep-selected row. Reproduce and + retain terminal visited position independently of shown counts, including + bounds, direction, EOF and zero-line positioning. Stop immediately at a + counted limit and do no excess regex work. Reuse existing facade cursors; + check show-cursor and cursor-file through the same finalization path. +- Replace single float/suffix vacuum-size parsing with native ordered compound + uppercase E/P/T/G/M/K/B grammar, decimal fractional handling and checked + u64 accumulation. This is cold CLI parsing; no SDK parser changes. +- JSON aggregation confuses a single binary byte array with repeated field + values. Preserve that distinction in the existing aggregation path and + check binary-first, multiple-binary and text-first values in every JSON + framing. No additional per-row occurrence map is necessary: this encoder + emits scalar strings/null or byte arrays, never scalar numbers. +- Rust converts native input paths lossily before facade opening. A concrete + two-file reproduction reads the replacement-named journal instead of the + requested journal, while native/Go read the requested four rows. Existing + native readers cannot be injected into the private facade. A narrow SDK + native-path exception versus an explicit CLI parity qualification is + presented to the user; dependent implementation remains pending that + decision. This is inherited behavior, not a regression from recent changes. + +Authority is systemd/systemd @8cc40e0c5e9234bf45084751ac53b1fbfe70b492: +src/shared/logs-show.c:600-604,1115-1169,1310-1318; +src/basic/string-util.c:704-841; src/basic/parse-util.c:157-284; +src/journal/journalctl-show.c:73-208,496-520. Private raw-sources and +native-path-proof pointers retain direct observations. Add shared controls +and retain red results before runtime repairs, then validate the complete +finite batch and repeat whole-scope review. No installation, service change, +new measurement framework or additional startup allowance is proposed. + +### Second Review Repairs Validated; Ordinary PR CI Update - 2026-10-18 + +The five CLI-private classes are repaired. Shared controls pass518/518, +including the explicitly qualified raw-namespace result, native cursor-file +contents, zero positioning, terminal after-cursor cases and all JSON framings. +Before repair, the first472-control expansion records90 failures on retained +binaries. One intermediate pretty-JSON comparison incorrectly parses multiple +objects as a single JSON document; streaming decode corrects the test without +loosening object/value comparison. The repaired binary-first nested arrays +match the native values in every framing. + +Supported-floor Go1.26.2 race and Rust1.91 CLI suites pass again; Rust has21 +unit and11 subprocess controls with no compiler warnings. Both134-case parser +suites,592/592 existing query/output controls and60/60 follow controls pass. +Follow adds skipped-suffix and no-grep-match cases that compare the actual +cursor-file content with the final native row after controlled child shutdown. +All28 portable smoke controls and CGO-disabled Go SDK compilation pass. +The complete-and-clean SOW audit, changed Python parsing, workflow actionlint, +16-page wiki structure and diff whitespace checks pass. The35 executed docs +examples remain unchanged by these prose-only documentation additions. + +The broader query run catches a Go checkpoint repair consequence: the raw +facade getter exposes an internal end marker after a partial-seqnum seek. +Use the existing TestCursor no-entry mapping on the relevant error path and +retain a pre-advance checkpoint only after a successful after-cursor test. +The fresh complete592-control pass validates the correction. Ordinary query +opens retain their original concrete facade; the small CLI cursor adapter is +allocated only when cursor output/persistence is requested. It calls the SDK +metadata cursor accessor rather than expanding another row. Rust retains a +reusable cursor string only for requested checkpoints. No extra regex call or +row traversal is introduced after a finite match limit. + +The matched-compiler release comparison uses Go1.27.1 and Rust1.97.1 on both +sides, matching the retained SDK-final binaries. Rust dynamically links PCRE2. +Three representative100,000-row commands use the existing five alternating +paired-sample runner and verify complete output equivalence before timing: + +| Command | Go baseline / candidate median | Rust baseline / candidate median | +| --- | --- | --- | +| Cat scan |0.546942 /0.548073 s|0.241363 /0.242167 s| +| Literal grep |0.822826 /0.514308 s|0.222886 /0.214755 s| +| Short output |0.663984 /0.666907 s|0.352176 /0.339566 s| + +The tiny adverse scan/Go-short median differences are inconclusive within the +observed sample variation; they are not a measured-equivalence claim. Primary +review finds unchanged native SDK traversal/decompression and removes a Go +UTF8 validation pass, temporary header values and unnecessary Rust clones on +ordinary output. The practical performance decision accepts this combined +representative evidence and review; no reproducible sustained slowdown is +established. The earlier comparison built with different compiler versions is +explicitly marked invalid and is not used as acceptance evidence. No new +benchmark framework or additional startup-cost allowance is introduced. + +Native-path SDK implementation still awaits the explicit user decision. +SDK runtime diff against25accee remains empty. Keep this SOW in-progress and +update the ordinary draft PR with the validated CLI repairs, quality-comment +corrections and Windows checkout fix so native CI can supply actual evidence. +This checkpoint does not satisfy final review, native Windows or SOW closure. + +### Actual Windows Build-Step Evidence - 2026-10-18 + +Normal PR run39143540947 at signed provisional0fc3d827 checks out Windows +successfully with the scoped index/sparse correction. PCRE2 prerequisites and +Go1.26.2 native compilation succeed. The Go CLI suite reaches one failure: +TestDirectoryInputDoesNotOpenRegularFileAsFile assumes the POSIX wording +"not a directory", while Windows returns its path-not-found OS diagnostic +for ReadDir on the existing regular file. The command does reject the source; +this is an inherited platform-specific assertion, not successful file fallback. + +Strengthen the same test before accepting the platform: create a valid journal +with the existing fixture helper, prove --file reads its expected MESSAGE, +then prove --directory on that same file fails without output. This directly +detects accidental file fallback on every OS without depending on translated +or platform-specific OS text. No runtime or SDK change is needed. Run the +focused local control and then the ordinary PR's complete Windows suite. +The first native build failure is retained, and Windows remains unqualified +until the real job passes. No special CI service is required. + +The same PR's Go analysis reports two unused initial header value assignments +and two integer-conversion warnings. The latter are false positives: the +shared mutable limit already bounds negative magnitudes to2^31 and positive +values to2^31-1, so every conversion fits even a32-bit int. Make those literal +bounds explicit in their respective branches to expose the existing proof; +do not change accepted values or suppress the check. Discard unused shortField +return values before the existing raw-unit lookups. A new markdown finding is +unquoted underscore-prefixed field names in the appended SOW; format those as +code. Run the affected CLI/parser controls and retain current CI results before +updating the task's provisional PR commit. SDK runtime remains unchanged. + +The strengthened directory control passes locally. The Go floor race suite, +518 shared semantic controls and both134-case parser suites pass after the +literal-bound/unused-assignment cleanup. Twelve direct native observations +cover both signed32-bit limits and their immediate out-of-range neighbors. +This cleanup adds no hot-path operation; the existing matched-compiler timing +and complete-output evidence remain applicable without another timing batch. + +Run39143540947 passes complete Linux, macOS ARM64 and FreeBSD checks at +0fc3d827; Windows's single wording assertion fails before Rust tests run. +The quality pipeline reports zero current Bandit/Pylint findings, one newly +fixed SOW markdown finding, and one unexplained Bandit tool error. The error +was also present in earlier PR runs; retain it separately from findings and +do not report the quality gate as passed. Its zero-result tool reports do not +yet close the three older annotations, whose latest instances still point to +an older merge commit. Current local source fixes those recorded cases. +All seven earlier inline threads were individually answered and resolved; +the newly reported equivalent findings will be answered in their own threads +after this correction is pushed. Final review and native-path scope remain +open. No SDK change, tool installation or service action is authorized here. +The same-failure search finds Rust's analogous wording assertion, but its +DirectoryReader emits that literal diagnostic before OS directory access; +it is platform-independent. No other CLI test relies on these POSIX OS error +phrases. The next Windows job still must execute Rust's full suite. + +### Ordinary PR Native Qualification And Remaining Path Decisions - 2026-10-18 + +Normal draft PR #12 at signed provisional837873a7 now passes all four native +jobs in run39144771573: Linux ubuntu24.04, macOS15 ARM64, Windows2022 and +FreeBSD14.4. Windows completes both language CLI suites and portable smoke +checks after the platform-independent directory assertion repair. The scoped +checkout correction is also proven by this actual native job. No separate +CI-only branch, manual dispatch, service or remote execution procedure is +needed. Every executable PR check is successful, neutral or intentionally +skipped; only the draft WIP check remains in progress. Release publication and +wiki publication are intentionally skipped on PRs. + +The current PR reports zero open analysis alerts and all14 inline review +threads individually answered and resolved. However, the green Codacy workflow +still logs one internal Bandit tool error and zero reported findings. Retain +that coverage limitation separately from the successful workflow status; +zero findings do not prove that every Bandit input was analyzed. This error +also occurred in earlier PR runs. Its cause is not established, and no tool +installation or unrelated analysis-framework change is proposed here. +Private native-ci-repair-4.json, pr-current-ci-complete.json, +pr-review-threads-4.json, pr-analysis-alerts-4.json and codacy-pr-ci-4.log retain +the results. Final external technical review remains outstanding; the earlier +transport failures supplied no verdicts. + +The user has two concrete path decisions pending, with no dependent SDK or +glob implementation started: + +1. Native Rust paths: the existing lossy conversion can open a replacement- + named journal instead of the requested raw-byte parent path. Recommend an + additive native-path facade API and native filename checks in this SOW, + preserving existing string APIs and reader traversal. The surgical + alternative explicitly rejects such inputs and documents the parity + qualification. Evidence is rust/src/cmd/journalctl/main.rs:748-756 and + rust/src/journal/src/facade.rs:88-138, plus the retained two-file oracle + reproduction. Go already supports literal native-byte paths. +2. File patterns: Go filepath.Glob and Rust's Unicode/lossy matcher differ + from native locale-aware file expansion. Under LC_ALL=C, the valid + unicode-?.journal pattern does not select a two-byte UTF8 basename in + native journalctl, while both candidates select it. Native brace-{a,b} + expansion selects both files; both candidates fail. A raw-byte bracket + class also demonstrates Go rejecting a native-valid pattern, and a + replacement-character class selecting additional raw-byte files. + Recommend native locale-aware Unix globbing confined to the CLIs, with + an explicit Unicode filename contract on Windows. The surgical alternative + retains a documented subset and rejects cases that could select the wrong + files. SDK purity, journal indexes and row traversal stay unchanged by + either file-pattern choice. The separate native-path choice controls any + Rust SDK API addition. + +File-pattern source evidence: go/cmd/journalctl/main.go:942 and +rust/src/cmd/journalctl/main.rs:800-862. Upstream authority is +systemd/systemd @8cc40e0c5e9234bf45084751ac53b1fbfe70b492, +src/journal/journalctl.c:424,1016 and src/basic/glob-util.c:102,160: locale +initialization plus native glob with NOCHECK, NOSORT and BRACE. Private +native-glob-proof-current.txt points to the retained literal, bracket, locale +and brace observations. Preserve ordinary text/unit matching independently +from the file-pattern decision. A limitation-only option must also cover +raw-byte names discovered through a Unicode directory or pattern. + +This is evidence and design tracking only. SDK runtime still matches25accee. +Keep this SOW in-progress, preserve the successful PR head, and fold this +record into the next authorized implementation/lifecycle update rather than +invalidating native CI with an evidence-only push. Final path decisions, +their implementation or explicit qualification, and final review remain +required before coherent closure. + +After recording this checkpoint, the local SOW audit reports initialization +complete and clean; diff whitespace checks pass. Evidence is retained in +audit-ci-complete-1.log. No implementation file changes in this checkpoint. + +### User-Requested Review Retry After Provider Restart - 2026-10-18 + +The user reports restarting the model provider and explicitly requests retrying +the six failed reviewers. This authorizes one additional attempt per reviewer +in existing session b3ee3abf5edc47409040187cb33fbeb3 despite the earlier automatic +retry limit. No successful reviewer is rerun; all six prior results were empty +transport failures. Preserve both previous attempts and use the same complete +SOW scope against SDK base25accee, with factual updates for current837873a7, +validated CLI repairs, actual native CI results and recorded coverage limits. + +The complete prompt is displayed before execution and retained privately as +whole-sow-review-retry-after-restart-20261018.txt. The user-requested recovery +of this interrupted review does not claim the current SOW is ready to close. +The two path-contract decisions remain pending, with no dependent SDK or glob +implementation authorized by this retry. Reviewers remain static and read-only, +with no tests, writes, delegation, dependency fetching or host-journal access. +Collect every result and independently validate any finding before action. + +All six requested attempts launch in parallel as turn3 of the existing session. +Collection confirms every invocation exits1 with zero stdout after32.9-36.1 +seconds. Each stderr again contains repeated reconnects and the temporary +high-demand message. These are invocation failures, not code findings or +technical verdicts. The continuing error's origin is not established by that +generic message; the user's provider restart does not prove recovery of the +entire request path. All18 attempts remain retained in the runner's session +logs. The result remains zero technical verdicts and six coverage gaps. +No further retry is started automatically. Final review and both pending path +decisions remain open; CLI/SDK implementation and the passing PR head are +unchanged by this recovery attempt. + +### User-Requested Review Retry After Provider Repair - 2026-10-18 + +After identifying a provider bug and asking to wait, the user explicitly +requests another retry because the model service should now be working. +Resume all six failed reviewers in session b3ee3abf5edc47409040187cb33fbeb3 +using the same full-scope, static-read-only prompt displayed before execution. +This is one further explicitly authorized attempt for each reviewer; retain +all earlier failures. Implementation remains at837873a7 with evidence-only +SOW/status edits. The pending path decisions and final acceptance gates are +unchanged. Collect and independently adjudicate every result. + +The repaired-provider attempt reaches active source review for multiple +reviewers. Qwen alone exits after5.98seconds with zero stdout and explicit +HTTP429 Too Many Requests, rather than the previous generic reconnect error. +After more than three minutes of backoff, retry only Qwen once under the +renewed user request and the skill's transient-failure policy. Reuse the +identical displayed full prompt. Leave the five running reviewers untouched; +retain Qwen's failed turn4 and do not automatically retry its replacement. + +Qwen's targeted turn5 also fails with HTTP429 after4.81seconds and no output; +its review coverage remains unavailable. Mimo's turn4 exits0 after308.53seconds +but produces no final agent message, a zero-byte last_message.txt and no +technical verdict. Treat that empty completion as a failed review, not a pass. +Retry only Mimo once for this concrete empty-output failure, reusing the same +displayed full prompt and preserving the other running reviews. No subsequent +automatic Mimo retry is allowed if this replacement fails or remains empty. + +### Recovered Review Results And Primary Adjudication - 2026-10-18 + +Minimax turn4 returns PRODUCTION GRADE after553.8seconds with five P3 notes. +Its result is preserved in the session; no successful reviewer is rerun. +Primary adjudication does not adopt that verdict as SOW acceptance: + +- Native paths and globbing: the underlying issues are already independently + reproduced and remain blocking gaps against the requested complete contract. + The P3/outside-implementation classification does not remove them from scope. + No product qualification or reject-on-invalid-native-path repair has shipped; + the two user decisions remain required before implementation and closure. + The cited Rust code performs lossy conversion, not canonicalization. +- Review coverage: the zero-verdict statement describes the earlier failed + attempts and is superseded by this actual review result. Remaining reviewers + and Qwen's explicit availability gap must still be collected/dispositioned; + no process success substitutes for a technical verdict. +- Performance: no concrete regression is established. Inconclusive timing + differences remain part of the combined representative evidence and source + review, rather than proof of measured equivalence. The user's engineering- + judgment rule remains controlling; no larger timing framework is required. +- Boolean whitespace guard: already implemented at + tests/interoperability/journalctl_v262_cases.py:187-190 for both supported + boolean options, with leading/trailing padding and help ordering. The518 + shared controls execute those cases. The suggestion adds no uncovered + behavioral requirement. PCRE whitespace is literal pattern content, not an + invalid-boolean rule; do not implement the review's conflation of the two. +- Bandit coverage: the internal tool error is real and remains recorded. + The latest retained PR alert inventory is empty, so the note about still-open + older annotations is stale. The actual combined SARIF contains no detailed + execution notification explaining the error. A focused syntax check of all + 116 tracked Python sources passes under Python3.12.3; this narrows the + investigation but does not replace Bandit coverage or establish its cause. + Evidence: codacy-current-combined.sarif and + python-all-ast-analysis-error.json in the private validation directory. + +No implementation change follows from this first recovered result. Other +running reviews remain independent; the active SOW remains in-progress. + +At the recovered-review checkpoint, GLM, Kimi, DeepSeek and Mimo's replacement +remain running; their durable jobs continue while the two path decisions await +the user. The refreshed complete-and-clean SOW audit and diff whitespace check +pass. Evidence: audit-review-recovered-1.log and review-recovered-status-1.log. +Collect the next terminal notification as new results for turn4/turn5, even +though its session ID matches the earlier failed notifications. + +### Recovered Round Collection And Timeout Continuations - 2026-10-18 + +The new terminal notification is collected as new results, not dismissed as +an earlier duplicate. The recovered attempt has the following actual coverage: + +| Reviewer | Final collected attempt | Result | +| --- | --- | --- | +| Minimax | turn4,553.8seconds | Usable PRODUCTION GRADE review; five notes adjudicated above | +| GLM | turn4,1801.1seconds | Execution timeout, no final verdict | +| Kimi | turn4,1801.2seconds | Execution timeout, no final verdict | +| DeepSeek | turn4,1801.1seconds | Execution timeout, no final verdict | +| Qwen | turn5,4.81seconds | HTTP429 after its targeted retry; no verdict | +| Mimo | turn5,682.2seconds | Final text is a serialized tool-call fragment, not a review | + +The three timeouts come from the runner's1800-second execution budgets, not +from the caller's shorter polling waits. They had been reading source before +termination. Use their single permitted timeout retry under the renewed user +authorization, continuing the exact existing client sessions in parallel. +The source remains unchanged; the full original scope is preserved, with an +instruction to reuse completed inspection and reserve time for a final report. +The complete continuation prompt is displayed before execution and retained as +whole-sow-review-timeout-continuation-20261018.txt. Do not retry any of these +three automatically again if its replacement fails or times out. + +Qwen and Mimo have already used their targeted retries in this repaired-provider +attempt. Neither supplies a verdict, and neither is retried automatically. +Mimo's malformed final text is not executed. Its exact cause is not established +by the output alone. Preserve both coverage gaps and all prior results. No +service, client configuration, dependency or implementation changes follow +from these invocation failures. Both path decisions remain pending. + +All three continuations launch successfully as turn5 and remain running at +the first collection checkpoint. The complete-and-clean SOW audit and diff +whitespace check pass after recording the terminal results and continuation +plan. Evidence: audit-review-timeout-continuations-1.log and +review-timeout-continuation-first-1.log. Their durable jobs continue; a later +terminal notification for this session must collect these new turn5 results. + +### Completed Continuations And Repair Analysis - 2026-10-18 + +All three continuations return usable NEEDS CHANGES reviews: GLM in1404.4s, +Kimi in444.3s and DeepSeek in382.6s. Together with Minimax this gives four +technical reviews; Qwen and Mimo remain the two recorded coverage gaps. +No further automatic invocation is needed in this round. The reviewers +independently confirm the already reproduced native-path and glob gaps; +their differing P1/P2 labels do not change the pending user decisions. + +New source-supported candidates are investigated before implementation: + +- Display timestamps currently choose the first source-realtime value and + accept all positive u64 values. Native selects the last C-string value, + parses its native integer grammar, and requires0 < value <2^55. Existing + last-value and C-string helpers can implement this without allocation; + Rust's temporary value vector can be removed. Native deliberately ignores + source-monotonic adjustment in commented code, so do not add that feature. +- Follow defaults combine pager-end and since with inconsistent precedence. + Native chooses10 first when follow has no since, otherwise1000 for pager-end, + otherwise all. Explicit lines/no-tail/cursor rules still apply. Resolve + initial backlog and the all-history branch from the same selected policy. +- No-full rendering ignores actual stdout width when COLUMNS is absent or + invalid. Reuse current native integer validation, then query stdout's size + once per renderer using CLI-private platform APIs, then fall back to80. + Existing x/sys and libc dependencies suffice; no SDK dependency change. +- Vacuum age uses only mtime. Native also considers access/change timestamps + and the earlier available creation time, including Linux user.crtime_usec. + Metadata reads belong only in explicit CLI maintenance. Inspect available + platform metadata before adding calls; preserve active/non-journal handling + and file identity checks. Header probes must preserve atime like native + O_NOATIME on Linux. Validate retained/deleted files and metadata side effects + on isolated synthetic copies, never host journals. Any measurable command + cost needs disposition; SDK/query traversal remains unaffected. +- Rust trims facility list items where native preserves significant spaces. + Retain native help/empty-item/number handling while removing the name trim. + The unpublished CLI package description must also reflect native PCRE2. + +Baseline inspection against25accee shows the source-realtime, vacuum-mtime +and Rust facility-trim patterns are inherited. The follow/pager and cached +width findings involve paths changed in this CLI SOW. Native references are +systemd/systemd @8cc40e0c5e9234bf45084751ac53b1fbfe70b492: +src/shared/logs-show.c:118-147,448-527; src/basic/terminal-util.c:1263-1308; +src/journal/journalctl.c:588-606,916-919; +src/libsystemd/sd-journal/journal-vacuum.c:59-101; +src/basic/xattr-util.c:419-477. Exact review results remain private in the +existing runner session and review-continuations-final-collection.log. + +The existing full pre-implementation gate still applies. First reproduce +these candidates using the pinned oracle and unchanged binaries, then extend +the existing shared output/follow/utility controls and repair only confirmed +behavior. No new measurement framework, compiler minimum, public SDK API or +source/path contract choice is introduced by this independent repair batch. +Use the supported-floor CLI suites, affected complete matrices, practical +performance evidence and native PR CI after material changes. Repeat the +complete primary audit before a later whole-SOW review; do not use reviewers +as an iterative debugging loop. The two path choices remain independent and +unapproved. The color-mode P3 and deliberately non-stock RAW-name behavior +are still being checked against the actual contract before disposition. + +### Reboot Checkpoint - 2026-10-18 + +The user explicitly requested a pause for a reboot. Implementation and review +work stop here. Preserve the working tree, including all untracked CLI platform +helpers. No commit, push, process termination or reboot is part of this pause. + +- SDK SOW-0159 is completed in signed commit + 25accee6cbca3a14ce14cd4ec3eb92d34dab3681. The current repairs do not change + SDK runtime code. +- CLI branch is fix/journalctl-v262-parity. Local and last recorded remote HEAD + are 837873a7a297015b71b91d5ca0cb34be48707ff0; draft PR #12 is open. Its four + native CI jobs passed before the latest uncommitted repairs. Those results + do not validate the latest working tree. +- The latest uncommitted CLI repairs address source-realtime selection and + parsing, follow backlog defaults, stdout terminal width, vacuum timestamp + metadata and Linux atime preservation, Rust facility whitespace and package + description. New Go terminal/vacuum metadata files and Rust terminal.rs and + vacuum_metadata.rs are required parts of these repairs. +- All external-review continuations have finished. Four reviewers returned + technical reviews; Qwen's HTTP429 and Mimo's malformed final output remain + explicit coverage gaps. No further automatic retry remains in this round. +- The focused native comparison records 126 observations. Before repairs, + there were 57 candidate differences; after repairs only the two forced-color + differences remain. This fixes 55 observed differences, not a claim that all + CLI acceptance criteria pass. Source timestamp, follow, terminal width, + facility and vacuum probes now match the pinned v262 oracle in these cases. +- Supported-floor Go CLI race tests passed. Rust CLI tests passed with 21 unit + and 11 subprocess tests; its final rebuild after formatting passed. Shared + committed regression controls, complete matrix reruns, cross-platform CI, + practical performance assessment, artifact updates and final review still + need completion for these latest changes. +- Two user decisions remain unanswered: native Rust path support versus an + explicit qualification, and native locale/brace file glob support versus an + explicit qualification. Dependent design changes have not begun. Forced + color output and invalid RAW-name behavior still need contract disposition. + +Private restart evidence under .local/sow-0121-v262-krauk7l5/: + +- recovered-review-red-1.log and recovered-probe-before-current.txt preserve + the before-repair comparison and frozen binaries. +- recovered-review-after-1.log and recovered-probe-after-current.txt locate + the latest comparison; each probe directory contains results.json. +- go-recovered-repairs-2.log, rust-recovered-repairs-1.log and + rust-recovered-build-2.log record the supported-floor validation. +- review-continuations-final-collection.log contains the completed review + collection. All current tests, builds and review workers have finished. + +On resumption, inspect this checkpoint and the preserved git diff first. Add +focused regressions to the existing shared harnesses, resolve the outstanding +contract decisions, then complete proportionate validation and whole-SOW +review. Capture current PR CI before any later push. Keep the SDK completed; +do not close the CLI SOW or goal until all remaining gates are actually met. + +### Resumption After Reboot - 2026-10-19 + +The user explicitly requested resumption. HEAD837873a7 and all recorded local +changes, including the untracked platform helpers, survived the reboot. +Continue the existing approved CLI repair batch and shared regression coverage. +The two outstanding path/glob design questions were presented again with +evidence, alternatives and implications; neither is approved by resumption +alone. No dependent SDK or glob design change begins without that decision. + +### Resumed Regression Coverage And Vacuum Cost Analysis - 2026-10-19 + +Shared regression controls now cover source-realtime duplicates, native integer +grammar and bounds, facility item whitespace, stdout PTY widths, follow/pager +precedence, access time and legacy creation attributes. Frozen pre-repair CLI +binaries reproduce 50 differences in the 594-control output batch; the repaired +commands pass 594/594. Existing query/output/utility controls pass 604/604 and +the expanded follow batch passes 66/66. One new test initially misclassified a +space-prefixed numeric facility as invalid; native accepts it, so the test now +compares terminal success instead of implementation-specific help text. + +Review of the pending atime repair also identifies a missing native fallback: +retry a header open without `O_NOATIME` when the first attempt fails, preserving +`O_NONBLOCK`/`O_NOFOLLOW`. This is corrected in both commands. Go 1.26.2 race tests +and Rust 1.91 unit/subprocess tests and rebuilds pass after that correction. +SDK runtime sources remain unchanged. + +The practical whole-command comparison retains 1000 private archive names as +hardlinks to one synthetic journal, with matching floor binaries before/after, +seven alternating paired samples and no deletions. Go median changes from +17.566636 ms to 24.840289 ms; Rust from 14.978942 ms to 18.756488 ms. Native v262's +three-sample median is 52.503579 ms. The sibling/native comparison does not excuse +the observed regression. Preserve the adverse samples under the directory named +by resume-vacuum-cost-current.txt; do not accept the current unconditional +creation-metadata calls as meeting the performance requirement. + +The repair follows from the existing native comparison contract, without a new +public API or relaxed result: cached filesystem timestamps can be read from +the existing metadata. Additional creation-time I/O is unnecessary when no +retention/size/count constraint can remove a nonempty candidate. When every +candidate has the same sequence ID, native sorting compares sequence numbers +alone; creation time can then affect only a nonzero retention cutoff, and a +cached timestamp already below that cutoff establishes eligibility without +another read. Mixed sequence IDs or unsequenced candidates still need complete +creation metadata when age can determine deletion order. Apply these checks +after collecting candidates, without retaining another metadata array or +adding an allocation. Add native controls for both same-ID sequence ordering +and mixed-ID creation-time ordering, then repeat the existing focused timing +comparison rather than adding a measurement framework. + +Forced-color probing confirms effects beyond priority text: short/cat, +verbose, all JSON framings, grep highlights and boot/invocation headers. The +user was presented a third decision: complete native color behavior within +this SOW, or explicitly qualify and separately track plain-output delivery. +No color implementation begins before that answer. RAW invalid field-name +differences remain outside the existing stock-compatibility promise: +docs/API-Overview.md and docs/Writer-APIs.md explicitly exclude RAW field names +from that promise; do not expand the SDK contract implicitly. + +### Validation Checkpoint And Remaining Decisions - 2026-10-19 + +Creation metadata now loads only when it can affect deletion or age eligibility, +using the existing native sequence-order rule. The new same-ID and mixed-ID +controls prove that the optimization preserves the right survivor. All 76 +utility controls pass; the complete query/output/utility batch passes 608/608. +The unchanged frozen binaries reproduce six differences in the 16 new metadata +controls, including the mixed-ID creation-time ordering case. Supported-floor +Go race and Rust CLI tests/rebuilds pass after the optimization. Shared output +controls pass 594/594 again, both parser batches pass 134/134, portable smoke +passes 28/28, and the Go SDK compiles independently with CGO disabled. + +The existing seven-pair retained-file control now measures Go 18.426862 ms +before versus 17.758796 ms after, and Rust 14.349798 ms versus 14.559356 ms. +The small adverse Rust median overlaps the individual sample ranges; this +does not establish timing equality. Review confirms no additional filesystem +I/O or allocation on that optimized path. Retain both the original adverse +results and optimized-results.json; do not discard the former comparison. + +A separate equal-result control with 1000 recent archive names and +`--vacuum-time=90days` still needs creation metadata for every candidate. It +measures Go 17.653034 ms before versus 24.696577 ms after and Rust 14.837644 ms +versus 18.616836 ms after. recent-retention-results.json retains all seven +alternating pairs. These are warm metadata controls using private hardlinks, +not a general large-archive throughput claim. The remaining cost is real and +is not accepted under the existing no-regression requirement. + +The user was presented decision 4 with the measured costs: continue optimizing +the CLI metadata/header path while preserving identity checks and atime +fallback (recommended under the strict gate), or explicitly accept only this +cold vacuum cost. No further filesystem-path redesign or performance exception +is inferred before the answer. The three preceding decisions remain native +Rust paths, native Unix file globbing and complete native color output versus +explicit qualifications. All four questions remain unanswered at this +checkpoint. Preserve completed independent work and do not start dependent +changes, a fresh whole-SOW review, a push or SOW closure before resolution. + +Validation evidence under .local/sow-0121-v262-krauk7l5/: resume-v262-3.log, +resume-query-2.log, resume-follow-1.log, resume-parser-1.log, resume-go-2.log, +resume-rust-2.log, resume-sdk-pure-1.log, resume-smoke-1.log and +resume-metadata-red.log. Docs and product scope now describe the corrected +source timestamp, follow, terminal width and maintenance semantics. The wiki +structure check passes; executed-example results and the final local audit +are collected separately before reporting the checkpoint. + +All 35 executed documentation examples pass in a new private workspace using +the existing repository caches; no older generated workspace was removed. +All 16 wiki pages validate. Rust/Go formatting, changed-Python syntax and diff +whitespace checks pass. resume-audit-1.log records the complete-and-clean SOW +audit verdict. The native CI result at837873a7 remains historical evidence, +not qualification of these local repairs. No commit, push or new review round +was started while the four decisions remain pending. All commands launched +for this checkpoint have finished. + +### Decision Context Clarification - 2026-10-19 + +The user requested context for the four presented choices. No option was +selected by that request. Distinguish actual technical-boundary approvals from +work already required by the approved contract: + +- Decision 1 asks for an additive native-path Rust facade extension inside the + CLI SOW instead of rejecting valid native filenames. The published SDK scope + and API surface are the approval boundary; no existing string API is removed. +- Decision 2 asks to extend the CLI-only native-call allowance beyond PCRE2 to + platform filename expansion. SDK purity and system journal linkage rules + remain unchanged. The alternative narrows file-pattern compatibility. +- Choice 3A implements the already approved full v262 output contract. Only + choosing 3B would introduce a new plain-output qualification. A separate + approval to repair the existing color parity gap is unnecessary. +- Choice 4A continues work required by the existing performance requirement. + Only choosing 4B would authorize a new vacuum-only cost exception. No such + exception is approved, and the measured slowdown remains an open finding. + +The earlier blanket implementation hold on all four questions was too broad. +Withdraw the self-imposed approval holds on 3A and 4A; they do not represent new +user decisions. Decisions 1 and 2 still await explicit answers before their +dependent API/native-call changes. Keep the original questions and measurements +above as historical evidence; do not report any unanswered option as selected. + +### Approved Path Scope And Vacuum Exception - 2026-10-19 + +After the contextual explanation and recommendation of 1A and 2A, the user +agreed and proposed accepting the small vacuum delta. Proceed with the two +recommended compatibility repairs and record that specific performance +exception. Native color completion remains part of the already approved full +v262 target. No further product decision remains for these four items. + +The accepted exception is explicit CLI age-based vacuum maintenance requiring +creation metadata for recent archive candidates. The retained warm control +measures approximately 7.04 ms Go and 3.78 ms Rust extra for 1000 archive names. +These measurements are evidence for this case, not universal timing bounds. +Keep the already implemented conditional metadata reads. Do not expand the +exception to SDK reads/writes, ordinary CLI queries, or unnecessary I/O in +other maintenance cases. Additional vacuum-path redesign is unnecessary merely +to eliminate this accepted delta. Preserve the original adverse evidence. + +Implementation plan within the existing pre-implementation gate: + +- Add native-path facade constructors for automatic, file, directory and + explicit-file-list opens, including existing options forms. Keep every old + string signature and delegate through the same readers. Use borrowed native + paths, preserve canonical deduplication and avoid allocations/canonicalization + on a single explicit file. Go strings already preserve native Unix bytes. +- Rust journal filename and whole-file Zstandard suffix checks must inspect + native encoded filename bytes. Reuse directory discovery, strict machine-ID + subdirectory checks, staging and cleanup. Wire the CLI to native facade + constructors without lossy string conversions. +- Confine Unix glob calls and thread-local locale selection to CLI modules, + using the already available platform C library and native BRACE/NOCHECK/ + NOSORT behavior. Restore the calling thread's locale and free result storage + on every exit. Preserve a literal-path fast path and existing deduplication. + Keep file expansion separate from unit/text patterns. Windows uses an + explicit Unicode filename contract. No new package or compiler minimum. +- Add shared native-oracle controls for raw parent paths, raw discovered + basenames, replacement-named decoys, braces, bracket/range matching, dotfiles, + escaping, no-match behavior and locale selection. Add focused SDK constructor + coverage for raw filenames, directory/list forms, flags/options and whole-file + compression; reuse existing fixture helpers. +- Document the additive SDK surface, CLI native-call allowance and accepted + vacuum cost. Run supported-floor suites, affected native/shared matrices, + independent pure SDK checks and ordinary native PR CI. Whole-SOW review + remains after complete local implementation and validation. + +### Native Filename Backend Consequence - 2026-10-19 + +The first new facade test fails before row iteration with "invalid filename". +The core path opener constructs a registry File through from_raw_path, whose +path.to_str gate rejects non-UTF8 paths. This is part of the approved native +reader-path blast radius; the facade and suffix changes alone are insufficient. +Evidence: native-path-sdk-1.log; journal-core/src/file/file.rs:568,592 and +journal-registry/src/repository/file.rs:263-285. + +Preserve the approved source-compatible design: retain existing text accessors +and constructors, add a borrowed native as_path accessor, and keep an optional +boxed native path only for newly supported non-UTF8 inputs. Normal paths retain +their current strings and allocation count. Raw paths use an unambiguous +reserved NUL-prefixed, hex-encoded Unix path key; that prefix cannot identify +a valid ordinary filesystem path. Core I/O must use as_path. Legacy software +cannot accidentally open a replacement-named file from the encoded key. +The metadata serializer retains its original fields and normal-path bytes; +deserialization reconstructs native bytes from the reserved key and rejects +malformed native keys. Do not serialize an ignored optional native field. + +Cache disposition: keep CACHE_VERSION at4 only if tests demonstrate unchanged +ordinary-file serialization and Hash input, and exact native-key round trips. +New raw paths were previously unsupported, and their encoded keys cannot alias +an ordinary valid path. No existing index contents or UTF8 cache keys change. +Do not force a broad cache invalidation for this additive path support. The +optional pointer adds one word to per-file metadata, not per-row iterators; +record that small storage consequence and retain the single-file facade's +removed Vec/HashSet/canonicalization work. No new dependency, compiler floor, +row traversal, decompression or allocation inside query iteration is required. + +### Color Implementation Evidence - 2026-10-19 + +Native source inspection confirms that the approved color contract includes +priority/audit text, first-match grep spans, verbose trusted fields and cursors, +JSON primitives, boot separators, and table headers. The implementation reuses +the existing renderers and PCRE2 match, selects color policy once per output +command, and adds formatting only when that policy enables colors. Buffered +reverse output retains spans only for colored grep; plain output retains its +existing entry buffers and does not repeat matching. Native tab/ANSI offset +adjustment and multiline/ellipsization rules are the oracle, including their +observable edge cases. Shared oracle controls cover environment selection, +priority levels, rendering modes, binary values, multiline and buffered grep. +No SDK parsing, indexing, compression, or query algorithm changes are needed. + +### Native Filename And Rendering Validation - 2026-10-19 + +Implementation and blast radius: + +- Additive Rust facade native-path constructors replace the CLI's lossy path + conversions. String constructors retain their signatures and now share the + single-file fast path before canonical-deduplication storage. Native journal + and whole-file Zstandard suffixes survive directory discovery. Core physical + file opens use the new borrowed registry File::as_path accessor. +- Metadata retains its three serialized fields and ordinary-file hash input. + New raw Unix paths use the documented reserved key and reconstruct their + original bytes on deserialization. Engine cache identity, serialized bytes, + map lookup, malformed-key rejection and replacement-name separation pass. + CACHE_VERSION remains 4 for these demonstrated compatibility reasons. +- Unix CLI globbing uses the platform C library, restores its temporary + thread-local locale and frees result storage. Literal paths bypass it. + Windows retains Unicode paths with brace alternatives and wildcard matching. + Raw header output and vacuum filename tie ordering retain original bytes. +- Color policy follows fixed, boolean and automatic modes, both stdout and + stderr terminal state, absent versus empty TERM, NO_COLOR and COLORTERM. + Linux-console headers remain plain. Short/cat priority and audit colors, + verbose timestamp/cursor/field styles, JSON primitives and boot separators + use the existing renderers. Buffered colored grep retains its own spans from + the existing PCRE2 call; it does not repeat the match. Plain matches do not + allocate an output offset buffer or store per-row highlight coordinates. +- Embedded DOCUMENTATION and CONFIG_FILE links use SYSTEMD_URLIFY. Optional + file links read command-local hostname lazily; no SDK code observes host + identity. Native's implicit host-catalog link fallback stays outside portable + mode under the existing parity-matrix catalog rule. The boot-color control + uses synthetic messages without catalog IDs so host databases cannot affect + its expected output. +- New controls also exposed pre-existing verbose continuation indentation and + negative short-delta differences on reverse output. Both implementations now + use native multiline indentation and saturating timestamp subtraction. + Cat projection order is normalized only for a one-row, single-line control: + native output-fields is an unordered set. Other cat controls compare complete + raw bytes, including multiline, NUL and repeated MESSAGE values. + +Reference evidence: systemd/systemd @ +8cc40e0c5e9234bf45084751ac53b1fbfe70b492; +src/basic/ansi-color.c, src/basic/terminal-util.c:1453,1844, +src/basic/string-util.c:704, src/shared/logs-show.c:208,533,790,1361, +src/shared/pretty-print.c:59,98,155 and +src/libsystemd/sd-journal/journal-vacuum.c:35. + +Supported-floor and executable evidence, retained under the private CLI +validation workspace: + +- Go 1.26.2 CLI race suite passes; Rust 1.91 CLI passes 23 unit and 11 + subprocess tests. Native-glob brace/Unicode/dotfile controls are included. +- Rust SDK passes all 201 library tests; registry passes 13 tests plus its + documentation test. The five engine cache tests pass, including both new + compatibility controls. Go's complete SDK package compilation passes with + CGO disabled independently from the CLI build. +- Expanded v262 oracle controls pass all 906 observations and cover native filenames, raw header output, + locale/glob rules, every renderer, colors, embedded links, grep spans, + terminal policy and the previous parser/query controls. A later retained + run replaces the over-strict unordered cat-projection expectation; this is + a test correction, not a changed product ordering guarantee. +- Existing query/output/utility controls pass 610 observations, including raw + vacuum tie ordering. Follow passes 66. Parser parity passes 134 cases per + language. Native portable smoke now passes 46 controls and is wired through + the existing four-OS workflow. This local smoke run does not certify a new + remote native build. +- All 35 executable documentation examples pass. All 16 wiki pages validate. + Changed Go/Rust formatting and Python syntax checks pass. Remaining final + audit/review/CI evidence is recorded after its actual completion. + +Artifact maintenance: AGENTS records the approved CLI native-call exception; +the journal compatibility runtime skill records filename/color controls and +plain-output discipline; product scope and the CLI parity matrix record the +current behavior and cache disposition. Journalctl and Reader APIs wiki pages +document native filenames, additive constructors, low-level path access, +colors/links and the accepted vacuum-only cost. No output/reference skill +exists or needs a new artifact. Both SOW status indexes reflect this state. +The original SDK SOW remains completed; this explicitly approved additive +native-path blast radius belongs to the current CLI SOW. + +Same-failure search covers facade opens, directory and compression suffixes, +core physical file operations, native header output, vacuum tie ordering, +streamed and buffered highlights, short-family variants, cat, verbose, all +JSON framings, tables and separators. Existing log-writer repository paths use +the strict UTF8 registry constructors and retain that contract. No SDK row +traversal, index population, journal bytes, compiler minimum or dependency +version changes are mixed into this batch. Added durable evidence uses +synthetic data and relative references; raw command output remains private. + +Lessons: filename bytes must survive the complete opener stack; stable cache +serialization and hash identity are separate compatibility checks; test native +terminal policy with both descriptors; retain match offsets with buffered +entries; distinguish unordered projection order from value correctness; and +keep optional host presentation helpers outside SDK runtime code. + +Follow-up mapping is unchanged: indexed invocation ranges and broad polling +scalability remain SOW-0122. Reopened SOW-0045/0071 and pending +SOW-0153/0156/0160/0161 remain separate. These repairs are implemented here; +no new deferred implementation item or product decision is created. + +### Final Portability And Plain-Output Audit - 2026-10-19 + +The primary audit identified three remaining consequences inside the approved +filename/rendering scope before final review: + +- Windows Go file selection used filepath.Glob, while Rust's existing Unicode + matcher accepts native-style negated classes and literal unmatched brackets. + Reuse the existing matching grammar through a small component walker for Go, + with the same brace and dotfile behavior. Unix still uses libc glob. Preserve + drive/root prefixes and literal-path bypasses; add shared native smoke cases + and focused fallback tests instead of a new glob dependency. +- Rust's optional file-link hostname used the Windows NetBIOS accessor, while + Go 1.26.2 os.Hostname uses ComputerNamePhysicalDnsHostname. Match that Windows + accessor and its grow-buffer protocol. Relative links must also preserve a + valid logical PWD, matching the native get_current_dir_name path and Go's + existing Getwd behavior. Verify with an isolated symlink cwd control. This + remains CLI-only optional presentation logic. +- Inspection of the supported-floor development binary found a separate call + to append_cat_field on the plain Rust cat path. Keep the original byte copy + inline and the plain Pattern::matches body unchanged. Add the offset getter + only after successful colored matches; avoid unconditional highlight stores. + The production-profile comparison uses the retained matching compiler and + profile. Preserve development timing outliers; they are not a production + equality claim and do not authorize a query slowdown. + +These changes implement the approved contract and remove inconsistent or +unnecessary work. They do not introduce a new product decision, dependency, +compiler floor, SDK host-observation path or measurement framework. + +### Final Local Performance And Readiness Disposition - 2026-10-19 + +The final primary validation pass is complete. Supported-floor CLI suites pass, +the latest expanded oracle matrix passes 906/906, and portable smoke passes +46/46 including the Windows fallback's negated classes and literal unmatched +brackets. The logical-PWD link control also passes against native journalctl. +The earlier 610 query/output/utility, 66 follow and 134 parser cases per +language remain valid; their query, follow and parser algorithms did not change +in the final presentation/fallback refinements. All 35 documented examples and +16 wiki pages pass. Rust SDK/registry/cache and independent pure Go SDK evidence +remain as recorded above. + +Performance evidence follows the user's engineering-judgment rule. Reuse the +existing 100000-entry fixture and small alternating whole-command comparisons; +do not create a new benchmark framework. Both compiler-matched production Rust +binaries use Rust 1.97.1, release opt3, fat LTO and one codegen unit. Identical +output hashes cover both full scans and a 75000-result grep. Median scan time +is 318.0ms before / 314.5ms after; grep is 304.7ms / 309.1ms. Sample ranges +overlap and paired results vary in both directions. Preserve that small +positive grep median and all development-profile outliers; these intervals do +not establish timing equality or a reproducible slowdown. Go's matched-floor +grep control has median wall time 674.5ms / 660.8ms and child CPU 964.2ms / +906.0ms. These controls are bounded observations, not universal guarantees. + +The engineering disposition rests on tracing the work as well as these +controls: SDK row/index/decompression paths do not change; native path metadata +adds no per-row allocation; ordinary literal opens avoid extra facade +canonicalization/storage; color policy is selected once; plain Rust cat keeps +its byte copy inline; plain regex matching retains its original body; Go's +uncaptured matches allocate no offset buffer; and buffered highlight storage is +enabled only for colored/linked grep. No second regex call, extra row scan, +DATA expansion or filesystem operation is introduced into ordinary query +iteration. No query slowdown allowance is granted. The accepted explicit +startup and age-based vacuum allowances retain their original narrow scope. +No further measurement work is needed merely to force noisy medians below 1.0. + +Same-failure, artifact, source-boundary and synthetic-data checks are complete. +Final SOW audit, whole-SOW review under standing authorization, ordinary PR CI +and coherent signed lifecycle closure remain. Local readiness does not claim +that the older remote native results certify the new helpers. + +Final local SOW audit reports "SOW initialization complete and clean" in +native-colors-audit-final-1.log. Whitespace, changed-language formatting and +Python syntax checks pass. The remaining follow-up words refer to implemented +work, superseded historical checkpoints, or the existing real SOW files mapped +above. No new unresolved product decision remains. Proceed to the authorized +whole-SOW static review; fresh native PR CI and lifecycle closure follow it. + +### Whole-SOW Collection And Native Qualification - 2026-10-19 + +The 97-file candidate matched the frozen review snapshot and was signed/pushed +as 1f84cca1 for ordinary draft PR #12 qualification. This changed no reviewed +file contents. Native run 39263286705 passes Linux and Windows; macOS and +FreeBSD both fail Rust compilation at file_glob.rs:64 because libc 0.2.189 +does not export GLOB_BRACE for those targets. Their Go command tests pass. +The earlier four-platform success at 837873a7 does not qualify this candidate. + +Review round 0ee095a6bd15499ab664a7633a7c140d is collected: + +- MiniMax returns PRODUCTION GRADE. Its five P3 notes cover fresh native CI, + review coverage, the existing internal analysis-tool error, accepted startup + cost and approved CLI-only host presentation calls. These are existing gate + tasks or documented allowances, not additional code requirements. Its + assertion that the round has zero usable verdicts is stale; the collected + MiniMax and DeepSeek reports are usable technical verdicts. +- DeepSeek returns NEEDS CHANGES. The macOS/FreeBSD binding omission is + independently reproduced by native CI. Its no-match unit-glob finding is + independently reproduced in both commands against the pinned v262 oracle. + Both are required repairs within this SOW's approved contract. +- DeepSeek's Rust forward-count finding is rejected: render_matching_entry + ignores render_entry's presentation boolean and returns true after a + grep-selected entry. Frozen binaries match stock for excluded +2 head rows + and a +1 head entry without MESSAGE, including output and exit status. + Neither a counter change nor a speculative counting repair is justified. +- GLM times out twice without a verdict. Mimo, Kimi and Qwen each return + HTTP429 on the original invocation and its single retry. These four gaps + supply no verdict. No further retry of this unchanged candidate is allowed. + +The unit-glob mismatch predates this repair program: SDK base 25accee contains +both addImpossibleMatch/add_impossible_match helpers and their unit-glob +callers. With `--unit=no-such-*` or `--user-unit=no-such-*`, stock exits 1 with +"Failed to add filter for units: No data available" and no output; both CLIs +exit 0 and print the no-entries marker. An absent exact unit correctly exits +0 and must retain that behavior. The matchable sentinel field is unnecessary. + +Quality run 39263286606 reports two Bandit_B108 findings at the color fixture's +literal MESSAGE/CONFIG_FILE values. These are synthetic log data, not temporary +file operations. Actual fixture paths use private repository scratch. Replace +the two payload literals with the standard /var/log/journal-sdk example prefix +without weakening the absolute file-link assertion. The separate pre-existing +internal tool error remains a coverage limitation; do not infer complete +analysis from zero reported findings or expand this PR into tool maintenance. +Raw reports and executable reproduction results remain under .local/. + +### Pre-Implementation Repair Gate - 2026-10-19 + +Problem and evidence: the Unix adapter assumed every libc binding exported a +non-POSIX flag; the unit filter used a synthetic match for an empty wildcard +expansion instead of native ENODATA behavior. Review current adapter/filter +callers, pinned v262 journalctl-filter.c:105-198, the locked dependency's target +modules, native CI logs and frozen-binary oracle reproductions. The earlier +primary audit omitted native binding qualification and empty unit expansions; +the internal audit must cover those surfaces before another whole-SOW review. + +Platform reference evidence: + +- apple-oss-distributions/Libc @ 71bbe350ab79eef58113991d817ccc6165061a64, + include/glob.h:104 defines GLOB_BRACE as 0x0080. +- freebsd/freebsd-src @ f19009020a33d07d7cad9393b4bf0a36f267cd07, + include/glob.h:94 defines the same BSD flag value. +- systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492, + src/journal/journalctl-filter.c:195 returns ENODATA when neither exact units + nor wildcard expansion added a unit. + +Affected surfaces and existing patterns: Rust CLI Unix glob flags, both CLI +unit-filter dispatchers and the shared oracle/portable smoke controls. Reuse +the approved native libc glob/locale path, ordinary command error handling, +FIELD/DATA unique enumeration and existing shared fixture writers. The SDK, +journal bytes, cache representation and compiler floors do not change. + +Risk and implementation: supply the BSD flag only for macOS/FreeBSD and retain +the GNU Linux binding. Remove the two sentinel helpers and return the native +no-data diagnostic only when neither system nor user unit expansion adds a +match. Preserve exact-unit and mixed exact/glob behavior. Replace only the two +synthetic color-fixture literals. No new dependency or runtime algorithm is +needed. Other Linux C-library profiles are not qualified by the four native +jobs; this repair does not claim new packaging coverage for them. + +Validation: supported-floor CLI suites; shared oracle cases for system/user +empty wildcards, combined empty wildcards, zero-line selection, exact absent +units, mixed exact/glob selection and a real sentinel-valued field; ordinary +portable smoke negatives; existing query/follow/parser controls; fresh native +Linux/macOS/Windows/FreeBSD CI and quality output. Compare error status, class +and absence of row output. No extra benchmark is warranted: the flag is a +compile-time constant and no-match unit selection exits earlier; successful +unit enumeration and query iteration retain the same work. + +Artifact and data plan: update the CLI parity/spec record, both status indexes +and this SOW with actual validation/review outcomes. Existing runtime-skill +rules already require native OS qualification; no new workflow or consumer API +is introduced. Use synthetic fields/paths only and retain raw logs privately. +Open decisions: none; these repairs deliver the approved native platform and +stock CLI behavior. Closure remains blocked until validation and review pass. + +### Native And Unit-Filter Repair Checkpoint - 2026-10-19 + +The BSD constant adapter and paired native no-data error are implemented. +The sentinel helpers have no remaining runtime callers and are removed; +the sentinel name remains only in the synthetic regression fixture. Exact +unit selectors and successful wildcard expansion retain their indexed path. +The two absolute link fixture payloads use the project example prefix. + +New shared controls reproduce eight failures on frozen 1f84cca1 binaries: +system/user/both empty wildcard selections and the zero-line case, in both +languages. All 924 comparisons pass on the repaired binaries, including the +ten exact/mixed-selector controls. Error comparisons also retain complete +stdout. All 52 portable smoke observations pass; the printed total was +updated with the three new cases per language. Supported-floor Go CLI race +tests and Rust CLI's 23 unit/11 subprocess tests pass. Parser parity passes +134 cases per language, query/output/utility passes 610, and all 16 wiki +pages validate. The timed follow batch is still running at this checkpoint. + +The source/performance audit finds no new successful-query work: no SDK +reader/index/decompression change, no additional unit-value enumeration, no +extra regex evaluation and no row-path allocation. GNU Linux uses the same +constant; BSD uses its header's compile-time constant. Empty wildcard queries +return before the former synthetic filter/query path. No new timing exception +or measurement framework is warranted. Existing performance evidence and the +user's narrow startup/vacuum allowances retain their scope. + +Same-failure inspection covers both system/user selectors, combined selectors, +exact absent names, zero limits, the sentinel field, both Unix binding targets +and both command error handlers. Metadata/spec/docs/index updates match the +implemented behavior. No compiler, dependency, SDK API, cache or journal-byte +change is added by this repair. Sensitive-data checks cover only synthetic +fixtures and sanitized summaries. Native CI and fresh whole-SOW review remain +gates; no completion is claimed from local success alone. + +Repair pre-push audit reports "SOW initialization complete and clean" in +native-platform-fix-audit.log. Whitespace and changed-language syntax/format +checks pass. The PR's prior three failed checks and two open analysis threads +were captured before pushing; both native failures share the repaired binding +omission and both analysis findings concern the replaced fixture literals. + +### Follow Collection And Focused Rechecks - 2026-10-19 + +The repeated follow batch records 62 passing observations and four failures. +All four failures originate in the synthetic live writer's readiness/completion +deadlines: Go boot-event output is exactly expected but the writer is stopped +at its deadline; stock sees only the first append before the writer deadline; +and the Rust live-append and Go cursor cases report writer-not-ready. A +read-only inspection of the task's exact log-owning process finds its current +fixture-writer child waiting for a filesystem log commit. This is evidence of +fixture I/O delay, not proof of a host-wide cause or a CLI regression. + +Retain the full failed batch. Recheck only its four failed reader/case pairs, +using the existing case table and run_follow_reader helper in fresh private +directories. All four pass without source, binary, timeout or expectation +changes, including stock's complete three-append output. The evidence is +62 initial passes plus four focused passes; do not mislabel the original run +as 66/66. No test timeout is relaxed and no broad rerun is needed. + +Both new analysis threads are answered individually and resolved after signed +repair commit 833e635a. The normal native run is 39267869667. Its outcomes and +the fresh whole-SOW review are recorded after collection, before closure. + +### Native Qualification And Final Primary Audit - 2026-10-19 + +Ordinary PR native run 39267869667 succeeds on signed source 833e635a: +Linux/glibc, macOS ARM64, Windows CLANG64/GNU LLVM and FreeBSD 14.4 all build +both commands and pass the 52 portable smoke observations. Pure Go SDK builds +remain independent. This qualifies the repaired BSD flag on both affected +platforms; no SDK compiler minimum changes. Other Linux C-library profiles are +outside this native qualification and no new packaging profile is claimed. +The earlier decision excluded developing separate static native profiles. + +Quality run 39267869602 confirms zero Bandit findings after the fixture-payload +change, while retaining the known internal tool error. Its sole current +finding is Markdown MD037 on the newly written wildcard example in this SOW. +The example is corrected to inline code. The final metadata push must obtain +the actual green quality result and clear the stale prior alert instances; +the current red check is not represented as passed. + +The primary audit and executable validation are complete for the repaired +runtime: 924 oracle comparisons; 610 query/output/utility controls; 52 native +smoke observations per OS; 134 parser cases per language; supported-floor Go +race and Rust unit/subprocess suites; 62 first-run follow passes plus four +focused passes, with original fixture-writer timeouts preserved. Prior SDK, +registry/cache, pure-Go and 35 executable-documentation evidence remains +applicable because those paths/examples did not change in this repair. All +16 wiki pages and changed-language syntax/format checks pass. + +The whole changed source and direct/indirect consumers were re-inspected after +the verified findings, including successful/empty/mixed unit paths, cursor and +zero-limit error timing, sentinel removal, Unix binding flags, platform build +separation, rendering, indexed access and error cleanup. No additional +blocking code defect was established. Complete this meaningful batch with a +fresh whole-SOW review under standing authorization. Keep the source frozen +during collection; do not repeat reviews merely for individual metadata edits. + +The final primary audit again reports "SOW initialization complete and clean" +in native-platform-fix-audit-final.log. No unresolved product decision remains. +The long-term-best ship recommendation remains to retain both commands under +the shared parity suite. Both now have four-OS native evidence; the historical +Go-first recommendation's broader-platform-evidence rationale is superseded. +This does not select a new packaging default or authorize a release. Existing +large-archive/polling work stays SOW-0122; the mapped paused/pending SDK work +remains separate. No additional implementation is deferred from this repair. + +### Whole-Unit And Invocation Audit - 2026-10-19 + +The repaired candidate 2b689a98 has clean ordinary PR checks, all four native +jobs and zero open PR analysis alerts. Both fixture-string threads were answered +and resolved individually. The quality tool's pre-existing internal error +remains a coverage limitation despite its green workflow and zero findings. + +The next whole-SOW review round is collected: MiniMax returns PRODUCTION GRADE; +DeepSeek returns NEEDS CHANGES; GLM reaches its 1800-second limit without a +verdict; Mimo, Kimi and Qwen each return HTTP429 twice. Do not retry this known +incomplete candidate. MiniMax's old remote-quality and prior-review-count notes +are stale. Its approved CLI-helper note is not a defect. DeepSeek's diagnostic +prefix note is nonblocking under the existing error-class comparison contract. +The unqualified musl/uClibc profile remains outside the accepted native matrix. + +DeepSeek's two P2 findings reproduce in both commands against the pinned v262 +oracle: suffixless/partial-suffix unit patterns are incorrectly given a service +suffix, and invocation unit discovery searches the full show-only field set. +The preceding primary audit was too narrow around units. Stop the review cycle +and audit the complete component before another review. + +Confirmed direct/indirect failures on frozen binaries: + +- `--unit=app-*` misses the fixture's slice, and `--unit=alpha.*` misses its + service. POSIX character-class patterns also fail. Exact empty selectors + incorrectly succeed. Native path, escaped and raw-byte unit arguments fail + to select their corresponding canonical unit names. +- Current-user conversion does not apply native duplicate removal when + system-unit selectors are moved to the user-unit list. A duplicate selector + therefore incorrectly makes invocation context appear to contain two units. + The separate system/user flag combination was checked and is not a new + last-setter repair: native combines those journal-type bits. +- Both wildcard and exact invocation unit selection include coredump/slice + groups that native invocation discovery excludes. Normal show filtering must + retain those full groups. +- Invocation first/last timestamps are restricted to the unit-discovery rows; + native bounds include all boot-scoped rows carrying that invocation ID. + A three-row fixture has native bounds T..T+2 but both CLIs report T+1..T+1. +- A row containing two invocation-ID fields creates an extra table entry. + Native chooses the first valid nonzero ID in its documented field order. + Explicit ID selection without an offset must still match any ID field. +- Invocation show queries ignore identifier and other non-unit filters. + Explicit-ID boot validation was checked separately and already agrees with + native; do not add a speculative boot-validation repair. +- Relative invocation follow changes its selected ID after another invocation + is appended. Native retains the initially resolved ID. The private live + control reproduces this in Rust with a normally completed writer and in Go + alongside a separately recorded writer deadline. + +These are CLI correctness gaps inside the approved v262 contract. They do not +reopen SDK SOW-0159. Reference: systemd/systemd @ +8cc40e0c5e9234bf45084751ac53b1fbfe70b492; +src/basic/unit-name.c:676-826, src/basic/path-util.c:355-429, +src/journal/journalctl-util.c:131-239, +src/journal/journalctl-util.h:9-27, +src/journal/journalctl-filter.c:429-481 and +src/shared/logs-show.c:1733-1819,1948-2363. +Raw oracle outputs and synthetic fixtures remain under .local/. + +### Pre-Implementation Gate - Unit And Invocation Repair - 2026-10-19 + +Problem/root cause: simplified name conversion, shared full unit fields/matches, +global unique-ID enumeration plus separately unit-restricted bounds, and +repeated invocation resolution do not implement native unit/invocation state. +The SDK's indexed matching, metadata getters and cursor seeks are available; +replacing them with row expansion or an unbounded journal scan is prohibited. + +Evidence and existing patterns: inspect the complete native routines cited +above, both CLI callers, SDK cursor/reset behavior, current shared query/follow +builders and frozen oracle reproductions. Reuse the approved CLI-native +globbing/locale pattern on Unix and the Unicode fallback on Windows. Reuse +existing SipHash primitives for long native unit path names where accessible; +keep any necessary Go helper private to the CLI. SDK public APIs need no new +capability or native dependency. + +Implementation plan: + +1. Normalize unit arguments once per command with native suffix/glob, byte + escaping, path conversion, empty/error and quiet-warning behavior. Preserve + raw Unix argument bytes and native user-list duplicate semantics. Cache + only command-local immutable selection results; do not change parser + terminal-action precedence or SDK APIs. +2. Match unit wildcards with native NOESCAPE behavior and proper character + classes; compile/prepare once outside unique-value callbacks. Enumerate + FIELD/DATA indexes, truncate C-string values where native does, and retain + literal fast paths. +3. Separate full show unit groups from invocation's base field/group set. + Keep the native current-UID rule. Apply non-unit show filters regardless + of whether invocation selection replaced boot/unit show filters. +4. Replace the unique-ID/sort/bounds approximation with native-direction + invocation discovery and indexed jumps. Use first-valid ID precedence, + boot-scoped global ID bounds, explicit-reference offsets, duplicate/broken + ID termination and bounded selection. Reuse readers and metadata getters; + use a private Go CLI adapter for existing facade function calls rather + than adding a public SDK API. Account explicitly for SDK match resets and + cursor positioning instead of assuming native internal cursor state. +5. Preserve performance on sparse-ID journals by deriving an indexed valid-ID + eligibility predicate from FIELD/DATA values before discovery. Do not scan + unrelated rows or expand MESSAGE fields. Resolve the invocation once for + follow, matching native state. + +Risk/blast radius: both unit show paths, invocation tables/relative selectors, +follow selection, unit-pattern helpers and shared regression fixtures. SDK +source, journal bytes, recovery policy, cache schema, compiler floors and +dependency versions remain unchanged. Rust may promote its existing +journal-core test dependency to a runtime dependency to reuse the already +linked native-name hash primitive; this introduces no new package/version. + +Validation plan: shared native-oracle fixtures for every confirmed failure; +unit naming boundaries, raw/Unicode names, long paths, glob classes and mixed +selectors; full/base groups; invocation ID precedence, overlap, bounds, +explicit/relative references and non-unit filters; a focused invocation-follow +control; supported-floor suites and existing full parser/query/follow controls. +Use representative whole-command timing and work tracing for changed query +work, including sparse-ID and many-ID inputs. No large measurement framework +or broader performance envelope is needed. Repeat complete primary validation +and whole-SOW review only after the cohesive repair is ready. + +Artifact/sensitive-data plan: update the parity/product specs, CLI docs, runtime +compatibility guidance and both status indexes. Use synthetic identities and +metadata; unit path conversion performs no host filesystem discovery. Raw logs +stay private. Reconcile SOW-0122's stale invocation description before coding: +required correctness and no-regression work remains here; large-input envelope +and broad polling work stay pending, without activating another SOW. +Open product decisions: none; native behavior and the performance/purity +constraints are already approved. This gate does not authorize unrelated +features, packaging profiles or release work. + +### Unit/Invocation Implementation Evidence - 2026-10-19 + +The first 290 new oracle observations reproduce 176 failures in the retained +2b689a98 binaries. The first repair reduces these to ten; the remaining failures +identify the native long-name hash's trailing NUL and broken-ID cursor restore. +After those repairs and expanded broken-ID/raw-field controls, all 348 +observations pass. Keep both adverse result sets. Additional naming boundaries, +fallback matcher and live controls are included in final validation below. + +The complete native traversal uses forward discovery for unlimited tables, +forward for oldest limits, and backward for newest limits. Bounds use the +boot-scoped ID predicate without unit restrictions. Broken-ID recovery restores +the reference boundary and temporarily clears unit/boot discovery matches, +matching the upstream routine, with local broken-ID and global duplicate guards. +The indexed eligibility predicate remains active to skip rows without valid ID +fields. Two readers are reused; metadata getters never expand MESSAGE data. +Sparse/no-ID inputs therefore do not become full row-expansion scans. + +The live audit also proves ordinary unit wildcard follow must retain its initial +expansion. The pinned v262 reader outputs initial-alpha/late-alpha; both +provisional commands additionally output new-beta. Expanded unit matches are +now cached once per command, alongside the once-resolved invocation ID. +This corrects behavior and avoids repeated FIELD-chain expansion during polling. +The original private follow probe accidentally selected the host oracle through +an old command-builder helper; exclude it from acceptance evidence. The three +shared command builders now honor an explicitly supplied stock_journalctl. +The repeated pinned probe confirms the wildcard failure and retains its +independent fixture-writer timeouts. No deadline is increased. + +Unix unit wildcard matching uses native fnmatch with NOESCAPE in a temporary +thread locale, prepared once per expansion. Both commands preserve raw selected +unit values. Windows uses a once-compiled byte matcher with POSIX character +classes, C-locale single-character collating elements, literal brackets and +literal backslashes; dedicated tests exercise this fallback on Linux too. +Normal unit names are mangled to ASCII; raw payload bytes retain byte matching. +Rust reuses the existing core SipHash primitive, including the native trailing +NUL; Go's small CLI-private primitive introduces no SDK API or dependency. + +Same-pattern audit additionally covers empty selectors, suffix-only and invalid +prefix names, oversized names, path simplification/hash boundaries, user-list +deduplication, ID precedence, overlapping/nested ranges, explicit secondary-ID +offsets, retained non-unit filters and follow state. The already disproved +line-count and system/user last-setter reports remain rejected. SDK source and +format contracts are unchanged. SOW-0122 is reconciled and remains pending. + +### Final Unit/Invocation Validation - 2026-10-20 + +The retained candidate `unit-repair-6/bin` passes 1,364/1,364 pinned v262 +comparisons, 610/610 existing query/output/utility controls, 134 parser cases +per language and 52/52 portable smoke controls. Supported-floor Go CLI race +tests and Rust CLI tests (24 unit and 11 subprocess cases) pass. Evidence under +`.local/sow-0121-v262-krauk7l5/`: `unit-repair-final-6-v262.json`, +`unit-repair-final-6-query.log`, `unit-repair-final-6-parser.log`, +`unit-repair-final-6-smoke.log`, `unit-repair-final-go-race.log` and +`unit-repair-final-rust-tests.log`. The final zero-line invocation-table shortcut +validates source/unit/boot before returning; it skips ID enumeration and reader +setup that cannot contribute any requested output. Ordinary queries without +unit/invocation selectors retain immediate fast exits. + +The full follow batch has 70 initial passes and two fixture-writer readiness +timeouts before either CLI starts. Both unchanged focused rechecks pass; no +deadline or expectation was changed. Preserve `unit-repair-follow-first.json`, +`unit-repair-follow-full.log`, and `unit-repair-follow-recheck.json`. The six +new pinned-oracle frozen-invocation/frozen-unit-glob observations pass in +`unit-repair-follow-after.json`. Subsequent changes affect raw warning output, +empty-selector/no-ID shortcuts and zero-line tables, not the exercised live +selection/cursor paths; the full follow batch is not repeated for those edits. + +Pure Go SDK checks pass (`unit-repair-pure-sdk.log`). All 35 executable docs +examples pass (`unit-repair-docs.log`), and all 16 wiki pages validate +(`unit-repair-wiki-final.log`). SDK code and contracts are unchanged by this +repair. Earlier native CI qualifies the preceding source only; ordinary PR +native CI must qualify this source before closure. Fresh whole-SOW review is +also pending; previous provider failures do not count as technical verdicts. + +Performance evidence uses existing whole-command controls, equivalent outputs, +supported-floor matched build profiles and five alternating pairs. The retained +`invocation-cost-m3v_wqu3/results.json` records baseline/candidate medians: + +| Control | Go baseline -> candidate | Rust baseline -> candidate | Stock v262 | +| --- | --- | --- | --- | +| 64 IDs / 4,096 rows, full table | 56.65 -> 10.50 ms | 74.69 -> 19.47 ms | 4.34 ms | +| Same fixture, oldest one | 61.65 -> 7.76 ms | 69.24 -> 4.68 ms | 2.79 ms | +| Two sparse IDs / 20,000 rows | 11.97 -> 7.49 ms | 13.02 -> 8.44 ms | 28.75 ms | +| No IDs / 20,000 rows, initial | 5.81 -> 6.47 ms | 4.66 -> 4.49 ms | 32.82 ms | + +The initial Go no-ID +0.67 ms observation is retained and investigated, not +waived. The absence shortcut avoids a second reader and predicates; syscall +counts fall from 12 to 8 pread64 calls and 13 to 9 fstat calls. Nine alternating +focused pairs yield 5.633 -> 5.658 ms with mixed directions; the later final +comparison yields 7.355 -> 6.510 ms (CPU 7.070 -> 6.186 ms). The small initial +increase is not reproducible. Evidence: `unit-repair-no-id-focused.json`, +`unit-repair-no-id-final.json`, and `unit-repair-no-id-{before,after}.strace`. +This is engineering judgment supported by work tracing and representative +measurements, not a claim that overlapping intervals prove exact equality. +The captured ru_maxrss includes the Python parent's fork peak and is excluded +from application memory/allocation claims. Ownership review instead confirms +two reused readers, metadata-only access, indexed eligibility, prepared matchers +and once-only follow expansion. The zero-line shortcut only removes work from +the measured implementation; no further timing framework is required. + +Artifact maintenance: both status indexes, parity/product specs, CLI docs, +compatibility skill and pending SOW-0122 are updated. AGENTS.md needs no change: +the approved scope, purity and evidence policies already cover this repair. +There are no output/operator skills to update. Sensitive-data checks use +synthetic fixtures and sanitized durable evidence; detailed logs remain private. +Same-failure coverage includes naming boundaries, raw bytes, fallback matching, +ID precedence, global bounds, broken IDs, equal timestamps, offsets, non-unit +filters and frozen follow selection. Broader performance-envelope work remains +explicitly tracked by pending SOW-0122; no new implementation SOW is activated. + +### Native CI And Final Review Progress - 2026-10-20 + +Signed candidate 587981b7ac5925f6f29946d8ccd416a16667b3ae is pushed to draft +PR 12. Native run 39401925131 passes Linux, macOS, Windows and FreeBSD. +Coverage test generation/upload, documentation examples and CodeQL complete. +All 16 existing review threads remain resolved. The full CI snapshot is retained +as `unit-repair-ci-completed-587981b.json` in the private validation workspace. +Codacy's sole Markdown finding is an extra blank line before the final evidence +section; the local correction removes only that line. The pre-existing Bandit +internal error remains a separate analysis coverage limitation. + +Codacy Coverage Variation reports -1.16 percentage points against a -1.0 limit. +The ordinary coverage scripts run language tests, not the external oracle +comparisons (`tests/coverage/run_go_coverage.sh:20` and +`tests/coverage/run_rust_coverage.sh:23`). The retained Rust report shows zero +hits on 319 invocation-module lines despite the separate executable oracle +controls. This is an instrumentation gap; behavioral test success does not +make the coverage check green. A user decision is requested: count representative +existing CLI controls in both coverage reports (recommended long-term-best, +without a new framework), or explicitly accept/document this PR's coverage +exception. No coverage-script implementation or exception is assumed before +that answer. Reports are retained under `unit-repair-ci-coverage-{go,rust}/`. + +Whole-SOW review session 38c2b9beccf04384a5e70a58fae8036e uses the fully displayed +prompt `whole-sow-unit-review-20261020.txt` and the 108-file snapshot +`unit-repair-review-snapshot.json`. GLM, MiniMax, Qwen and DeepSeek are still +reviewing. Mimo and Kimi returned HTTP 429 without a verdict, including their +single permitted retries; these are coverage gaps, not negative technical +verdicts. Source files remain identical to the snapshot; subsequent local +changes are this progress record and the Markdown blank-line correction. +The SOW remains in-progress until review disposition, the coverage decision +and coherent lifecycle/commit closure are complete. + +### Review Timeout Collection - 2026-10-20 + +Session 38c2b9beccf04384a5e70a58fae8036e turn 1 ends without a final technical +verdict from GLM, MiniMax, Qwen or DeepSeek: each reaches its 1,800-second +execution limit with zero final-output bytes. These are incomplete reviews, +not approvals or verified defect reports. The collection is retained in +`unit-review-final-collection.log`. Each receives its single permitted retry +as a continuation of the saved inspection, with unchanged whole-SOW scope and +the fully displayed prompt `whole-sow-unit-review-retry-20261020.txt`. +Mimo and Kimi have exhausted their retries and are not invoked again. +The coverage choice remains unanswered; no exception or coverage-script change +is authorized by elapsed time or review completion notifications. + +### Final Review Disposition - 2026-10-20 + +Session 38c2b9beccf04384a5e70a58fae8036e completes its permitted continuations. +MiniMax returns PRODUCTION GRADE after 458.4 seconds with three P3 notes and +no blocking finding. GLM, Qwen and DeepSeek each reach their second 1,800-second +limit without final output. Mimo and Kimi have already exhausted their HTTP 429 +retries. The round therefore supplies one technical verdict and five coverage +gaps, not six approvals. No further automatic retry is permitted. Retained +collection: `unit-review-retry-final-collection.log`. + +Independent adjudication: + +- Optional-argument concern: disproved. The reviewer inspected an older mirror + rather than the supplied pinned source. systemd/systemd @ + 8cc40e0c5e9234bf45084751ac53b1fbfe70b492, + `src/journal/journalctl.c:512-539,620-628,673-686` explicitly peeks/consumes a + parseable separate boot or line argument, while case-sensitive uses only its + attached optional argument. Both parsers follow that rule. Seven executable + comparisons against the pinned v262 binary cover separate boot 0/1, a short + option bundle, short/long line limits and bare case-sensitive forms; all Go + and Rust exit codes, output and diagnostic-presence classes match. Evidence: + `unit-review-optional-argv-check.json`. No code change is warranted. +- Rust introspection data: accepted existing monorepo dependency, no current + defect. `parser.rs:378` embeds the shared Go-owned JSON; Cargo.toml declares + `publish = false`. Standalone CLI source packaging is outside the approved + outcome. A speculative packaging redesign is rejected from this SOW rather + than creating unsolicited work. +- Coverage variation: confirmed policy decision remains open. Language-test + instrumentation includes the CLI but does not include external oracle runs. + Thus its unit-test-only percentage is valid for what it measures; it does + not describe all executed behavioral validation. The reviewer cannot grant + the requested user exception or authorize coverage changes. + +The supplied pinned upstream source is available locally; the reviewer's +statement that upstream is unavailable applies only to its chosen older mirror, +not this project's reference. Its suggestion to prune private historical logs +is unrelated and rejected from scope: retained adverse evidence is useful and +file deletion is not authorized. No implementation finding remains from the +returned review. Coverage disposition and coherent final SOW closure remain. + +### Coverage Decision And Pre-Implementation Gate - 2026-10-21 + +The user selects 1A: count representative existing CLI controls in both coverage +reports, without introducing a new test framework. The coverage exception is +not accepted. Root cause: the reports measure language tests but omit external +CLI subprocess controls. Evidence: both coverage scripts, CI reports and the +existing portable smoke and unit/invocation fixture tables. + +Plan: reuse the portable smoke runner with one or both language binaries; +replay a small native-qualified subset of the existing unit/invocation controls +there. Build coverage-instrumented CLI binaries, run those controls, and merge +hits into each language's existing report. Preserve pure Go checks, all existing +workspace tests, report normalization and test-source exclusion. Go uses native +covdata plus its existing text profile; Rust uses cargo-llvm-cov's shared profile +collection and final report. Rust coverage needs the existing Go toolchain for +synthetic fixture generation, explicitly configured in CI. Do not install tools +on the workstation or change released binaries, runtime dependencies or floors. + +Blast radius: coverage scripts, shared portable controls, coverage workflow and +coverage documentation. Product runtime code is unchanged. Main risks are lost +unit coverage, stale profile contamination, and measuring fixture-writer work +as CLI work. Use private per-run profiles; validate final reports retain SDK +coverage and contain new CLI invocation hits. Fixture generation remains outside +CLI instrumentation. Verify expected outputs against the pinned v262 oracle, +execute both complete report-generation paths, and inspect real CI results. +Existing representative performance evidence remains applicable because only +test binaries are instrumented. Record source/tool failures separately. + +Artifact plan: update coverage README, SOW and status evidence; no product spec +or consumer-doc change is needed for test-only instrumentation. Use synthetic +fixtures and private logs; no sensitive input. No remaining design decision +blocks this approved coverage implementation. + +### Coverage Implementation And Validation - 2026-10-21 + +The portable smoke runner accepts either or both language binaries and replays +13 native-qualified unit/invocation cases from the existing shared fixture +table, alongside its original 26 controls per language. Both instrumented +commands pass all 39 controls; the paired run passes 78. Each smoke run owns +its fixture and builder directories. No SDK or CLI runtime source changes. + +Go runs the original pure SDK and instrumented language suites, then builds an +atomic instrumented CLI, runs the portable controls, converts native covdata, +and merges source blocks before existing filtering. It rejects incompatible +profile modes or statement counts. All 13,279 unit-profile blocks survive with +unchanged statement counts and nondecreasing hits. Invocation coverage rises +from 2/321 to 221/321 statements; unit naming from 0/125 to 79/125. SDK reader +coverage remains present (500/601 to 502/601 statements). Final Go report is +69.9 percent statement coverage; Codacy's separate line-based PR threshold must +still be checked remotely rather than inferred from this number. + +Rust runs workspace tests and portable subprocess controls in one explicit +cargo-llvm-cov profile directory before LCOV export. The integration tests +already build the instrumented CLI. Invocation coverage rises from 0/319 to +240/319 lines; unit naming from 0/105 to 83/105. Covered SDK crate lines remain +present (16,509 in the preceding CI report, 16,555 in this local report). +The Rust coverage job explicitly sets up Go for synthetic fixtures. Python uses +only standard-library modules, with an optional PYTHON interpreter override. + +Retained evidence under `.local/sow-0121-v262-krauk7l5/`: +`coverage-go-verified.log`, `coverage-rust-verified.log`, their corresponding +report directories, `coverage-merge-verification.json`, and the unchanged +source hashes in `coverage-validation-source.json`. ShellCheck with sourced +helpers and shell syntax checks pass. The full fixed scripts execute end to +end; SDK tests and report exclusions remain enabled. + +Adverse observations remain recorded. The first concurrent Go instrumented +run hits the existing live-writer test's 45-second deadline. The full sequential +rerun passes without changing deadlines, test expectations or runtime code. +The first Rust attempt exposes differing default profile directories between +test and show-env operations, followed by an invalid run after the script was +edited while Bash was reading it. That run is rejected. Explicit directory +selection and a stable, unchanged-script rerun produce the accepted report. +No failed attempt is counted as validation success. + +Performance impact: only test binaries are instrumented; shipped source and +build behavior are unchanged, so existing runtime performance evidence remains +applicable. The smoke runner and coverage README document the new collection. +No product spec, consumer documentation, runtime project skill or output skill +change is needed: this implements the approved reporting decision without a +new product contract or reusable workflow beyond the coverage README. Existing +SOW-0122 performance scope remains pending and separate. The user coverage +choice is resolved by implementation, not by an exception. + +### Coverage CI Result And FreeBSD Matcher Finding - 2026-10-21 + +Candidate b12c8fffb2457fdb8627947aa08230d3e17cf09a passes both coverage +jobs and upload in run 39581079338. Codacy coverage variation passes at ++1.99 percentage points. The approved coverage decision is delivered. +Native run 39581079490 passes Linux, macOS and Windows, but FreeBSD fails +the newly included unit-name-8---unit control in both commands. The pattern +`[[:alpha:]]lpha.service` must match the fixture's `alpha.service` units; +the native FreeBSD matcher does not provide that named-character-class +behavior. Both command implementations call libc fnmatch directly: +`go/cmd/journalctl/unit_pattern_unix.go:20` and +`rust/src/cmd/journalctl/unit_pattern.rs:28`. + +This is a runtime compatibility gap exposed by expanded validation, not a +runtime regression caused by coverage instrumentation. The existing Linux +oracle results do not establish this behavior on FreeBSD. Retained evidence: +`coverage-native-freebsd.log` under the current private validation directory, +lines 1101 and 1140, and native CI job 119158555362. A controlled C-locale +probe of the FreeBSD source also differs from glibc for a single-character +collating-element pattern; it is source-level supporting evidence, not a +native FreeBSD execution claim. + +The previously approved native-matcher design cannot supply the required +behavior by directly calling FreeBSD fnmatch. No runtime repair has been +implemented. Pending design choice: permit a FreeBSD-specific compatibility +path for unsupported unit-pattern syntax, preserving ordinary native paths, +or explicitly accept and document platform-native pattern differences. The +former preserves the requested parity outcome and is recommended; the latter +would change the CLI compatibility contract. A replacement must be checked +for locale, raw-byte and malformed-pattern behavior before implementation; +a generic regex substitution has not been established as equivalent. +SOW closure remains blocked on this finding and final review disposition. + +### Coverage Review Collection - 2026-10-21 + +Collected session 38c2b9beccf04384a5e70a58fae8036e, turn 3 for GLM, +MiniMax, Kimi, Qwen and DeepSeek, and Mimo's already-used retry, turn 4. +GLM returns NEEDS CHANGES for the independently reproduced FreeBSD matcher +gap above. MiniMax and Kimi return PRODUCTION GRADE but rely on older native +qualification for unchanged runtime code; those verdicts do not override the +new native failure. Qwen and DeepSeek each time out after 1,801.3 seconds +without verdicts. Mimo's retry fails with HTTP 429. These three failures are +review coverage gaps, not approvals. No further review is launched while the +verified matcher blocker awaits a design decision and repair. + +Nonblocking notes are adjudicated as follows. Failure output could show more +detail, and awk can leave a partial raw profile after its nonzero exit; neither +can turn a failed check into a successful upload. Hard-coded smoke selection +and case-name coupling are intentional representative sampling; current names +and counts are verified, and a missing name fails loudly. Additional empty-hit +or per-run Rust retention checks are optional reporting defenses, not evidence +of current lost coverage: retained final reports and successful CI establish +the delivered result. A further reporting framework is rejected from this +minimal coverage change. Automatic scratch deletion is rejected: evidence is +retained intentionally and deletion is not authorized. Existing local stress +test timing sensitivity remains recorded, with the unchanged sequential run +and isolated CI passing. Standalone Rust CLI packaging is not in scope; its +shared introspection input is deliberate and the crate is not published. + +Stale observations are corrected by current evidence: Codacy variation now +passes at +1.99 points, and git signature verification reports G for b12c8ff. +The pre-existing malformed-JSON smoke diagnostic can fail with a traceback; +it still fails closed and is not required to repair this coverage outcome. +These optional diagnostic and housekeeping changes are rejected from the +current change, not deferred implementation obligations. Full collected output +is retained privately as coverage-review-collection.log in the validation +directory; no reviewer transport log is copied into durable evidence. + +### FreeBSD Compatibility Repair Gate - 2026-10-21 + +The user approves option 1A: repair FreeBSD unit-pattern compatibility, keeping +ordinary native paths. The user also replaces external-model review with an +internal subagent applying the final-review skill. Do not launch further +external reviewers. Review the whole final implementation and causal blast +radius after primary validation, and independently adjudicate findings. + +Root cause and evidence are recorded above. Affected callers include system +and user unit expansion, invocation selection and frozen follow expansion. +SDK behavior, file-pattern globbing and compiler floors are unchanged. +Implement a CLI-private prepared compatibility matcher only for unsupported +extended bracket syntax on FreeBSD. Investigate native POSIX regex compilation +as the existing-libc implementation of classes and collating elements, with +glob translation and explicit locale/raw-byte handling; do not assume regex +grammar is interchangeable with glob grammar. Reuse ordinary fnmatch for all +other patterns. Share the compatibility implementation between the commands +to prevent language drift; retain per-command prepared state and teardown. + +Before accepting that implementation, differential-test the fallback against +glibc fnmatch for C/UTF-8, positive/negative/malformed brackets, literal +backslashes, Unicode, raw bytes and wildcard boundaries. Compile the same +fallback in primary-host tests so FreeBSD-only code is not untested locally. +Extend the existing portable controls and use native FreeBSD CI to qualify +both command integrations. Compare actual work and focused timings for added +costs; no new benchmark framework. If the libc regex approach fails semantic +qualification, resolve it within the approved compatibility-path outcome +before adoption, rather than accepting a parity exception. + +Artifact impact: update unit-pattern contract notes, compatibility skill, +portable tests, CLI build prerequisites if needed, SOW and status evidence. +Use only synthetic fixtures and ignored local probe output. No sensitive +inputs, production journal access, service changes or installations. + +### FreeBSD Repair Implementation And Primary Validation - 2026-10-21 + +Both CLIs now use one CLI-private compatibility implementation for extended +unit brackets on FreeBSD. Ordinary patterns allocate no compatibility state +and retain native fnmatch. The fallback prepares byte/wide pattern forms once, +reuses its value buffer, handles named classes through native character +classification, and preserves malformed-bracket and multibyte/byte retry +behavior. It does not scan journal rows or change indexed unit discovery. +Go owns the state through the existing matcher lifecycle; Rust owns the same +C implementation through an RAII wrapper. Linux/macOS compile the wrapper for +direct fallback tests while their production matching stays native. Windows +keeps its existing byte matcher. File globbing and SDK code are unchanged. + +The initial POSIX regex translation is rejected after 688 mismatches in +237,150 bracket comparisons: rejecting an entire malformed regex differs from +fnmatch's successful-member and unterminated-bracket handling. The replacement +direct wildcard matcher has zero differences across 355,725 comparisons in +C, C.UTF-8 and en_US.UTF-8 for the retained corpus. This is targeted evidence, +not a claim that different operating systems share identical locale databases. +The 237,150-case C/UTF-8 corpus also passes address/undefined-behavior checks. +Byte ranges use byte collation, rather than treating invalid bytes as Unicode +characters; the broader locale probe exposed and corrected that distinction. + +Supported-floor Go CLI tests pass with race detection; Rust passes 25 unit +tests and 11 subprocess tests. Pure Go SDK build passes with CGO disabled. +All 1,372 pinned-v262 observations pass, including new system/user bracket +invocation discovery and selection controls. Expanded portable smoke passes +96/96 locally. Native FreeBSD qualification remains pending before acceptance. +Wiki validation passes for 16 pages; all 35 executable documentation examples +pass. The raw example spelling initially triggered the wiki-link validator; +consumer prose now names the feature without introducing a false wiki link. + +Performance evidence follows the changed work: ordinary matching retains its +native loop and avoids compatibility allocations; SDK/query traversal is +unchanged. The previously incorrect FreeBSD extended-bracket path has no +correct pre-change timing baseline. A small alternating C-level comparison +with working Linux/glibc matching measures about 48 ns versus 37 ns per value +in C locale, and about 93 ns versus 125 ns in UTF-8. These are isolated matcher +costs, not whole-command measurements or FreeBSD timing claims. The C-locale +difference is about 0.011 ms per 1,000 distinct unit values; the UTF-8 path is +faster in this control. Pattern allocations occur once and the value buffer +grows only when needed. No performance framework or ordinary-query exception +is introduced for this compatibility repair. + +Rust adds a direct CLI build dependency on the already locked cc 1.4.0 +(declared Rust minimum 1.63); no dependency version or SDK compiler floor +changes. Existing CLI prerequisites already require a C toolchain. The shared +source belongs to the Go command directory and is consumed by the unpublished +Rust command, consistent with the existing shared CLI contract input. +Specs, CLI consumer prose and the compatibility skill explain the FreeBSD +path. No output skills exist; no SDK docs or public SDK contract changes. + +Evidence under the current private validation directory: +freebsd-go-cli-final.log, freebsd-rust-cli-final.log, +freebsd-final-v262.json/log, freebsd-portable-final.log, +freebsd-differential-direct.log, freebsd-differential-locales.log, +freebsd-differential-asan.log, freebsd-pattern-cost.log and +freebsd-doc-examples.log. The rejected translation observations are retained +in freebsd-differential.log. Internal final review follows native qualification. + +### FreeBSD Native Qualification And Locale Edge Cases - 2026-10-21 + +Native run 39659065166 on signed candidate e3ed6ec passes FreeBSD, Linux and +Windows, including all 96 portable controls. macOS fails the new direct +fallback Unicode-class test. Its default MB_CUR_MAX macro observes global +locale state unless the extended-locale header is included; the fallback now +includes xlocale.h on Darwin. This affects direct fallback validation there, +not ordinary macOS production matching, which still calls native fnmatch. + +The remaining primary audit adds accented/case equivalence-class controls. +The compatibility path compares primary weights from native wide collation +keys (level separator 1 on the qualified FreeBSD/glibc implementations), and +preserves glibc's byte-retry consumption for multibyte equivalence matches. +All 718,200 retained comparisons across C, C.UTF-8 and en_US.UTF-8 now agree +with local glibc 2.39, under address/undefined-behavior instrumentation. Both +CLI suites pass again; the explicit UTF-8 tests also require the selected +locale rather than silently skipping an unavailable locale. Latest logs: +freebsd-differential-final.log, freebsd-go-cli-qualified.log and +freebsd-rust-cli-qualified.log. Native rerun qualifies these final additions. + +Inspection references (reference code is not copied into the implementation): + +- freebsd/freebsd-src @ bd31ee522f8eb5bc2a3208accc2f6976e5d45b5d: + lib/libc/locale/collate.c:394-419, lib/libc/gen/fnmatch.c, + lib/libc/regex/regcomp.c:1182-1193. +- apple-oss-distributions/Libc @ 71bbe350ab79eef58113991d817ccc6165061a64: + include/_stdlib.h:123-139, include/__xlocale.h:27, + locale/xlocale.c:142-195. +- bminor/glibc @ 04e750e75b73957cf1c791535a3f4319534a52fc: + posix/fnmatch.c:234-263 and posix/fnmatch_loop.c (bracket grammar). + +Static analysis reports two generic strlen terminator alerts in the new +private header. Both are false positives after inspecting every caller: +patterns come from Go C.CString or Rust CString, and both value wrappers append +a NUL before calling C. Localized Flawfinder directives record these exact +invariants; no global rule is disabled. The local level-4 scan reports no hits. +The pre-existing Bandit internal error remains separately recorded. Raw logs +stay private (freebsd-first-macos.log, freebsd-first-codacy.log and +freebsd-flawfinder.log). Ordinary coverage jobs and Codacy variation pass on +e3ed6ec; new runtime review still awaits final native qualification. + +Native run 39660136110 passes all four platforms on f705c21, including the +Darwin thread-locale fix and accented equivalence controls. Coverage also +passes. The sole new analysis finding is a missing blank line before the +reference list in this SOW; it is repaired. The pre-existing Bandit execution +error is unchanged. Later primary checks add unterminated named/collating +brackets and self-matching malformed patterns. Those controls expose and fix +trailing-range and unfinished-special-member handling. Range members now +delegate to a bounded, escaped native fnmatch pattern, retaining the platform's +ordinary range ordering rather than approximating it with full collation-key +ordering. This prevents mixed native/compatibility patterns from disagreeing +within the same locale. Native ranges and primary equivalence weights serve +different contracts and must not be conflated. + +The final 1,026,675-comparison corpus passes under address/undefined-behavior +instrumentation. Both CLI suites pass again; permanent tests include the +reproduced malformed cases and compare mixed class/range patterns with the +platform's ordinary range matcher. Collation-key resize and bounded conversion +failure paths are checked before use. Latest retained evidence: +freebsd-differential-final-audit.log, freebsd-go-cli-final-audit.log, +freebsd-rust-cli-final-audit.log and freebsd-flawfinder-final.log. Final source +is frozen for native qualification and independent internal review; no more +external-model review rounds are authorized. + +### Internal Final Review And Timestamp Rendering Repair Gate - 2026-10-21 + +The user-requested internal final-review subagent reviewed frozen revision +6752fc7 and returned FAIL with one proven P2 failure class. Independent probes +passed 931 option combinations and 450 rendering combinations. Calendar output +bypasses the existing CLI timezone resolver: Go time.Local ignores POSIX TZ +rules; both ordinary rendering providers ignore custom TZDIR and TZif leap +records and apply transition-free footers prematurely. Rust can combine an +incorrect clock with a libc-derived abbreviation. Entry output, verbose output, +headers and boot/invocation tables share the affected paths. Raw epoch output +is correct. This is an existing parity gap, not caused by the FreeBSD repair. +Evidence: go/cmd/journalctl/output.go and main.go formatHeaderTimestamp; +rust/src/cmd/journalctl/output.rs; private final-review-6752-cli zones/leaps +reproducers and results. Native CI 39661284599 passes all four platforms on +6752fc7; coverage, CodeQL, documentation and SARIF jobs pass. The SARIF log +still contains the pre-existing Bandit execution error, an analysis coverage +limit rather than proof of complete Bandit coverage. + +Repair applies the approved portable timestamp semantics already implemented +for bound parsing. Reuse each CLI's timezone resolver and parsed TZif metadata +for read-only calendar conversion; retain parser state semantics unchanged. +Resolve the rendering timezone lazily once per command process, so cat/JSON/raw +clock output does not load timezone data and rows do not reread files. Borrow +zone names and search transition/leap arrays without rebuilding parser names +or allocating a timezone per row. Calendar output uses one consistent source +for civil fields, zone offset and abbreviation, including leap corrections. +Preserve stock's local header/table behavior with --utc and raw short-unix +values. No SDK/API/dependency/platform architecture change is needed. + +Validation covers the reproduced POSIX/custom TZDIR/leap/footer controls in +both languages against pinned v262, all calendar modes, tables/headers, UTC, +raw epoch controls and ordinary named zones. Run focused CLI suites and existing +full oracle, inspect changed work for allocation/I/O cost, and re-review the +whole SOW through the same internal reviewer. Specs/compatibility instructions +will record the parser/rendering consistency requirement. Detailed evidence +remains in ignored scratch; fixtures are synthetic. No new user design decision +is required to apply the existing approved timestamp contract. + +### Timestamp Repair Implementation And Primary Evidence - 2026-10-21 + +Both renderers now reuse command-local timezone resolution and TZif metadata, +with read-only rendering state initialized on first calendar output. Go caches +fixed locations per TZif kind once and keeps ordinary formatting on time.Time; +only files with leap records need the explicit leap conversion. Rust borrows +names and produces calendar fields from the same offset source. Neither path +rebuilds parser abbreviation state or reads files per row. SDK code is unchanged. +Empty/malformed TZ names and native offset-component limits are retained during +resolver fallback. Source: bminor/glibc @ 04e750e75b73957cf1c791535a3f4319534a52fc, +time/tzset.c:141-208 and 303-309. Valid POSIX providers remain in use; named-zone +fallback remains available on platforms without an installed zoneinfo database. +The expanded fixture exposed a separate small-epoch presentation gap: short-unix +now uses stock's ten-column seconds width; its epoch value stays unchanged. + +Primary evidence: time-render-check-final.log passes 668 controls covering +parsing and all calendar displays, including raw epoch, UTC, tables/headers, +custom TZDIR, POSIX rules, historical transitions, leap hits and footer handling. +time-final-v262.log passes 1,372 existing v262 observations. time-portable.log +passes 102 controls, adding POSIX/custom-directory/leap rendering to native CI. +Supported-floor Go race and Rust suites pass; all 35 executable documentation +examples and 16 wiki pages pass. The SOW audit is complete and clean. Private +invalid-zone probes cover 60 render/bound cases without differences after repair. + +Performance review identified an initial Go UTC-rendering increase (119.66 to +128.51 ms for 20,000 rows). It is retained in time-cost.json and was not accepted. +Ordinary Go rows now use cached locations and the original standard formatter; +explicit leap conversion is confined to files with leap metadata. A subsequent +alternating comparison removes that increase; isolated final evidence is retained +separately. Rust avoids the previous separate libc abbreviation lookup. No new +performance framework, SDK cost or ordinary-query exception is introduced. + +Artifact maintenance: parity spec, CLI consumer prose and compatibility skill +now require parsing/rendering timezone consistency. No AGENTS change or output +skill is needed; existing policy covers this repair and no output skills exist. +Synthetic fixtures and sanitized records satisfy the sensitive-data gate. This +repair and its performance evidence remain in SOW-0121, with no correctness +item transferred to the broader pending scalability SOW-0122. + +The final isolated five-pair comparison (20,000 rows, equivalent output) records +UTC medians Go 117.10 -> 119.91 ms and Rust 111.48 -> 100.05 ms; New York medians +Go 117.98 -> 119.08 ms and Rust 107.06 -> 99.04 ms. Stock is 34.82/34.98 ms. +Go paired directions are mixed (UTC three increases/two decreases; New York +three increases/two decreases), and the preceding comparison has lower candidate +medians in both zones. This does not prove equality; it does not reproduce the +initial sustained formatter cost either. Code review confirms one existing-style +transition lookup/format, cached locations and no per-row filesystem work or +zone allocation. These representative results and work tracing establish the +performance disposition without expanding a measurement framework. Raw samples +and initial adverse results remain in time-cost*.json. + +### Default Daylight Rule Repair Gate - 2026-10-21 + +Internal re-review of eff0231 confirms the original timestamp repairs and +independent 931 option/450 rendering probes pass, but returns FAIL for omitted +POSIX daylight transition rules. Rust treats XYZ5ABC as fixed standard time; +both providers bypass a custom TZDIR/posixrules file. This changes displayed +times and local query boundaries by an hour in reproduced ordinary fixtures. +The correction stays within the existing resolver contract: intercept rules +that supply a DST name without dates, load/transform the default TZif once using +its standard/universal transition flags and caller names/offsets, and use the +native US transition defaults only if the default file is unavailable/unusable. +Explicit-date rules and ordinary rendering paths are unchanged. Retain the +original footer offset behavior and override its displayed names as native +conversion does. TZif parsing gains the two existing per-kind flags; this is +CLI-only metadata, not journal SDK work. Native offset components also undergo +unsigned-short conversion before their documented clamp, covering two extreme +input discrepancies proven by review. No dependency or architecture change. + +Reference: bminor/glibc @ 04e750e75b73957cf1c791535a3f4319534a52fc, +time/tzfile.c:459-613 and time/tzset.c:248-268,317-329. Private default-dst and +default-rules evidence remains under final-review-eff0231-cli. Validation adds +ordinary/default-file/missing-file, historical/future and explicit-rule controls +to the existing timestamp suite, including local bound selection and rendering. +No per-row file access or new benchmark framework is introduced. Whole-SOW +internal re-review and native CI remain required after the repair. + +### Default Rule Implementation And Primary Validation - 2026-10-21 + +Both resolvers now prepare omitted-date rules from TZDIR/posixrules, with +standard/universal transition flags, native name/offset replacement and a +checked timestamp adjustment. An unusable/missing default file uses explicit +native US dates. Components wrap through the native unsigned-short conversion +before clamping. Installed named zones and explicit-date rules retain their +existing paths. Rendering only sees the resulting cached metadata; no ordinary +per-row path or SDK code changes in this repair. + +Expanded checks exposed two details that reading upstream comments alone missed: +native default-file conversion starts with zero remembered daylight offset, +and footer conversion's tm_zone retains the footer abbreviation even though +its parser abbreviation pair is restored to user names. The implementation +matches observed pinned-oracle output, including historical/future dates; the +prior gate's planned displayed-name override is superseded by this evidence. +A first synthetic universal-flag fixture violated TZif's universal-implies- +standard constraint; the permanent flag fixture is valid and exercises both +branches. The initial 916-case failure log is retained with final passing output. + +Default timestamp controls pass 1,052/1,052, including local query boundaries, +ordered UTC/local bounds, all calendar modes and tables, omitted/explicit dates, +custom/missing default files, flags and future footer behavior. Existing v262 +controls pass 1,372/1,372; portable smoke passes 106/106. Supported-floor Go CLI +race and Rust 25-unit/11-subprocess tests pass. Wiki validation passes; prior +35 executable examples remain unchanged, and normal CI verifies them again. +Evidence: default-timestamp-final.log, default-final-v262.log, +default-portable.log and default-{go,rust}-tests.log in the private validation +directory. Native run 39665146987 and coverage/analysis pass on preceding +eff0231; the final default-rule changes require their own native qualification. +No further timing run is needed: this repair changes resolver setup and only +the previously incorrect omitted-date path; ordinary hot rendering is identical +to the measured candidate. The same whole-SOW internal reviewer rechecks the +final revision. Specs, consumer prose and the compatibility skill record the +missed default-file requirement; no new follow-up is deferred. + +### Final CI Integer Conversion Disposition - 2026-10-21 + +Internal whole-SOW review passes bed8c1152fb6 with no remaining P0-P2 finding. +Native CLI run 39667258392 passes Linux, macOS, Windows and FreeBSD; both +coverage jobs and documentation examples pass. CodeQL flags the intentional +uint64-to-uint16-to-uint64 TZ component conversion. Native compatibility requires +modulo 65536 before clamping; rejecting overflow would break tested parity. +Replace the narrowing cast with an explicit 0xffff mask: equivalent for every +uint64 input, no allocation or algorithm change, and directly expresses the +required bounded result. Re-run the existing timestamp controls and internal +impact review, then collect final CI. This is an implementation clarification +of the approved native rule behavior, not a new parsing policy. + +### Final Closure - 2026-10-21 + +- Acceptance: paired file-backed v262 CLI behavior is implemented; SDK format, + compiler floors and pure SDK dependency boundaries remain unchanged. The + SDK scope completed separately in SOW-0159 after prerequisite SOW-0108. +- Validation: 1,372 v262 observations, 1,052 timestamp observations, 106 portable + controls, supported-floor CLI suites and documentation checks pass. The + final explicit mask repeats all 1,052 timestamp observations with zero + differences. Native run 39667258392 passes all four supported platforms; + coverage run 39667258353 and CodeQL workflow 39667258404 pass. The aggregate + CodeQL annotation concerns intentional narrowing; the equivalent explicit + mask resolves its cause without changing the native compatibility contract. + Normal PR CI revalidates the combined closure revision. +- Independent final review: the user-selected internal final-review process + passes bed8c1152fb6 with no P0-P2 blocker. Its independent 931 option and 450 + rendering comparisons, retained failure reproducers and 42 additional + timezone-history comparisons pass. Closure and the explicit mask receive + independent impact review before publication. No external model was invoked. +- Same-failure search: shared timestamp parsing/rendering, query bounds, + headers, boot/invocation tables, TZif flags/footer/leaps/default files, + unit matching and native platform paths are covered by the recorded audits. +- Performance: retain all adverse observations and accepted narrow startup/ + vacuum exceptions. Cached timezone state avoids per-row file access and + location allocation; explicit masking is equivalent to the prior cast. + The representative measurements and hot-path review above establish the + disposition without claiming inconclusive intervals prove equality. +- Analysis limitation: Codacy run 39667258303 passes its aggregate job but + Bandit reports one execution error; complete Bandit coverage is not claimed. + This is retained infrastructure coverage evidence, not a hidden pass. +- Sensitive data gate: fixtures and durable evidence use synthetic identities + and sanitized summaries; raw logs stay in ignored .local/. +- Artifact gate: specs, consumer CLI documentation and runtime compatibility + skill were updated with shipped behavior and lessons. AGENTS.md already + states the approved CLI dependency/target boundaries and needs no further + change. No separately distributed operator skill exists to update. Both + status indexes and SOW-0122's dependency reference are reconciled. This SOW + moves to done with implementation in the same signed commit. +- Follow-up mapping: all confirmed CLI correctness findings above are fixed; + historical pending/decision wording is execution history. Broader large-input + invocation/polling/archive optimization remains in real pending SOW-0122, + unactivated. Independent SDK work remains in paused SOW-0045/SOW-0071 and + pending SOW-0160/SOW-0161; none is silently included or required to repeat + this accepted scope. No new implementation obligation is deferred. +- Lessons: oracle-check calendar rendering separately from bound parsing, + qualify native dependencies on each target OS, and preserve scanner execution + limitations even when aggregate jobs pass. These are reflected in the + compatibility skill and the evidence above. + +The final lifecycle audit reports complete and clean (closure-audit.log). + +## Regression - 2026-10-23 + +### What Broke And Evidence + +The user approved reopening after internal final-review risk analysis of +45e0de2832f3bb886551896287b6ddc1e3f5edb4. Twenty-one valid, nonempty archived +journals span three sequence IDs and backward realtime ordering. Every filename +agrees with its header/ENTRY identity and timestamps; pinned v262 verifies all +21. With --vacuum-files=2, stock retains two and exits zero; Go retains a +different pair and exits zero; Rust exits 101 with a non-total-order sort panic +and retains all 21. Earlier empty-file cleanup may already mutate a mixed +archive directory. Synthetic reproducers/results remain under ignored +.local/risk-cli-review/ (write_valid.go, valid-names.json, valid_check.py, +valid-result.json). No live host journal was accessed. + +Rust main.rs:1644,1753 and Go main.go:3381,3454 use sequence order within one +ID and realtime across IDs. These comparisons can cycle. The comparator and +Rust sort already exist in the PR base: this is an inherited defect missed by +the requalification, not a new SDK regression. Earlier vacuum controls lacked +valid cross-ID clock-rollback ordering. The independent SDK and upgrade reviews +pass their scopes; their verdicts do not override this maintenance blocker. + +### Repair Gate And Plan + +The user approved investigating native ordering and a shared Rust/Go repair. +No archive-deletion policy has yet been selected. Inspect pinned upstream +comparison/sort/enumeration and library guarantees; determine whether matching +one oracle's accidental order is a portable contract. Go also pre-sorts directory +names through os.ReadDir, unlike native enumeration, so changing only its sort +algorithm does not establish parity. Present any irreducible deletion-policy +choice with concrete tradeoffs before implementation. Do not catch the panic +and continue, or silently choose a different retention order. + +Validate the selected policy with the retained valid 21-file fixture, ordinary +monotonic archives, multiple sequence IDs, corrupt-style names, and applicable +count/size/age combinations. Trace added sorting/allocation work and use a +focused existing-style timing control if needed; do not expand a benchmark +framework. Preserve the user-approved narrow startup/vacuum costs. Update the +parity spec, consumer maintenance documentation, compatibility skill and both +status indexes; rerun the local audit and internal full-scope review before +closure. All fixtures are synthetic and raw evidence stays ignored. No SDK +source or unrelated SOW is activated by this repair. + +### Native Ordering Investigation And Open Decision - 2026-10-23 + +Pinned reference: systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492, +src/libsystemd/sd-journal/journal-vacuum.c:35-49,153-160,293 and +src/basic/sort-util.h:31-36. Native enumeration feeds qsort through its wrapper; +Go os.ReadDir sorts names first. POSIX qsort requires total ordering (Open Group +qsort specification, https://pubs.opengroup.org/onlinepubs/009695399/functions/qsort.html). +The native comparator violates that prerequisite for a cycle such as A=(ID1, +seq1,time300), B=(ID1,seq2,time100), C=(ID2,seq1,time200): A precedes B, B precedes +C and C precedes A. A read-only in-memory libc experiment over all six input +permutations produces three different output orders: ABC, BCA and CAB. This +experiment performs no journal operations or deletions. Emulating one libc's +sorting algorithm alone cannot establish portable stock parity. + +Open policy decision, before code: recommend a deterministic shared order that +preserves increasing sequence within each history and chooses the oldest-time +available history head across histories (with explicit deterministic tie +breaks). This defines behavior where native comparisons are contradictory, +rather than promising the oracle's incidental survivor pair. It requires a +narrow documented parity exception and targeted cost validation; the plan must +cover missing sequence IDs, ties, empty-file handling and count/size/age stop +conditions before implementation. Alternatives are conservative refusal on +clock-rollback histories before deletion, or freezing a selected libc/order +emulation with its portability and maintenance limitations. None is approved +by the earlier approval to investigate. The local lifecycle audit reports +complete and clean after reopening; no production source has changed. + +### Approved Decision 1A And Implementation Gate - 2026-10-24 + +The user selected 1A: deterministic sequence-preserving ordering in both CLIs. +Sort each sequenced history by sequence number, breaking equal sequence numbers +by filename; treat each candidate without sequence metadata as an independent +history. Merge available history heads by adjusted realtime, then sequence ID +when both heads have one, then filename. To make ties a total order, sequenced +heads precede unsequenced heads at equal realtime; names compare native Unix +bytes / Unicode Windows names in both commands. This explicit tie convention +extends the approved deterministic policy. Filename is unique within a directory. + +Implementation uses a contiguous history sort and a min-heap of history ranges, +consuming only candidates needed by the existing retention loop. No journal +payload decoding or new metadata syscall is added. Time is O(n log n + k log h) +for n candidates, h histories and k consumed candidates; auxiliary state is +O(h). A single history needs no heap comparisons. Empty cleanup, protected files, +metadata adjustment and count/size/age stop conditions retain their existing +contracts; an age stop does not skip an earlier-sequence file to delete its +later successor. Contradictory cross-history comparisons are the approved +parity exception. Test ties, unsequenced archives, reversed insertion order, +clock rollback, ordinary stock-compatible order and real command survivors. +Add portable controls and targeted sorting/command cost evidence. Update docs, +spec and compatibility skill. No dependencies or SDK changes are required. + +### Decision 1A Implementation And Local Validation - 2026-10-24 + +Both commands now use a valid contiguous history sort and a lazy min-heap of +available heads. The old pairwise cyclic comparators are removed. Missing +sequence metadata is represented by singleton histories. Tests enumerate all +120 permutations of a cyclic/tied/mixed candidate set in each language. Shared +portable controls generate the original 21 valid archives using caller-provided +head sequence numbers; reversed creation order, count two/ten, size, age and +combined limits pass. A recent earlier sequence protects its old successor +under age-only limits, while an eligible old sibling history is removed. + +Local validation passes 130/130 portable observations, 78/78 existing utility +oracle observations, Go CLI race tests, Rust 26 unit/11 subprocess tests, +35 executable documentation examples and 16 wiki pages. Synthetic fixtures +and raw results are under .local/risk-cli-review/; no SDK source changed. + +Performance evidence uses seven alternating measured runs over 1,000 nonempty +archive names, with one, 50 and 1,000 histories, against retained binaries and +stock. The first Rust many-history median increased (14.67 to16.26ms); retained +as adverse evidence in vacuum-cost-initial.json, not accepted as equality. +An isolated repeat yields Go before/after20.43/20.53,20.41/20.44,20.76/20.41ms; +Rust16.85/16.83,16.46/16.55,16.56/16.81ms; stock75.91-79.49ms. The original large +increase did not reproduce. These small mixed/reduced differences do not prove +equality; work tracing shows unchanged file/header/syscall work, one in-place +sort, O(h) heap metadata and lazy O(log h) selection. No per-row payload work or +new metadata syscall is introduced. Go baseline is retained final45e0de2; +Rust retained unit-repair-6 precedes unrelated timestamp changes, so its +comparison is representative, not an exact whole-binary before/after claim. +No additional performance waiver is requested. Final internal review assesses +this evidence and the complete changed behavior before publication. + +Consumer maintenance docs, parity spec and compatibility skill record the +approved exception and future tests. AGENTS.md needs no change: this is a +CLI behavior decision within its existing dependency boundary. No distributed +operator skills exist. Pending0122 remains unactivated and owns only broader +scalability work. Internal review/native qualification and lifecycle close +remain outstanding; previous completion/PASS is superseded for vacuum ordering. + +### Regression Closure - 2026-10-24 + +Decision 1A is implemented in Rust and Go. Internal final-review passes exact +repair fcd918bcaab05b29e2bd18cf297ce2273b2059d8, with matching reviewed hashes. +Independent checks add 10,000 randomized candidate sets per language against +an eligibility model and 12 real-command controls using the original 21 verified +journals (two languages, two creation orders, limits 2/10/21). No remaining +P0-P2 failure or reproducible blocking slowdown was established. The whole +relevant scope includes maintenance, platform/native dependency boundaries, +shared fixture generation, documentation and previous CLI findings. Review +findings are fixed, not waived; final lifecycle changes receive impact review. + +Native run 40076505728 passes Linux, macOS, Windows and FreeBSD with 130 portable +observations across both commands. Both coverage jobs in 40076505657 pass; +documentation examples and wiki validation pass. The Codacy SARIF job initially +failed fetching remote configuration with an upstream Internal Server Error, +before source analysis; a retry failed identically. Coverage generation passes +but its Codacy upload reports Request URL not found / no data sent. Neither +failure establishes an implementation finding; the external configuration/API +path remains unverified and no CI configuration is changed under this repair. +Go/Rust CodeQL run 40076505661 passes. The earlier Bandit execution-coverage limitation +remains explicit; aggregate success alone does not remove it. Normal PR CI +revalidates the final lifecycle amendment. + +Acceptance evidence also includes 78 existing stock-oracle utility observations, +Go CLI race tests, Rust 26 unit/11 subprocess tests, 120 insertion permutations +per language, 35 documentation examples and 16 wiki pages. The local SOW audit +reports complete and clean. Cost results and their uncertainty remain above. +Same-failure review covers mixed/unsequenced histories, ties, backward clocks, +heap successor replacement, age stopping and reversed enumeration; the old +cyclic comparators no longer serve either command. + +Artifact maintenance: consumer maintenance docs, parity spec and compatibility +skill are updated; AGENTS.md and output/operator skills need no changes for the +reasons recorded above. Both status indexes and SOW-0122's prerequisite +reference are reconciled. Synthetic identities and ignored raw evidence satisfy +the sensitive-data gate. The lesson is to test valid cross-history clock rollback +and require a total ordering rather than treating one libc output as a portable +contract. Historical pending decisions are resolved by 1A; broader scalability +remains SOW-0122, with unrelated SDK issues still explicitly tracked by their +own SOWs. No new deferred repair remains. Implementation and completed SOW +lifecycle are amended into the same signed repair commit. + +## Regression - 2026-10-26 + +### Pre-Implementation Gate + +The user approved handling the confirmed reviews and CI findings without another +design decision. At a1144dcbf537, Go invocation listing treats facade TestCursor +false,nil at EOF as a positioned mismatch (invocation.go:345-353), then reads +missing data. Existing invocation-bounds portable smoke fails all four OS jobs. +Check position using cursor metadata before returning a candidate, retaining the +approved facade false,nil EOF contract and avoiding row expansion. + +Rust file_glob.rs:190-222 resurrects an exhausted wildcard candidate list. +Stop expansion when a wildcard yields no candidates; test literal and parent +suffixes. Go already preserves empty candidates. Both CLIs concatenate Windows +paths into invalid file URIs. Normalize drive/UNC paths using standard file URI +forms, percent-encode Windows path bytes, retain Unix output behavior, and render +unsupported device-path forms as plain text. This is CLI-only formatting; no SDK +or dependency change. Test drive, UNC, relative resolution and escaped characters. + +Query-matrix helpers silently mix PATH journalctl with injected stock commands. +Require one explicit version/feature-validated v262 binary, propagate it to every +case and version report, and update documented invocation. Fix relative Go CLI +example paths. Parser rejection tests must name a fixture instead of host defaults. +These harness corrections preserve existing cases rather than creating a new +framework. Prior explicitly injected oracle results are not relabelled invalid. + +Validation: Go/Rust CLI unit tests, existing portable smoke including invocation +bounds, focused v262 oracle cases, query harness unit/plumbing controls, wiki/docs, +compile checks and native OS CI. Review same patterns (TestCursor callers, empty +glob branches, all Windows link surfaces, every stock command builder). No live +host journals. Keep source/fixture/cache outputs under .local and raw analysis +exports ignored. Preserve plain-output hot paths; URI encoding runs only when +links are enabled. No new synchronization or per-entry SDK work. + +Artifact impact: update relevant parity/product specs, compatibility skill and +consumer docs only where behavior or test invocation changes; both SOW indexes +track this lifecycle. Runtime purity/CGO boundary unchanged. Three internal final +reviewers will assess SDK blast radius, CLI and validation after local fixes; +external models remain disabled by user instruction. Codacy now reports13 minor +Markdown findings and one Bandit execution error, not the old configuration +failure. Investigate actual causes before editing CI; unrelated baseline problems +must be explicitly dispositioned, not swept into this PR. No open product choice +blocks the confirmed minimal-complete fixes. + +### Local Correction Evidence - 2026-10-26 + +SDK prerequisites are committed as f50676f and5224c25 under completed0159: +typed facade cursor absence, both CLI EOF consumers, and filtered cursor-seek +exhaustion/reversal. The full explicit-v262 static matrix initially found one +remaining Go sequence-cursor mismatch (3033/3034); after the proven fix it passes +3034/3034. The fresh scratch run selects one full-feature v262 oracle and records +its path/version. No host journal is used; follow was not rerun because this +batch changes no follow loop. Existing native CI will replay portable behavior. + +Rust wildcard expansion stops immediately on an empty candidate set. Both +Windows file-link implementations normalize drive/UNC paths and percent-encode +path bytes only when links are enabled; device forms remain plain text. Unit +controls cover Unicode, spaces, URI delimiters, drive and UNC paths, unsupported +forms and native Windows relative resolution. Linux executes pure URI helpers; +native Windows integration remains explicitly pending CI. Go CLI unit tests and +Rust 28 unit/11 subprocess tests pass. Oracle-plumbing unit tests pass4/4; +parser parity passes; wiki validation and all35 verified docs examples pass. +The paired portable suite includes permanent cursor EOF controls (134 checks). + +Same-pattern searches cover both file-link builders, exhausted wildcard paths, +all query-matrix stock command constructors and imported timestamp/v262 helpers. +No PATH fallback remains; version and feature validation precedes tool builds. +The parser conflict probe explicitly supplies a synthetic filename. Consumer Go +examples account for go -C changing the working directory; README enumeration +restrictions and pager-end/follow specification are reconciled with existing +behavior. Compatibility skill documents explicit oracle propagation. + +Performance review: plain-output and Unix formatting paths are unchanged apart +from a compile-time platform choice. Windows URI work occurs only for requested +links. Glob early exit removes unnecessary work; the oracle/docs/test corrections +have no SDK runtime cost. No benchmark framework or new slowdown waiver is needed. +Codacy's13 Markdown findings were formatting-only in0159 and corrected in057a746. +Bandit execution failure is pre-existing (retained earlier runs and prior SOW +evidence); no detailed scanner diagnostic establishes a new source issue. It +remains an analysis-coverage limitation, not a passing scanner claim. No CI +configuration changes or software installation are required. + +Whole-workspace cargo fmt check reports pre-existing formatting differences in +reader.rs, repository/file.rs and reader_hardening.rs; changed CLI modules are +formatted. These are not runtime findings, and unrelated formatting churn is +excluded. Three internal final reviews will assess the SDK blast radius, CLI +behavior and qualification/artifacts before this SOW closes. No external models +are authorized. No merge or release is authorized. + +### Reverse Cursor Initial Position - 2026-10-26 + +Independent CLI final review found24 differences in54 reverse-cursor controls. +Both CLIs retain the SDK's eager forward seek candidate for reverse output; +native v262 selects the initial candidate in the requested direction +(systemd/systemd @8cc40e0c5e9234bf45084751ac53b1fbfe70b492, +src/journal/journalctl-show.c:76-80). A cursor before filtered rows returns a +later row; one after all matching rows loses earlier results. + +Pre-implementation analysis: retain the SDK's existing forward seek contract. +For reverse CLI positioning, inspect cursor metadata and step backward once when +forward seek exhausts or selects a strictly later location. Compare only the +same sequence/boot/time ordering components as the SDK; x remains exact-match +only. Then apply existing after-cursor exclusion. Reuse public cursor parsers; +CLI-local presence/monotonic extraction adds no SDK API or row scan. Metadata +work occurs once per reverse cursor seek, not per emitted row. Test both cursor +modes and cursor-file through shared v262 controls before/between/after filtered +rows, including partial sequence/time and mismatched x. Preserve genuine errors, +checkpoint semantics and forward behavior. Internal final review remains FAIL +until this behavior is corrected and the full shared matrix is rerun. + +### Directional Seek Design Decision Pending - 2026-10-26 + +The provisional one-predecessor plan above is disproved, not implemented. The +reviewer proved repeated realtime ties require selecting the last equal entry; +unknown sequence identity with no time starts reverse from the tail, unknown +boot/monotonic behaves differently, and contradictory full cursor fields expose +sequence precedence over the eager forward seek's time starting point. The +experimental CLI adjustment is retained only as ignored scratch; production +sources remain at the previously validated revision, which still has this known +reverse-cursor gap. Full matrix PASS does not cover those newly found cases. + +The user was asked decision1: A (recommended long-term-best) add explicit +direction-aware indexed cursor seek APIs in both SDKs, preserving current APIs, +and use them in CLIs; B change existing facades to deferred native seek semantics, +which affects callers expecting immediate positioning. This is an SDK API design +fork requiring user choice under AGENTS.md. Do not implement either until the +answer is recorded. If A is approved, reopen0159 as the SDK prerequisite and +pause0121, then validate both languages and complete the CLI integration. No +row-scan workaround or performance waiver is proposed. PR12 remains draft; +final CLI review is FAIL and native CI qualification is still outstanding. + +### Read-Only Directional Feasibility Check - 2026-10-26 + +While decision1 remains pending, source inspection confirms reusable indexed +primitives. Rust cursor.rs:179-195 already applies directed_partition_point +with strict/inclusive timestamp bounds for forward/reverse ties; filtered +positioning resolves the resulting entry offset through native postings +(cursor.rs:303-327). Go stores entry offsets and has lower/upper realtime +searches (reader.go:963-978), but those generic helpers suppress metadata read +errors, so a new directional path must use the existing error-propagating +filtered-search pattern rather than copy that error suppression. + +Pinned upstream evidence is systemd/systemd +@8cc40e0c5e9234bf45084751ac53b1fbfe70b492, +src/libsystemd/sd-journal/sd-journal.c:844-863 and945 onward: matching sequence +identity takes precedence; boot/monotonic positioning can fall back to realtime +or adjacent boot; otherwise seek uses the direction's boundary. Filtered queries +use DATA postings directly, not a global-row walk. The external reference +checkout HEAD has advanced, so evidence was read with git show at the pinned +commit, not assumed from its working tree. No API or runtime implementation was +changed while awaiting the user's choice. This supports feasibility of optionA, +not a completed design or permission to implement it. + +### Decision1A Approved - 2026-10-26 + +The user approved explicit directional seek APIs in both SDKs, preserving +existing API behavior and Netdata execution/memory. SOW0159 is reopened as the +SDK prerequisite. This SOW pauses implementation until that prerequisite is +validated and reviewed; existing CLI/oracle/docs corrections remain preserved. + +### Directional CLI Integration Gate - 2026-10-26 + +The approved1A SDK prerequisite passes full local suites and three internal +final reviews. Resume this SOW after its coherent SDK commit. Both commands +must use explicit directional seek for initial cursor/after-cursor positioning, +invocation anchors and forward checkpoint repositioning. Preserve exactness +checks and no-extra-step eager positioning. Update the Go CLI interface/test +double, add shared native controls for partial/full cursors, ties and filters, +and portable reverse controls. Rebuild both commands, run the explicit pinned +v262 query matrix, CLI suites and portable controls; then obtain three internal +whole-SOW final scopes and check CI before publication. No new product decision +or Netdata caller change is required. + +Directional integration initially reproduced six invocation-boundary failures +in the existing oracle controls. Eager backward seeking had already selected +a neighboring eligible row, but the older-invocation walker stepped again. +Apply the existing forward exactness/absence check in both directions: advance +only from an exact anchor; return an already selected neighbor immediately. +Pinned systemd logs-show.c discover_next_id retains its boundary before one +directional step. This is the caller adaptation required by approved1A, not a +new SDK behavior or scope change. Retain the first failing matrix and rerun. + +### Final Directional CLI Validation - 2026-10-26 + +Both commands now consume explicit directional seeking for cursor starts, +invocation anchors and forward checkpoint repositioning. The invocation +exactness correction resolves all six initial failures. Fresh pinned-v262 +matrix: 3,514/3,514 observations pass; the failed first run is retained. +Permanent shared controls add full/partial cursor keys, repeated realtime ties, +boot bounds, absent/incomplete IDs, contradictory components, sparse filtering +and both at/after directions. Native-independent portable controls now pass +146/146, including six reverse cursor forms per command. Parser controls pass +134/134 per language. Go CLI tests and Rust's 28 unit plus 11 subprocess tests +pass. The four oracle-selection tests pass. Prior wiki validation remains valid +for unchanged prose: 16 pages and35 executable examples pass. + +No follow-loop implementation changed; prior live-follow qualification remains +applicable. Source search finds no remaining old eager-forward cursor calls in +either CLI. New calls use native indexes; normal output/Netdata paths gain no +persistent state or per-row work. Windows hyperlink generation stays confined +to hyperlink output, and unmatched glob fallback exits before cwd-relative +resurrection. Explicit oracle selection prevents a host version from silently +qualifying v262. Full internal final review now covers the complete0121 surface +and its SDK prerequisites; publication awaits those verdicts and CI capture. + +### Review Regression Closure - 2026-10-26 + +Three internal final-review scopes pass the complete reviewed surface: CLI +native behavior/cursor consumers; SDK and CLI runtime costs/purity/portability; +and evidence/docs/oracle/CI qualification. Independent checks include both CLI +suites and144 additional native comparisons of cursor/after-cursor/cursor-file, +directions, line bounds, filters and clock-skew directories, including final +cursor-file bytes. No blocking finding remains. The invocation double-step +finding is corrected and covered by the retained failing-then-passing matrix. + +Performance reasoning follows changed work: directional initialization uses +native metadata/indexes; existing SDK methods, reader layouts and Netdata paths +are unchanged. URI allocation is confined to requested Windows hyperlinks; +empty glob results now stop earlier. No new benchmark framework, dependency, +synchronization, source repair or host identity probing is introduced. + +Artifact gate: CLI docs/README, parity spec, oracle-selection runtime guidance, +shared/portable tests and both lifecycle indexes are updated. AGENTS.md and +operator skills require no changes because platform/purity/review rules remain +unchanged and no operator skill is exported. SOW0122 links to the completed +prerequisite and retains only its approved broader performance scope. SDK0159 +is completed in signed3ee6203. No implementation item is silently deferred. +Lesson: eager directional seek callers must distinguish an exact anchor from +an already-selected neighbor before taking another step. Qualify checkpoint +bytes as well as output and ensure every oracle helper uses the pinned binary. + +Sensitive evidence uses synthetic fixtures, upstream pins and ignored logs; +no operational identifiers or credentials enter durable records. Final audit +must report complete and clean before this coherent implementation/lifecycle +commit. Native Windows/macOS/FreeBSD/Linux jobs will qualify the pushed head; +local portability controls do not substitute for those jobs. Existing Bandit +execution failure remains an explicit analysis coverage limitation, not a +runtime defect or a reason for unrelated CI configuration changes. PR12 remains +draft, and no merge or release is performed. + +### Final Table Formatting Correction - 2026-10-26 + +Individual PR-thread adjudication confirms Go table json-pretty used the +standard colon spacing while native uses spaces on both sides. Parsed-JSON +matrix comparison did not establish spacing parity. Reuse the existing native +value formatter for table rows without allocating an intermediate interface +slice. Keep compact JSON unchanged. Add a focused table-format regression and +check pretty structural spacing in native comparisons; rerun affected tests +and review this correction before final closure. This is an in-scope CLI +formatting correction, not an SDK or Netdata change. + +The stronger separator check also exposes the same Rust table path: serde's +pretty printer used standard spacing. Reuse Rust's existing native array/value +formatter through a slice helper, without cloning the table Value array. The +new matrix assertion covers both languages and preserves key-order neutrality. + +The table formatting correction is complete in both languages. The final +strengthened oracle report passes3,514/3,514 observations; Go CLI and Rust28+11 +tests pass again. Rust reuses the existing array formatter with a borrowed +slice, avoiding a cloned Value array. Compact output is unchanged. Both earlier +failed spacing runs remain recorded. Final impact review covers this correction +and lifecycle records before publication. Optional PR-thread suggestions for +harness aggregation, historical ineligible release sources and test-only +cleanup are rejected with concrete scope/evidence, not silently deferred. + +### Publication Analysis Disposition - 2026-10-26 + +Pushed7432b0c native Linux CLI and documentation checks pass. Codacy reports +zero Markdown findings and one B404 import finding in the new oracle unit test, +plus the previously recorded tool execution error. The import creates only +CompletedProcess mock values; every execution path is patched. Add a narrow +inline rationale consistent with existing test tooling, rerun all four tests, +and preserve the separate execution-error limitation. This comment-only +clarification changes no runtime or test behavior. Remaining native jobs were +captured before pushing the clarification; no CI configuration is changed. diff --git a/.agents/sow/done/SOW-0159-20261014-rust-go-sdk-hardening-and-reader-compatibility.md b/.agents/sow/done/SOW-0159-20261014-rust-go-sdk-hardening-and-reader-compatibility.md new file mode 100644 index 00000000..791b132e --- /dev/null +++ b/.agents/sow/done/SOW-0159-20261014-rust-go-sdk-hardening-and-reader-compatibility.md @@ -0,0 +1,3921 @@ +# SOW-0159 - Rust/Go SDK Hardening And Reader Compatibility + +## Status + +Status: completed + +Sub-state: directional cursor prerequisite completed 2026-10-26 after local +validation and three independent internal final review scopes passed. The Go +closed-reader finding was reproduced, corrected and independently retested. +CLI integration resumes in SOW-0121. Historical records remain below. + +## Requirements + +### Purpose + +Improve the robustness and correctness of both SDKs while maintaining or +improving their existing performance and journal-native access paths. + +### User Request + +The user requested two separate SOW workstreams: SDK and journalctl. The user +explicitly requires performance to remain as-is or improve: do not hurt +performance. SOW-0159 owns SDK work; reopened SOW-0121 owns journalctl work. + +### Assistant Understanding + +Facts: + +- Both SDKs can loop indefinitely during ordinary field/value enumeration on + cyclic FIELD/DATA links. Strict verification has separate chain checks. +- Rust silently returns the first frame of a concatenated Zstandard DATA + payload. A synthetic 1014-byte message was returned as 506 bytes with success; + Rust verification rejected its DATA hash. Go returned all 1014 bytes and + verified the same file successfully. +- Both SDKs already bound decompressed DATA output and read unknown-size single + Zstandard frames. These are regression protections to preserve. +- Go rejects opening a file whose final ENTRY header was lost, although three + earlier entries survive. Rust recovered those entries. With a missing later + ENTRY array, Go returned no entries; Rust returned 350 entries before error. +- The binary journal schema, object layouts, and feature-bit assignments are + unchanged between systemd v260 and checked upstream main. Sealing's OpenSSL + migration preserves the encoded state/key/tag contract. +- Existing compatibility verification deliberately accepts an acyclic ascending + FIELD DATA chain that strict index verification rejects. Unconditionally + adopting newer upstream direction checks would change that contract. + +Inferences: + +- Shared traversal safeguards and complete frame decoding address concrete + correctness risks without requiring a file-format redesign. +- Reader recovery can improve access to intact data, but its termination and + diagnostics policy is a product decision rather than a validator cleanup. + +Unknowns: + +- Which bounded chain-termination mechanism preserves the measured hot paths + across all affected reader, Explorer, facade, and writer callers. +- Final public recovery/status acceptance and cost; the policy itself is + resolved by the recorded user decisions. +- Performance of the eventual candidates; diagnostic debug runs are not + release-performance baselines. + +### Acceptance Criteria + +- AC1: both languages terminate safely on self-links, multi-object cycles, + invalid links, and lost chain tails in every reachable affected ordinary API. + Audit hash lookup, FIELD enumeration, unique visitors/stateful enumeration, + directory caches, Explorer/index callers, and writer lookup reuse. +- AC2: Rust consumes complete supported Zstandard streams in DATA reading and + verification. Audit whole-file `.journal.zst` staging and every shipped, + reachable decoder for the same failure. Go passes identical fixtures without + unnecessary decoder changes. +- AC3: shared fixtures cover known/unknown-size single frames, concatenated and + skippable frames, binary/repeated fields, malformed/truncated frames, trailing + garbage, aggregate decompression bounds, and error-buffer cleanup. +- AC4: implement the approved paired crash-recovery contract and demonstrate + forward/reverse iteration, time seeks, exact filters, file/directory input, + and readable-prefix/error behavior on the same crash-truncation corpus. +- AC5: strict index verification, writer-excluded capture, append-open, retention + inventory, and writer-failure guarantees remain strict. Rejected append-open + leaves bytes and extent unchanged. Normal live-growth behavior remains valid. +- AC6: preserve borrowed payload lifetimes, mmap/window behavior, reusable DATA + paths, indexed selection, existing acyclic compatibility allowances, and + synthetic-identity/runtime-purity contracts in both languages. +- AC7: every affected valid-input performance surface meets the performance + gate below. No reproducible regression is accepted, and one case's speedup + cannot offset another case's slowdown. +- AC8: local conformance/interoperability, same-failure scans, artifact updates, + external review or explicit waiver, and the complete-and-clean SOW audit pass + before implementation close. Actual stock versions/features must be recorded. + +## Analysis + +Sources checked: + +- SDK commit `dafcdac4b971d04e48768a1f831974f9252061e2` (`v0.10.0` source tree). +- `go/journal/reader_unique.go:18`, `reader_entry.go:330`, `reader.go:260`, + `reader.go:606`, and `writer_compression.go:148`. +- `rust/src/crates/journal-core/src/file/file_iterators.rs:44`, + `file/file.rs:506`, `file/file_payload.rs:451`, and + `file/object_compression.rs:112`. +- `rust/src/journal/src/verify_graph/io.rs:12` and + `reader_helpers.rs:266` for additional decoder surfaces. +- `go/journal/verify_index_test.go:394`, + `.agents/sow/specs/product-scope.md`, and `rust-reader-performance.md`. +- `.local/sdk-reader-audit-l8whqk5m/sdk-reader-probes.json`: generated synthetic + evidence only; durable summaries above do not depend on retaining this path. +- `tests/benchmarks/README.md`, existing reader/writer benchmark runners, + `tests/interoperability/run_compression_matrix.py`, and `run_verify_matrix.py`. + +Initial investigation state: + +- Fresh Rust/Go debug binaries reproduced both ordinary chain hangs under + 1.5-second child-process timeouts. Both passed the full-message control and + unknown-size single-frame read/verify checks. These runs establish defects, + not performance or full v262 interoperability certification. +- Available PATH stock journalctl is systemd 255.4. Source comparisons use + pinned v260.1/v262/current revisions; a full-feature target-version runtime + oracle remains required for implementation acceptance. + +Risks: + +- Adding full verification or entry scans to ordinary open/query paths would + hide corruption at unacceptable cost and is outside this approved scope. +- Per-row visited sets, extra decompression passes, disabled native decoding, + whole-file copies, or repeated metadata parsing can regress hot paths. +- Recovery can accidentally relax append/retention safety or silently conceal + incomplete data unless its contract is explicit. + +## Pre-Implementation Gate + +Status: design decisions resolved for traversal, streams and default recovery +with normal EOF/status. Implementation resumed on 2026-10-16 after SOW-0108 +passed its gates and signed combined commit; the remaining work below uses +that repaired baseline and preserves strict-path policy. + +Problem / root-cause model: + +- Ordinary index traversals trust link progression; separate strict-verifier + protections do not make those calls terminate. Rust's first-frame sizing and + single-frame fallback do not establish complete stream consumption. Go eager + ENTRY-array loading discards a readable prefix when later references are lost. + +Evidence reviewed: + +- The source paths, synthetic probe results, existing compatibility tests, + live/stable contracts, benchmark surfaces, and upstream references below. + +Affected contracts and surfaces: + +- Both public SDKs/facades, shared traversal consumers, compressed DATA, + whole-file compression where reachable, verification, recovery, live reads, + snapshots, writer lookup reuse, and consumer-visible error/lifetime behavior. +- CLI-only parsing, rendering, grep, and action dispatch belong to SOW-0121. + +Existing patterns to reuse: + +- Journal-native hash/FIELD/DATA chains and entry posting arrays; existing strict + graph checks as evidence, without running that graph pass on ordinary reads. +- Native single-frame Zstandard fast paths, bounded streaming, reusable row + arenas, cold/cached unique-value paths, and current mmap bounds separation. +- Traversal candidate: three-word lazy Brent guard, activated only after a link + departs from normal monotonic progression. It accepts all acyclic orders, + uses no extra reads/allocations, and preserves successful hash-hit early exits. + An exhaustive scalar-model check covered 83,184 rooted functional graphs of + one through five nodes in both directions against a visited-set oracle. + This is correctness evidence only; retained release binaries precede edits, + and per-path release measurements decide whether the candidate is accepted. +- Stream candidate: retain the known-size native attempt unchanged; replace + single-frame fallback with bounded complete-stream decoding and per-frame + checksum/size checks. Reuse one frame decoder. Expose a hidden core helper for + verifier/staging reuse; whole-file staging remains streaming and has no DATA + size cap. Unpublished legacy FFI crates are outside the shipped surface. +- Recovery implementation contract: copyable per-file status contains reason, + first unavailable offset, declared entry count and surviving count. Directory + and facade diagnostics identify files; collections allocate only on explicit + diagnostic requests. Status records observed loss, not a full integrity proof. + Preserve raw header counters. Go retains its existing loaded offset prefix; + Rust derives effective counts/tail from ENTRY-array slots only after a proven + missing-tail access, using native array slots and fixed ENTRY headers. Recovery + views clamp partial final arrays and DATA posting tails; strict object parsing + is unchanged. Refresh invalidates degraded bounds on growth/publication. + On the error path, validate the readable native ENTRY-array slots and fixed + ENTRY headers so an interior hole followed by a readable entry remains an + error. Binary sampling alone cannot establish a contiguous intact prefix. + This cold pass expands no DATA/rows, uses bounded scratch space, and adds no + ordinary valid-file work. Measure its damaged-file cost separately. + Recovery diagnostics report observed loss; they do not turn indexed queries + into full integrity checks. Do not preflight unrelated arrays for every query. + A damaged secondary posting index must return an error when its missing tail + could still contain matching entries inside the recovered global population; + do not silently certify an incomplete filtered result or scan/decode all rows. + Apply the same completeness check before recovery is observed: a positive + DATA posting count requires its inline posting and all declared array slots. + Native array traversal must distinguish a completed count from a missing + required next link. Unused DATA with zero count and zero inline pointer stays + valid. These scalar/error-only checks add no global preflight or row scan. + Exact-filter cursor construction may classify a tail error after its DATA + guard drops and retry once after a changed global prefix is established. + Refresh all retained filter leaves before consuming the final expression; + earlier leaves can predate recovery discovered by a later leaf. Keep failure + retryable. Do not replay an entire controlled Explorer query: emitted callbacks + and shared sampling state cannot be undone. An unknown damaged secondary tail + remains an error there until global loss is observed through reader access. + Wrong types, cycles, misalignment, malformed nonzero sizes, compression errors + and general I/O errors remain errors. Strict verify/snapshot/append/retention + paths explicitly retain strict validation. Prefix computation/status work is + independent of the approved terminal policy. Enable ordinary-reader recovery + with normal EOF and explicit status; preserve strict callers' false policy. + +Risk and blast radius: + +- High for damaged-file termination, silent payload truncation, and recovery + semantics. Changes must cover the actual shared callers in both languages + without absorbing unrelated refactors or writer optimization projects. + +Sensitive data handling plan: + +- Use synthetic identities and repository-local fixtures only. Do not inspect + host journals or private corpora. Keep generated journals, profiles, logs, + dependency caches, and oracle builds under this task's `.local/` directory. + Durable evidence contains sanitized counts, relative paths, and public commits. + +Implementation plan: + +1. Resolve the decisions below, re-ground against the current source, enumerate + shared call sites, and retain release baseline binaries/corpus before edits. +2. Add paired corruption and stream fixtures and missing benchmark cases. +3. Implement the approved bounded chain protection and Rust stream completion + while preserving existing fast paths; cover reachable sibling decoders. +4. Implement the approved ordinary-reader recovery behavior separately from + strict validation. Keep corruption handling bounded and observable. +5. Run correctness, performance, profiling, and cross-language gates; update + artifacts and obtain the required whole-SOW review disposition before close. + +Validation plan: + +- Run `go -C go test ./journal` and the affected Rust workspace package suites, + including public SDK, core, facade, and verifier tests. Use existing local + toolchains and redirect every cache/temp output inside the repository. +- Extend and run compression/verify, directory/mixed-directory, and relevant + live/growth matrices. Add shared ordinary-reader corruption cases rather + than assuming verifier tests cover those APIs. +- Validate sealed output/read/verify against full-feature stock targets. Retain + historical unkeyed, compact/regular, XZ/LZ4/Zstd, and FSS fixtures. +- Apply the performance gate; inspect same-patterns in every reachable shared + caller, then run `.agents/sow/audit.sh` and inspect its complete verdict. + +Artifact impact plan: + +- AGENTS.md: retain native-index, language-parity and purity rules; record + decision 11's approved representative performance-evidence guidance. +- Runtime project skills: update compatibility guidance for adopted recovery, + stream-completion, and corruption-validation procedures when implemented. +- Specs: update product scope and Rust performance evidence for shipped behavior; + planning must not describe unimplemented recovery as current reality. +- End-user/operator docs: update affected reader/recovery/error guides and any + changed performance guarantees after implementation; use docs-authoring skill. +- End-user/operator skills: none currently exist; verify at close. +- SOW lifecycle: this is the sole current/in-progress implementation after + prerequisite SOW-0108; keep SOW-0121 paused until this SDK SOW closes. +- SOW-status.md: update canonical and root indexes at preparation and each state + change; commit implementation/lifecycle completion together at final close. + +Open-source reference evidence: + +- `systemd/systemd @ c0a5a2516d28601fb3afc1a77d7b42fcfe38fced` (`v260.1`): + `docs/JOURNAL_FILE_FORMAT.md`, `src/libsystemd/sd-journal/journal-def.h`. +- `systemd/systemd @ 7f073ff56f14e9b6ff7052998b640df88a359358` (`263~devel`): + `src/libsystemd/sd-journal/journal-file.c:583`, `:1004`; + `src/basic/compress.c:841`; `src/test/test-fsprg.c:303`; `NEWS:422`. +- `systemd/systemd @ 62c5c888f434067683e72f678b1bb995873f275b`: + `src/basic/compress.c`, unknown-size/streaming Zstandard decoding. + +Open decisions: + +1. Resolved on 2026-10-16: the user authorized read-only external review after + complete local validation, including later rounds after material fixes. + The prerequisite and subsequent whole-SOW reviews are complete, with reviewer + coverage gaps explicitly recorded below. +2. Resolved on 2026-10-16: the user requires Go to have complete parity with + Rust for the ordinary-filter decision. Implement native indexed Go filtering + with equivalent Boolean, positioning, refresh and failure behavior; no + recovered-only workaround or language-specific error exception is selected. + Preserve the existing performance gate and language-idiomatic API surfaces. +3. Resolved on 2026-10-17: repair reverse-filter skipped matches in both SDKs, + returning the nearest preceding match after filter installation, forward + EOF and direction changes. The user approved the long-term-best outcome. +4. Resolved on 2026-10-17: retain the last successful position in both SDKs + after a nonrecoverable selected-entry load error. Repair followed by + iteration must retry the failed candidate instead of skipping it. +5. Resolved on 2026-10-17: use binary posting search for explicit filtered + realtime seeks in both SDKs, reporting errors from headers/slots actually + visited. Ordinary sequential posting traversal remains unchanged apart + from the separately approved reverse-position correction. Performance + acceptance is still required; no slowdown exception was authorized. +6. Resolved on 2026-10-17: performance evidence uses engineering judgment, + representative high-level tests and careful review, with targeted + measurements for credible costs. Do not mechanically expand a large + framework or block every uncertain timing interval. See decision 11. +7. Resolved on 2026-10-17: user selects 1A for Rust counted-array history. + Keep Node at four words (32 bytes on 64-bit targets), replace cached next + with the root and read next at array transitions. See decision 12. +8. Resolved by the existing product-scope cache policy: apply the required + journal-engine cache generation bump for changed native index results. + This introduces no new cache architecture or performance waiver; the + one-time migration and unchanged hit path are recorded below. + +Resolved decisions: preserve current acyclic compatibility and strict-verifier +separation; execute SDK first, then the CLI SOW with complete v262 behavior. +Recover intact prefixes by default, end with normal EOF and expose status. +Repair prerequisite SOW-0108 first while preserving this working change set. +Local performance acceptance is recorded below under decisions 11 and 12; +independent whole-SOW review is adjudicated below, including coverage gaps. + +## Implications And Decisions + +1. Approved on 2026-10-14: two separate workstreams, both Rust and Go, with SDK + scope here and journalctl scope in originating SOW-0121. +2. Approved hard constraint: preserve or improve performance. This overrides + any generic permission to accept a documented slower path for this work. +3. Recommendation classification: long-term-best, implemented minimal-complete. + Fix demonstrated gaps and their reachable consequences; broader performance + redesign, dependency refresh, format redesign, and refactoring are separate. +4. Existing baseline remains v260.1; newer source/fixtures supplement it. No + compiler-floor increase, CGO, system journal linkage, implicit host identity, + automatic writer lock, or core subprocess execution is authorized. +5. Sealing backend replacement is not required by the observed format changes. + Preserve interoperability and test it rather than porting an OS dependency. +6. Approved on 2026-10-15: complete both workstreams sequentially, SDK first. + Recover intact entries by default and expose recovery status. Iteration's + terminal signal was clarified by decision 8 below. +7. Approved on 2026-10-15 for SOW-0121: advance the complete CLI contract to + systemd v262. SDK binary-format baseline remains v260.1, with newer stream + and crash-recovery behavior covered here. +8. Approved on 2026-10-15: recovered SDK iteration ends normally after returning + the intact prefix; callers inspect recovery status for observed tail loss. +9. Approved on 2026-10-15 for SOW-0121: native PCRE2 in the CLIs only, including + a Go CGO exception, subject to the unchanged performance gate. SDK runtime + dependencies and purity remain unchanged. +10. Approved on 2026-10-15: pause this SOW, repair the original SOW-0108 Explorer + lifetime regression first, preserve current work and measure allocation + costs, then resume this SOW and SOW-0121 sequentially. +11. Approved on 2026-10-17: use engineering judgment for performance acceptance. + Representative high-level tests and careful code review may establish that + a change does not negatively affect performance. Do not build or expand a + large measurement framework to prove every small detail. Preserve or improve + performance, investigate credible observed regressions, and use targeted + measurements where code review or existing evidence leaves a real concern. + The earlier self-authored requirement to block every inconclusive timing + comparison is superseded; statistical uncertainty is not proof of equality + and is also not, by itself, evidence of a regression. +12. Approved on 2026-10-17: keep Rust Node at 32 bytes on 64-bit targets, + replacing cached next with the starting array offset. Accept the current + header lookup at array transitions; preserve within-array row movement + and validate representative performance. Root ordering state uses a spare + alignment bit, avoiding per-row history allocation or a larger cursor. + +## Plan + +1. Approve contracts and measured candidate designs; freeze baseline evidence. +2. Close chain/stream gaps with shared tests and per-path performance evidence. +3. Deliver the selected recovery outcome and cross-language validation. +4. Review, maintain artifacts, audit, and close only after all gates pass. + +## Performance Gate + +- Capture baseline before runtime edits using release binaries, pinned compiler + and dependency versions, an immutable synthetic corpus, identical output + checksums, format/compression/cardinality, mmap/window strategy, bounds, + publication cadence, and host conditions. Record baseline variance. +- Require unchanged or improved affected valid-input behavior. There is no + permitted reproducible slowdown percentage or cross-case averaging exemption. + Apply decision 11: combine careful code review, representative high-level + controls and retained measurements; do not require every possible case or + every uncertain timing interval to demonstrate a statistically significant + speedup before accepting the change. +- When measurement is needed, reuse existing runners, alternate baseline and + candidate with warmups, and retain samples and semantic counters. Measure the + quantities relevant to the actual change, such as elapsed cost, allocations, + work counts or syscalls. Investigate credible adverse observations with + targeted controls or profiles. Do not expand a framework mechanically or + claim exact equality from an interval that crosses one. +- Cover FIELD catalogs, low/high-cardinality unique visitors and cached restart, + exact indexed filters, forward/reverse payload scans, reused compressed DATA, + live/snapshot and file/directory inputs, snapshot visits, and verification. + Add writer/Explorer cases when shared lookup or decoder changes reach them. +- Preserve zero uncompressed Rust row allocations after warmup, Go allocation + baselines, borrowed slices, native single-frame decoding, reusable buffers, + and requested-scope decompression. Do not add full ordinary-open verification, + entry-scan substitutes, per-row visited allocations, or whole-file buffers. +- Reuse `tests/benchmarks/run_reader_core_benchmarks.py` for regular/compact, + forward/backward, file/open-files production surfaces. Extend existing drivers + for field/unique enumeration and compressed fixtures: those surfaces are not + all measured by the current runner. +- Reuse existing Go/Rust benchmarks when they answer a concrete remaining + question. Shared writer changes need a blast-radius review; run targeted + writer controls with matching publication settings when a credible cost + remains unresolved. +- Compare equivalent work against the fastest sibling and stock where practical; + record versions and reasons for unavailable/non-equivalent comparisons. A + previously truncated payload or omitted row is not a valid faster baseline. +- Render available before/after reports through + `tests/benchmarks/report_benchmarks.py`. Every measurable regression blocks + acceptance; redesign and rerun instead of silently relaxing the requirement. + +## Implementation And Review Plan + +Implementation: + +- One project-manager writer, sequential edits, paired language outcomes, and + shared fixtures. Approved public default recovery and normal EOF/status are + enabled and paired validation is complete as recorded below. +- SOW-0156 retains the broader fixed-header audit, SOW-0155 the unconfirmed + large-writer-offset concern, SOW-0137 residual writer optimization, and + SOW-0098 legacy-core consolidation. Avoid unrelated changes to those scopes. + +Reviewers: + +- Standing read-only external review is authorized on 2026-10-16 after complete + local implementation/validation, including rounds after material fixes. Use + the approved six reviewers and system-wide reviewer skill; do not infer + acceptance here from the separate prerequisite's reports. + +Repository boundary block for every external-reviewer prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Failure handling: + +- Record correctness, performance, review, and audit failures. Do not close, + weaken strict paths, narrow compatibility, or change the approved design + without resolving the evidence and any required user decision. + +## Execution Log + +### 2026-10-15 + +- The user authorized implementation of both SOWs as one goal, sequentially. +- Recorded default recovery/status and CLI v262 decisions. Started release + baseline capture before runtime changes; no compiler-floor or native-library + policy changes are authorized. +- Recovery validation exposed a Rust seek discrepancy in `file/cursor.rs`: + reverse realtime seeks used the forward partition, and forward seeks beyond + the last row returned the tail. Repair the shared native-array partition in + this scope so recovered and intact files both select the first timestamp at + or above the request forward, or the last at or below it backward. Test both + filtered/unfiltered directions and out-of-range requests; no row scan is added. +- Initial release A/B discovery used retained baseline/candidate binaries on + identical 100,000-row regular/compact corpora, alternating order, one warmup + and three measured pairs. Checksums matched, but Rust core paths flagged + roughly 5-13% slower cases and some SDK/Go cases also flagged slowdowns or + noise. These are rejected discovery results, not performance acceptance. + `Node` had grown from four to seven words, copying chain state on every row. + Restore its original size; only a non-increasing ENTRY-array link invokes a + cold stack-local bounded chain check. Normal arrays still follow their direct + native path. Keep lazy guards on FIELD/DATA iterators, and move unordered + bookkeeping/recovery retries off the common successful path. Remeasure every + affected case; do not offset one slowdown against another improvement. +- Built an isolated full-feature `systemd/systemd @ + 8cc40e0c5e9234bf45084751ac53b1fbfe70b492` v262 oracle. Its journalctl reports + `+OPENSSL +PCRE2 +LZ4 +XZ +ZSTD`. Sources, private header extraction, builds, + dependency downloads and logs remain under the task's ignored `.local/` + directory. No system installation or service change occurred. +- Added matching public SDK `inspect-reader` probes to the existing conformance + adapters, plus `tests/interoperability/run_reader_hardening.py`. Its first + chain/stream run passed 120 SDK observations over the same regular/compact + corpus. It exercised self/multi-object cycles, ascending acyclic FIELD DATA, + DATA-hash lookup, ENTRY arrays, known/unknown/concatenated/skippable/empty-first + Zstandard frames, corrupt later checksums, truncation, trailing bytes, output + caps, binary/repeated fields and whole-file staging. All 20 applicable stock + v262 verification checks agreed. Public recovery cases remain gated until + terminal policy is selected and default recovery is enabled. +- Focused private recovery-enabled tests pass forward/reverse/time/exact reads, + raw counter preservation, same-size and physical repair, pending directory + merge rows, malformed/interior holes and arena bounds. Rust additionally + retained the historical no-RTC population of 10,757 entries. Full suites are + not accepted: the current deliberately unwired public recovery policy leaves + one historical Rust drain failure and one Go directory-fixture failure. +- Retained alternating before/after measurements and standard reports. The + ten-pair CPU-pinned diagnostic still flagged Rust core-next and several + compact/unique cases; candidate acceptance remains failed. Kernel policy + rejected unprivileged hardware profiling; no kernel setting was changed. + Installed Callgrind profiling found 1.213B baseline versus 1.163B candidate + instructions for core payloads, with similar simulated cache misses. These + counters do not explain native timings or waive the performance gate. + Removed an avoidable per-row population check from `JournalCursor`; empty + population checks now occur when resolving positions. Remeasure this change. +- DWARF layout inspection found that the new cold recovery state shifted Rust's + header map from byte 376 to 456 and window manager from 392 to 472. Move the + cold state after existing mappings and recheck layout/timings. This is a layout + candidate, not a claim that it explains the regression. +- Same-pattern investigation added sparse posting coverage: a known lost ENTRY + beyond the recovered population is safe to omit even when that DATA value + does not occur in the final surviving row. An unreadable posting segment with + unknown targets remains an error. Share the recovery-aware DATA-header cursor + construction between exact filtering and Explorer facets/histograms so a + missing array pointer cannot be silently converted into an empty result. +- Paired sparse-value and missing-array tests now pass, including strict index, + snapshot capture, and append rejection. The Rust cold-state layout candidate + restored the original header/window-manager offsets in emitted DWARF. Run + alternating measurements before attributing any timing change to layout. +- Preserve the approved traversal checks while removing duplicate work in the + measured unique path: return the matched FIELD's DATA head from the existing + bucket visitor, and reset only the cycle guard's inactive marker. Initialize + the remaining guard words on its cold activation path. These implementation + candidates still require functional and before/after performance validation. +- Restored layout alone did not remove the measured regressions. Disassembly + found a new out-of-line DATA constructor staging a 48-byte object on the stack + and copying it into the return guard, whereas the baseline constructs its + return directly. Inline that constructor without removing type, size, arena, + alignment, or zerocopy validation; measure the resulting binary separately. +- DATA inlining removed its out-of-line temporary and improved measured core + payload scans by about 5% regular and 13% compact. Some cursor-only cases + still regressed. The remaining 75-byte ENTRY and ENTRY_ARRAY constructors + stage analogous 40-byte temporaries. Apply the same targeted inlining there, + keep their checks, and remeasure rather than accepting the residual slowdown. +- Cursor-only timing remains unresolved after constructor inlining. Emitted + code showed that `OffsetArrayObject::capacity` still stages an array and + repeats variant/alignment/length extraction already performed to read its + slot. Inline this small getter so the compiler can share that work. Retain + recovery dispatch and zero-slot checks; inspect and measure the candidate. +- The capacity candidate removes reproducible regressions in the current + targeted ten-pair run, but several confidence intervals remain inconclusive. + Recheck those cases with twenty pairs/twenty child measurements per phase; + do not treat absence of a flag as a performance pass. +- Allocation probes compare the same warmed payload scans on both immutable + corpora: Rust baseline/candidate each recorded zero allocations in all three + measured passes. Go matched its baseline at five allocations per complete + regular scan and zero per compact scan. Each scan returned 100,000 rows, + 3,200,000 fields and 81,700,000 payload bytes. This is allocation evidence, + not a timing or general performance waiver. +- Updated the product/performance specs and reader/language/snapshot wiki pages + for complete streams and native-index errors. The wiki structure validator + passed all 16 pages. Public recovery docs and executable-example validation + remain required after the terminal contract is selected. +- Mid-implementation `.agents/sow/audit.sh` reported complete and clean. Repeat + after final artifacts, review disposition and lifecycle updates before close. +- Followed disassembly evidence for two remaining duplicated-work candidates: + inline the new FIELD-head visitor, and read each traversed next-link once + before both cycle checking and iterator advancement. This avoids repeated + zerocopy header validation. Internal cursor materialization uses a required + nonzero-offset getter to remove an emitted second zero test; the public + Option-returning getter and the actual zero-slot validation remain intact. + Functional and retained-binary timing validation are still required. +- Added directory-catalog benchmark coverage to the Rust helper, matching Go's + existing directory surface. Retained comparison tooling also supports + directory inputs and content/name fingerprints, including hardlinked fixture + files without repeatedly hashing the same inode. Rebuild matching helper + sources against both frozen baseline and candidate SDKs for that comparison; + retain the existing file-run binaries unchanged. +- Public API impact check found that the new `JournalObject::EXPECTED_TYPE` + member must have a default to preserve downstream custom implementations. + Make the hint optional/default absent, and set it for every built-in object. + Built-in typed reads keep their wire-type check; custom implementations keep + their previous contract. The new public `CyclicObjectChain` error variant + needs an unreleased migration note for exhaustive Rust enum matches. +- Expanded public chain/stream/facade/directory tests passed 232 observations + and 20 stock v262 verification checks. Fresh-open malformed posting tests + also pass in both SDKs. Added cold exact-filter recovery after releasing the + DATA guard and refreshed retained filter leaves before consuming them; + all ten focused Rust recovery tests and Go's paired suite pass. +- A full release comparison stopped at a reproducible crash in the unchanged + Go Explorer baseline. Its completed SOW-0108 lifetime claim is contradicted; + the prerequisite scope question remains open. The benchmark runner now saves + raw output and completed samples/comparisons incrementally, so later failures + do not lose the earlier evidence. +- Compact diagnostics flag several forward Rust payload/facade cases 2-4% + slower. The candidate helper also adds directory-catalog code, so rebuild + matching helper sources for the controlled comparison. Read-only disassembly + found no extra runtime Option/type-hint test, but forward `Cursor::next` now + saves two more registers than the baseline. Reverse cursor and payload + visitor instructions are unchanged. Outline the rare array transition to + isolate its checks from same-array movement, then inspect emitted code and + measure it. Preserve every missing-link, cycle and bounds error. +- Go's compact high-cardinality visitor also regressed in the diagnostic run. + Retained CPU profiles show time concentrated in DATA-header/accessor parsing; + compiler diagnostics confirm the guard is inlined. Recovery fields had been + inserted before the reader's cached layout/row state. Move that cold state to + the struct tail, restoring the existing hot-field positions. This is a layout + candidate, not a proven cause. Measure it on both corpora with matched helpers. +- A traced Rust compact snapshot payload scan returned identical complete + counters and exactly 124 syscalls before/after, including 22 mmap calls, + 11 munmaps and 7 statx calls. This supports unchanged syscall behavior for + that case only; traced timings are not latency acceptance evidence. +- Rebuilt private SDK-backed CLI/writer integration tools and target-version + C readers. Stock v262 closed-file checks pass: 32 general, 72 DATA-compression, + 20 compact and 18 binary-field checks. All 18 live writer/feature cases pass + across both writers and stock/Rust/Go polling readers, including active + libsystemd readers, all 100 final entries, structure and sealed-key checks. + These are SDK integration checks, not full v262 CLI acceptance. Final public + recovery wiring still requires a fresh complete validation pass. +- Directory/mixed-directory matrices also pass against stock v262. Whole-file + compressed source fixtures were retained outside the watched directories + rather than deleted; fixture generation and SDK reader paths are unchanged. +- Matched-helper measurements accept neither residual small Rust SDK slowdowns + nor inconclusive cases. Inspection found a full `Location` checkpoint copied + before every SDK row, solely for cold selected-ENTRY failure handling. Keep a + scalar prior resolved offset on successive rows; copy a full anchor only for + nonresolved seek positions. Reconstruct the exact prior Location on error. + This preserves filtered EOF getters, direction changes and live repair. + Rejected the alternative of anchoring recovery at the global surviving tail: + a sparse filter can end on a different matching row, so that shortcut changes + post-EOF getters and can return a duplicate when reversing. Add that concrete + counterexample to the focused tests before measuring the compact checkpoint. +- The compact-checkpoint test passes with all eleven focused Rust hardening + tests. Independent read-only inspection confirms that every Location variant + reconstructs the same pre-error anchor as the full checkpoint; this is source + correctness evidence, not performance acceptance. +- The completed matched-helper run has 36 comparisons: 14 improvements, + 18 inconclusive results and four compact Rust regressions. The flagged + payload/facade cases are approximately 0.9-2.3% slower. Both Go high-cardinality + visitor cases improve after restoring the reader's hot field layout. The + compact-checkpoint candidate is not included in those measurements. +- A subsequent current Rust build failed because the archived baseline and + current checkout shared a Cargo target directory; Cargo reused baseline core + artifacts while rebuilding the current SDK. Keep the retained baseline + binaries immutable and use a separate target directory for current builds. + The failed build is not validation evidence and did not alter source or + dependencies. Rebuild current tools in the isolated target before measuring. + +### 2026-10-14 + +- Prepared the SDK scope after source inspection and isolated synthetic probes; + recorded the user's language-parity and performance requirements. +- No SDK, dependency, public spec, or runtime implementation changed. + +## Validation + +### Explorer Recovery Ownership Qualification + +SOW-0108's prerequisite repair has an early-owned-key shortcut for ordinary +all-candidate views whose visible ENTRY count equals the declared count. Its +standalone walker always visits a positive inline posting. This SDK SOW adds +recovered posting clipping, so equal counts alone cease to prove contribution: +missing unused ENTRY_ARRAY capacity can leave every declared ENTRY visible while +recovery still suppresses a DATA object's postings. + +The implemented Go facet shortcut checks actual recovery state, rather than +global count equality alone. Degraded views retain count-first ownership; +reader_recovery_facet_test.go covers the equal-count/incomplete-array case +with postings outside the recovered population. This qualification belongs +here rather than the prerequisite SOW-0108 patch. Results/error semantics are +unchanged. No unconditional zero-unmatched-allocation claim is made for +recovered reads. + +This section records current acceptance evidence. Dated execution sections +below retain earlier failures, open questions and rejected experiments as +history; the final dispositions here and at the end supersede those states. + +Acceptance criteria evidence: + +- AC1-AC3: paired traversal guards and complete Rust stream decoding are + implemented locally. The expanded shared chain/stream corpus passed 232 SDK + observations and 20 applicable stock v262 verification checks. Exhaustive + small-graph tests also cover reset/reuse of the lazy cycle detector. + Added fresh-open exact/Explorer checks for positive posting counts with lost + inline/head/next pointers, plus missing required global-array links, facade + calls and directory catalog paths. Short counted-array controls also cover + descending-prefix revisits, public Rust Nodes, reconstructed cursors, valid + backward links, unused terminal links and writer rejection before mutation. + Both regular and compact layouts pass. +- AC4: default intact-prefix recovery, normal EOF and explicit status are + enabled in both public SDKs. Shared probes cover file/directory SDK/facade + reads, sparse posting tails, repair/retry and unknown-target errors. Complete + nearest-match and error-position decisions are approved and implemented, + including directory filter mutation, native sequential discovery and repair/retry. Private-policy-only wording in the dated + history is historical. Whole-SOW independent review is adjudicated below. +- AC5-AC6: full language suites retain strict verification/capture/append-open, + raw counters, malformed/interior-hole rejection, lifetime, retention and live + growth protections. Both compiler floors and Go's additional targets pass. + The final v262 verifier/directory/mixed checks pass 63/34/42 observations. + Earlier 32 general, 72 compression, 20 compact, 18 binary, 18 live-feature + and 552 shared hardening observations retain their original provenance; + later equivalent position/constructor/guard edits do not change their + writer, codec, FSS or publication algorithms. They are not relabelled as + runs of a later binary. +- AC7: accepted locally under decision 11, including the final counted-array + controls for decision 12. Eight final Rust cases give three improvements and + five inconclusive intervals, with no regression flag. Both focused final Go + writer cases are inconclusive with no regression flag. Four-word Nodes, + unchanged within-array work and reviewed allocation/native-I/O paths provide + the accompanying engineering evidence. This is not a claim of exact equality. + Earlier evidence retains its original source provenance. + All 32 explicit filtered realtime comparisons improve on the original SDK + baseline; the intentionally interrupted broad stage retains 28 completed + comparisons without an adverse flag. Representative filter setup, + ordinary materialization/visitation, unique enumeration, directory, + compression/staging and writer controls are combined with native-path, + allocation/work-counter and generated-code inspection. Credible Rust + setup/ordinary and Go entry/writer costs prompted equivalent cost removals; + the final targeted repeats do not reproduce those flags. Earlier adverse + blocks remain recorded. No case's speedup offsets another's slowdown, and + uncertain intervals are not treated as proof of equality. Exact writer + observations, final source provenance and limits are recorded below. +- AC8: local validation, docs/specs, artifact reconciliation and whole-SOW review + disposition are complete. The closing complete-and-clean audit and signed + combined implementation/lifecycle commit accompany this completed record. + +Tests or equivalent validation: + +- Rust's supported 1.91 floor workspace passes, including 200 public SDK and + 98 core tests; the allocative-feature core also passes 98 tests. Final-source + evidence is external-findings-fatcvrlo/rust-floor-validation. Existing + ignored documentation examples remain recorded. This covers default recovery, + frame streams, strict paths, lifetimes, counted cycles and repair/retry. +- Go's supported 1.26.2 all-package suites pass 1,995 amd64, 1,996 386 and 1,995 + race test events (parent/subtest nodes, not distinct leaves), excluding only + the unchanged TestGoWriterLiveStockReadersStress. The retained isolated + 1,200-entry/fsync-per-entry test passes in 38.03 seconds with its original + 45-second deadline; an earlier full run's deadline failure remains a failure, + with contention only an unproven working theory. After the final internal + directory call correction, 170 directory/recovery/cycle events pass on each + of amd64, 386 and race. The final directory mutation and lazy-discovery + controls are included in all three final all-package runs; focused final + Go directory checks also pass 306 events. Evidence: external-findings-fatcvrlo/ + go-floor-validation. Earlier Windows/macOS/FreeBSD builds retain their + provenance; subsequent common-code edits are checked on 386 as well. +- Final release-build-8 tools pass 552 shared SDK observations and stock-v262 + verification/directory/mixed matrices (63/34/42). The exact source/binary + hashes, commands and results are retained in internal-review-go-floor, + internal-review-rust-floor-2, directory-transition-floor, release-build-8, + shared-validation-8 and final-integration-8 under the task's ignored workspace. +- The required generation-4 engine migration passes Rust 1.91 index/engine + package suites and existing cache-key migration tests. The new archived + indexer test covers regular/compact missing links and short cycles with a + healthy 13-row control. The original SDK executable instead succeeds with + four rows for the regular missing-link case; it stops at that assertion, + so no stronger baseline-execution claim is made. Evidence: cache-migration. + The final requested-field test covers 20 native-chain/pointer failures plus + two absent-map facet targets, with actual engine cache omission and a healthy + sibling. Before repair, its first cycle case incorrectly succeeds; the + added absent-map control separately reproduces the inherited panic. After + repair all cases pass, including the complete final index/engine suites. +- After the unreleased cache upgrade note, all 35 verified examples pass again, + and the static wiki validator passes all 16 pages (docs-validation-8). +- Earlier pre-enablement historical-fixture failures and the isolated retention + precondition failure remain retained with their dispositions. Passing later + tests do not erase them or the final amd64 stress deadline failure. +- Release A/B runs use immutable 100,000-row regular/compact journals, frozen + pre-change SDK binaries, alternating order, warmups, exact row/field/byte + checksums, retained raw samples and case-specific repetitions recorded in + each manifest. The final targeted reader block uses seven measured pairs; + writer controls use ten. CPU-pinned diagnostic + runs use five child measurements per phase. Standard benchmark summaries and + reports are retained under the task's ignored `.local/` directory. +- Preparation validation on 2026-10-14: `git diff --check`, explicit whitespace + checks including new files, gate/state/index checks, and the original CLI + narrative-preservation comparison passed. `.agents/sow/audit.sh` reported + `SOW initialization complete and clean`; its durable-artifact scan found no + sensitive-data patterns. + +Real-use evidence: + +- Public SDK reader/verifier paths were exercised through file-backed commands + on synthetic fixtures. The isolated stock v262 oracle has OpenSSL, PCRE2, LZ4, + XZ and Zstandard enabled. No host journal or production service was used. + +Reviewer findings: + +- Read-only investigations confirmed traversal/decoder gaps, recovery-policy + isolation, sparse-posting boundaries and code-generation differences. Fixed + the confirmed sparse-tail rejection and missing DATA-array cursor ambiguity; + paired regression tests pass. These investigations are not final whole-SOW + implementation review approval. +- An independent read-only planning consistency check found no blocking scope, + performance-gate, lifecycle, or decision-recording flaws. Runtime design + choices are approved as recorded. Standing whole-SOW external review was + executed after complete local validation as recorded below. The prerequisite's + separate reviewer reports do not substitute for this SDK review. +- The first whole-SOW SDK round returns two positive reports and one + NEEDS CHANGES report; three reviewers have no verdict after permitted + retries. Both verified blockers are reproduced and repaired, with the full + primary audit and final floor/integration/docs checks passing again. Its + verdicts do not accept the materially repaired candidate. The fresh complete + round returns Minimax PRODUCTION GRADE; GLM, MiMo, Kimi and DeepSeek reach + their execution limits without verdicts, and Qwen returns HTTP 429 twice. + All notes are independently adjudicated; no verified blocker remains. These + five coverage gaps are retained, without inventing consensus or approval. + Exact dispositions and the decision against another long retry batch are below. +- Historical prerequisite, now resolved by signed commit 4c49205: the frozen + pre-repair Go Explorer benchmark crashed + on the valid 100,000-row regular corpus. Independent inspection traced a borrowed facet value across + posting reads that can unmap its storage; histogram has the same pattern. + `go/journal/explorer.go` copies the value only after those reads. Commit + `a8119457` introduced rolling temporary slices under completed SOW-0108. + The original SOW explicitly claimed Explorer lifetime preservation. A scope + clarification was answered with repair-first execution. SOW-0108's combined + implementation/review/validation/lifecycle commit precedes this resumption; + its reports are not substituted for this SDK SOW's final review. + +Same-failure scan: + +- Guards cover FIELD/DATA hash lookup, FIELD/DATA iteration, global/posting + arrays, Explorer and shared writer lookups. Recovery-aware DATA cursor + construction is shared by exact filtering and Explorer facets/histograms. + Strict constructors were inspected independently in both languages. Final + shared-caller scans cover strict constructors, decoder callers, directory + merging and retry, copied-window invalidation, post-Close access and both + ordinary and explicit-time posting paths. Directory mutations and Go mode + changes preserve proven sequential state; representative later-file damage + controls prove nearer rows and repair/retry remain usable. Strict index + consumers propagate encountered required chain/posting failures; an absent + global-map target returns an error on its existing branch. Inherited source + map membership/captured-tail analysis is separately tracked in SOW-0161. + Existing lifetime/live-growth + suites and shared fixtures validate those surfaces. Unfiltered Go refresh + and realtime-probe issues remain separately tracked in SOW-0045. + +Sensitive data gate: + +- Changed source, tests and tracking text use synthetic identities, counts, + relative evidence and public commits. Generated journals, logs and profiles + remain ignored under `.local/`. The final changed-artifact scan is clean; + closure text and the moved SOW are included in the closing audit. + +Artifact maintenance gate: + +- AGENTS.md: performance-evidence guidance now records decision 11's use of + careful review, representative tests and targeted measurement. Existing + purity, native-index, no-regression and review requirements remain in force. +- Runtime compatibility skill: updated with default recovery, normal EOF/status, + fresh-query, callback-lifetime, filter position/time and directory retry + controls and decision 11's representative performance-evidence workflow. +- Specs and end-user/operator docs: reader, compression and native-index error + contracts are updated and labelled unreleased where applicable. Default + recovery/status and normal EOF are documented. All 35 verified examples and + 16 wiki page checks pass. The final cold guard changes no documented API. +- End-user/operator skills: none are installed for this project. +- SOW lifecycle: completed and moved to done with the implementation in the + same signed commit, after prerequisite SOW-0108's signed commit 4c49205. + SOW-0121 remains separately paused until this commit succeeds; its activation + and runtime changes belong to its own sequential implementation scope. +- SOW-status.md: canonical and root summaries reflect completed SOW-0108 and + SOW-0159, then paused SOW-0121, with decision 11 and final review coverage. + +Specs update: + +- Updated: product reader contract, Rust hot-path/decompression contract, + snapshot byte-extent wording and required engine cache generation 4. + CLI v262 specs remain owned by SOW-0121. + +Project skills update: + +- Updated: paired damaged-file corpus, bounded cycle checks and + ordinary-recovery versus strict-capture validation workflow. + +End-user/operator docs update: + +- Updated: Reader APIs, language API pages, Indexed Snapshots, reader options + and native-index error propagation, plus the unreleased generation-4 cache + migration. All 35 examples and 16 wiki page checks pass; the final equivalent internal cost removals change no example/API. + +End-user/operator skills update: + +- No output/reference skills exist; the final skill catalog confirms that no + external consumer skill is affected. + +Lessons: + +- Safe strict verification does not prove safe ordinary index traversal. +- Single-frame decoder success does not prove complete logical DATA consumption. +- File-level position tests do not establish directory mutation semantics; a + replay expectation can encode the original bug. Preserve native sequential + discovery as well as the returned row sequence. +- New low-level errors require auditing consumer catches and cache insertion. + +Follow-up mapping: + +- SDK defects, reachable consequences, and performance acceptance belong here. +- CLI defects belong to SOW-0121; general CLI scalability remains SOW-0122. +- Pre-existing healthy EOF/refresh, directory seek/getter state and unfiltered + realtime-probe error suppression remain in paused SOW-0045. +- Unchanged Go XZ dictionary allocation remains in pending SOW-0160. +- An inherited source-map membership/captured-tail lead is tracked in pending + SOW-0161. It generates no new traversal error to propagate; source predictions + remain unexecuted and do not justify adding certification to healthy indexing. +- The narrow Rust constructor identity optimization is implemented here. + Broader cursor/getter experiments, decoder outlining and the private Go + hash-lookup terminal-zero relocation are rejected from this delivery; + their evidence is retained. No unimplemented optimization is required for + the accepted outcome or left as an untracked promise. + +## Outcome + +Traversal/stream repairs, default public recovery/status and native filtering +are implemented and validated locally in both languages, including approved +position/time and directory retry corrections. Performance acceptance uses +the user's approved representative evidence and careful review, with final +targeted controls after equivalent cost removals. The final whole-SOW report +is positive, every note is adjudicated and five reviewer coverage gaps remain +explicit. Local validation and artifact maintenance pass. This completed record +and implementation are committed together after the closing audit. Publication +and the separately approved journalctl implementation are outside this SDK commit. + +## Lessons Extracted + +Measure the real production path and verify complete output before interpreting +performance. Keep ordinary compatibility and strict integrity policies separate. + +## Followup + +No additional SOW is required to close this work. All valid remaining items map to +SOW-0121, SOW-0122, SOW-0045, SOW-0160 or SOW-0161 as recorded in Validation. Rejected +optional experiments are not implementation requirements. + +## Regression Log + +No regression against this completed outcome has been reported. Reopen this +SOW and append a dated regression if later evidence contradicts its outcome. + +## Prerequisite Measurement - 2026-10-15 + +A separate healthy-positioning finding is tracked in reopened/paused original +SOW-0045. Unchanged HEAD skips an append after empty/EOF Refresh or a unique +query that refreshes offsets. The approved normal EOF/status decision here +concerns recovery termination; do not silently absorb that independent state +machine repair. Preserve this SOW's recovery-specific positioning validation +and cross-check interactions when implementation resumes. + +SOW-0108's isolated header analysis identifies a required performance correction +in this paused implementation: `go/journal/reader.go` currently acquires the +16-byte ENTRY-slot header with unconditional `readSlice`. A header crossing two +already-cached windows then takes an extra positioned file read, unlike the +previous chunk-copying path. In one matched primitive comparison, mmap crossing +headers take 740.90 ns versus 58.06 ns for the original path. Removing heap +allocation alone does not satisfy the performance contract. + +The user selected option `1a`, hybrid native slices plus an owned buffer, for +the active prerequisite on 2026-10-15. Its complete reader performance gate +remains required. When this SOW resumes, use the approved acquisition strategy while +preserving this SOW's separate bounds/ENTRY-shape/zero-header/recovery checks. +Do not transfer those additional checks into the prerequisite repair. + +Add warmed crossing headers, non-power-of-two/tiny/default windows, retained +rows, closed readers and empty/small/large opening allocation costs to this +SOW's Go performance validation. The private primitive checks pass 64 cases; +they do not certify the full paused SDK or its reader-construction cost. This +note tracks dependent work only; SOW-0159 remains paused and no runtime source +is changed by this update. + +## Review Authorization - 2026-10-16 + +The user selected option A, granting standing read-only external-review +authorization for SOW-0108, this SDK SOW and SOW-0121 at their complete local +validation milestones, including further rounds after material fixes. Use the +six named reviewers and the system-wide reviewer skill. Earlier unauthorized +review notes describe the prior preparation checkpoint and are superseded by +this decision. Keep the approved one-SOW sequence and performance gate; this +authorization does not resume SDK implementation before SOW-0108 closes. + +## Resumption - 2026-10-16 + +The prerequisite is complete in signed commit `4c49205393c7`. Its exact +reviewed tree, signature verification and all 80 preserved working-file hashes +pass parent and independent post-commit checks. The preceding paused execution +log remains historical evidence; clarification-pending wording in the current +gate/acceptance summary is reconciled with the approved normal EOF/status policy. +No new product/design fork is identified by the paired read-only preparation. + +### Approved Remaining Implementation And Risk Review + +- Ordinary entry points still disable recovery: `go/journal/reader.go:230` and + `rust/src/journal/src/lib.rs:437`. Enable these existing policy switches. + File, directory and facade status APIs already exist; use their current + language-native contracts rather than adding another status model. +- Preserve strict explicit policies in Go verification/capture and Rust sealed + verification/IndexedSnapshot foundation. Ordinary bounded snapshot reading + remains distinct from excluded-writer IndexedSnapshot capture. Append-open, + retention and strict index validation retain their existing rejection rules. +- Apply the already recorded recovered-view qualification to Go's all-candidate + facet key shortcut. Equal visible/declared counts do not prove every posting + contributes after incomplete unused array capacity. Check actual recovery + state, preserve count-first ownership for degraded views, and cover the + equal-count/no-contributing-posting allocation case. +- Preserve the prerequisite's hybrid header acquisition and this SOW's separate + alignment/arena/entirely-zero-header/ENTRY-shape/physical-extent validation. + No additional header-acquisition redesign is authorized or needed. +- Promote appropriate focused recovery tests to public constructors and run the + shared adapter corpus through files, directories, facades and whole-file + compressed input. Check normal EOF/status, lazy detection, counts/raw header, + reverse/time/exact reads and repair/refresh. Retain strict rejection and + byte/extent preservation controls. Prior private-policy passes do not certify + public defaults. +- Risks remain semantic relaxation of strict paths, recovered cursor/filter + completeness, temporary DATA lifetimes, live refresh and added hot-path work. + Reuse existing internals/tests and keep all changes within the approved scope. + +### Baseline And Validation Plan On Resume + +Freeze repaired HEAD before further runtime edits, retaining separate baseline +and current build targets and matching complete helper sources. The repaired Go +Explorer is now a valid default-window control. Preserve every older baseline, +failed run and historical context limit; do not overwrite or reinterpret them. +Rust runtime has no prerequisite change, but use the same final helper source +against baseline and candidate to avoid benchmark-link context differences. + +Run focused public recovery suites first, then complete language, stock-v262, +shared corruption/stream/recovery, live/growth, strict capture/append, historical +and docs/example gates. Measure damaged-file recovery cost separately from +valid-input performance. Re-run every affected native/index/catalog/payload/ +facade/directory/snapshot/verifier/writer surface with allocation evidence and +matched retained binaries. Current reproduced SDK regressions remain blocking; +no tolerance or aggregate offset is introduced. Investigate inconclusive cases +before any performance acceptance claim. + +Maintain product/performance specs, the compatibility skill and affected wiki +pages for observed status, default prefix recovery, normal EOF and remaining +errors. Existing user decisions and standing external-review authorization apply. +All fixtures, caches, reports and profiles remain repository-local and synthetic; +no installation, host journal, service or out-of-repository write is authorized. + +### Resumed Public Defaults And Validation - 2026-10-16 + +- Enabled existing ordinary policies at `go/journal/reader.go:230` and + `rust/src/journal/src/lib.rs:437`; strict verification/capture keep explicit + disabled policies. Promoted focused fixtures to public constructors. +- Focused tests pass 353 Go leaves and 11 Rust cases. All 35 executable wiki + examples pass against the current source in a fresh isolated workspace; + the 16-page structure check passes. Full suites remain unaccepted. +- Go recovered equal-count facet qualification is implemented. Both layout + fixtures return the expected missing-value facet and counters. Ten paired + allocation-only samples per layout compare identical current SDKs with/without + the qualification: 525,394 to 1,104 bytes/op and 15 to 14 allocations/op. + This eliminates the unmatched 512 KiB value copy. Concurrent validation makes + those latency samples diagnostic only; the old rejecting SDK is not a valid + damaged-file timing baseline. +- Shared public validation exposed an obsolete missing-global-next assertion: + the declared million-entry array now returns its four intact entries, explicit + recovery status and normal EOF. Updated that expectation and retained strict + verification rejection. A later sparse lost-array case reveals the actual + ordinary-filter fork described in Open decisions: both recover 350/1005 + entries; Go returns one row and Rust returns an unknown-posting-tail error. + No dependent runtime change is authorized by this observation alone. +- Current Go Reader allocation shape is 760 bytes / 768-byte heap class, above + the repaired baseline's 640-byte class. Recovery/chain fields require an + internal storage-cost investigation; steady scan counts do not prove opening + allocations unchanged. Preserve the approved copyable status/API and measure + cold state storage and lazy exceptional chain state before accepting it. +- Read-only disassembly identifies two Rust code-generation candidates within + the approved cold-path design: borrow the 32-byte Node at the cold next-array + boundary to avoid duplicate staging, and inline the small healthy cursor + dispatch while retaining cold recovery. Test each immutable private candidate + independently, inspect emitted code, and measure before adopting. No bounds, + type, size, cycle, prefix or filter-completeness check may be removed. +- The first full Rust workspace run fails one unchanged five-second retention + timing precondition after 55 passing log-writer integration cases. The failure + occurs before close: the expected pre-expiry archive was already pruned. + Investigate elapsed-time sensitivity and an isolated unchanged-source rerun; + do not attribute it to recovery, change retention behavior, or hide the run. + +### Internal Storage Candidate Before Measurement - 2026-10-16 + +Test a private Go representation candidate before adopting it: keep immutable +recovery status/tail state behind a pointer allocated only after observed loss; +retain the previous pointer in refresh rollback snapshots; keep successful status +reads copyable and allocation-free. Make stateful unique-chain storage lazy only +on an exceptional non-monotonic link, preserving its full three-word guard and +reset semantics. Store the three fixed layout sizes (4/8/16/64/72) as byte-width +private cache fields and widen at their existing accessors. Pack the recovery +policy flag with existing boolean fields. These internal choices preserve APIs, +file bytes and validation, and aim to retain the baseline Reader heap class. + +Risks are nil-state access, refresh rollback aliasing, directory dirty detection, +restart after a previously unordered chain, compiler layout and cache effects. +Keep recovery state immutable after construction, compare observed values rather +than pointer identity, and test regular/compact, amd64/386, healthy/damaged opens, +unique restarts, directory repair and failed refresh. The candidate must pass +allocation and per-path timing checks before adoption. This storage investigation +is independent of the pending ordinary-filter backend decision. + +### Benchmark Evidence Instrumentation Plan - 2026-10-16 + +Extend both existing reader helpers with separate `sdk-digest` and +`facade-digest` preflight modes. Their ordered checksum consumes every payload +byte and length plus existing row/time markers; normal timing modes retain their +sampled checksum. Collect final file-access/mmap counters in the diagnostic SDK +mode before closing, retaining the existing Go pre-scan counter scope. These +preflights validate complete bytes separately from performance measurements; +they do not add a public SDK telemetry API, a dependency, or per-payload work to +normal timing modes. Directory/facade access statistics remain unavailable at +that public layer and must not be invented. Rebuild matching helper sources +against both frozen runtime trees before final measurements; retain earlier +binaries/results as historical candidates rather than reusing their timing. + +The first private storage candidate passes selected tests and restores the +640-byte class, but read-only review establishes two costs: first supported +ascending stateful enumeration allocates a guard, and unchanged degraded refresh +allocates replacement status state. It is not accepted. Test a second private +representation retaining the guard inline: encode Brent's checkpoint schedule +with checkpoint plus a monotonically increasing step counter, updating at powers +of two. This is the same lazy Brent mechanism and needs the exhaustive graph +oracle plus counter-boundary checks. Explicit overflow ends traversal; a valid +physical journal cannot have that many distinct minimum-sized objects. Preserve +normal ordered links' inactive fast path and every early hash-hit allowance. +Reuse immutable recovery state only when newly derived status and tail values +are equal; do not mutate rollback-visible state. Measure both unusual valid +ascending chains and repeated damaged refresh, rather than hiding these costs. + +V2's measured shape is 640 bytes / 704-byte heap class: the selected test reports +shape but does not itself enforce equality. It is not accepted. Same-pattern +source inspection finds `Reader.direction` is write-only: six assignments in +`reader.go` and two in `explorer.go`, with no reads, constructor literals or +external supported access. Remove this private dead bookkeeping in a third +private candidate; preserve the independent `DirectoryReader.direction` and +Explorer source-direction fields, which are read. This eliminates unnecessary +per-row writes and aims to restore the 640-byte class without allocating chain +state. Compile every caller and check amd64/386 shape, direction/recovery tests +and matched performance before adoption. No public direction API is removed. + +### Decision Checkpoint - 2026-10-16 + +The ordinary-filter choice remains pending. SOW-0159 remains in-progress and +SOW-0121 remains paused; neither is complete. No native-filter implementation or +language-specific failure exception has been selected. Runtime storage/cursor +experiments remain private and are not adopted by passing selected tests alone. + +- Current public runtime: full Go suite passes 1,175 leaf tests (1,314 test + nodes), uncached; focused public recovery/header/opener coverage passes 353 + leaves. Rust public reader-hardening passes 11 cases; the full workspace has + the retained five-second retention-precondition failure. Its unchanged binary + passes the same exact test in isolation, which does not certify a clean full + workspace run or establish the original failure's cause. +- All 35 executable wiki examples and the 16-page structure check pass. + Whitespace checks and the changed-artifact in-progress SOW audit are clean. + This is not a completed-SOW validation/review/commit gate. +- New full-byte SDK/facade preflights compile in both languages and pass 16 + checks against independent native ENTRY/DATA parsing, both layouts and both + directions. Only the two benchmark helpers differ from the initial runtime + source manifest; library runtime source is unchanged since those full/focused + functional validations. Final matched baseline helpers still need rebuilding. +- Initial compact Rust timing: one regression, two inconclusive cases and one + improvement over the four historical flags. Borrowed Node: four inconclusive. + Plain inline hint: one regression and three inconclusive; disassembly retains + the same hot boundaries. Forced inline: four inconclusive, with 856 additional + text bytes and the Result staging/outlined dispatch still present. None + satisfies the complete SDK performance gate. Keep all sources and raw samples. +- Go private V1 restores class 640 but adds an ascending-chain allocation and + repeated recovery-state allocation; rejected. V2 removes those costs but + measures class 704; rejected. V3 additionally removes write-only Reader + direction bookkeeping: amd64 size 632/class 640 (baseline 624/class 640), + 386 size 524/class 576 (same baseline). Focused public recovery, 386 exhaustive + chain/ascending and allocation-shape checks pass. Matching opening/header, + ascending-chain, degraded-refresh and all remaining path timing/allocation + gates remain required before adoption. +- Next narrow Rust experiments, if needed: extract missing-cursor native + re-resolution into one cold helper while retaining both direction predicates, + effective bounds and materialization; separately synthesize `Ok(true)` at the + healthy return instead of forwarding the full Result. These are code-generation + candidates within the approved cold-path design, not accepted optimizations. + +Private execution evidence is rooted at the resumption workspace referenced by +`.local/sow-0159-20261015-i54ghwxo/sdk-current-resumption.json`. All current child +build/test/benchmark processes have reached a recorded terminal state. Retain the +required user's filter decision before dependent implementation, then resume the +complete local SDK gate and authorized whole-SOW review before the CLI SOW. + +### User Filter Decision And Resumption - 2026-10-16 + +The user requires Go to match Rust completely for the pending ordinary-filter +choice. This selects native indexed filtering, including unknown-secondary-tail +errors, rather than recovered-only preflight or different language failures. +Equivalent operations must preserve Boolean grouping, same-field alternatives, +empty/binary/repeated values, sparse/dense/absent matches, bidirectional stepping, +realtime/cursor positioning, directory merging, live refresh and retry semantics. +Use Rust's native posting/cursor design as the behavioral reference, with Go's +existing global offset vector and hash/array access primitives. Analyze the +complete file/directory/facade movement and invalidation paths before edits; +do not flatten grouped expressions, scan/decode unrelated DATA, or materialize +all matches simply to avoid implementing a cursor. Existing raw Go movement +helpers retain their documented role; filtered Step/facade operations correspond +to Rust filtered iteration. Compiler minimums and SDK purity remain unchanged. + +Earlier pending-decision checkpoint text above describes historical state, not +a current blocker. Independent Rust code-generation and Go storage investigations +can continue while the native-filter implementation is prepared and validated. +No SOW is completed by this decision; complete SDK validation/review/commit still +precedes resuming the CLI SOW. + +### Native Go Filter Implementation Plan - 2026-10-16 + +Read-only file/directory/facade and Rust-reference analysis is complete. Keep +pending and compiled expressions behind the existing Reader filter pointer. +Resolve exact raw DATA payloads through the native hash table; each leaf retains +its DATA identity, inline posting, scalar ENTRY_ARRAY node metadata and current +slot/value. Cache node metadata for reverse traversal, never all matching rows. +Union/intersection advance posting cursors directly. Copy scalars before nested +accessor reads so mmap eviction and ReadAt scratch reuse cannot invalidate them. +Healthy first-match reads stay lazy; recovered posting qualification follows +Rust's cold scalar-prefix pass and rejects unknown required secondary tails. + +Preserve Rust's pending-expression lifecycle, including replacement after a +successful build, invalid empty-builder errors and retry after lookup failures. +Go conjunction/disjunction methods will return errors, matching their Rust +counterparts; existing call statements can ignore the added result, while the +facade and internal consumers must propagate it. This is a consequence of the +approved equivalent failure semantics, not a separate filtering feature. +Missing DATA hash tables are errors for ordinary exact lookup. Retain existing +Explorer compatibility behavior by qualifying its helper at the ordinary-filter +boundary instead of globally changing Explorer fallbacks. + +Filtered file stepping commits only a successful position and preserves the +last matching row at EOF/error. Head/tail/realtime/cursor seeks and raw/Explorer +movement must reconcile cached posting positions. Cursor seeks use metadata +without expanding unrelated DATA and explicitly move to tail on exhaustion. +Refresh rebuilds retained DATA leaves after a successful changed view, without +mutating rollback-visible state; a missing DATA leaf stays false until a new +filter build, as in Rust. Existing recovered-empty DATA identity is retained so +same-size repair can expose its surviving/new postings. + +Forward filter mutations to every directory child and invalidate candidates. +Filtered candidates use child native stepping; raw/filtered mode changes must +discard prefetched candidates and reseek from the last emitted merge key. +Preserve ordering, duplicate suppression, realtime bounds, recovery restart and +the unfiltered sequential path. Keep strict snapshot/verify policy, compiler +floors, dependencies, runtime purity and paused SOW-0045 raw-refresh scope intact. + +Validation adds direct/facade/directory Boolean and bidirectional results, +metadata-only native access, bounded healthy first-match work, malformed and +missing postings, general I/O retry, filtered EOF/realtime/cursor state, refresh +rollback and same-size repair. Extend shared recovered-tail expectations from +fixture structure (lost-entry sparse succeeds; unknown lost/partial-array sparse +tails error). Freeze ordinary-filter benchmark helpers into both baseline and +candidate; measure startup, first match, traversal, restart, realtime seek, +allocations and unfiltered opening/header paths before acceptance. + +Reference inspection also confirms these ordinary-filter details: healthy +resolved posting populations remain fixed until a new builder is installed; +missing DATA remains false on recovery refresh; selected valid ENTRY objects do +not require membership in the global ENTRY array. The Go port keeps an optional +native selected offset in filter state for that last case, preserving existing +unfiltered Reader storage. Healthy duplicate/skipped posting values follow Rust's +needle traversal; recovered posting qualification still rejects non-increasing +values. Snapshot recovery can reread repaired bytes inside the original extent +and declared population. Add a separate cold prefix-rediscovery path, bypassing +the ordinary unchanged-header shortcut, without enabling snapshot growth. + +The Go CLI and shared adapter need mechanical interface/error propagation for +the new operator results so the SDK chunk compiles with all consumers. Keep the +existing collectBoots indexed-error scan fallback in this SDK chunk; its broader +CLI policy belongs to the already paused SOW-0121 parity audit. No v262 CLI +implementation has started. + +Private Go V3 opening/header comparison has 13 timing regressions, 12 +inconclusive cases and two improvements over 27 cases; it is not accepted. +All median allocation counts match; median byte costs match except one lower +candidate result. The apparent individual byte increase is the same roughly +5,320-byte process event observed in both builds, not an established new steady +allocation cost. Read-only codegen evidence identifies the expanded ENTRY +validator (191 to 645 text bytes, frame 16 to 72), rather than changed native +slice acquisition. Preserve every bounds/layout check and test smaller hot +validation plus cold diagnostic construction in a separate private candidate. + +The smaller validator reduces the measured overhead but still fails the gate: +12 regressions, 14 inconclusive cases and one improvement. Test one further +private candidate caching the validated arena end and effective ENTRY extent. +`Reader.path` is private write-only state (constructor assignment, no reads); +remove that unused string to fund the two scalar caches without increasing the +Reader allocation class. Recompute caches after size/header refresh and rollback. +Keep original arena-versus-physical failure classification on the cold path, +and retain location/alignment/type/size/layout checks on the healthy path. +Selected tests, codegen and matched per-case results remain prerequisites to +adoption; these optimizations do not waive the no-regression contract. + +Initial native-filter validation passes 176 focused Go leaves (194 nodes) and +536 shared hardening observations, 268 per language. Both layouts, normal and +recovered forward/reverse reads, facades, directories and unknown sparse +secondary tails are included. New direct controls cover metadata-only selection, +posting/build I/O retry, snapshot repair, loss observed after open, EOF anchors, +orphan/duplicate posting behavior, mode transitions and malformed unrelated +compressed payloads. Subsequent fixes and final source still require the full +functional and performance gate. + +### Newly Demonstrated Filter Anchor Decision - 2026-10-16 + +The user-facing question distinguishes complete behavior parity from copying a +newly demonstrated skipped-match result. The paired public-API probe positions +at the last of 1,005 rows, installs `COMMON=all AND SPARSE=some`, then steps +backward. Matching timestamps are 1000, 503000 and 1005000. Current Rust returns +1000; current Go returns the nearest preceding match 503000. After an intervening +forward EOF, both return 1000, skipping 503000. Private source, executable logs, +fixture/library hashes and results are retained under the post-decision +`filter-anchor-evidence/` directory. Rust's `JournalCursor::set_filter` preserves +the public location but leaves fresh posting cursors at their initial heads; +`previous_until` can then accept the first posting without moving to the bound. +An AND expression can also leave unevaluated children at earlier positions after +forward EOF, affecting the next direction change. + +Question 1 offers A, repair both SDKs in this SOW with shared controls and the +strict performance gate (long-term-best recommendation), or B, preserve current +Rust behavior in Go and document/track the limitation (surgical for Rust but +introduces the skipped result in Go). Do not implement the dependent policy +until the user answers. The original native-index decision remains approved; +independent storage/codegen/performance and harness work continues. + +The extent-cache candidate passes 185 selected test nodes and retains +size 632/class 640, but its 27-case timing still has nine regressions and 18 +inconclusive results. It is not accepted. Test one further private candidate +extracting reader-access bounds diagnostics: keep overflow-safe containment and +the supported-int-size limit in a small hot check, move error construction into +a noinline helper. Once containment holds, the old combined end/size MaxInt test +is equivalent to checking size alone. Retain identical error precedence and +high-offset small-read support, including 386 coverage. This tests whether +inlining removes the remaining common accessor overhead without weaker checks. + +The extracted accessor-bounds candidate improves 12 of 27 construction/header +cases; eleven remain inconclusive and four regress (small mmap opens in both +layouts, and pinned headers in both access backends). Median allocation counts +and bytes remain unchanged. This candidate remains unaccepted. Read-only +disassembly confirms the native opener, accessor constructors and file-header +reader match baseline instructions. The scratch header begins at unaligned +Reader offset 0x17b. Test a separate private candidate moving the existing +16-byte buffer before three one-byte layout caches, yielding offset 0x178 while +preserving subsequent offsets, total size and allocation class. Alignment may +reduce split stores on the pinned path; that effect is a hypothesis until +measured. Retain complete controls, including ordinary header paths affected by +moving the byte caches. Array-chunk bounds/error code and recovery dispatch +remain further inspected opening costs; no required validation is removed. + +Matched baseline/current Go and Rust helpers, plus the private Rust cold-resolve +candidate, now build from retained immutable source manifests. Their filter +output preflight uses an independent native fixture-byte parser and full ordered +payload checksums, with binary matches, both directions, rebuild/restart/time +seeks and file/directory/open-files surfaces. A benchmark audit found the former +different-field AND case used identical posting populations. Strengthen it to +overlapping alternatives whose intersection differs from either operand and +their union, and validate benchmark semantic counters independently before +timing rather than accepting agreement between two potentially wrong outputs. + +The alignment experiment has ten improvements, fourteen inconclusive cases and +three regressions over all 27 controls. The pinned mmap case still regresses; +alignment alone does not establish a repair and remains unaccepted. Next test +the inspected error-interface join in accessor bounds checks by expressing the +same containment predicate as a boolean and constructing errors only in its +failure branch. This preserves physical/int-size limits and error precedence. + +For a separate opening candidate, reuse the arena bound already validated by +the array-header reader, move all-zero/error classification to cold helpers, +and replace redundant checked additions/division with equivalent bounded +arithmetic. Successful array-header validation proves offset+24 and the object +extent fit the arena; `toRead <= capacity` proves its byte product cannot +overflow. Compare that product directly with MaxInt. The 4/8-byte item widths +permit mask/shift arithmetic. Keep physical whole-object checks and recoverable +tail classification unchanged. Isolate global-array failure recovery in a cold +helper if it reduces the measured healthy frame cost; retain complete selected +recovery/bounds/386 validation and all construction/header timing controls. + +Opening inspection also finds each array-header read allocates a temporary +24-byte copy even when the complete header fits a cached native window. Within +the private opening candidate, use an immediate borrowed header only for the +same fitting power-of-two window condition already used by ENTRY headers. +Parse all header scalars before nested reads. Preserve the original copying +readAt path for crossing, tiny and non-power-of-two windows; no new persistent +Reader storage, API or alias lifetime is introduced. Measure allocation and +timing on both layouts/backends, including pinned/crossing/recovery controls, +before considering adoption. + +The first complete native-filter Go run finishes with 1,210 passing test leaves +and one failing leaf. `TestReaderHeaderOnlyOpenSkipsEntryArrayAndKeepsIndexedUnique` +finds five global offsets loaded by an unchanged-header unique query. The new +offset-population freshness check incorrectly treats a metadata-only nil slice +as a stale full reader. Preserve nil for metadata-only opens; use a nonnil empty +slice for loaded empty populations, without adding Reader storage/allocation. +Qualify the freshness predicate with that distinction. Also include actual +offset-population length changes in Refresh's result: a fresh filter group can +publish new header counters before the global vector is rebuilt, so comparing +only header/size/status can falsely report no change and skip a new raw row. +Exercise initially empty/nonempty readers, explicit refresh, raw continuation, +and successful/failed group builds. This repairs consequences of the new native +group visibility path; paused SOW-0045's unrelated raw unchanged-header repair +contract remains outside this chunk. + +Focused source inspection additionally finds selected offsets already present +in the cached global vector receive only a 64-byte ENTRY header read. Unlike +Rust row loading, that does not establish the complete ENTRY body's bounds or +item alignment after live truncation/modification. Validate the selected +header's scalar alignment, arena extent and physical extent before committing +every filtered result, including ordinary global and orphan offsets. Reuse +the already-read scalars; do not reread item bodies or decode DATA. Preserve +arena errors versus recoverable physical tail loss and the cold zero-header +classification. Add public controls for retained-header/partial-body tail loss +and malformed item alignment after open in both layouts/backends. + +Further native-path inspection identifies two related recovery consequences. +Snapshot group/recovery refresh currently clamps to its last observed physical +size, permanently narrowing the original bound after shrink. Preserve a separate +immutable size captured at open, alongside current physical size. Clamp repair +visibility to the original cap and original declared population, so restoring +bytes within that cap can work while later appended entries remain excluded. +This adds one scalar without increasing the current Reader allocation class; +carry it into the private compact-storage candidate and remeasure that class. +The original-size field is immutable and does not need refresh rollback state. + +A realtime seek can load a stale global vector and discover recovery after the +filter expression was built. Refresh marks the expression stale, but the same +step currently evaluates the old leaves. Rebuild the retained DATA leaves on +that observed transition before using them, preserving lazy qualification and +the pending realtime bound. Add first-call EOF controls for a newly published +matching DATA whose ENTRY was not completed. These fixes implement the existing +repair/first-call parity contract; they do not resolve the separate pending +cursor-nearestness decision. + +Ordinary movement errors must not reread DATA populations: marking every error +as expression-stale can widen an installed healthy filter after an append, +contrary to Rust's captured-population behavior. Invalidate only owned walk/node +cache state on those retries, preserving DATA identity, inline/head and captured +counts; reserve DATA requalification for actual recovery transitions. Commit +chain-guard changes only after successful node reads. Cover transient selected +ENTRY failure followed by append/retry in both backends. + +Recovery can also change midway through a multi-leaf refresh. An earlier leaf +can be clipped to the old prefix while a later missing DATA header discovers a +complete repair. Repeat whole-expression requalification when its observed +recovery status/tail changes during the pass; install only a pass with one +consistent view. Keep intermediate children private and failures retryable. +This is a cold-path consistency loop, with no additional healthy per-row work. +Add a deterministic truncation/restore AND control and retain its failed +counterfactual evidence before accepting the fix. + +Snapshot reference clarification: the preceding analysis incorrectly inferred a +frozen global declared population from Rust's public header snapshot. Modern +core header mappings and new exact DATA builders can see publications inside +the original byte extent. The user's Rust-parity instruction selects that +existing byte-bound contract, not a stronger immutable logical snapshot. Remove +the unsupported population claim from consumer docs and use fresh internal +header counters for native group/recovery refresh while retaining the original +byte cap. Already installed healthy DATA populations remain fixed as above. +Use the same bounded native vector refresh for filtered realtime/orphan +normalization under snapshot bounds; ordinary snapshot raw refresh policy is +otherwise unchanged. + +Rust's cold snapshot prefix discovery also used only its original byte cap, +even after an observed post-open failure caused by physical shrink. Compute its +discovery extent as min(original cap, current physical length), preserving the +immutable cap itself. This adds no healthy per-row work and allows the same +observed prefix/repair policy in both languages. Test actual shrink/restore +after open in addition to at-open truncated fixtures; a zero status remains a +lazy observation and does not promise detection of every external modification. + +The same-failure search finds Rust also refreshes Boolean children in place +without checking whether a later leaf changed the recovery view. Apply the +same whole-view stability check to both a compiled expression and the pending +builder's top-level groups. Keep recursive leaf refresh internal to one pass, +retain DATA identities across retry, and repeat only when status/tail metadata +changed. This repairs the existing observed-repair contract in both languages; +it does not alter the pending cursor-nearestness policy or healthy traversal. + +The stable-pass loop also needs persistent retry state: a DATA refresh can fail +after earlier branches were replaced and after ENTRY discovery clears recovery. +The next build would otherwise skip requalification and retain clipped branches. +Add a deterministic pending-builder counterfactual with two rows: compile +COMMON and TARGET while healthy, truncate before TARGET, attempt a recovered +build, restore ENTRY bytes while leaving TARGET's header zero, retry, then repair +that header. The final AND must find the second row. Cover an installed +expression's analogous failed refresh. Preserve pending state across errors and +clear it only after a complete stable pass. Inspect private cursor padding for +flags before increasing public/private object sizes or adding allocations; +measure healthy filter overhead before acceptance. No public cursor fields or +FilterExpr variant shapes may change. + +The private array-opening candidate finishes all 27 controls with five +regressions, eleven improvements and eleven inconclusive results. Reader +allocation grows from 640 to 704 bytes and latency regresses, so this +candidate is not accepted. Test a separate storage-only variant: put the +immutable initial byte cap in the accessor that observes it, funding that scalar +by narrowing its two normalized 0/1/2 access-mode caches to bytes. Replace the +Reader's duplicate staged-file cleanup path with a Boolean and derive the same +name from the owned file at close. Ordinary files remain unstaged. Preserve +staged cleanup/close behavior, snapshot bounds, mode statistics, allocation +classes and 386 high-offset small-read support; validate source-derived +equivalence and all construction/header controls before adoption. No runtime +installation, host journal operation or wider design change is involved. + +The pending-builder control reproduces false EOF before the repair. The retry +implementation will keep refresh, recovered-population and unfinished-position +flags in existing InlinedCursor padding. A file-local sticky observation flag +avoids walking expression trees on healthy files that have never needed +requalification; damaged/repaired files inspect all leaves so public expression +composition cannot hide a pending subtree. Requalify all builder groups as one +batch, retaining pending flags after any error. Keep backward positioning pending +through fallible tail preparation without rereading DATA counts on a tail-only +retry. Preserve flags through cursor copies. Measure actual sizes and healthy +filter timings; source layout estimates alone are not acceptance evidence. + +The storage-cap variant restores measured reader allocation from 704 to 640 +bytes, with one allocation per operation. Its 27 construction/header controls +show eleven improvements, sixteen inconclusive results and no detected +regressions; inconclusive remains unaccepted as equivalence. Fifty-three +selected recovery/access/staging/high-offset test nodes pass on each of amd64 +and 386. The sparse-file control reads a small range above 2 GiB and preserves +physical-bound versus integer-limit error precedence without allocating that +large range. Integrate the measured internal changes into the native-filter +source for whole-source functional/performance validation; retain every native +filter freshness, selected-body, orphan and snapshot correction. This is a +candidate integration, not final performance acceptance. + +Rust's complete core suite now passes all 96 tests. New retry controls cover +pending and installed filters, forward/backward, both layouts, public Boolean +composition, temporary zero DATA population metadata, and a failed backward +preparation followed by append. The latter must keep the captured population +and return the old second row rather than the new third row. These functional +results do not establish the healthy timing or allocation gate. + +The same-failure search confirms Go's transactional expression replacement does +not cover a previously compiled pending group: recovery can clear while another +group fails, with no installed expression to mark stale. The new pending-AND +control fails in both layouts/backends before repair. Mark the builder stale on +every successful recovery transition, including before installation, and honor +that flag in the pending build branch. Keep it until the complete build succeeds. +Also retain a known Match's DATA identity when a refreshed population is zero; +initially absent/empty matches keep their existing false semantics. Reuse Go's +transactional tree replacement and existing builder flag, adding no Reader +storage or healthy per-row scans. Add restoration controls for both consequences. + +The first Go retry controls failed a fixture precondition: ordinary Refresh +correctly skipped an unchanged healthy header/extent, so they had not yet +established recovery. Those failed attempts are retained, not counted as bug +reproductions. Corrected controls use observable physical shrink. The matched +counterfactual now returns false EOF after repair in all eight layout/backend +leaves (pending groups and known-empty DATA); current source passes all 188 +selected nodes. The complete integrated Go suite passes 1,459 nodes, with no +failures. Matched filtering helpers pass 480 small-fixture/full-payload +observations and 180 independent large-corpus golden observations. + +Compiler-derived Rust layout controls confirm the retry flags add no size to +InlinedCursor (128 bytes), FilterExpr (136) or JournalCursor (280). Whole SDK +changes do increase JournalFile from 552 to 640 bytes and JournalReader from +608 to 656; these are recorded costs, not allocation/timing acceptance. Public +constructor/filter timing and actual allocation still require measurement. + +Consumer documentation now explicitly covers pending-builder retries, facade +handle operator error results, and FlushMatches rewinding all reader surfaces. +The older Go API guide's immutable-state snapshot wording is corrected to the +approved original-byte-extent contract, with healthy filter populations kept +separate from fresh group visibility. No example fence or harness syntax changes. + +The documentation blast check finds direct flush-to-head coverage weak or +masked outside Go's file reader. Extend the existing paired public adapter +probe with a clear-after-two-sparse-matches operation. It must then return the +complete unfiltered sequence from its first row, without an explicit seek. +Run it on file/directory SDK and facade surfaces in both layouts/languages. +This validates the approved Rust-parity consequence and does not add SDK APIs. + +Integrated timing remains unaccepted. Current Rust has two compact facade-scan +regressions (median ratios 1.0214 and 1.0289). The separate cold-resolve candidate +has four inconclusive results, not equivalence. The integrated Go header matrix +has ten improvements, thirteen inconclusive cases and four regressions; the +private older-source result is not substituted for this integration result. + +Accessor inspection finds hardware division repeated to compute native window +bases, including crossing/pinned reads. Test a private integrated candidate +caching the power-of-two condition in existing access-mode padding; use +offset & -windowSize for that case and retain division for arbitrary sizes. +Construction already rejects zero window sizes. Keep public sizes/statistics, +non-power-of-two behavior and overflow/bounds precedence unchanged. Measure +all 27 controls and actual object/allocation sizes, with high-offset/386 tests, +before integrating it. This is internal cost removal for the approved strict +performance gate, not a new access-mode policy. + +The Rust footprint control identifies ordinary-reader storage costs beyond the +retry flags: inline recovered-prefix metadata and three-word guards retained by +two iterator options. Test a private combined candidate with the existing +cold-resolve codegen change, the equivalent two-word Brent schedule already +validated in Go, and a private recovery cell backed by a lazily initialized +boxed Cell. Its get/set API returns the same Copy values; never-recovered files +allocate no recovery box, clearing an uninitialized cell allocates nothing, +and later recovery/repair reuses the first box. Keep the active recovery check +on the cell's value, not whether storage was initialized. No unsafe code, +borrowed recovery references, public type fields or behavior changes are needed. +Measure actual sizes, healthy/damaged allocation costs and affected timings; +run the exhaustive guard model and complete core/recovery controls before any +candidate integration. Additional cold allocation is recorded explicitly. + +The geometry candidate passes 198 focused nodes on each of amd64 and 386 and +all 264 new independent output observations (192 small/full-payload, 72 large +golden controls). Its header matrix has twelve improvements, fourteen +inconclusive results and one compact empty mmap-open regression, median 1.0111 +with interval [1.00046, 1.03132]. All results remain diagnostic. A longer paired +empty-open control is running because the earlier integration flagged the +other layout, making repeatability unresolved. + +Bounded disassembly identifies a concrete constructor consequence: baseline's +public OpenFileWithOptions wrapper is inlined into the caller, while the current +extra private wrapper prevents that and adds a 0x40-byte call frame. +configureLayout itself remains inlined, and both policy constructor frames are +0x3c0. Test a separate private geometry candidate having the public function call +openFileWithPolicy(path, opts, true, true) directly. Preserve the metadata-only +and strict private callers. This removes a redundant layer without changing the +approved recovery/load policy; verify actual inlining, controls and timing. + +The Rust cold-storage candidate passes all 96 core tests after supplying the +missing private-checkout FSS fixture; the initial fixture failure is retained. +Compiler-derived sizes are JournalFile 568 and JournalReader 640, with posting, +expression and cursor unchanged. Separate allocator-instrumented binaries link +the matching benchmark dependency fingerprints, leaving normal timing binaries +unchanged. On both healthy 100,000-row layouts, open requests five allocations: +924 bytes baseline, 1,012 current inline-recovery source, and 940 candidate. +First scan requests three allocations/736 bytes; warmed scans request zero +allocations/bytes in all three builds. On the 1,004-row recovered prefix, +candidate open requests six allocations/1,162 bytes versus current five/1,154; +the extra lazy box adds one request and eight total requested bytes there. +Both have identical ordered full-payload counters and zero warm allocations. +These counts describe allocation requests, not allocator usable sizes or elapsed +performance. Cold/healthy timing gates remain open. + +Current root adapters pass all 552 paired hardening observations against the +full-feature v262 oracle, including the sixteen new flush-to-head observations. +The first shared-run attempt only lost the copied executable bit and is +retained separately; rerunning the same binaries with executable permissions +passes. All sixteen wiki pages validate and all 35 examples compile/run after +the consumer contract corrections. + +The longer twenty-pair Go empty-open controls are all inconclusive; allocation +counts/bytes match exactly. Rust cold-storage's four historical compact cases +are also all inconclusive. Neither set establishes equivalence or satisfies the +strict timing gate. Integrate Rust's independently validated cold-storage, +two-word guard and cold-resolve changes for whole-source validation while +retaining every pending-retry flag and current adapter probe. The complete core, +ordered output, footprint and allocator controls justify testing this combined +candidate, not accepting its final performance. Keep all prior measurements and +source/binary identities. No cursor-nearest policy change is included. + +The flat Go public opener now actually inlines into its benchmark caller, +removing the extra wrapper frame. Its 198 focused nodes pass on each of amd64 +and 386. Integrate that one-line call-path correction and the separately +validated cached window geometry for complete-source validation. Only the +power-of-two arithmetic path changes; arbitrary window sizes retain division. +Source comparison confirms all other Reader/accessor behavior is identical. +Final timings, allocation classes and wider suites remain acceptance gates. + +A bounded Rust disassembly comparison finds that the current FileReader healthy +step makes a per-row call to step_inner, with six saved registers and a 0x158-byte +stack reservation. The baseline inlined this logic into step_valid. Test one +private candidate adding an inline hint to step_inner while preserving the cold +recovery and missing-cursor helpers. Verify actual code generation and ordered +outputs before measuring; no filter policy, retry semantics or public API changes. + +The subsequent history check confirms plain/forced step_inner hints were already +measured earlier. The newly started combined-source build is retained for +code-generation inspection only; do not repeat its timing absent a distinct +actual change. Two separate, narrower candidates address demonstrated work: +OffsetArrayObject::len/is_empty can inline their capacity checks so callers do +not recompute the same kind/alignment/length checks after an outlined call; +CurrentRowView::clear_pins can keep its no-active-pins branch in a tiny inline +wrapper and put the existing fallible unpin work in a noinline helper. The +current false path saves seven registers before checking its Boolean, and row +advance/load can call it twice. Preserve every validation, error and row-pin +lifetime; active unpin is ordinary facade work and must not be marked cold. +Build independent candidates from the validated cold-storage source, inspect +actual code generation, then measure only concrete differences. These internal +cost removals serve the existing strict performance requirement. + +The plain inline hint leaves the same call boundaries, function sizes and Result +staging in the combined candidate; do not time it again. The array-length variant +recompiled successfully, but the following clear-pins build incorrectly reused +that artifact (identical executable hash, no SDK compilation in its log). That +output is excluded as a clear-pins candidate. The shared Cargo target can reuse +relocated source fingerprints when independently prepared variants have older +mtimes than the preceding build. Prepare a new clear-pins source after that +build, apply its change then, retain a new manifest and require the log to show +the expected source compilation plus distinct code-generation evidence. Existing +cold-storage layout/allocator/core evidence is independently tied to its actual +compiled type sizes and behavior; do not infer a general validation failure. + +### Completing Matched SDK Performance Coverage - 2026-10-16 + +The read-only coverage audit finds ReaderBoundsSnapshot scans do not measure +IndexedSnapshot capture/visitors, and the existing Rust example omits payloads +visited by Go's snapshot benchmark. Add matched file-only helper modes in both +languages: indexed-open, indexed-exact, indexed-field and indexed-all, with full +payload digest companions for the three visitors; verify-index and verify-file +count successful complete operations. Reuse one existing --match FIELD=value +for snapshot declarations and selection, and catalog-repetitions for complete +open/operate/close lifecycles. Snapshot callbacks add realtime and every payload. +Report capture counts and successful opens separately from rows visited. Reject +irrelevant live/backward labels; verifier output identifies its public SDK +defaults and rejects custom access/window labels it cannot honor. + +Keep fixture creation, independent native-byte/full-payload preflight and process +startup outside the timer; include SDK open, traversal/verification and cleanup. +Use bounded multi-array fixtures for compatibility verification to avoid its +existing repeated-posting quadratic cost. Run plain/codec layouts and single +native Zstd frames with repeated DATA reuse; the existing zstd-known hardening +fixture includes skippable padding and cannot certify the native single-frame +path. Whole-file single-frame staging can use existing SDK/facade modes against +the original decompressed population. Old Rust's truncated concatenated output +is invalid as a performance baseline. Keyed verification's HMAC branch is +unchanged and shares the affected graph decoder, so retain sealed keyed +correctness and measure verify-file/index without adding a key-handling mode. + +Copy the exact helper changes into fresh baseline and candidate sources before +final timing; qualify all new counters independently. Extend the existing +comparison runner with opt-in auxiliary cases, preserving its default matrix. +Update benchmark documentation with effective options and timer boundaries. +The same audit requires current writer raw/structured mixed-cardinality cases, +a repeated-compressed-DATA writer lookup control and reader refresh after +publication; immutable live bounds alone do not establish growth-branch cost. +These are validation consequences of existing shared lookup/recovery changes, +not new SDK APIs or unrelated optimization scope. Root remains the only writer. + +The distinct array-length hint removes one call and array staging but retains +some repeated alignment/kind checks; four compact scan controls remain +inconclusive. Keep it private rather than adopting an unneeded extra change. +The fresh pin-clearing split puts false checks inline in advance, entry loading, +DATA restart and DATA clear; only the active path calls the unchanged fallible +unpin operation. Its four diagnostic scan comparisons have one improvement and +three inconclusive results (median ratios 0.98533, 0.98174, 0.98323, 0.99342). +Both distinct variants pass 264 independent ordered-output observations. +Integrate only the pin-clearing split for whole-source/floor validation and the +complete reader matrix. This is not a passed final timing gate. + +Go's flat-opener 27-control matrix has eight improvements, sixteen inconclusive +results and three regressions: compact empty ReadAt open, ReadAt eviction and +ReadAt pinned headers. Keep every sample and allocation metric. The latter two +also flagged earlier integrated source and require targeted code-generation +investigation with the complete current validation preserved; removing the +wrapper did not establish the performance contract. + +Complete source validation at this checkpoint passes Rust's 602 workspace tests +(five existing ignored), Go's 1,459 amd64 nodes, 1,460 386 nodes and 1,459 race +nodes. Go 1.26.2 passes 326 focused nodes on each architecture; Windows, macOS and +FreeBSD test builds pass. Runtime/compiler floors are unchanged. The new Rust +pin split and benchmark-helper coverage still require their own final checks. + +The Go header comparison measures ReadAt eviction at 1,047 to 1,061 ns/op and +pinned headers at 789.25 to 809.20 ns/op; generic header parsing retains a nil +error result join in the valid path. Test a private scalar candidate: explicitly +reject buffers shorter than 16 with errInvalidJournal, copy type and size, and +keep the same type, minimum size, shape, arena and physical error ordering. +Generic parsing performs only that length check and those scalar extractions; +its unused flag has no effect here. After size >=64, size&(itemSize-1) is equal +to (size-64)&(itemSize-1), since both supported item sizes 4 and 16 divide 64. +Preserve offset-before-subtraction checks and cold zero-header classification. +Validate both architectures and bounded code generation, then measure affected +controls; no validation is removed and borrowed bytes never cross another read. + +The complete Rust 1.91 compiler-floor workspace now passes, including the +pin-clearing integration and new auxiliary helper compilation. The exact test +counts and ignored cases are retained with the complete log. + +The Go ENTRY scalar candidate's complete 27-control matrix has fifteen +improvements and twelve inconclusive results, with no flagged regressions. +Its validator shrinks from 107 to 94 instructions and 372 to 330 text bytes; +the same 32-byte frame and Reader/offset spills remain. Read-only adjudication +confirms all short/type/zero/shape/arena/physical outcomes and borrowed lifetimes. +Integrate only this equivalent validator for combined-source measurement; +twelve inconclusive cases do not establish the complete performance gate. + +The auxiliary counter preflight passes 72 initial observations, then 128 after +adding the timed Go ReadAt and Rust whole-file snapshot strategies. All snapshot +visitors have sampled and full-byte counterparts, capture counts are independent, +and both verifiers use their documented defaults. Stock v262 verifies each +bounded 1,005-row mixed-cardinality source before preflight. All four matched +SDK/helper binaries pass; original hashes, complete stdout and process records +are retained. Compressed/staged fixtures still require their own full-byte +qualification against the declared native original before timing. + +The complete matched Rust matrix finds additional metadata-stepping and +fresh-filter construction slowdowns; final acceptance remains blocked. Bounded +actual disassembly confirms a healthy per-row nested step_inner call with six +saved registers and 0x158 stack bytes, while baseline inlined that resolution. +The prior forced-inline experiment moved the call into a downstream resolver, +so an attribute alone is not a new remedy. + +Prepare a distinct private source candidate: public JournalCursor::step resolves +its location directly and matches the Result. Only Ok(Some(location)) commits +the location and returns true; EOF and the unchanged error enter the existing +cold step_recovery. Keep the latter's single step_inner retry, including its +second-error/EOF behavior. Do not use an initial question-mark propagation that +would bypass recovery. Resolvers mutate only filter/array state, and the public +row loader retains its existing rollback. The short duplicated resolution +sequence keeps the healthy body at its public inline boundary; verify the entire +downstream call graph before any timing. This changes no nearest-match policy. +Build only after the current serialized timing sequence, then run focused/core +controls and independent output preflight before measurement. + +The private per-file storage audit finds no dead field that can remove Rust's +sixteen extra healthy allocation-request bytes without changing live state or +introducing a wider storage design. All recovery latches, historical-header, +file ownership, publication and statistics fields have consumers. Retain that +measured footprint cost explicitly; do not delete a live field to improve sizes. + +The complete reader matrix also flags fresh facade filter construction in the +regular layout (paired median ratios 1.07846 and 1.11349). Read-only bounded +disassembly finds new outlined closure/qualification calls and repeated +128-byte cursor staging; no added healthy syscall or per-build vector allocation +was found. Test a separate private healthy-constructor candidate alongside the +direct-step candidate. Preserve the public hidden constructor and consuming +qualification signatures. A private unit-returning qualifier mutates only owned +local cursors, returns immediately without recovery, and calls an outlined +recovery helper otherwise. Preserve provenance before empty-prefix clipping, +captured posting counts, pending flags and restoration of the inline position. +Drop borrowed DATA before qualification or any recovery read. The constructor's +error helper must preserve the original error when recovery is unchanged and +perform exactly one nonrecursive retry when recovery changes. Build after the +serialized measurements, inspect the complete actual call path, and validate +recovery/ordered outputs before paired timing. This is an implementation of the +existing accepted retry contract, with no change to the pending cursor policy. + +The remaining compressed lookup control uses private matched writer helpers, +not a relabeled standard workload. Name it +`mixed-cardinality-long-reuse-32-fields`: repeat the 14-byte HOST_CLASS value +128 times, retain 32 fields and the 512-byte compression threshold, and report +2,606 logical bytes per row (817 + 127*14 + 11 profile-name bytes). Keep public +SDK options, append APIs, hash sizing and publication cadence identical within +each baseline/candidate pair. Build plain and Zstandard variants. Independently +require exactly one compressed DATA object, standard single-frame bytes without +padding, content size 1,803, complete decompressed byte identity and references +from every ENTRY. Full-byte reader checks precede timing for every mapping. +The standalone comparator must retain append-only timer boundaries and reject +incorrect metadata; existing standard workload checks remain intact. + +The bounded directory/OpenFiles control uses two separately generated 1,005-row +original-profile files, with constructor-supplied distinct synthetic file IDs. +The second file starts at sequence 1,006, realtime 1700000000502500 and monotonic +50050250, retaining 500/50 microsecond increments and common machine/boot/seqnum +identity. Freeze native bytes and validate both files plus complete 2,010-row +forward/reverse results before timing. Do not patch file IDs in completed keyed +files, and do not describe two identical input files as multi-file coverage. + +The complete matched Rust file matrix finishes all 268 comparisons with +unchanged semantic counters: 121 improvements, 134 inconclusive results and 13 +flagged regressions. Facade forward stepping with whole-file mappings flags +both layouts and both bounds; fresh facade filter construction also flags both +layouts. Preserve the entire run. The private direct-step/constructor candidates +are diagnostic repairs to investigate those costs, not accepted replacements. +The combined SOW audit is clean at this in-progress checkpoint; that does not +satisfy the remaining acceptance or review gates. + +Retained-reader publication controls will use a frozen synthetic Go publisher +and distinct baseline/candidate raw and filtered reader handles opened before +publication. The writer commits one changed batch outside reader timing and +waits for acknowledgement. Measure Go Refresh/SeekHead/drain and Rust +seek-head/drain under their actual public APIs; Rust has no FileReader Refresh +method. Fresh-filter cases flush/add/seek/drain without an earlier Refresh on +that handle. Explicit seeking normalizes pre-existing flush-position behavior. +Run unchanged physical allocation and a committed uncompressed 9 MiB DATA +growth case; assert physical/declared geometry and independent ordered digests. +Use a new publication per sample, not repeated unchanged polling. New recovery +status and old Go's incomplete reload outcomes cannot form a successful paired +baseline; keep repair costs as candidate diagnostics with their actual outputs. + +### Failed-Candidate Positioning Evidence - 2026-10-16 + +Read-only adjudication finds a second position-policy difference. Rust +`rust/src/journal/src/lib.rs:635-656` advances the core reader before loading the +selected ENTRY. `reader.rs:135-136` leaves that candidate installed when recovery +does not apply. The new type check reports `InvalidObjectType`; that error is +ineligible for tail recovery. After a malformed second matching ENTRY fails, +restoring its bytes can therefore let getters expose the second row without a +successful iteration result, and a subsequent advance can skip it. This commit +ordering predates this SOW. Recovery-eligible anchor restoration is separate. + +Go's native implementation commits at +`go/journal/reader_filter_step.go:63-68` only after the selected header/body +passes. It retains the earlier successful position on that error. The user's +complete parity requirement makes a common policy necessary. Question 2 offers +last-successful state in both SDKs or the existing Rust failed-candidate state +in both. The former is recommended long-term-best; the latter minimizes Rust +change but preserves the unreturned-row/skip behavior. These source-grounded +sequences still need paired executable controls after the serialized timing +batch. No error-position policy change has been implemented. + +The retained baseline Go source confirms this is a consequence of the native +port: its `Step` called `Next` before `GetEntry`, and returned payload/header +errors without restoring `entryIndex`; `StepBack` did the same. Its existing +ENTRY header reader already rejected the wrong type. Thus both old Go and Rust +retain the failed candidate, whereas new Go retains the successful row. This +strengthens the need to disposition the port's changed error position, without +assuming permission to change Rust's existing behavior. + +The same read-only scan found a pre-existing directory realtime-seek/getter +invalidation gap, independently of filtering. Track it in paused SOW-0045's +related healthy-position section; no runtime change or new prerequisite is +included here. Orphan getter routing, raw normalization, successful cursor/time +positioning and retained filtered EOF yielded no further concrete source-level +defect in this bounded scan. Executable validation remains distinct from that +source-only finding. + +### Go Reader Timing Cost Investigation - 2026-10-16 + +The initial complete Go file comparisons flag several allocated-entry scans +around 1-2 percent and high-cardinality stateful unique enumeration around +4 percent. Actual matched binaries retain identical `readEntryAt` instructions, +calls and 712-byte frame, with the same 128-byte Entry allocation. Reader sizes +624/616 both occupy the same 640-byte amd64 heap class. No per-row allocation +increase was found in those paths. + +The new unfiltered Step/StepBack call `errors.Is` even when raw movement returns +nil; baseline branches on nil first. Prepare a private candidate restoring the +healthy early return while retaining wrapped EOF/general error classification +and native filter dispatch. Measure this separately before combining changes. +Two further equivalent private controls may restore the valid-global-index +GetEntry path with a cold orphan/error fallback (28 to 50 instructions currently) +and outline only stateful unique cycle-error clearing/formatting (272 to 288 +stack bytes currently). Retain the complete Brent guard and all valid unordered +chain behavior; do not outline the whole compatibility traversal as a shortcut. +Source/codegen observations identify actual work, not its causal timing share. +Validate public orphan/error/unique contracts and both architectures before +adoption; use matched helpers, output checks and affected timing controls. + +For the pure unknown-size single-frame DATA control, prepare a separate private +fixture-generator fork of the frozen Go writer. Omit only its post-encoding +content-size-header insertion; retain its stream encoder, public writer path, +threshold, payloads, identities and references. This fork is a generator, never +a timed baseline/candidate SDK. Independently require one standard frame with +UNKNOWN content size, complete frame extent and exact 1,803-byte decompression, +then require stock verification and matching full-byte reader results before +using it as a valid-input compatibility timing control. + +### Remaining-Control Harness Review - 2026-10-16 + +A read-only review found three defects in the private remaining-control +drivers before their results could be used as evidence. Native sequence-header +qualification reversed head/tail offsets; the correct offsets are 168/160. +The shared writer command builder rejects the intentionally private long-reuse +workload before execution, so its private comparator now constructs only that +workload's equivalent arguments directly. The retained-reader coordinator's +blocking response read bypassed its later timeout, and cleanup could stop at +the first unresponsive child. Response reception is now bounded, handles +partial/multiple lines per child, and cleanup visits every captured process +handle while retaining failures without signaling processes. + +Static re-review confirms those repairs; syntax checks pass. The fixture +preparer was already loaded in a waiting interpreter, so its first attempt +retains the prior offset check. Preserve that diagnostic failure and run the +corrected preparer into a new output directory after it exits. Release the +queued dependent controls only through the corrected attempt's independently +qualified manifest. These harness repairs change no SDK source or accepted +positioning policy, and prepared controls are not validation results. + +### Filtered Realtime-Seek Cost And Policy - 2026-10-16 + +Completed regular-layout Go comparisons show approximately 1.05 seconds versus +0.048 seconds per 1,000 filtered realtime seeks, about twenty-two times slower, +across SDK/facade, live/snapshot and both directions. The compact cases seen so +far reproduce that cost. The previous Go path binary-searched the global ENTRY +offsets and checked rows from that point. The native port binary-searches the +global offsets, then linearly walks the matching DATA posting list from its +head/tail to the target (`go/journal/reader_filter_posting.go:195-260`). + +Rust currently has the same linear filtered alignment via +`InlinedCursor::next_until/previous_until` (`offset_array.rs:869-923`), despite +having a separate directed binary partition primitive at `:925`. This is not +evidence that Rust already uses binary posting search for ordinary filtering. +Switching explicit time seeks to binary posting search in both SDKs is a +behavior decision: a skipped malformed slot need not be read, and a probed slot +past an early answer can fail sooner. User question 3 explains this change in +error discovery, its performance benefit and the linear alternative. Keep +ordinary sequential stepping unchanged and do not implement the dependent +search policy before the user replies. + +A separate read-only ownership audit confirms that both Rust path openers copy +an already-absolute path into a temporary PathBuf, then copy it again into the +retained repository path string. Test a private equivalent candidate that +borrows absolute inputs through `File::from_raw_path`; relative inputs retain +the existing current-directory join and error ordering. Both public signatures +and the retained path bytes stay unchanged. This removes a redundant temporary +allocation on every absolute-path variant, including historical/recovered +files, without moving cost to another path. For the recorded 103-byte fixture +path, five current requests total 940 bytes; eliminating that temporary predicts +four/837, which still needs executable measurement. Retained requested storage +remains sixteen bytes above baseline (837 versus 821); cumulative allocation +requests, retained/peak storage and allocator usable sizes are different +quantities. No broader storage redesign or public behavior change is included. + +The complete matched Go file matrix finishes all 220 comparisons after +3,797.46 seconds, with unchanged semantic counters: 164 improvements, +35 inconclusive results and 21 regressions. Sixteen flagged cases are explicit +filtered realtime seeking across both layouts, APIs, bounds and directions. +Four allocated-entry scans and one high-cardinality stateful unique case form +the remaining five flags. This distinguishes the repeatable linear-seek cost +from the smaller call-path/code-generation investigations without accepting or +offsetting either group. The complete Rust/Go file matrix therefore contains +488 comparisons: 285 improvements, 169 inconclusive results and 34 flags. +None of those aggregates is a performance acceptance verdict. + +After the current serialized measurements and private candidate validation, +capture actual generated call paths and bounded Go CPU profiles for entry +reads, high-cardinality stateful unique enumeration and explicit filtered time +seeks. Use frozen matched helpers and verify counters; keep allocator/profile +instrumentation out of ordinary timing binaries. The absolute-path candidate's +checks must also assert unchanged open reallocation count and first-scan +allocation cost on recovered inputs, not only healthy inputs. + +The bounded plain auxiliary matrix completes forty comparisons in 121.10 +seconds: thirteen improvements, twenty-six inconclusive results and one +regular-layout Go VerifyFile flag (paired median ratio 1.04389, interval +[1.00706, 1.05715], approximately 0.178 versus 0.182 seconds). Include that +verification path in the bounded Go profiles; it is not waived as noise. +Ten direction-specific standard before/after reports render successfully. +The first combined-direction report request was rejected by the existing +report key contract; its failed attempt is retained, and the supported +direction-specific summaries are used without changing reporting code. + +Both private Rust reader candidates and all nine matched writer/generator +controls compile successfully. The loaded first fixture attempt fails exactly +at its stale reversed sequence-offset check; all eighteen preceding generator +and stock-verification subprocesses return zero. A corrected new-output attempt +then qualifies all twenty fixtures, including exact decompression and logical +row equality, native byte identity within each language/codec pair, distinct +multi-file IDs, disjoint ordered sequence/time ranges and stock verification. +Its qualified manifest releases the queued probe builds. These are fixture and +build results only, not SDK performance acceptance. + +The executable failed-candidate probe passes all twelve observations (old Go, +new Go and Rust; regular/compact; forward/backward), with fixture bytes fully +restored after each sample. After repair, old Go/Rust getters expose the failed +middle match and advancing skips it; new Go getters retain the prior successful +match and advancing retries the middle match. This confirms user question 2's +source analysis. No common policy has been implemented while the reply is pending. + +The first complete private Go journal-package validation fails eight top-level +tests, all with concrete missing-fixture/helper errors: historical Zstandard +journals, shared FSS vectors and the libsystemd binary-field helper source were +absent from the relocated source parent. Preserve that attempt; it is not an +SDK correctness verdict. Copy the exact repository fixture/FSS/binary/live +helper trees, verify their hashes, and rerun both architectures plus the queued +controls into a new validation directory. The dependent absolute-path build +stops before compilation on the failed prerequisite and is queued anew against +the repaired validation. Lesson: a private source copy needs the tests' relative +fixture/helper closure, not just language source files; Rust's FSS fixture alone +does not satisfy Go's full package closure. + +Read-only inspection of frozen actual code justifies focused timing of all +five proposed cost removals. Go Step/StepBack retain their 48-byte frame but +skip errors.Is and its spills/sentinel loads on successful movement. GetEntry +shrinks from fifty to twenty-six instructions with its sixteen-byte frame +unchanged. Stateful unique enumeration drops from 201 to 158 instructions and +288 to 272 frame bytes; guard arithmetic and payload work remain unchanged. +Rust direct-step retains one outlined resolver call and unchanged posting +helpers, but removes the intermediate successful `Result` transport; +resolver frames shrink while the outer frame grows sixteen bytes. Rust's +constructor candidate removes two healthy call layers, while its two 128-byte +cursor copies remain. Do not describe those copies as eliminated or claim +timing wins from instruction/frame counts alone. + +The repaired Go package passes 1,380 amd64 and 1,381 386 test nodes. Rust's +ninety-six core tests also pass. The first Rust core command mistakenly used +the dependency alias `journal-core`; the manifest's package selector is +`systemd-journal-sdk-core`. Correct that harness selector without rerunning +successful Go checks. Public recovery validation then passes twelve cases and +fails only on a missing relocated historical fixture directory; copying and +hash-verifying the complete fixture tree produces thirteen passing public +cases. Preserve all setup failures and reuse only genuinely passed source/binary +results; fixture additions do not modify SDK code. + +The first unknown-size generator does not actually produce an unknown-size +frame: independent libzstd qualification reports content size 1,803. Merely +omitting the writer's header-insertion helper is insufficient because the Go +encoder optimizes a short buffered stream on close. Reject that mislabeled +fixture before any reader timing. Prepare a new generator-only source and +binary that writes the payload in two portions and explicitly flushes between +them, forcing the encoder's streaming header before final length is known. +Keep normal writer hashing, offsets and object bookkeeping; do not patch a +completed journal's object layout. Independently require UNKNOWN content size, +one complete standard frame, full decompression, all ENTRY references and stock +verification in both layouts. This remains a private fixture generator, never +a timed SDK baseline/candidate. + +An existing local Valgrind 3.27.1 installation provides software call/instruction +profiling despite unavailable hardware perf events. After retained-reader timing +finishes, profile the frozen Rust baseline/current/direct-step/direct-builder +helpers serially on core offsets, facade stepping, payload visits and fresh +filter construction. Disable remote symbol lookup and keep all outputs/caches +inside the private task directory. Validate semantic counters. Whole-process +software instruction/branch simulation and per-function call costs are diagnostic +evidence, not native elapsed-performance or hardware-event measurements. + +The explicit-streaming generator and qualification succeed in both layouts: +each compressed DATA object holds one 59-byte standard UNKNOWN-size frame, +decompresses to the exact 1,803-byte payload and is referenced by all 1,005 +ENTRY objects. Stock v262 verification passes. The failed known-size attempt +remains intact; only the newly qualified paths enter reader controls. +Both Rust candidates also pass all 192 small full-byte filtering observations +and 72 large independent-golden observations before diagnostic timing. + +The post-matrix checkpoint audit again reports complete-and-clean SOW +initialization, with SDK-0159 solely in progress and all three user behavior +choices retained. That is a framework checkpoint, not completed acceptance. +Extract the private-validation setup lesson into the journal compatibility +skill: preserve the tested source's relative fixture/helper closure, verify +copied bytes, and select Cargo packages from their manifest names rather than +folder names or dependency aliases. This changes the validation method only. + +### Retained Readers And Equivalent Cost Controls - 2026-10-16 + +The retained-reader publication control passes all 416 observations across +regular/compact layout, same-file/growing-file publication, Go/Rust and +unfiltered/filtered readers. Its sixteen paired timing comparisons record +twelve improvements and four inconclusive results, with no flagged regression. +Publication occurs outside the measured reader interval. Inconclusive timings +remain unproven under the performance contract; correctness agreement alone +does not close the gate. + +The remaining independent reader preflight passes 3,264 observations across +known/unknown-size compressed DATA, whole-file Zstandard, directory/OpenFiles, +both layouts and the four frozen baseline/candidate helpers. Full-byte native +goldens and strict verification qualify those inputs before native timing. +These passed preflights do not establish the unexecuted timing results. + +The private Rust absolute-path borrowing control passes all 42 allocation and +path-equivalence checks. On the recorded healthy inputs, open requests change +from the original SDK's five allocations/924 bytes, through the current +five/940, to four/837; first-scan and warm-scan costs remain unchanged. Recovered +open changes from current six/1,162 to five/990 on the longer recorded path. +Retained recovery storage still adds sixteen bytes per file; fewer cumulative +allocation requests do not prove reduced retained or peak memory. Relative and +absolute paths, both reader APIs, bounds and mmap strategies preserve full-byte +results. Native timing is still required before adopting this private control. + +All twelve Go CPU profiles and sixteen Rust software call/instruction profiles +complete with independently checked semantic counters. Rust direct stepping +removes about twenty-three simulated instructions per row while its underlying +offset and ENTRY operations remain unchanged. Direct filter construction +removes two call layers and about thirty-one instructions per build; the two +128-byte cursor copies remain. Those profiles justify focused native timings +and are not elapsed-performance evidence. + +The first focused Go successful-step comparison finishes eight cases: two +improvements and six inconclusive results, with identical counters and no +flagged regression. The serialized getter, unique, Rust step and Rust builder +comparisons continue. None of these equivalent private candidates has yet +been adopted into the root SDK. The three pending user behavior choices remain +unchanged and no dependent policy has been implemented. + +The in-progress getter comparison flags four facade-stepping cases in both +layouts/directions at approximately ten to thirteen percent, although that +measured facade path calls the realtime getter rather than GetEntry. Keep the +candidate unadopted and inspect normalized generated code before attributing +the cause. A smaller GetEntry body is not sufficient acceptance evidence. + +Queue the already prepared matched writer controls only after all five hot-path +timing stages finish. Then compile a separate non-instrumented Rust open/header/ +drop probe against fingerprint-matched original and absolute-borrow libraries; +check the borrowed library hash against its validated build. Compare both +layouts, small/large healthy inputs, core/SDK, live/snapshot, both mmap strategies +and absolute/relative paths with ten alternating measured pairs and two warmups. +Keep header counters identical, record process resources, and isolate timings +from build/profiling/test work. Relative paths are an explicit unchanged-path +control. These queued measurements do not depend on pending filter policies. + +The five focused hot-path controls finish all ninety-four comparisons with +matching counters: ten improvements, seventy inconclusive results and fourteen +flags. Getter splitting flags four facade cases; Rust direct stepping flags +four compact cases; direct construction flags six restart cases. Keep those +candidates out of the root SDK. Go successful-step and unique cycle-error +outlining have no flags in their bounded deltas, but remain unaccepted as full +SDK performance equivalence. Fewer instructions do not override native timing. + +The unique-profile/codegen investigation identifies avoidable work that both +baseline and current perform: stateful unique traversal needs object metadata +and next-FIELD offset, but the generic DATA parser loads five unused scalars +and transports/copies the complete 64-byte result through multiple frames. +Prepare a private unique-specific 24-byte result control without the rejected +getter split, and compare it against the generic accessor and original SDK. +Preserve the exact 64-byte access request in both layouts, accessor/short-buffer +errors, DATA type and declared-size thresholds (64 regular, 72 compact), flags, +next link, borrowed-byte lifetime and unique state transitions. Add differential +accessor checks including short buffers and injected errors, then run existing +cycle/restart/compression tests and both Go architectures before timing. Do not +add alignment, arena, posting or zero-tail checks on this existing compatibility +path. The unused-work finding is not proof of the earlier four-percent flag's +cause, and the candidate must pass actual timing before root integration. + +The read-only getter follow-up compares forty-one generated functions. All +forty outside GetEntry have the same normalized operations, calls, frames and +spills; recursive source comparison confirms only reader_entry.go changed. +Some unchanged functions and interface tables move. In particular, the two +iterator-reset method slots move from sharing one 64-byte block to straddling +a boundary. This establishes a layout difference, not causation; preserve the +four adverse native measurements and the candidate's rejected status. + +The matched writer matrix finishes all thirty-two comparisons and 768 retained +observations, with native byte identity within each language and stock-verified +validation samples. It records eight improvements, twenty-two inconclusive +results and two flags: Go regular/plain/per-entry-publication structured writes +at ratio 1.01960, interval [1.00269, 1.05581], and Rust compact/long-Zstandard/ +no-intermediate-publication structured writes at 1.02943, [1.00901, 1.06592]. +Both remain open. Investigate changed shared validation/chain dependencies; +do not weaken checks or infer causation from a short timing difference. + +Read-only pre-execution review catches a private differential-test setup error: +synthetic Go Readers must configure their cached layout after setting compact +flags. Fix both before tests run, so the checks exercise actual 64/72-byte +declared-size thresholds. The new accessor itself preserves the intended +contract. Rust open-probe package names, fingerprint linkage, APIs and fixture +paths also check out. Qualify its header observations independently from native +header fields: both small inputs declare five entries/tail sequence five; +both large inputs declare 100,000/100,000. The probe's digest is therefore zero +on these particular inputs and is not a full-row correctness check; the earlier +independent full-byte preflights provide row validation separately. Retain the +active binaries/driver and postcheck all observations against native metadata. + +Read-only writer codegen review finds two bounded equivalent controls to test. +Go findData/findField already branch on a nonzero next-hash link, then repeat a +terminal-zero guard/boolean merge outside that branch. Move only guard calls +under those existing nonzero branches, after depth updates; updateHashChainDepth +keeps its guard before depth increment. A zero next link returns true without +state mutation today, so skipping that redundant call preserves checks, errors +and mutation order. Larger guard frames/initialization remain and the earlier +timing flag is not thereby explained. Preserve acyclic unordered and cyclic +behavior; validate both architectures and byte-identical writer controls. + +Rust's new complete-stream fallback is inlined into decompress_zstd_payload, +growing that function from 0xcd6 to 0x67a3 bytes and displacing native success +far into its body. Both baseline and current enter with roughly eleven-KiB +frames and stack probes; the current frame is smaller, so do not describe this +as an added-frame regression. Test inline(never) on the existing private +streaming-with-cap helper only, leaving native-first dispatch, output cleanup, +complete-stream validation and cap unchanged. Keep ordinary unknown-size streams +as a valid performance case; do not mark that helper cold. Build the candidate +from the root SDK, excluding rejected step/builder candidates. Check actual +generated native frames/calls, core/public recovery/compression suites, known +and unknown DATA/full-file compressed reader preflights and matched compressed +writer/read timings before any root integration. Neither control changes a +public contract or depends on pending filter policies. + +The repaired unique-header control passes eleven targeted tests on each Go +architecture and all sixty-four compressed unique observations, including +unknown-size DATA, whole-file compression and both access strategies. Its final +combined comparison against the original repaired SDK records twenty +improvements and four inconclusive results across twenty-four cases, with no +flagged slowdown. Intermediate outline/header deltas retain one small flag each +on other paths; they remain diagnostic measurements, not separate shipped +baselines. Apply the validated successful-step early return, unique cycle-error +outline and narrow header together to the root SDK as an equivalent cost +removal. Preserve generic DATA parsing for other consumers, the rejected getter +split, and all pending filter policies. Overall performance acceptance remains +open; the complete original-baseline gate still applies after integration. + +All 1,536 Rust open observations pass the independent native-header postcheck, +including unchanged qualified fixture hashes and frozen binary hashes. The +sixty-four comparisons record two improvements and sixty-two inconclusive +results, with no flagged slowdown. This confirms the scoped observation contract +and allocation findings, not elapsed equivalence or a full-row digest claim. + +The integrated Go narrow helper's actual amd64 code uses a 24-byte frame versus +the generic wrapper's 168 bytes plus its separate 64-byte parser frame. The +enumerator falls from 272 to 144 bytes. Five unused scalar loads and the full +64-byte header copies disappear. The payload callback and visitation have +unchanged normalized code/allocation calls; static code does not substitute +for measured allocation counts. Exact runtime files match the tested frozen +candidate, and the root accessor contract test is copied unchanged. + +Writer-cost candidates pass 105/106 selected Go test nodes on amd64/386, +ninety-six Rust core tests, thirteen public reader-hardening tests and all +ninety-six independent full-payload compressed reader observations. This is +selected Go writer/append/chain coverage, not every writer-related test. +Their native timings are ongoing. + +Review catches a disassembly harness defect: objdump demangling with raw nm +symbols exits successfully while emitting empty bodies. Preserve the exact +loaded initial coordinator source and reject those captures as body evidence. +The corrected separate collection uses raw symbols without demangling and +requires an actual symbol label/instructions. All four captures pass. The +outlined Rust native decoder now has ninety-seven static instructions and +reserves forty stack bytes after six pushes, replacing the large inlined +fallback frame/probes. The fallback remains a tail call on native failure. +This is actual codegen evidence; the unchanged caps/cleanup and decoded bytes +are separately tested, and elapsed tradeoffs remain unaccepted until measured. + +All thirty-two writer-cost deltas finish with two improvements, twenty-eight +inconclusive results and two flags. Neither private writer change is accepted. +The compressed-reader deltas also flag several cases, including whole-file +inputs whose DATA is uncompressed. Read-only checks confirm all 183 shared Rust +source/Cargo/config files match except the intended inlining annotation; nested +FFI build outputs do not affect these package builds. Both release logs show +fresh SDK/helper compilation, pinned equivalent flags/features and frozen +matching binary hashes. No source/helper/stale-link mismatch explains the flags. + +Before further micro-tuning, queue a bounded same-binary timing diagnostic after +the active compressed sequence. Compare each frozen executable against itself +on representative entry/unique/facade/core/verification paths, plus fresh/restart +filters with 250,000 repetitions to lengthen very short samples. Use the same +alternating ten-pair/two-warmup method and independent counters. This measures +the harness/host variation floor; it does not waive adverse candidate results, +introduce a slowdown tolerance or turn inconclusive results into accepted +equivalence. Keep the required zero-slowdown contract and all pending product +choices unchanged. + +The first compressed sequence stops on a harness scope error: low-level +JournalFile core modes receive an entire .zst archive and correctly reject its +non-journal magic. The SDK layer stages archives; low-level core modes require +native journal bytes. Preserve the 218 completed reader comparisons (93 known, +93 unknown, 32 archive SDK cases), exclude sixteen unsupported core/archive +combinations across both layouts, and resume the remaining supported cases +without rerunning completed observations. The first same-binary coordinator +correctly refuses the failed prerequisite before launching any diagnostic. + +The resumed archive indexed-open control reveals an unbounded workload choice: +1,000 full archive decompressions take 25-26 seconds per sample, roughly ten +minutes per paired case. Prepare twenty opens per sample with the same ten +measured pairs/two warmups and equal before/after counters; native file contents +and the zero-slowdown rule stay unchanged. User question 4 requests permission +to terminate only this task's verified active benchmark child under captured +driver PID 2731853, retain completed results, and rerun unfinished controls. +Server instructions require approval before killing processes; no signal has +been sent and the existing run remains active while permission is pending. + +Prepare one isolated Rust reader-data-cursor identity control from root source, +excluding the rejected stepping/builder changes and the unaccepted decoder +outline. Retain the factory Result through the DATA guard scope. Borrow only +its discriminants; Err/None return before reading recovery state. Healthy +Ok(Some(_)) returns the intact Result when recovered_last_entry is None; an +active prefix delegates to the unchanged consuming for_reader. Some(None) +continues to identify an active empty prefix. The outer error/recover/rebuild +wrapper and public layouts/signatures remain unchanged. The recovered branch +performs one extra cached status read but no discovery or I/O. Inspect actual +copy/frame behavior, then validate core/public recovery/filter controls and +longer paired fresh/restart timing before root integration. This is equivalent +constructor cost removal, not a pending position/time-seek policy change. + +The regular archive stage finishes naturally with all fifty-three resumed +comparisons, and compact known-size DATA also finishes. The previously named +regular archive driver has exited, so question 4 about terminating its child is +withdrawn; no process was signalled. Before the compact archive stage starts, +preserve the original loaded wrapper and bound future staged indexed-open +controls to twenty opens per sample. This affects only a future benchmark +invocation: completed observations and the currently running native unknown-size +DATA stage keep their loaded scripts. Preserve ten alternating measured pairs, +two warmups, matching counters and separate original/candidate results. + +All four resumed compressed-reader stages finish, with every supported case +retained. The identical-binary diagnostic then fails before timing: selecting +only verify-file removes the snapshot declaration required by the independent +auxiliary oracle. This is a private harness setup failure, not an SDK result. +Preserve the failed output. Repair only oracle setup in a separately named +wrapper by deriving goldens from the original complete auxiliary declaration; +retain the selected timed cases and executable arguments unchanged. Run the +repaired diagnostic in fresh output directories before further timed controls. + +The complete private Rust decoder-outline comparison covers 542 supported +reader cases: 19 improvements, 410 inconclusive results and 113 adverse flags. +All retained operation counters match; sixteen low-level core/archive cases +are excluded because the low-level API accepts native journal bytes only. +Reject the outline as an elapsed-performance candidate; it has not entered +root source. Reduced static frames/instructions do not overrule these results. +No original-baseline SDK performance blocker is closed by this experiment. + +The repaired identical-binary diagnostic retains selected timed cases/arguments +and uses the full original auxiliary declaration only for independent oracle +setup. Read-only review confirms this separation and identical executable +hashes in both sides of the manifests. Queue isolated constructor build/tests +only after both native diagnostic stages finish; no build/profile/test overlaps +the measurements. The timing classifier itself has no equality verdict, so +a confidence interval crossing one is neither a detected regression nor proof +of unchanged performance. No actual user choice specified rejecting every +such interval; that requirement appears in this SOW-authored interpretation. +Do not silently convert it to a pass: present the measurement-method limitation +with concrete identical-binary evidence when available. + +Read-only review catches a second private diagnostic setup mistake: paths +passes the 100,000-entry corpus to Go VerifyFile despite the existing bounded +verification rule. verify_graph.go rebuilds repeated posting lists per entry; +this fixture invokes its known quadratic path. Four ordinary regular-layout +same-binary cases finish with overlapping intervals before the first verify +sample. Preserve them. Prepare a corrected four-stage plan: compact ordinary +paths, independent regular/compact verification on qualified 1,005-entry inputs, +and the longer fresh/restart filter controls. The ordinary path stages carry +no auxiliary verification cases. Confirm the bounded headers before execution. +User question 4 is a new approval request for the current diagnostic child +under captured driver PID 2871669, not the naturally completed archive driver. +Stop only the verified SDK verification child using a stable process handle, +then let its timing/coordinator parents collect failure normally. No signal or +replacement workload is authorized by an unanswered question; server process +approval comes from the user-provided conversation instructions. + +Read-only spot-check finds no concrete defect in the prepared targeted stop +or bounded continuation: driver/time/SDK ancestry, start times, argv and +executable are rechecked around pidfd acquisition; the continuation requires +the old failed result/stop record, preserves four completed cases by hash and +serializes all stages. Both replacement verification headers declare 1,005 +entries. Scripts remain unexecuted while user question 4 awaits approval. +The three behavior choices remain pending; both requested SOWs remain open. + +### Execution Blocker Audit - 2026-10-16 + +The same unanswered user decisions remain across three consecutive goal turns: +reverse-filter nearestness, failed-entry position, explicit filtered time-seek +search/error discovery, and permission to stop the owned oversized diagnostic. +The latest direct process/session inspection at 21:42 UTC confirms the same +verification child is still live with its saved parent/start identity. Four +completed comparisons are retained; replacement controls and constructor +validation have not started. The process is not assumed terminal from missing +output. No permission has arrived, so no signal or dependent policy edit is +allowed. Remaining implementation and serialized acceptance cannot proceed +without those decisions. Mark the thread goal blocked; keep this SOW +in-progress and SOW-0121 paused, preserving all work and the full objective. + +### Operational Question Withdrawn - 2026-10-17 + +At 04:34 UTC, direct session collection shows the diagnostic coordinator +ended after its existing 1,800-second subprocess timeout. The waiting candidate +build also ended before starting validation. Direct process inspection confirms +the captured driver, time wrapper and SDK child are all absent. Operational +question 4 therefore no longer requires a user decision; no new termination +action is needed. The three SDK behavior choices remain pending. Preserve the +failed diagnostic and four completed cases; do not treat its terminal state as +performance acceptance or claim the constructor tests ran. + +### Approved Three-Fix Resumption - 2026-10-17 + +The user explicitly approves all three recommended SDK fixes and asks whether +they were reproduced and whether they are inherited or newly introduced. +Prior evidence includes an executable reverse-anchor probe, twelve paired +error-position observations, and sixteen adverse Go explicit-time comparisons. +Those are discovery controls, not a claim that permanent regression tests pass. + +Root-cause and blast-radius update: Rust retains stale posting heads after +filter installation/AND early exhaustion; FileReader commits its core location +before row load; the native Go port uses a linear posting seek after the +global time anchor. Existing Rust anchor and load ordering predate this SOW; +new validation adds observable failure triggers and the Go port changes error +position. Confirm inherited behavior separately against the frozen baseline. + +Implementation/validation plan: preserve current source bytes, add permanent +paired layout/direction controls and run them before runtime edits. Correct +nearestness using existing bounded cursor reset/alignment paths; restore the +prior location/cache after nonrecoverable row-load errors without cloning +whole cursors per row. Add explicit-time binary posting search using captured +counts and existing checked node/slot access, preserving zero/singleton lazy +paths, recovery identity, chain guards and retry invalidation. Tests must cover +EOF/direction changes, filter installation at a resolved row, repair/retry, +boundary times and visited-slot failures. Add a probe-count bound rather than +a timing assertion to verify reduced search work. Run both SDK/facade surfaces, +layouts and Go backends, relevant recovery/filter suites, then isolated release +comparisons. Include direct baseline executable controls for provenance. + +No changes to the format, runtime purity, writer contract or ordinary sequential +filter error discovery are authorized by these fixes. Maintain docs/specs for +changed reader behavior after tests establish it. All synthetic evidence stays +inside this repository, with existing local caches and no live host journal. +The three decision blockers are resolved; SOW-0159 remains the sole in-progress +implementation and SOW-0121 remains paused. The earlier diagnostic processes +are absent, so no termination permission or signal is needed. + +Frozen-baseline execution confirms Rust reverse skips and invalid-size error +bookmark leakage in both layouts. Old Go reaches the middle match on direct +reverse but repeats the final match after forward EOF; the native port changed +that sequence to skipping the middle. Original outputs are retained under +approved-fixes/baseline-provenance. Permanent pre-fix controls reproduce Rust +anchor/position failures and all sixteen Go explicit-time read-count failures +(517/522 reads for 1,005 postings). Raw unfiltered Go Step is lazy and does not +load ENTRYs; its error-position controls use dense/sparse filters, while Rust +also covers unfiltered selected-row loading. + +The bounded position repair uses a private Rust leaf-direction tag beside +existing flags, invalidated on installation or explicit resolved-position +changes. Prepare head/tail only when a visited leaf changes direction; preserve +recovery flags and captured counts. Go stores direction beside walkValid, +invalidates before fallible reset, and commits anchors after ENTRY validation. +Restore Rust's old location after unsuccessful selected-row recovery without +cloning a full cursor. A compact ReadAt repair control also exposes copied +window staleness after malformed ENTRY bytes are repaired. Use existing cold +window invalidation before retry, including after intervening getters, while +retaining captured populations and avoiding Stat/header refresh on this path. + +The focused post-fix controls pass: 190 Go test events across six new test +groups and eighteen Rust public hardening cases. Each of the sixteen Go +explicit-time probe controls now reads 55 times, versus 517/522 before the fix. +This establishes reduced search work, not elapsed performance acceptance. +Duplicate-time tests identify the exact selected sequence number, cover live +and snapshot bounds, and cross posting-node boundaries. Visited zero slots +error and permit repair/retry; unvisited malformed slots do not prevent an +explicit time query, but ordinary sequential traversal still reports them. + +Retry also invalidates copied windows after a failed Go filter build, not only +after a selected ENTRY error. Rust public cursor requalification rebuilds at +the head and must retain that new direction rather than copy the old backward +tag. A new core control reproduced the latter failure before its correction; +the full workspace and optional allocative-feature checks remain to run. +The nested AND/OR direction test was corrected to force the intended early +branch exhaustion; its corrected form passes after implementation, but was +not executed against the pre-fix source. Keep that distinction in the evidence. + +Private evidence for this resumption lives under approved-fixes-k_n92jni: +before-tests/before-2 retain expected failures, baseline-provenance retains +fifteen successful build/probe commands against the frozen original SDK, +before-requalification retains the new core failure, and after-2 retains the +successful focused controls. Full Go amd64/386/race and Rust 1.91 workspace +plus allocative-feature validation have now started against current source. +No native timing runs overlap those checks. + +The first whole-source checkpoint passes Go amd64 (1,650 test events), Go +386 (1,651), and the complete Rust 1.91 workspace, including 185 SDK and 97 +core tests. The optional allocative-feature command initially fails offline +because build inputs are absent from the local cache, rather than because of +a compiler/test failure. Reuse checksum-matched existing cache archives and +fetch only already-locked missing build inputs into the repository cache; +the retry passes all 97 core tests and three existing ignored documentation +examples. No dependency version, compiler floor or system installation changes. +Go race validation remains running at this checkpoint. + +Same-failure inspection extends the approved position repair to directory +movement. Sequential traversal commits the next file index before that child's +ENTRY load; merging can prefetch a different row in the selected child before +another child fails. Restoring file-local state alone does not preserve the +directory's last successful payload/metadata selection. This is an inherited +directory state issue, not the separate seek/getter gap tracked in SOW-0045. + +Apply the already-approved last-successful-position contract across those +wrappers. Add regular/compact, SDK/facade, forward/backward, filtered/unfiltered +and Go backend controls for sequential file transitions and merged direction +changes, including repair/retry and complete payload/cursor identity. Preserve +the previous selection with scalar state and existing cold row-reset paths; +avoid full cursor cloning, row scans and new steady-state allocations. Retain +unaffected prefetched candidates and captured filter populations. Go's lazy +unfiltered movement also needs rollback when directory key loading fails. +No new error category, recovery policy or sequential posting policy is added. + +Source inspection also finds fresh Rust filter installation invalidates every +leaf immediately before Head/Tail/realtime positioning resets those leaves +again. Limit installation invalidation to resolved locations, matching the +existing set_location pattern. Preserve all resolved-anchor controls and +measure fresh/restart behavior with the final release source. + +Go race validation also passes all 1,650 test events. The corrected directory +fixtures use a shared seqnum identity with disjoint sequence ranges for the +sequential fast path, and overlapping independent identities for merge cases. +The first fixture attempt did not establish the intended sequential path and +is retained as a harness failure. The corrected pre-fix controls reproduce +wrong payload selection/unset cursors; after scalar rollback, all 56 included +Go layout/backend/API/filter/direction cases and 32 Rust cases pass. + +Directory rollback keeps the previous child/index, clears that child's cached +candidate and row state, and retains other candidates plus operational bounds. +Go snapshots raw movement's realtime anchor as well as index/override. A private +directory retry flag fits beside the existing flags and invalidates copied +windows on the next movement, after any intervening getter. Its final layout +and elapsed cost still require measurement. No steady-state heap storage is +added. The pre-existing Go unfiltered backward time-probe error suppression is +now explicitly tracked in paused SOW-0045, with that one unsupported error +trigger excluded from these position tests rather than counted as a pass. + +Two final directory controls are required before freezing source. A first +backward sequential error has no committed row; restoring Go's unset index +while retaining its prepared direction would make retry return EOF. Reset +initial sequential direction on that cold error path. Rust also needs to +retain merge-active routing after consuming an explicit realtime seek; +otherwise retry can enter the sequential fast path and skip an already +prefetched candidate. Add the corresponding private flag alongside existing +directory flags, clear it on explicit boundary resets, and preserve it through +merged success/error/recovery. Test initial boundary and pending-time errors, +repair, complete remaining order and EOF in both languages and facades. + +Both final controls fail before those corrections and pass afterward: sixteen +Go initial-tail cases return premature EOF, while Rust's pending-time case +skips a retained candidate. Cold Go initial-error direction reset and a Rust +merge-active flag resolve them. The flags fit existing directory padding; +no persistent candidate-offset fields or full cursor copies are introduced. +The final-source workspace passes Rust 1.91 (187 public SDK tests, 97 core), +the optional allocative feature (97 core tests), Go amd64 (1,756 test events) +and Go 386 (1,757). Final race checks remain running at this checkpoint. + +Final release adapters pass all 552 shared chain/stream/recovery observations +against the pinned full-feature v262 oracle. All 35 consumer examples pass; +the wiki validator checks 16 pages and git diff whitespace checks are clean. +The local SOW audit reports initialization complete and clean; this is a +framework checkpoint, not SDK performance acceptance or SOW completion. + +Final layout/allocation measurements confirm the latest flags add no size: +Go Reader is 616 bytes/class 640 on amd64 (baseline 624/class 640) and 524/class +576 on 386 (unchanged); DirectoryReader is 168/class 176 and 84/class 96 +respectively, unchanged. Native Go filterBuilder is 120/class 128 on amd64 and +68/class 80 on 386, versus the old scanning builder's 72/class 80 and 36/class +48. This fixed representation cost belongs to the approved native-filter port; +do not describe it as unchanged builder memory. Rust posting/filter/cursor +remain 128/136/280 bytes and DirectoryReader remains 352. Core JournalFile is +568 versus baseline 552, and JournalReader 640 versus 608. Healthy SDK open +still makes five allocation requests, totaling 940 versus 924 requested bytes; +the sixteen extra bytes retain recovery state. Both 100,000-row layouts keep +identical content counters, first-scan allocation requests and zero warm-scan +allocations. These are requested-byte measurements, not peak/usable allocator +memory or elapsed-performance proof. + +The user has been asked one evidence-method decision: keep the no-tolerated +reproducible-slowdown requirement while using a fixed finite comparison plan, +or retain this SOW's stronger self-authored rule that blocks every inconclusive +case. Four identical-binary controls have overlapping intervals and no adverse +flags; they demonstrate uncertainty, not candidate equivalence. No gate change +or slowdown waiver is inferred while the answer is pending. Continue approved +local validation and matching baseline/final-source measurements independently. +The frozen plan retains ten alternating measured pairs/two warmups per case, +all 488 original file cases without repeating explicit-time cases, bounded +auxiliary verification, directory/OpenFiles, compressed/staged and writer +controls, plus one independent confirmation block for adverse cases. Preserve +every block and investigate reproducible costs; never sample until favourable. + +The final race run also passes all 1,756 Go test events. Release builds use +matching baseline/candidate compilers (Go 1.27.1 and Rust 1.97.1); the supported +minimums remain Go 1.26.2 and Rust 1.91, tested separately. The thirty-two +explicit filtered realtime comparisons all show improvements against the +original SDK baseline. The complete remaining-file matrix is still running; +the time-seek result does not establish acceptance of unrelated paths. + +Read-only same-failure inspection finds one new cold retry-loop issue before +close-out: DirectoryReader.Close retains child readers whose accessors have +been cleared. A first filtered movement returns the existing invalid-journal +error and arms retryCopies; a second movement would call the cleared accessor. +This originates in the new directory rollback code, rather than the baseline. +The minimal-complete repair preserves the existing closed-reader error and +guards copied-window invalidation when the accessor is absent, inside the +error-only retry loop. Add repeated closed-directory movement controls for +both directions and SDK/facade APIs. The nil accessor is independent of file +layout and access backend, so do not duplicate this control across those +dimensions. Reproduce +before the guard, then validate the guard. No Rust equivalent is needed: +Rust reader destruction consumes ownership and exposes no post-Close method. +Do not overlap those test/build processes with the active native timing +sequence. Record the pre-guard binary provenance of that sequence and rebuild +the final source before subsequent directory measurements. + +Decision 11 supersedes the earlier exhaustive performance execution plan. +The user authorized stopping only the captured broad benchmark driver; after +identity verification, its interrupt completed and the coordinator exited. +All 32 time-seek comparisons and 28 completed ordinary file comparisons remain +retained, with no regression flag in those completed cases. The interrupted +stage is intentionally partial, not a failed SDK operation or a completed +matrix. No further combinatorial stages are required mechanically. + +Read-only performance inspection confirms that ordinary movement retains +native arrays/reusable row buffers, compressed values remain outside metadata +work, new directory rollback uses scalar checkpoints, and the closed-accessor +guard adds no executed healthy-path operation. Fixed feature memory costs +remain as recorded. Target the actual remaining concerns: fresh/restarted +filter construction, representative sequential/merged directory movement and +the two earlier adverse writer observations. Retain existing stream/recovery, +allocation and work-count evidence; do not build another framework to certify +each layout/backend/API combination. Unadopted constructor-identity, decoder, +getter and cursor experiments are rejected from this implementation unless a +credible reproduced cost requires reconsidering them. The abandoned additional +identical-binary continuation is superseded by this evidence-method decision. + +The closed-directory control reproduces the second-movement nil-accessor panic +before the guard (closed-retry-before). The guard preserves invalid-journal +errors without changing successful traversal. Its post-fix focused checks are +the next validation step; prior full-source suites are not relabelled as +having executed this final guard. + +The focused final-source comparison isolates a real Rust setup cost: regular +100,000-row facade/snapshot/windowed/forward fresh filters take 1.04870 times +the baseline cost, with a paired 95 percent interval [1.04488, 1.06466]. The +same installed-filter restart improves (0.98348, [0.96674, 0.98590]). This is +evidence to revisit the narrow constructor-copy candidate, rather than expand +the full matrix or accept an observed slowdown. Go's corresponding fresh and +restart controls improve to ratios 0.12883 and 0.01039 respectively. + +Apply only the existing reader_data_cursor closure optimization to current +Rust source: retain the factory Result intact, release the DATA guard at the +inner scope boundary, and return Err/None or healthy Ok(Some(cursor)) without +the consuming for_reader round-trip. Read-only verification confirms healthy +for_reader returns its input unchanged, the cached recovery-status read has no +side effects, and active empty/nonempty prefixes still take the unchanged +qualification path. Preserve the outer error/recover/rebuild wrapper and all +later direction/population corrections. No public signature, file format, +error policy, allocation policy or query algorithm changes. The recovered +branch adds one cached status read; it does not discover recovery or perform +I/O. Validate existing core/public recovery and filter retry controls, then +repeat this actual setup workload with matching helpers and complete counters. + +The focused ordinary comparison also reproduces smaller observations: +Go materialized SDK-entry ratio 1.01989 [1.00369, 1.02332], Rust whole-file +facade-next 1.05624 [1.01301, 1.16967], while Go borrowed payload visitation +improves to 0.90686 [0.89302, 0.93333]. Inspect actual executed work before +adding tests or changing algorithms. Disassembly finds Go's readEntryAt frame, +instruction count and allocation/map/string/copy calls unchanged, with only +a byte-sized cached payload-offset load changed in the DATA path. The ordinary +GetEntry path now checks the special filtered orphan index before valid range +checks. Make the ordinary unsigned range check first and retain the identical +special -2/override behavior only on the out-of-range path. The unsigned check +rejects negative indices on amd64 and 386 because slice lengths fit MaxInt. +This removes ordinary-position branches without changing errors or access. + +Rust's scalar position checkpoint is already inlined; no Result allocation or +getter call remains to remove. The selected-ENTRY error recovery body is still +expanded into step_valid, adding cold recovery locals and restore/invalidation +blocks to its frame. Mark that existing recover_after_error body cold and +non-inlined, preserving the body, checkpoint and every rollback/retry rule. +Neither change adds a public API or weakens validation. Run affected Go/Rust +controls and repeat only the representative observations against the original +baseline. Do not infer that these few instructions explain every timing delta. + +Release-build-5 confirms the intended generated paths. Go GetEntry's normal +prefix falls from 17 instructions/four conditional branches to 14/two; +header cache-fill position work falls from 15/five to 12/three. The original +GetEntry prefix was nine/two, so do not claim complete instruction identity. +Rust's recovery helper is now called only after failed ENTRY loading, and +step_valid shrinks by 1,054 code bytes. Its stack reservation increases from +0x4f8 to 0x508 rather than shrinking; no frame improvement is claimed. Healthy +DATA cursor construction bypasses for_reader, although the outer 128-byte +result transfer remains. Elapsed acceptance still needs the focused repeat. + +The final-source existing integration runners pass all 63 strict verifier, +34 directory and 42 mixed-directory checks against systemd v262. Retained +directory fixtures are reused without regeneration/deletion; command mapping +selects the pinned oracle explicitly. Previous 32 general, 72 compression, +20 compact, 18 binary, 18 live-feature and 552 shared hardening observations +remain valid with their original provenance. They are not relabelled as new +runs. No writer, codec, FSS or publication algorithm changed in the final +position/constructor code-generation edits. + +Final Rust 1.91 workspace validation passes, including 187 SDK and 97 core +tests; the optional allocative feature also passes 97 core tests. The final +Go amd64 full run passes 1,760 test events but its live stock-reader stress +test fails when the writer exceeds the existing 45-second deadline. The same +control previously passed in 12.94 seconds. This is not a reader assertion +failure and is retained as a failed full run, not reported green. Concurrent +release linking/workspace testing is a working theory for contention, not a +proven cause. Run that unchanged 1,200-entry/fsync-every-entry control alone, +without other task builds/tests/measurements. Do not relax its timeout or +change writer behavior to obtain a pass. The failed Go chain did not execute +its final 386/race commands; validate their affected read paths explicitly. + +The unchanged isolated stress control passes in 38.03 seconds with the original +45-second deadline: all 1,200 entries are written and observed in order. This +resolves the functional test failure but does not establish the cause of the +earlier timeout or explain the duration difference from the prior 12.94-second +run. Retain both results and avoid attributing environment variability to an +SDK code change without evidence. Final 386/race commands select affected +reader/filter/entry/snapshot/recovery/Explorer/facade/chain tests and omit the +unrelated Go live-writer group; the preceding complete suites and this isolated +durability control retain their separate meanings. + +The final representative reader controls no longer reproduce the earlier +setup/ordinary costs after the equivalent code-generation corrections: Rust +fresh setup is 0.99693 [0.94300, 1.09289], installed restart improves to +0.97172 [0.87909, 0.98150], Rust facade-next is 1.00501 [0.94628, 1.02716], +and Go materialized entry access is 1.00389 [0.99594, 1.04393]. Uncertain +intervals are not claims of equality; the unchanged native algorithms, +allocation/work counters and removal of redundant healthy-path work provide +the additional decision-11 evidence. Final Go 386 and race selections each +pass 1,153 affected reader/filter/entry/recovery/directory test events. + +The Go regular/plain/publish-1 structured writer confirmation still reports +1.04883 [1.02723, 1.09979]. The Rust compact/long-Zstd writer observation +does not reproduce (1.01045 [0.97594, 1.06459]). A 100,005-row append-only Go +profile produces byte-identical journals and takes 5.87984 seconds before and +5.79988 seconds after this SOW, but that larger control does not establish +the smaller case's cause. DATA header reads and publication dominate; regular +DATA array traversal also carries added guard control flow. + +The private hash-lookup terminal-zero relocation is not adopted. Its latest +two original-baseline controls are inconclusive, and an older private delta +flagged long-Zstd/raw/publish-1 at 1.01864 [1.01040, 1.04408]. Retain both +results rather than choosing the favourable block. A distinct equivalent +cost removal is justified in appendToDataEntryArray: the existing branch has +already proved the next offset nonzero. Test `steps != 0 || next <= offset` +at that call site and invoke the unchanged advanceUnordered only when true. +This is exactly advance(current, next, false) under that nonzero precondition, +including continued tracking after an unordered link. It removes the ordinary +boolean success-result merge without changing traversal, error, allocation, +I/O, layout or writer publication contracts. Validate existing writer/append/ +cycle controls on amd64 and 386, inspect generated code, then repeat only the +two affected representative Go writer observations. No new writer cache, +framework or algorithm is proposed. + +The ascending-link specialization passes 62 existing writer/append/guard test +events each on supported Go 1.26.2 amd64 and 386. Saved Go 1.27.1 disassembly +confirms the ordinary link skips boolean result merging: ten instructions +become six useful instructions plus one alignment NOP, with three conditional +branches becoming two. The 72-byte frame and guard initialization remain; +active/unordered links retain the original checks. This is instruction/work +evidence, not an attribution of the earlier timing magnitude. + +The final two original-baseline writer comparisons retain ten alternating +measured pairs and two warmups: regular/plain/publish-1 structured is 0.98828 +[0.88800, 1.08311], regular/long-Zstd/publish-1 raw is 0.96598 +[0.92051, 1.01694]. Neither flags a regression; both remain statistically +inconclusive. Every 10,005-row/32-field output matches the baseline bytes and +stock v262 verification passes. The earlier 1.04883 block is retained against +its previous source, not erased or averaged with these samples. Acceptance +under decision 11 combines the targeted repeats, larger append-only control, +equivalent reduced control flow, unchanged I/O/allocation/publication paths, +and writer correctness tests. It is an engineering judgment that no remaining +reproducible writer regression is established, not a slowdown waiver or proof +of exact equality. No further combinatorial timing stage is required. + +Final writer evidence is writer-ascending-guard (source/build/test/codegen +provenance) and writer-ascending-control (48 warmup/measured observations). +Reader/interoperability evidence remains release-build-5: only writer_arrays.go +changed afterward. Other private lookup-relocation candidates remain rejected. +Original-baseline executable provenance covers the Rust reverse/bookmark +failures and Go direct/EOF anchors. Inherited original-Go fatal-load and broad +directory-origin conclusions rely on source inspection; those exact original +baseline executable controls were not run. Corrected nested Boolean coverage +was not executed before its fix. Do not strengthen those provenance claims. + +### Internal Final-Review Corrections - 2026-10-17 + +The pre-review framework audit is complete and clean, but whole-surface +read-only review finds three contract gaps before external review. The earlier +local-readiness statement is superseded until these repairs are validated. + +- Both directory FlushMatches implementations rewind their children but retain + a directory-level pending realtime seek. AddMatch; SeekRealtime(later); + FlushMatches; Next therefore skips the beginning despite the documented and + approved head-reset contract. Clear only the pending directory seek in flush; + do not clear it in general filter mutation, which must preserve explicit + positioning. Test SDK/facade, sequential/merged and both file layouts. +- Sequential directory EOF discards the last selected anchor. Reversing then + boundary-resets and repeats the endpoint instead of returning its nearest + predecessor/successor. The native Go port newly enables filtered sequential + directories; Rust has the same inherited inconsistency. Extend the existing + scalar rollback to filtered EOF only: retain the directory key/index and + selected child's position, discard only its candidate, and preserve other + candidates. Rust needs a hidden, nonallocating core filter-presence query on + this EOF-only path; do not duplicate filter state or change raw/unfiltered + semantics. Test repeated EOF in both directions, then reversal, SDK/facade, + sequential/merged and regular/compact. +- Rust recovery discovery bounds snapshot shrink by the physical file length, + but recovered_posting_count uses the original snapshot cap. With COMMON's + first secondary array truncated after its first valid slot, two global rows + survive; a later slot read reports UnexpectedEof instead of normal prefix + EOF. Bound cold recovered posting qualification by reader_extent, retaining + the original growth cap. Inspect the partial recovered-array view for the + same physical-bound mismatch. Add paired language snapshot controls and + repair/retry; preserve strict constructors and healthy paths. + +These are consequences of already-approved rewind, nearestness and intact- +prefix/normal-EOF contracts, with no new user design choice. Reproduce them +before implementation, fix minimally, then run affected suites and review the +complete final surface. The added work is in cold flush/EOF/recovery paths; +no new framework or combinatorial timing batch is required. + +The flush same-failure search also finds Rust FileReader retains its cached +CurrentRowView after the core cursor is rewound. Getters can therefore return +the old row until another movement. This is inherited, but is within the +approved paired flush/head-reset contract. Clear that existing row cache with +the same best-effort path used by seek_head, then clear matches. Add paired +file/directory SDK/facade getter controls before the fix; no extra native path, +allocation or new reset contract is needed. + +The Go review additionally identifies resolved-anchor backward preparation +that eagerly reads an AND leaf Rust never visits, and repeated resolved EOF +that restarts an entire posting walk. Reproduce both with an absent final AND +leaf/damaged earlier array and an accessor work counter. On resolved resets, +arm backward leaves lazily using the unused node=-2 sentinel, materializing +only in the existing zero-value seek branch. Preserve eager explicit Tail +preparation and binary realtime behavior. Record a successful resolved reset's +offset/direction as its walk anchor before seeking, including EOF; the existing +error invalidation still clears it. This adds no per-leaf field or healthy +cached-value branch. + +Go partial-array recovery also needs the existing main array reader's MaxInt +byte-count check before allocation. An accessor-backed oversized extent can +prove the current panic on both architectures without creating a huge file or +allocating its body. This is a cold bounds check, not a streaming redesign. + +Finally, count-bounded array walkers can revisit an earlier array and finish +their declared population before lazy Brent detects the cycle. This violates +the controlled-error claim for actually traversed cycles. Reproduce a short +A-to-B-to-A population first; unused terminal next links remain untraversed. +Investigate cold checks against already visited native-array prefixes, avoiding +healthy preflight, per-row sets or reads of unneeded future headers. Resolve +the shared reader/posting/recovery/writer blast radius before acceptance; +do not treat bounded duplicate output as successful cycle handling. + +Refined minimal EOF repair: preserve only the committed directory key while +unsetting its public index, matching the existing merged-EOF state. The +existing direction/key check then selects merge-based reversal at the saved +anchor. Children already retain their filtered EOF positions; no child reset, +index checkpoint extension or per-row key copy is required. Rust's hidden +filter query runs only at sequential EOF against the first child; public +directory filter mutators apply to every child and retain pending builders. + +Refined snapshot repair avoids per-slot metadata calls or new cold state. +WindowManager already retains the original open size in retained_size; only +writer-owned sync/growth can subsequently change it. In the Snapshot arm of +reader_extent, cache min(retained_size, physical length) in file_size. Existing +recovery discovery performs that one cold metadata read; posting qualification, +partial array length and get_slice header bounds then share the observed +physical extent. Repair can restore up to the original cap. Keep LiveFile and +WriterOwned behavior unchanged. No healthy layout, allocation or read cost +changes. + +The short-cycle controls reproduce required revisits in Go global/posting +reads and both writer helpers, plus Rust SDK reads, public bare Node.next and +both writer helpers. The first Rust SDK attempt had a test-only dependency +alias typo; the corrected control reproduces both failures. Rust's unused +terminal self-link control already passes; Go incorrectly checks that link. + +Go repair uses a shared cold native-header prefix check only before following +a required nonzero link at or below the highest visited array offset. It checks the target +against head-through-current, never future nodes. The helper retains a local +termination guard for malformed/unreachable prefixes. Native filter leaves +already retain node metadata, so compare against that vector on exceptional +links without I/O or separate Brent state. A high-water scalar is necessary: +after an accepted descending edge, an ascending edge can revisit an older node. +The descending-prefix/ascending-revisit control reproduces this omission in +the first cold-prefix candidate; do not accept only first-backedge coverage. +The scalar replaces the previous two-word guard and changes only array +transitions, not ordinary within-node movement. Remove the now-unused bounded-walk +guards; unbounded hash/FIELD guards remain. Readers ignore terminal next links +once their captured population is complete; writers still validate links they +must follow to choose a new writable slot. Use existing error categories. + +Rust's public bare Node lacks a starting point, so the user has been asked the +irreducible representation choice: replace cached next with head at the same +32-byte size, or add a head field and grow to 40 bytes. No dependent Node +representation change is implemented while that answer is pending. Continue +the independent Go repair and validation. + +The user selects **1A on 2026-10-17**: keep Node at 32 bytes, replace cached +next with the chain head, and read the next link at array transitions. Pack +the exceptional-order flag in an unused alignment bit of that head word. +After a descending/equal edge, check every required transition against the +visited prefix, including an ascending revisit of an earlier higher node. +Arbitrary non-head cursor reconstruction conservatively arms that flag. +Within-array row movement and cursor size stay unchanged. Validate the Node +size, short cycles, descending-prefix revisits, acyclic backward links and +unused terminal links, then representative reader performance. + +Use a shared cold Rust prefix helper with a native-header callback. Writer +walkers retain head/highest scalars in place of the two-word Brent guard; +drop array guards before exceptional prefix checks. Recovery uses its fresh, +physical/captured-extent array-prefix reader and validates only required links +after subtracting the consumed count. Preserve each writer's existing tail +stop predicate and cached-tail fast path. This is the already-approved +counted-cycle repair, independent of the Node representation decision, with +no new healthy reads or allocations in these writer/recovery loops. + +A final same-failure audit identifies nested rollback ownership in both directory +implementations. After sequential traversal advances to another child, a newly +observed recovery restarts merging through the public rollback wrapper while +the public key still belongs to the previously selected child. A later probe +error therefore restores the scanning child and discards its prefetched row; +the outer wrapper then restores the selected child. Repair/retry can skip the +nearest surviving row. Reproduce three disjoint filtered files with rows +1..9, select row 4 backward, lose row 3's header and damage row 8's type. The +merged probe must fail while preserving row 4; after repairing row 8, retry +must return row 2, then row 1. Go's new control reproduces row 1 instead of row +2 before the fix. Run the paired Rust control before changing runtime code. + +The minimal repair is to call the existing inner merge routine at the internal +recovery restart only: Go stepSequentialReader and Rust sequential forward and +backward. The outer public movement wrapper already owns rollback. Public +entry points retain their wrapper, other prefetched candidates remain intact, +and no new per-row checkpoint or state is introduced. This is a consequence of +the approved selected-row/error-retry contract, not a new positioning policy. +The same-failure caller search finds only those three nested wrapper sites. + +The paired nested-rollback control reproduces the same row-2 skip in Rust. +Both implementations pass after changing only their internal recovery restart +calls to the inner routine. Go's final supported-floor directory/recovery/cycle +selection passes 170 test events on amd64, 386 and race; Rust's complete public +SDK library passes 197 tests on Rust 1.91. Independent source recheck confirms +one public rollback owner, retained other candidates, and no new healthy work. +The short-cycle repairs also pass the Rust 1.91 whole workspace (196 SDK/98 +core before this final directory control), allocative core (98), and Go's +all-package amd64/386/race suites (1,819/1,820/1,819, with only the unchanged +live-writer stress control excluded as explicitly documented above). + +Final release-build-7 retains matched helper source and binary hashes. Its +shared hardening run passes all 552 observations; verifier, directory and mixed +matrices pass 63/34/42 stock-v262 observations. Existing verified examples are +unaffected by these final internal representation/rollback corrections: there +is no changed public signature or example flow. The Node shape assertion now +uses two u64 and two usize words, preserving 32-bit test portability as well as +the approved 64-bit 32-byte size. Specs and compatibility guidance record +required counted revisits, ignored unused terminal links and single-owner +rollback validation. + +The final performance confirmation reuses existing runners only: Rust core +forward/backward, facade-next and dense filtering on regular/compact fixtures, +plus the two previously sensitive Go writer cases. The normal Rust path keeps +four-word Nodes, unchanged within-array movement and no new allocation; an +uncached transition reads its current native header. Cached reverse Node chains +retain their existing reuse. Go's normal transition replaces lazy-guard state +with a high-water scalar and introduces no reads or allocations. Exceptional +acyclic backward links remain accepted and may rescan only their visited prefix; +no future suffix is preflighted. Directory rollback corrections run solely at +recovery transitions and add no healthy check, allocation or I/O. Earlier +writer-ascending results refer to their earlier source; the counted-array +replacement requires its own two targeted controls and does not inherit those +binary hashes or code-generation counts. + +Final counted-array performance confirmation passes semantic counters, immutable +fixture/binary checks and stock writer verification. Rust's eight cases retain +seven measured pairs and two warmups: regular core-next forward/backward and +compact dense filtering improve; the other five comparisons are inconclusive. +No case has a regression flag. Go's two writer cases retain ten pairs and two +warmups: regular/plain/publish-1 structured is 1.01477 [0.94232, 1.05638], and +regular/long-Zstd/publish-1 raw is 0.97970 [0.96239, 1.00383]. Both remain +inconclusive. All 10,005-row/32-field writer outputs match original-baseline +bytes and stock v262 verification passes. Evidence is counted-performance and +writer-counted-control, with writer-counted-guard source/build provenance. + +Decision-11 acceptance combines these final targeted repeats with unchanged +within-node work, the approved Node32 shape, no new healthy-path allocation, +reviewed native reads and the earlier retained representative results. There +is no established remaining reproducible regression; no speedup offsets an +adverse case and no uncertain interval is called proof of equality. The final +array repair replaces the earlier writer guard, so its earlier ten-to-six +instruction observation is historical, not claimed for the final binary. +The sole extra normal Rust work is the approved native current-array header +lookup at an uncached transition; the row path and cached reverse nodes retain +their shape. Independent internal source audits find no remaining blocker in +the repaired counted-array and directory-retry surfaces. Whole-SOW external +review remains the final independent technical gate. + +### Required Engine Cache Migration - 2026-10-17 + +The final artifact audit traces a reachable legacy cache consequence. At +baseline 4c49205, Node::next treats a missing required array link as successful +EOF (offset_array.rs:59-76), and List::collect_offsets returns a partial vector +(:327-349). Core open validates mapping bounds without a graph pass. With no +source timestamp field or facets, FileIndexer builds a partial histogram and +FileIndex from those offsets (file_indexer.rs:150,185-224,473-501). Count-bounded +cycles can similarly supply duplicates. The engine stores successful indexes +(indexing.rs:549-552); offline archived indexes remain fresh indefinitely +(file_index.rs:94-102), and compatible cache hits return without reopening the +journal (indexing.rs:392-398). This is source-proven reachability, not a claim +that any real cache contains such an index. + +The existing product-scope rule requires CACHE_VERSION to change whenever +indexer inclusion, computation or serialization changes. Apply that established +policy here: increment journal-engine's generation from 3 to 4, leaving the +schema, key construction and cache algorithms unchanged. Old keys miss; valid +files rebuild on demand once after upgrade, while malformed files now fail +indexing and can be retried later. Old cache storage follows normal cache +eviction, with no new deletion operation. Cache-hit and steady-state +reader/writer work remain unchanged. Go has no equivalent persisted engine +cache in this source tree; its explicit IndexedSnapshot contract is unchanged. +This is a required consequence of the approved native-array repair, not an +optional cache redesign or a new user policy exception. + +Stream completion alone does not require this bump: FileIndexer skips compressed +facet DATA and parses source timestamps from raw payloads; query-time payloads +decode dynamically. Ordinary public recovery also remains separate from the +core FileIndexer open path. Record generation 4 and the one-time rebuild in the +spec and unreleased consumer migration note. Add a small archived-file indexer +regression with a healthy control, required missing link and short cycle; +validate it and the existing cache-key/engine tests on Rust 1.91. No reader or +writer timing repeat is required for an unchanged cache algorithm and a key +generation constant. + +The original-source archived FileIndexer control reproduces successful indexing +of only four of 13 rows for the regular missing-link case. It stops at that +first assertion; the baseline did not execute the other variants. Current +Rust 1.91 index/engine package suites pass, including the four malformed +variants in the new test, healthy archived freshness and existing v3/v4 key +non-reuse. All 35 verified documentation examples pass again and all 16 wiki +pages validate. Malformed archives now fail indexing rather than being promised +a successful rebuild; valid archives incur the documented once-on-demand cost. + +The generation constant belongs only to journal-engine, which is not a +reader-core benchmark, conformance adapter or journalctl dependency. The final +reader/writer source used by release-build-7 is unchanged; cache-migration +retains the separate final optional-engine test/source provenance. No cache +algorithm, persistent schema shape, compiler floor or public reader path is +changed by this migration. Internal artifact recheck finds this consequence +covered. The complete candidate is locally ready for its authorized whole-SOW +static external review, subject to the final audit result below. + +The final pre-external audit reports `SOW initialization complete and clean` +with changed-file sensitive-data scanning enabled. `git diff --check` passes; +source, tests and durable artifacts use synthetic data and contain no user's +personal name. Runtime implementation is frozen for the whole-SOW review. + +### Whole-SOW External Review - 2026-10-17 + +After complete local validation and the complete-and-clean audit, the full +neutral prompt was displayed and the six authorized static reviewers started +in parallel. Session: `ef5096c38bfe40ea80265cb9344505bc`. The prompt covers the +complete SOW, tracked and untracked changes against prerequisite 4c49205, +indirect consumers, performance evidence and strict/pure SDK contracts. Review +is read-only with no tests, builds or delegation. Runtime source is frozen. +This SDK round is distinct from prerequisite SOW-0108's historical round. +Reviewer results and independent dispositions will be recorded before close. + +Qwen's initial invocation failed before returning a review with HTTP 429 +Too Many Requests. Its sole permitted retry uses the identical displayed +prompt and the same reviewer session. Successful reviewers are not rerun. +The other five reviewers continue independently; no technical verdict is +inferred from a transport failure. + +Qwen's one retry also failed before producing a review with HTTP 429. Both +attempts are retained in the session artifacts; no further retry is permitted +or planned. This reviewer supplies no technical verdict and leaves a stated +coverage gap. The five other requested reviewers remain independent of that +transport outcome, and their results will be preserved individually. + +GLM and Minimax return `PRODUCTION GRADE` for the complete SDK candidate. +Independent disposition of their observations: + +- GLM's unfiltered realtime error suppression is verified in the unchanged + raw search and frozen baseline. The existing SOW-0045 mapping covers it; + filtered explicit realtime search propagates visited errors here. +- GLM's raw automatic-refresh error position observation is also present in + baseline Next/nextAfterTailRefresh: the index increments before a refresh + error returns. SOW-0045's queued failed-refresh rollback and healthy-position + analysis covers that contract. This is not substituted for the selected + filtered-row retry guarantee implemented and validated in this SOW. +- Minimax labels two Rust realtime observations P2 while explicitly describing + correct behavior and no production defect. Reject those defect labels. + InlineCursor::head retains the native inline first posting; backward + partition index zero correctly returns it when the next posting exceeds the + anchor. The caller retains the resulting posting cursor intentionally. + Existing requested-direction, reverse-anchor, duplicate-bound and visited-slot + tests cover that positioning contract. Additional test-only suggestions do + not establish an uncovered behavior or justify redundant controls. +- Minimax's Go P2 sentinel suggestion is rejected. node=-1 with value=inline + denotes the real inline position. seek returns that value without another + read when it satisfies the requested bound; it reads the first array slot + only when moving beyond it. The -2 sentinel denotes an unmaterialized + backward tail and is inspected only when value is zero. Changing node alone + in this nonzero-inline branch would provide no claimed benefit. +- Minimax's two P3 observations explicitly identify no defect. Trailing Zstd + garbage is already an expected-error shared case; finite native-file bounds + make the guard's u64 overflow limit unreachable for distinct valid objects. + No new runtime code, test framework or optional cleanup is warranted. +- The remaining Minimax notes summarize already retained history, rejected + private candidates, superseded evidence policy and mapped SOWs. They add no + uncovered defect. Its descriptions of a known-broken stress test or proven + contention root cause are not accepted: the full-run deadline failure and + isolated 38.03-second pass remain recorded, with the cause unproven. The + final all-package counts explicitly exclude only that stress control. + +The other three available reviewers remain in progress. No successful reviewer +is rerun because of another reviewer's transport failure. + +MiMo, Kimi and DeepSeek each reach the configured 1,800-second execution +limit with no stdout review or verdict. The session metadata records timeout +and owned-harness termination; their source-inspection logs are retained. +Each receives its sole permitted retry in the same session with the identical +full-scope prompt, displayed again before execution. These are independent +timeout retries, not a new debugging round. Runtime source is unchanged and +the two successful reviews are preserved without rerunning them. Qwen has +already exhausted its retry and is excluded. + +### External Finding Verification And Repair Gate - 2026-10-17 + +Kimi returns NEEDS CHANGES with directory-filter anchor loss and Rust indexer +traversal-error suppression findings. Root source inspection confirms that +both directory mutation paths reset the emitted key, contradicting the +already-approved reseek-from-emitted-key and nearest-installation contract. +Prior installation controls exercise files; directory controls install before +their first row or exercise raw/filtered movement without changing the filter +at a committed directory row. This omission means the earlier primary audit +was incomplete. Do not close on the two positive reports. + +Add paired controls using the existing six-row sequential/merged directory +fixture. Select row 3, install/replace a match or commit a conjunction or +disjunction, then require rows 4..6 forward or 2..1 backward. Cover SDK/facade, +regular/compact and the Go access backends. Run before changing runtime code. +Repair by separating mutation invalidation from FlushMatches' full reset: +retain the committed key/index, discard prefetched candidates and revalidate +the changed expression through existing merge-key positioning. Preserve +pending explicit seeks, error/retry state and boundary behavior when no key +exists. Keep healthy per-row state/work unchanged; mutation-only reseeking is +required by the existing contract. Same-failure inspection covers all three +operators in both languages, raw/filtered transitions and recovery restarts. + +Independently verify the indexer finding and reproduce its actual requested +facet/source-timestamp cases before any indexer repair. Preserve intentionally +documented per-object tolerance unless a new native traversal failure must +propagate. Any required consequences belong to this existing SDK SOW; no new +product policy or compiler/native dependency change is proposed. Let the two +remaining current review attempts finish under their existing limits; do not +launch another review until repairs, complete internal validation and audit +are finished. Then review the complete candidate again under standing +authorization, rather than using reviewers as an iterative debugger. + +The paired directory controls reproduce endpoint replay: Go's 128 subcases +fail; Rust stops at regular/sequential/SDK/forward, returning 1000 instead of +4000. Backward Go controls return 6000 instead of 2000. The original reset +implementations predate this SOW; this is an inherited omission inside the +approved installation contract. An earlier Go test itself expects replay +after mutation and needs its expected sequence corrected. + +The mutation audit also covers a consumed initial realtime seek after load +failure: before clearing its bound, restore the target to pending when no +committed key exists, without overriding a newer pending seek. Invalidate +before fallible child group loops. Existing key-based merge preparation and +full-key comparisons provide the repair, with no new per-row field or branch. + +The Rust indexer finding is source-verified. Unchanged baseline FIELD DATA and +posting-list handlers suppress new cycle/required-link errors, yielding +successful incomplete generation-4 indexes. Reproduce healthy global arrays +with requested MESSAGE facets or repeated source timestamps and damaged FIELD +DATA links/postings. Propagate CyclicObjectChain through FIELD DATA setup and +iteration handlers; propagate failed required posting collection by operation. +Reuse reader_inlined_cursor with the borrowed DATA header to reject +positive-count missing inline/head pointers while retaining zero-reference +orphans. Preserve unrelated per-object tolerance, invalid timestamp fallback, +compression/size/cardinality limits and per-file batch failure isolation. +Test regular/compact facet/source cases, missing links, required cycles and +missing positive-count pointers, plus healthy/tolerated controls and actual +cache omission. Reuse native headers and cache helpers; no dependency, cache +schema or steady-state algorithm change is required. + +MiMo and DeepSeek also exhaust their one retries at the execution limit with +no verdicts. Preserve every attempt; these reviewers and Qwen leave coverage +gaps. GLM and Minimax are positive; Kimi identifies the verified blockers, +so the candidate is not accepted. Complete repairs, the full primary audit +and validation before the next whole-SOW review. No further retry belongs +to this round. + +The first repair passes all 128 Go mutation cases, the paired Rust mutation +matrix and both consumed-realtime-seek controls. All 20 requested facet/source +index damage cases and the actual engine cache-omission test also pass. +Source review identifies a necessary refinement before acceptance: forcing a +resolved non-overlapping directory into permanent merging adds all-file work +and can expose a distant ENTRY error before nearer healthy matches. Retain +sequential state only when non-overlap remains proven, merging has not begun, +a direction and committed key exist, and the selected file index is valid. +Clear candidates in that cold mutation branch; preserve the existing general +key/seek invalidation for every other state. Existing reversal, mode-change, +recovery and rollback checks remain authoritative. Add a paired three-file +control: select row 4 under COMMON, damage unvisited row 7, replace COMMON, +return rows 5 and 6, then report the error while retaining row 6 and permit +repair/retry to row 7. Run the control against the first repair before changing +the helper. This bounds the consequence directly without a timing framework, +new state or a new per-row branch, consistent with performance decision 11. + +Both first-repair controls reproduce the unnecessary distant-file error before +row 5; the guarded repair passes both layouts, repair/retry and the existing +directory suites. Go has the same invalidation inside its existing raw/filtered +mode-change branch. Apply the identical proven-sequential predicate there, +retaining direction only in that state while always updating mode and clearing +candidates. This covers first filter installation as well as replacement and +avoids a Go/Rust native-path discrepancy. Existing file offset normalization, +reversal and directory rollback provide the position guarantees. Extend the +same representative control with installation after raw movement, reproducing +the mode-change issue before its repair; do not add a timing matrix. + +Same-failure inspection finds an inherited indexer panic when a requested +facet posting points to an offset absent from the global ENTRY map. A DATA +object's own aligned offset in its inline posting reproduces the existing +missing-map branch without requiring any extra validation work. AC1's +controlled invalid-link outcome includes this affected index consumer. Replace +the panic on that cold branch with the existing MissingOffset error and add a +regular/compact reproducer before the repair. Preserve normal membership +lookups, allocation and traversal costs. Qualify migration prose to encountered +required native-chain/posting failures; documented value tolerance is retained, +so it must not promise universal malformed-file verification. + +The absent-map facet control reproduces the inherited panic before repair and +passes after returning MissingOffset on the existing branch. A separate source +map can admit an unknown target without a native traversal error. That inherited +cross-index membership/captured-tail lead is tracked in pending SOW-0161, with +its unexecuted prediction explicitly distinguished from measured failures. +Strict bidirectional membership remains a verification responsibility; no new +source-map certification pass or live-index behavior is introduced here. + +### Post-Repair Primary Audit And Validation - 2026-10-17 + +The complete SDK candidate is locally validated again. Go 1.26.2 all-package +amd64/386/race runs pass 1,995/1,996/1,995 test events, excluding only the +unchanged live-stock stress control whose earlier isolated pass and full-run +deadline failure remain qualified above. Rust 1.91 workspace passes, including +200 SDK and 98 core tests, plus all 98 allocative core tests. The final index +and engine tests cover 22 requested-field damage cases and actual damaged-file +cache omission with a healthy sibling. The 306-event focused Go directory suite +and 14 Rust directory tests pass. Before controls retain endpoint replay, +unnecessary distant-file errors and the absent-map panic with exact provenance. + +Release-build-8 retains 359 source hashes and 11 rebuilt tool hashes. All 552 +shared observations and 63/34/42 pinned-v262 verify/directory/mixed checks pass. +All 35 documentation examples and 16 wiki pages pass. Final Rust test and +release manifests match current source; the Go-only runner captured the Rust +indexer immediately before its cold panic repair, so its Go source is exact +and the separate final Rust run certifies that later Rust file. Do not treat +that irrelevant cross-language manifest difference as a Go execution claim. + +The primary audit covers the complete original implementation and new consumer +consequences, all three directory operators, failed initial time seeks, native +sequential versus merged positions, raw/filtered transitions, recovery restarts +and rollback, field/posting error catches, checked positive-count pointers, +cache generation/insertion and strict-versus-tolerated boundaries. The inherited +source-map membership question is separately recorded in SOW-0161 without a +new certification pass. No further verified in-scope blocker remains locally. + +Performance decision 11 applies: directory invalidation reuses proven sequential +state inside cold mutations/mode switches, adding no steady-row condition or +checkpoint. Indexer changes propagate existing errors and check borrowed DATA +header scalars once per requested value; the absent-map repair changes only +an existing error branch. No extra container, row scan, decompression or healthy +posting-array read is added. Earlier measured native-reader/writer controls +remain applicable with this bounded-work and source review evidence. No speed +equality is inferred from an uncertain timing interval. + +The changed-file scan covers 120 durable artifacts without findings; all 97 +changed/new task files pass the personal-name check. The audit reports complete +and clean, and whitespace checks pass. Specs, user docs, compatibility skill, +status indexes and follow-up mapping reflect the repair. Runtime source is +frozen for the new whole-SOW external round; only review/closure evidence may +be appended while it runs. This is a material-fix review after complete internal +audit and executable validation, not a partial debugging review. + +The new full-scope static round starts at 09:35 UTC in session +076f2256bda9495486f72a229e32851a, with the complete neutral prompt displayed +before execution. All six authorized reviewers start in parallel. Qwen's first +attempt returns HTTP 429 before any verdict; its sole retry uses the identical +displayed prompt in the same session. Other reviewers continue independently. +No source edits, executable review workloads or CLI implementation run during +this round. + +Qwen's second attempt also returns HTTP 429 with no stdout or verdict. Both +6-second attempts remain retained; its retry allowance is exhausted. This is +a reviewer coverage gap, not an implementation verdict. The other five +reviewers remain independent and in progress. + +Release-build-8/retained now preserves immutable copies of all 11 tool binaries +and all 359 manifested Go/Rust source files, with verified SHA-256 identities. +The task's reusable bin directory can therefore be rebuilt during the later +CLI SOW without losing this SDK's final executable/source evidence or the +current CLI baseline. This is evidence retention only; reviewed source is +unchanged. + +Minimax returns PRODUCTION GRADE after 1,680.1 seconds. Independent +adjudication rejects unnecessary changes in its nonblocking notes: + +- The recovered Go posting helper has only two callers, both of which enforce + the symmetric count/inline check before calling it. Repeating the check in + the callee changes no reachable behavior and adds no missing protection. +- Removed path fields are private implementation storage, not a public API. + File.Name retains the original staging name, exactly as the previous stored + string did, including after external renaming. No correction is needed. +- Rust refresh_recovery_batch explicitly clears every recovery_pending flag + after a stable pass. Ordinary movement checks the flags but does not rebuild + every recovered leaf on each row. Failed pending refresh remains retryable by + contract. Reject the claimed repeated-rebuild scenario; no optimization is + required for acceptance. +- The unrelated-note claim that an unfiltered-reader issue is excluded from + final Go counts is incorrect. The only excluded test is the unchanged live + stress control. Actual inherited raw refresh/seek/probe issues remain mapped + to SOW-0045, without changing the tested selection contract here. +- SOW-0161 remains a separately recorded inherited membership investigation. + Inconclusive measurements and future cache-version requirements are already + qualified in the canonical evidence; neither is a new defect or a reason to + expand the framework. Use recorded case identities rather than the report's + imprecise performance-case labels. + +No verified blocker or additional required change follows from this report. +The four other reviewers subsequently reach their execution limits without +verdicts, as recorded in the terminal disposition below. + +### Final Review Disposition And Closure - 2026-10-17 + +Fresh session 076f2256bda9495486f72a229e32851a is terminal. Minimax returns +PRODUCTION GRADE after 1,680.1 seconds. GLM, MiMo, Kimi and DeepSeek each reach +the 1,800-second execution limit (observed 1,800.7-1,800.8 seconds), returning +no stdout and no verdict. Qwen returns HTTP 429 after 6.3 seconds and again +after its sole retry of 6.2 seconds. Every attempt and result is retained. +These are five coverage gaps, not five approvals; earlier positive reports +apply only to their earlier candidate. + +The current positive report is independently adjudicated above. No verified +in-scope blocker remains after the complete internal audit and executable +validation. The reviewer policy permits at most one retry for a transient +failure; it does not require another long retry batch. No additional retry +is launched for the four timed-out reviewers. This decision preserves their +coverage gaps and uses the completed current report with the full primary +validation, rather than treating failed invocations as verdicts. No review +waiver or unanimous production-grade conclusion is claimed. + +The final source remains identical to release-build-8's 359-file manifest. +The retained directory contains verified immutable copies of those sources and +all 11 tools, preserving both SDK provenance and the later CLI baseline. +Only review/closure records change after source freeze. Follow-up search maps +all live remaining work to SOW-0121, SOW-0122, SOW-0045, SOW-0160 or SOW-0161; +rejected optional experiments have no remaining delivery promise. SOW-0071's +independent static-guard regression remains separately paused. + +The completed SOW moves to done with both status indexes updated. The closing +audit reports SOW initialization complete and clean; whitespace and changed-file +personal-name checks pass. The independent read-only closure check finds no +verified unresolved blocker and identifies one stale SOW-0045 scheduling +sentence, which is corrected. Stage explicit paths and create the combined +signed implementation/lifecycle commit after the final artifact check. +SOW-0121 may activate only after that commit succeeds. No push is authorized. + +## Regression - 2026-10-25 + +### Evidence And Pre-Implementation Gate + +The user approved repairing confirmed PR12 findings and individually adjudicating +all60 review threads. SDK repairs execute before CLI SOW-0121; no parallel +implementation SOW is active. At2c12221 the public Go directory facade reproduces +SeekCursor(t beyond newest) success followed by Next returning the first row. +facade.go:563 resets an already exhausted reader with SeekTail, clearing the +directory direction. Preserve observed EOF instead, validating forward EOF and +backward tail behavior for file/directory and filtered/unfiltered callers. +Do not broaden this into the separately paused healthy-refresh SOW-0045. + +Rust reader_helpers.rs:265 maps every streaming copy error into decompression +failure, including propagated source/destination I/O. Restore I/O classification +for actual transport/storage errors, retaining malformed/truncated stream errors +as decompression failures. Review core stream decoding because decoder wrappers +can themselves erase source errors. Use injected failing Read/Write tests, not +actual disk exhaustion or production files. Normal decoding must keep existing +streaming behavior and allocations. No dependency or compiler changes. + +Inspect other SDK comments with concrete reproducers before accepting them. +TestCursor EOF direct API inconsistency is a separate pending user decision; +facade semantics remain native-compatible. NUL native path construction/serde +consistency and small consumer/probe documentation issues are within reviewed +blast radius; avoid optional refactoring or blanket benchmark/test expansion. +No implementation of an unresolved policy is authorized. Existing completed +SOW specifications and prior failure controls are reused. Risks are position +reversal semantics, decoder error taxonomy and consumer error matching. Fixtures +are synthetic, scratch/logs under.local only. Update relevant specs/docs/skills, +status indexes and thread evidence; perform focused plus representative SDK +validation and internal final review before signed coherent closure. + +### User Decision - Direct Cursor EOF + +The user selected 1A: both Go direct Reader.TestCursor and +DirectoryReader.TestCursor return false with ErrEndOfEntries when no current +entry exists at EOF. The facade normalizes that error to false, nil, matching +native cursor-test behavior. Preserve errors unrelated to EOF. Cover both +reader types and filtered/unfiltered exhaustion, then successful reverse steps. + +### Repair Validation And Remaining Decision + +Implemented the approved direct cursor EOF contract and facade normalization. +The directory reader uses a private EOF sentinel, so it explicitly translates +that sentinel to the public ErrEndOfEntries. The four facade regression cases +exercise file/directory and filtered/unfiltered EOF, repeated forward steps, +direct/facade cursor tests, and reverse selection of the final row. + +Rust stream errors now inspect decoder error sources on failure only. Injected +source PermissionDenied and destination StorageFull retain I/O categories; +truncated streams remain decoding failures. Healthy streaming loops and buffers +are unchanged. Native raw path construction rejects NUL consistently; the probe +preserves array-shaped field output on failure. Consumer documentation clarifies +EOF, strict verification and caller-excluded indexed snapshot capture, and the +adapter help lists inspect-reader. + +Validation on the current repair source: + +- Pure Go journal and adapter suites pass (137.480 s and 0.005 s). +- Rust core, registry and SDK suites, including doctests, pass. +- Shared reader-hardening suite passes 552 observations against pinned v262. +- All wiki pages validate and executable documentation examples pass. +- Internal final-review of this bounded repair diff finds no blocking defect; + its verdict excludes the separately investigated shared-cursor issue below. +- Healthy per-entry work is unchanged. New error-source traversal occurs only + on decoding failure, cursor error normalization occurs on metadata calls, + and path validation occurs during construction. No broad benchmark expansion + is needed for these changes. + +Review item 13 is now reproduced using advanced public core JournalCursors +sharing one JournalFile: both read two surviving entries, repair restores the +third, cursor A sees it, and cursor B remains at EOF after shared recovery clears. +The reproducer uses doc-hidden recovery initialization, not independent high-level +readers. Per-filter observed recovery revision versus an explicit-rebuild-only +advanced contract is awaiting a user design decision. Do not close this SOW or +advance CLI implementation until resolved. Item 32 is disproven as stated: +filtered EOF retains its last selected index; the questioned past-end branch +represents explicit tail/raw state, so moving it back would alter positioning. + +Raw synthetic logs and reproduction sources remain under .local/pr12-sdk-triage. +The published PR remains at 2c12221; its native jobs pass, while Codacy upload +and remote-configuration failures remain separately recorded, not counted as +successful checks. No new push has occurred. + +### Scope Correction - Corruption Handling Versus Synthetic Restoration + +The user clarified that SDK behavior must mimic systemd: readers extract readable +data without modifying source journals; corruption is not an invitation to repair +source bytes in place. The prior per-filter revision recommendation and associated +1A/1B question are withdrawn pending realistic supported-operation evidence. +The shared-cursor reproduction restores damaged bytes through a separate test +writer. It proves behavior under synthetic restoration, not normal journald +corruption handling. It is not sufficient evidence for added runtime bookkeeping. + +Upstream systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492: +src/shared/journal-file-util.c:487 rejects or renames/replaces corrupt files; +src/journal/journald-manager.c:907 selects rotation on corruption; +src/libsystemd/sd-journal/sd-journal.c:1699 opens readers O_RDONLY. +The format concurrency contract separately allows temporarily inconsistent +structures during ordinary publication. That must not be conflated with rewriting +corrupt entries. Audit shipped source-write behavior and repair-specific promises, +tests and runtime costs before any further recovery-design implementation. +No per-filter recovery revision implementation has been made. + +### Approved Scope Audit - 2026-10-25 + +The user approved the correction audit. Preserve default read-only intact-prefix +recovery and ordinary publication semantics; do not treat externally rewriting +corrupt entries as a supported lifecycle or a reason for runtime overhead. +First remove consumer/spec promises that instruct repair/retry and correct the +runtime skill's test interpretation. Retain synthetic fault-injection evidence +only as such, not as proof of a journald publication sequence. Review existing +requalification/state rollback before removing it: these mechanisms can also +protect initial recovery, error atomicity and Boolean-query completeness. +No new source-repair API, synchronization, revision counter or per-entry check +is authorized. Independent read-only audits cover runtime causality and test +classification. Validate docs, test evidence and any actual code changes, with +an internal final review of the corrected scope. Historical SOW evidence remains +intact and is superseded by this explicit scope correction. + +### Recovery Scope Audit Results + +Independent runtime and test audits confirm no source-byte reconstruction in +reader recovery or writer reopen. Readers use read-only descriptors/mappings; +whole-file decompression writes a separate staging file. Writer open rejects +observed invalid committed bounds/tail rather than reconstructing them. This +is not a claim of full journald writer-lifecycle parity: core ONLINE reopen and +optional directory policy remain existing explicitly scoped SDK behavior. + +Corrected product scope, Reader-APIs, Go API and the compatibility skill to remove +promised automatic refresh after arbitrary external repair. Fault-restoration +controls remain labelled synthetic robustness evidence; they do not establish +an upstream requirement. Renamed the misleading same-size Publication Go test +to SyntheticRestoration. The shared 552-observation suite constructs static +mutations before opening readers and remains valid static recovery evidence. +Actual Writer.Append publication and injected transport-error retry controls +remain distinct. Do not mechanically rename synthetic restoration as publication. + +Retain initial prefix discovery, effective counts/tail metadata, posting clipping, +stable Boolean qualification, pending-builder/error rollback and directory-change +notification: these serve initial damaged-file reads without later reconstruction. +Retain explicit header refresh and copied-window invalidation: real appends into +preallocated space can change index bytes without changing file size. + +A concrete cost needs an EOF-refresh policy decision: Rust cursor.rs:90 calls +reader_recovery.rs:419 prefix rediscovery on every EOF while recovery is active; +Go reader_filter_step.go:87-98 calls reader_filter_recovery.go:32-69, reloading +ENTRY arrays. Unchanged damaged archived files can therefore incur repeated work +proportional to their readable population. No locks or atomics are involved. +No runtime change to that policy has been made in this correction. + +Upstream journal-file.c:2421-2435 initializes ENTRY before linkage; :2289-2298 +publishes global ENTRY linkage after a memory fence; :2159-2163 and :2193-2210 +publish slots/links before counts; :2303-2324 update tail and DATA links. Zeroing +and restoring committed ENTRY bytes is not this writer sequence. Weak storage +visibility still warrants care before suppressing live rediscovery. The earlier +shared-cursor generation proposal is rejected for its unsupported reproducer; +it is not an unresolved request for user approval. + +### Native EOF Evidence And Validation + +Both baseline v260.1 sd-journal.c:1011-1016 and CLI target v262 +sd-journal.c:1016-1021 skip an exhausted file when direction and declared entry +count are unchanged. This is concrete evidence against treating unchanged-EOF +full-prefix rescans as a systemd requirement. A new user decision proposes the +native eligibility rule for both SDKs, preserving initial recovery and changed +count/direction handling. Implementation awaits that decision; this replaces +neither healthy-filter captured-population policy nor the separate SOW-0045 scope. + +Scope-correction checks: 16 wiki pages validate, all executable documentation +examples pass, whitespace checks pass, and focused Go synthetic-restoration, +actual preallocated Writer.Append and accessor-I/O retry controls pass (0.444 s). +Runtime behavior remains unchanged by the scope-correction edits. + +Internal scope-artifact review found three remaining repair/retry promises in +Go-API and Rust-API wiki pages. Those are corrected; the broader consumer/spec/ +skill search now finds only explicit exclusions or unrelated software/cache +repair references. Re-review passes with no blocker in this bounded correction. +Wiki and whitespace checks pass after the final prose edits; executable example +blocks are unchanged from the passing example run. The SDK SOW remains open for +its outstanding implementation/review work and the native EOF policy decision. + +### User Decision - Native EOF Eligibility + +The user selected 1A: apply systemd's unchanged direction/count EOF rule to both +SDKs. Include Go raw Next as well as filtered Step; preserve explicit Refresh, +initial damaged-prefix recovery, movement reversal, seeks and changed counts. +Keep EOF memo state outside healthy Go readers in the existing recovery +allocation. Rust's public JournalCursor layout must not change: put terminal +memo state in the private JournalReader wrapper and move automatic EOF refresh +there, retaining core cursor initial-error recovery. The low-level cursor is a +positioning primitive, not a separate automatic repair-following service. +Clear terminal memos on successful movement, seeks and filter changes; never +cache failed refreshes. Compare declared header population, not recovered count. +No cross-thread synchronization or per-filter recovery revisions are introduced. +Validate actual layout, bounded read work at repeated EOF, unchanged source bytes, +actual Writer.Append, reversed movement and both filtered/raw access paths. +Synthetic restoration controls must no longer demand automatic visibility after +an unchanged terminal poll; retain their initial recovery/rollback assertions. + +### Native EOF Implementation And Evidence + +Implemented the approved rule. Go stores direction/count/validity in its existing +exceptional recovery allocation; raw Next and filtered Step consult it before +repeating terminal work. Explicit Refresh, seeks, filter changes and successful +movement invalidate it. Recovery equality ignores these traversal memo fields, +so directory notifications and rollback compare only actual prefix state. +The healthy filtered path checks the existing recovery pointer before invoking +the counter-read helper. Unchanged recovered EOF reads the declared count once +without rebuilding copied windows or scanning the prefix. + +Rust JournalReader owns a private 16-byte optional direction/count memo; it checks +terminal state before cursor traversal. Automatic recovery EOF refresh moved from +the public JournalCursor into that wrapper. The public cursor layout is unchanged, +and its initial-error recovery remains. No mutex, atomic, per-filter revision, +new heap allocation or source-file writes were introduced. + +Validation: + +- Full Go journal/adapter suite passes (118.935 s); after early-gate refinement, + focused recovery/filter suite passes (7.114 s). +- Rust SDK 202 tests and core 99 tests pass, with doctest results preserved. +- Rebuilt shared adapters pass all 552 reader-hardening observations against the + pinned v262 oracle. All executable wiki examples pass; wiki/whitespace checks pass. +- New Go test covers raw/filtered readers in all three access selections, + bounded reads over ten unchanged EOF polls, identical source bytes, reversal, + unchanged-count synthetic restoration, and a real appended entry changing count. + Repeated EOF allocates zero times per poll in all six cases. +- Equivalent Rust raw/filtered control preserves source bytes and validates + unchanged EOF, reversal and real append count changes. +- Go Reader remains 616 bytes; its exceptional recovery state is 120 bytes + (the added memo resides only in already allocated recovered state). Rust + JournalReader is 656 bytes including its 16-byte memo; JournalCursor is 280 + bytes and unchanged. These are local 64-bit layout observations, not API sizes. +- Same-failure search includes Go raw EOF, filtered EOF, explicit Refresh, + seeks/filter mutation and Rust wrapper/core ownership. Synthetic restore tests + preserve their initial damaged-prefix assertions and now test unchanged EOF + before explicit repositioning; they no longer demand repair-following polling. +- Independent internal final-review passes the current implementation, including + targeted Go/Rust reruns. No external reviewers invoked. + +Performance evidence is bounded-work and allocation evidence plus code review, +not a claim of measured throughput equality. Removing population-sized repeated +scans dominates the constant counter check on recovered EOF. Healthy paths do +not scan or allocate for this memo. No broad measurement framework was added. +Remaining unrelated PR findings and CLI work keep this SOW/PR work open; this +entry does not assert entire PR readiness or clear remote Codacy service failures. + +### Publication Checkpoint - 2026-10-26 + +The user explicitly requested committing and pushing the validated current work, +then three independent internal final reviews of different PR aspects. This is +an interim implementation checkpoint, not SOW completion or merge approval. +Preserve the current/in-progress lifecycle because remaining PR findings and +CLI work are not completed. External-model reviews remain waived in favor of +internal reviewers as previously requested. The three review scopes are SDK +correctness/recovery/performance, CLI compatibility/portability, and validation/ +consumer contracts/CI; each reviews the entire PR surface relevant to its scope. + +Before push, the published head remains 2c12221. Native Linux/macOS/Windows/ +FreeBSD checks and Go/Rust coverage generation passed. Codacy SARIF generation +and coverage upload retain the previously diagnosed remote-service failures; +no unrelated CI configuration change is included. Local validation and internal +review evidence for this checkpoint are recorded above. + +### Regression Closure - 2026-10-26 + +The user authorized fixing the remaining review findings. Independent SDK review +of a1144dcbf537 passes after Go focused suites, Rust core99 and SDK202 tests. +Prior shared552/docs35/wiki16 checks and bounded-work/allocation evidence remain +recorded above. No SDK-specific blocker remains. The latest publication did expose +a CLI consumer of the changed facade EOF result; do not claim whole-PR readiness. +That fix, Windows glob/URL defects, explicit oracle qualification and CLI example +paths are tracked by reopened SOW-0121 before proceeding sequentially. + +SDK review items: cursor EOF, compressed-stream I/O classification, NUL-path +construction, array-shaped probe diagnostics and public help/docs are implemented. +Shared advanced-cursor synthetic restoration is rejected as a production repair +requirement; native EOF semantics supersede it. Unknown posting completeness, +strict cache traversal and required facet errors remain intentional. Optional +cleanup/extra test cross-products, redundant assertions and test-only dependency +relocation are rejected as unnecessary to the approved outcome; generic DATA +decoder diagnostics remain an existing limitation, not silently accepted data. + +Artifact gate: specs, runtime compatibility skill, consumer docs and SOW indexes +are updated. AGENTS.md needs no change: existing systemd/purity/performance rules +already cover this outcome. No exported operator skill exists. The lesson is to +validate a realistic upstream trigger before treating synthetic restoration as a +required lifecycle. Scratch/raw reports remain ignored; durable evidence is +sanitized. No source repair or new host probing was introduced. Existing broader +follow-ups remain SOW-0045/0071/0122/0160/0161; new CLI findings map to real reopened +SOW-0121. The user-requested interim checkpoint explicitly split implementation +publication from this lifecycle handoff. Run the local audit before committing +this transition; SDK and CLI implementation are not executed in parallel. + +Final lifecycle audit reports complete and clean. Current Codacy Markdown +findings in this record were corrected before the handoff commit. The unchanged +Bandit execution error predates this repair and remains an explicit coverage +limitation in SOW-0121; no speculative CI configuration change is made. + +## Regression - 2026-10-26 - Facade Cursor EOF Classification + +### Pre-Implementation Gate + +The approved public TestCursor EOF correction exposed a missed consumer contract: +Go SdJournalGetCursor forwards private errEndOfEntries (facade.go:287), while +TestCursor now correctly reports false,nil. Invocation listing and ordinary +cursor output cannot identify EOF using public errors. The user approved fixing +confirmed review findings; normalize facade GetCursor EOF to existing ErrNoEntry +and use that metadata-only result in both CLI consumers. Preserve all other +errors. Direct getters and Rust behavior remain unchanged. This is a prerequisite +to the paused CLI regression; preserve its uncommitted Windows edits untouched. + +Reuse existing errors.Is and cursor regression fixtures. Test initial/beyond-tail +facade getters for file/directory inputs, positioned cursors and real errors; +run CLI invocation and cursor controls. No payload expansion, synchronization, +new allocation or file mutation is added. Raw evidence stays ignored under +.local; tests use synthetic journals. Update facade contract spec and status +indexes; no operator action or dependency changes are needed. Review the narrow +SDK correction before closing this prerequisite and resuming CLI work. No open +design choice is needed to expose the existing public no-entry error. + +The shared beyond-tail control also reproduces Rust CLI `--cursor` reporting +`unset cursor` from its entry getter. Its cursor getter already exposes NoEntry. +Apply the same metadata-only error fallback to the Rust CLI, preserving the +original error unless cursor metadata confirms absence. This is the same +consumer failure class, with no Rust SDK change. Portable smoke now includes +both cursor options and updates its observation count. + +Runtime verification disproved the initial assumption that every Rust facade +cursor getter already normalized absence: directory seek-tail can leave its +selected file at UnsetCursor, mapped to generic Other. Normalize UnsetCursor +only in facade get_cursor to NoEntry, matching Go, and add file/directory +regression controls. Do not broaden the global error mapper or suppress I/O. +The earlier focused reviewer PASS is superseded pending this runtime correction. + +### Facade Cursor Regression Closure - 2026-10-26 + +Both facade cursor getters now expose typed absence; unrelated errors remain +unchanged. Both CLI cursor consumers and Go invocation listing use metadata +checks on mismatch/error paths. Go focused pure-SDK cursor tests and full CLI +unit tests pass; Rust all 19 facade tests and CLI 27 unit/11 subprocess tests +pass. Shared portable smoke passes 134/134 across both commands, including new +future cursor and after-cursor controls. Pinned v262 independently returns empty +success for both options. Internal final review passes the corrected scope after +the earlier Rust assumption was disproved by execution. No payload expansion, +locking, file writes or successful SDK path reads were added; no benchmark +framework or throughput claim is warranted for this error-only correction. + +Same-failure search covered both production Go TestCursor consumers and Rust +cursor getter delegation through directories. Specs document the public getter +classification. Existing consumer docs promise native-style cursor behavior; +this repairs that promise without a new workflow or example, so no docs or +operator-skill update is required. Runtime project skills already require EOF, +facade, directory and cross-language controls; no new workflow rule is needed. +AGENTS.md remains applicable. Both lifecycle indexes are reconciled. Synthetic +fixtures and ignored logs satisfy the sensitive-data gate. The lesson is to +execute error-mapping tests through delegated file readers, not infer directory +behavior from its outer bounds check. Remaining Windows/oracle/docs findings +stay in the real reopened SOW-0121. No untracked deferred SDK repair remains. +The local audit reports complete and clean; implementation and completed +lifecycle are committed together before resuming CLI work. + +## Regression - 2026-10-26 - Filtered Sequence Cursor Exhaustion + +### Pre-Implementation Gate + +The explicit-v262 query matrix passes 3033/3034 observations; Go's filtered +single-file partial sequence cursor beyond the tail returns the last row. +SdJournalOpenFiles selects a direct Reader for one path. Filtered Step preserves +its committed row at EOF, and the duplicated facade cursor-seek loop returns +without invalidating that row. Direct Reader.SeekCursor already clears position +with SeekTail at exhaustion. Reuse that existing seeker for the single-reader +facade; retain the directory loop and its exhausted-direction guarantee. This +preserves normal traversal EOF semantics and adds no synchronization or payload +work. The user authorized confirmed fixes; no new product choice is needed. + +Pause CLI work and preserve its edits. Add file/directory filtered future-sequence +controls, reproduce before the fix, verify forward EOF and backward last row, +then rerun existing cursor tests and the failed oracle case. Review the narrow +SDK change before closing; rerun the full CLI matrix after resuming0121. Existing +specs already require native seek behavior; no contract change is intended. +Keep synthetic fixtures and ignored raw evidence, update both lifecycle indexes, +and record unchanged docs/skills rationale at close. + +The pre-fix regression also proves directory reversal skips the last match: +its retained currentKey anchors reverse movement even though seek found no row. +On directory cursor-seek exhaustion only, reuse clearDirectoryPosition to clear +that anchor while retaining the exhausted forward direction and child state. +Repeated Next must stay exhausted; Previous now resets from the tail. Ordinary +filtered iteration EOF retention is intentionally unchanged. + +### Filtered Sequence Cursor Closure - 2026-10-26 + +The regression test failed before the fix for both stale file position and +skipped directory reversal. It now passes for a file, one-file directory and +overlapping two-file directory. Focused pure-Go facade/cursor and related +filter/directory tests pass. The original matrix command now returns empty +success. Independent internal final review passes 12 combinations: single-file, +deduplicated directory and overlapping merge, filtered/unfiltered, future time +and sequence cursors; each checks absent cursor, three repeated forward EOF +calls and reversal to the last row. No extra child seeks are necessary: existing +direction-change handling supplies the tail boundary after clearing the anchor. + +Same-pattern analysis covers direct versus facade seekers and both directory +traversal strategies. Ordinary filtered EOF behavior remains unchanged. Reusing +the direct seeker removes duplicated work; the directory correction changes two +fields only after an unsuccessful seek. No benchmark or performance waiver is +needed. Existing specs, consumer docs, AGENTS.md and runtime skills already +require these contracts and controls, so this correction needs no further +artifact change beyond tests, evidence and lifecycle indexes. No operator skills +exist. Synthetic evidence is retained ignored; no personal or operational data +is added. The lesson is to distinguish failed seek boundaries from ordinary +iteration EOF. Remaining CLI validation is owned by reopened0121; no SDK item +is deferred. Audit must remain complete and clean before this coherent commit. + +## Regression - 2026-10-26 - Directional Cursor Seeking + +### User Decision + +The user approved1A: add explicit direction-aware indexed cursor seeking in +both SDKs and consume it in both journalctl commands. Preserve existing seek +APIs and their immediate forward-positioning behavior. Netdata retains its +existing Explorer pagination and exact-row payload expansion paths. No shared +synchronization, persistent reader state or per-row allocation is authorized +for this correction. Use existing index machinery, with focused cost evidence +where new work warrants it; do not build a broad benchmark framework. + +### Pre-Implementation Gate + +Problem: eager forward cursor positioning cannot select reverse query starts, +including timestamp ties, missing or filtered-out cursor rows and partial +sequence/boot cursors. Evidence: PR12 independent final CLI review and pinned +v262 adversarial probes, recorded in0121. Existing SDK seekers also prioritize +a realtime start before sequence comparison; a new directional API must use +native per-file key precedence instead. No source journal mutation is involved. + +Affected surfaces: idiomatic file/directory readers and optional facade +extensions in Go/Rust, then CLI integration under0121. Existing APIs and Netdata +call graphs remain unchanged. Reuse ENTRY-array partitioning, DATA posting +indexes, boot metadata and directory merge candidates. Investigate exact +sequence/boot fallback and corruption behavior before implementing each path. +Risk: wrong tie side, inconsistent cross-file candidates, filter-state rewind, +suppressed metadata errors, or scans where indexes suffice. Retain successful +reader layouts and avoid adding state to existing reader structs. + +Plan: implement explicit SeekCursorDirection/seek_cursor_direction with existing +Direction types; eager positioning at the requested inclusive boundary. Reuse +per-file native indexed search and existing match expressions. Compose directory +candidates with existing ordering. Add facade extension without changing native +named APIs. Confirm both languages before resuming0121 integration. +Validation: shared v262 cursor cases forward/reverse, full/partial keys, ties, +filters and directories; error/recovery controls and absence/reversal tests; +existing SDK/Netdata tests and layout/work comparisons. Raw evidence and scratch +stay ignored under.local; fixtures use synthetic identities. Update API docs, +specs and relevant compatibility guidance, status indexes and closure evidence. +Three internal review scopes remain required; no external models or release. +Open decisions: none;1A resolves the API design fork. CLI0121 is paused while +this prerequisite is implemented and reviewed; preserve its current edits. + +### Directional Cursor Validation - 2026-10-26 + +Implemented explicit file/directory/facade extensions in both languages without +changing existing seekers or reader struct fields. Native per-leaf sequence, +boot/monotonic and realtime bounds reuse ENTRY/DATA indexes; directory initial +selection honors exact sequence identity before cross-history ordering and +repositions obsolete sibling candidates. No source writes, locks, dependency +changes or Netdata call-path changes are introduced. + +Pinned upstream evidence: systemd/systemd @ +8cc40e0c5e9234bf45084751ac53b1fbfe70b492, +src/libsystemd/sd-journal/sd-journal.c:767-975, 1032-1047, 1169-1186, +1310-1405 and src/libsystemd/sd-journal/journal-file.c:3732-3803. +The current workstation mirror is newer; only the pinned source and built +v262 oracle qualify this evidence. Incomplete sequence/monotonic pairs are +ignored when another valid location component exists, matching native parsing. + +Validation: 1,384 synthetic native comparisons per language match, covering +forward/reverse and at/after cursors, partial keys, ties, sparse Boolean filters, +multiple boots, rotated histories and cross-file clock skew. These development +probes are supplementary to committed regression tests, not a final review +verdict. Committed tests cover file/directory boundaries, exact sequence versus +boot ordering, both layouts, Go mmap/ReadAt, transient I/O retry, unrelated DATA +avoidance, and read-only recovery. The Go indexed 1,005-entry control uses 72 +accessor reads; it does not expand unrelated payloads. Existing iteration and +Netdata work are unchanged, so no broad timing harness is needed. + +Full pure-Go journal suite passes (113.7 seconds). Rust SDK/core suites pass +206 and 99 tests respectively, with three pre-existing ignored core doc tests. +Consumer wiki validation passes all 16 pages and all 35 executable examples. +SOW audit reports complete and clean. Focused tests are rerun after adding the +incomplete-key control. Three independent internal final reviews are pending; +this section does not claim completion or publication. + +Artifact maintenance: product-scope spec, Go API reference and Reader APIs wiki +now document the directional extension and eager selected-row contract. +Existing compatibility skill already requires cursor direction, error/recovery, +index usage and performance checks, so this SDK change needs no new workflow +rule. AGENTS.md and operator skills are unaffected. Lifecycle indexes identify +0159 as active and0121 as paused. Evidence uses synthetic IDs and ignored local +logs. CLI consumption and permanent oracle integration remain in reopened0121, +which must resume only after this SDK gate closes. + +Internal Go final review reproduced a closed filtered reader panic in the new +seek operation: pending filter construction dereferences the closed accessor. +Reuse the existing stepFiltered nil-accessor guard before any new seek work. +Test raw, pending and compiled filters through file/directory facades with +repeated forward/backward calls after Close. This is an error-path completion +within the approved API scope; no ordinary iteration or reader layout changes. + +The closed-reader test reproduced the panic before the guard and passes after +it, including all six raw/pending/compiled file/directory combinations and +repeated opposite-direction calls. Directional plus Netdata focused tests pass +again (1.025 seconds). Native review rebuilt both probes and expanded to 1,432 +cases per language, all matching v262. Rust final review passes, including the +additional failed-seek/getter/retry audit; native contract review passes. +Go final review is checking the guard and remaining state cases before closure. + +### Directional SDK Closure - 2026-10-26 + +All three independent internal scopes pass the final implementation: Go SDK, +Rust SDK and native contract/evidence. The Go reviewer independently confirms +closed-reader tests and a positioned transient-I/O probe for raw and filtered +readers: error propagation, retained committed row, ordinary movement and +explicit seek retry remain consistent. No unresolved SDK finding remains. +Rust edition-2024 formatting and final evidence/status edits do not change +behavior. Same-pattern inspection covers file/directory/facade entry points and +pending/compiled filters. Source changes remain additive to explicit seeking; +ordinary hot paths and Netdata retain their previous execution and layouts. + +The lesson is to qualify cursor initialization per native index and direction, +then test continuation and failure boundaries rather than only the first row. +Existing compatibility guidance already captures those requirements. Sensitive +data and artifact checks are clean; all external evidence uses pinned upstream +identity. SDK completion and code are committed together. CLI integration, +shared CLI oracle cases and native CI qualification belong to reopened0121; +no SDK task is deferred without an owner. The lifecycle audit is rerun before +commit and must retain its complete-and-clean verdict. diff --git a/.agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md b/.agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md index a618f557..00cb17d3 100644 --- a/.agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md +++ b/.agents/sow/pending/SOW-0122-20260621-journalctl-performance-followups.md @@ -4,7 +4,9 @@ Status: open -Sub-state: pending activation after SOW-0121 close. +Sub-state: pending activation; prerequisite SOW-0121 is completed, including vacuum ordering. This separate +scalability opportunity does not replace mandatory performance preservation in +the user-requested SDK/journalctl workstreams prepared on 2026-10-14. ## Requirements @@ -24,18 +26,24 @@ should be tracked before release-quality performance claims. Facts: -- SOW-0121 brought Rust and Go file-backed `journalctl` correctness parity to - the shared stock-oracle parser, directory, and query matrices. -- `--list-invocations` still derives invocation ranges by iterating matching - entries instead of using a more systemd-like indexed discovery path. +- SOW-0121 passed its shared stock-oracle parser, directory, and query matrices + at its earlier close. New combinations failed the 2026-10-14 audit; the + originating SOW repaired those failures, then reopened for the 2026-10-26 + CLI review findings, now corrected and independently reviewed. +- SOW-0121's 2026-10-19 unit/invocation audit supersedes the old row-scan + description. Its provisional implementation enumerated indexed unique IDs + and reopened readers for unit-restricted bounds. That approximation had + confirmed correctness gaps. Its completed invocation repair delivers native directional + discovery, global ID bounds, reader reuse and performance evidence. - `--follow` uses portable polling over explicit file/directory inputs. That preserves non-Linux correctness and purity, but may be more expensive than a daemon-backed Linux journal wait path on very large inputs. Inferences: -- These are performance and scalability risks, not observed SOW-0121 - correctness regressions. +- Broader input-size envelopes and portable polling costs are scalability + opportunities. Confirmed invocation correctness gaps and the performance of + their repairs were resolved in SOW-0121 and are not deferred here. - A production-grade release decision should either improve these paths or record measured evidence that the existing shape is acceptable for the target package. @@ -50,8 +58,8 @@ Unknowns: - Measure `--list-invocations`, `--follow`, reverse output, and large file/directory selection on representative synthetic fixtures. -- Replace entry-scan behavior with journal-native index/facade paths where the - journal format can answer the request safely and measurably faster. +- Evaluate remaining large-input costs using journal-native index/facade paths + against SOW-0121's delivered indexed invocation baseline. - Preserve SOW-0121 parser and query parity results. - Record any intentionally retained slower path with measured cost and release implications. @@ -67,8 +75,13 @@ Sources checked: Current state: -- Correctness parity is validated by SOW-0121. +- Earlier invocation correctness and performance repairs remain accepted. + SOW-0121 is completed after CLI review fixes, including reverse cursor seeking; + completed SOW-0159 owns the approved directional SDK primitives. - Performance work remains for invocation listing and follow scalability. +- This SOW remains pending and is not a third active workstream in the new + two-SOW request. Broader optimizations here are not prerequisites for measuring + or preserving performance of fixes in SOW-0121 and SOW-0159. Risks: @@ -82,9 +95,9 @@ Status: pending activation Problem / root-cause model: -- Some portable CLI actions still use correctness-first row iteration or - polling where an optimized implementation should prefer journal-native - indexes, cursor seeks, and bounded state. +- Portable polling and the large-input envelope need measurement after the + active correctness work. Activation must inspect the delivered indexed + implementation rather than assume the historical invocation algorithm. Evidence reviewed: @@ -146,7 +159,8 @@ Artifact impact plan: Open-source reference evidence: -- None checked yet. Activation should inspect `systemd/systemd` v260.1 paths +- None checked yet in this pending SOW. Activation should inspect the CLI's + `systemd/systemd` v262 target paths for invocation listing and follow behavior. Open decisions: @@ -200,6 +214,21 @@ Failure handling: - Created as the explicit follow-up mapping for SOW-0121 residual performance work. +### 2026-10-14 + +- Updated the dependency/reference after SOW-0121 regression reopening. The + user requires performance as-is or better for both requested workstreams; + their correctness and no-regression gates cannot be deferred to this SOW. +- Kept broad invocation/follow scalability work pending without activation or + merging it into the two requested scopes. + +### 2026-10-19 + +- Reconciled the historical invocation description with the confirmed native + correctness repair in SOW-0121. This SOW retains only broader envelope and + polling opportunities; it does not defer required behavior or performance + preservation and remains pending. + ## Validation Pending activation. diff --git a/.agents/sow/pending/SOW-0160-20261015-go-xz-reader-dictionary-allocation.md b/.agents/sow/pending/SOW-0160-20261015-go-xz-reader-dictionary-allocation.md new file mode 100644 index 00000000..a3c8f61c --- /dev/null +++ b/.agents/sow/pending/SOW-0160-20261015-go-xz-reader-dictionary-allocation.md @@ -0,0 +1,282 @@ +# SOW-0160 - Go XZ Reader Dictionary Allocation + +## Status + +Status: open + +Sub-state: measured performance opportunity tracked from SOW-0108. Analysis and +implementation are not activated. Preserve the approved SOW-0108, SOW-0159, +SOW-0121 execution order; this item is not a prerequisite for that chain. + +## Requirements + +### Purpose + +Assess whether Go XZ DATA decompression can reduce repeated dictionary +allocation while preserving format compatibility, reader isolation and speed. + +### User Request + +The user requires SDK/journalctl changes to maintain or improve performance. +This separate opportunity was discovered while measuring the approved reader +lifetime repair; no decoder redesign was requested or authorized. + +### Assistant Understanding + +Facts: + +- `go/journal/writer_compression.go:173` creates a fresh `xz.NewReader` for + each XZ DATA expansion, then bounds decompressed output with `readAllLimited`. +- SOW-0108's immutable 2,048-row compressed corpus exposes approximately + 8.6 GB/high-cardinality query and 17.2 GB/singleton query of cumulative + allocation in affected XZ cases. These are per-operation allocation totals, + not peak resident memory. Baseline and repaired implementations share this + decoder path and its cost; this is not a new repair regression. +- Existing pending/current SOWs do not own this specific optimization. + +Inferences: + +- Decoder construction/dictionary allocation is a plausible reusable-work + opportunity. Reuse may retain excessive memory or cross reader lifetimes; + lower allocation alone would not establish a better implementation. + +Unknowns: + +- Whether the compatible dependency API permits bounded reuse, and whether + reuse, a dependency improvement or retaining current behavior is best. +- The real-use frequency of XZ journals and the acceptable retained-memory + tradeoff. Investigate measurable costs before requesting a design choice. + +### Acceptance Criteria + +- Reproduce allocation sources with synthetic fixtures and explicit timer + boundaries; report cumulative allocation and retained memory separately. +- Present measured alternatives and a user-approved design before coding. +- Any implementation preserves Go/Rust observable behavior, pure SDK runtime, + complete compressed-stream handling and compiler/dependency constraints. +- Preserve or improve affected latency, allocation and memory paths, including + repeated readers, concurrent independent readers and non-XZ controls. + +## Analysis + +Sources checked: + +- `go/journal/writer_compression.go:173` and indexed DATA expansion in + `go/journal/explorer.go`. +- SOW-0108's 2026-10-15 regression evidence, immutable codec corpus and retained + before/candidate allocation observations under ignored task scratch. +- Pending/current SOW inventory; SOW-0159 owns regression prevention, while + SOW-0122 owns separate CLI invocation/follow scalability. + +Current state: + +- No decoder source change is included in SOW-0108. Its lifetime/header repair + must satisfy its own performance gate independently of this opportunity. + +Risks: + +- Pooling could exchange allocation for retained memory, contention or complex + ownership. Stream completeness and malformed-input behavior must remain + consistent. A dependency change could affect compiler floors or portability. + +## Pre-Implementation Gate + +Status: needs-user-decision + +Problem / root-cause model: + +- Repeated independent XZ DATA decoding constructs new decoder state. Existing + allocation measurements justify investigation, but do not select a design. + +Evidence reviewed: + +- The SDK construction call and matched allocation controls described above. + A complete dependency/API and lifetime analysis is required on activation. + +Affected contracts and surfaces: + +- Go compressed readers and all callers of XZ DATA expansion; preserve Rust + parity and cross-language fixture behavior. Writers are affected only if a + shared dependency or helper actually changes. + +Existing patterns to reuse: + +- Immutable codec fixtures, matching indexed/traversal checks, per-DATA output + limits, complete-stream controls and frozen paired benchmark helpers. + +Risk and blast radius: + +- Decoder ownership, memory retention, concurrency, dependency/compiler floors, + and allocation/latency regressions in XZ and shared decompression callers. + +Sensitive data handling plan: + +- Use synthetic fields and identities. Keep raw profiles/fixtures in ignored + repository-local scratch; commit only aggregate findings and source pointers. + +Implementation plan: + +1. After activation, profile constructor/dictionary/output costs and inspect + compatible reuse/dependency alternatives without changing production code. +2. Present measured options and obtain the user's design choice. Implement + only the selected minimal-complete design, or reject the opportunity with + evidence if every option has worse costs or incompatible behavior. + +Validation plan: + +- Cover regular/compact layouts, historical XZ fixtures, multiple blocks and + streams, read errors, concurrent readers and repeated opens/closes. Measure + time, allocations and retained memory against current Go and the Rust sibling; + run non-XZ controls and relevant interoperability checks. + +Artifact impact plan: + +- AGENTS.md: no workflow change is proposed. +- Runtime project skills: update compression/performance guidance if new reuse + or validation rules become necessary. +- Specs: update only actual operational guarantees changed by an approved design. +- End-user/operator docs: explain any changed options or memory guarantees; + an internal improvement may require no usage changes. +- End-user/operator skills: none currently exist in this repository. +- SOW lifecycle: keep open/pending until explicitly activated; no current-chain + scope is transferred here. +- SOW-status.md: list this opportunity in both canonical and root indexes. + +Open-source reference evidence: + +- This tracking record cites project source and retained project measurements; + no external repository checkout is used as evidence. On activation, record + dependency identity and checked revision for any external source analysis. + +Open decisions: + +- Activation and the decoder design are not authorized. No immediate user + answer is required while the requested SDK/journalctl chain remains active. + +## Implications And Decisions + +1. Track independently instead of expanding the approved lifetime/header repair. + No pooling, decoder replacement, new dependency or public option is approved. + +## Plan + +1. Retain the measured opportunity and source pointers in this pending SOW. +2. Await activation, then complete analysis and present design options before code. + +## Implementation And Review Plan + +Implementation: + +- The project manager owns any future analysis and implementation sequentially. + No runtime implementation is part of creating this tracking record. + +Reviewers: + +- No external review is authorized. Follow the system-wide external-reviewers + skill after complete local validation if this SOW is later activated. + +Repository boundary block for every external-reviewer prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Failure handling: + +- Retain unfavorable measurements and failed controls. Do not accept a lower + allocation count that increases latency, retained memory or compatibility gaps. + +## Execution Log + +### 2026-10-15 + +- Created from the project SOW template as SOW-0108's concrete follow-up mapping. + No runtime code, dependency or current execution order changes. + +## Validation + +Acceptance criteria evidence: + +- This is an open tracking record; optimization acceptance is not claimed. + +Tests or equivalent validation: + +- Source/ownership search and existing matched codec measurements establish + the lead. New implementation tests are neither required nor run for this record. + +Real-use evidence: + +- Existing Explorer XZ queries reproduce the cumulative allocation cost with + synthetic on-disk journal files; consumer prevalence remains to be measured. + +Reviewer findings: + +- Read-only ownership analysis found no existing specific follow-up SOW. + No final implementation review has occurred. + +Same-failure scan: + +- The shared decoder serves multiple SDK callers; their actual blast radius + must be enumerated before selecting or implementing a design. + +Sensitive data gate: + +- This record contains only synthetic aggregate measurements and source paths. + +Artifact maintenance gate: + +- AGENTS.md: unchanged; no workflow rule changes. +- Runtime project skills: unchanged for this tracking-only record. +- Specs: unchanged; no product behavior or guarantee changes. +- End-user/operator docs: unchanged; no new user behavior exists. +- End-user/operator skills: none currently exist. +- SOW lifecycle: open under pending; not an implementation prerequisite. +- SOW-status.md: canonical and root indexes include this opportunity. + +Specs update: + +- No implementation or shipped contract changes exist yet. + +Project skills update: + +- Existing compression/performance workflow already governs later analysis. + +End-user/operator docs update: + +- No usage or memory policy is changed by tracking an opportunity. + +End-user/operator skills update: + +- No output/reference skills are present or affected. + +Lessons: + +- Separate cumulative allocated bytes from peak or retained memory, and do not + mix an unchanged decoder opportunity into a lifetime regression's acceptance. + +Follow-up mapping: + +- SOW-0108's XZ dictionary-allocation opportunity is tracked here. Required + SDK/CLI regression-prevention work stays in SOW-0159/SOW-0121. + +## Outcome + +Open; no optimization has been authorized or implemented. + +## Lessons Extracted + +The allocation-accounting distinction above is retained for future analysis. + +## Followup + +This SOW is the concrete tracking destination; it creates no further workstream. + +## Regression Log + +No completed outcome exists; this record tracks an unchanged baseline opportunity. diff --git a/.agents/sow/pending/SOW-0161-20261017-file-indexer-source-posting-membership.md b/.agents/sow/pending/SOW-0161-20261017-file-indexer-source-posting-membership.md new file mode 100644 index 00000000..5012cd20 --- /dev/null +++ b/.agents/sow/pending/SOW-0161-20261017-file-indexer-source-posting-membership.md @@ -0,0 +1,203 @@ +# SOW-0161 - File Indexer Source Posting Membership + +## Status + +Status: open + +Sub-state: inherited source-analysis lead; not activated or implemented. + +## Requirements + +### Purpose + +Determine the correct membership and captured-tail contract for source-timestamp +postings used by native metadata indexes, preserving Go/Rust behavior and speed. + +### User Request + +Follow-up tracking from the requested SDK compatibility and performance work. +This record does not authorize an additional implementation workstream. + +### Assistant Understanding + +Facts: + +- Rust FileIndexer collects source postings without dereferencing their ENTRY + targets, then fills missing timestamps from global ENTRY offsets. +- The algorithm predates SOW-0159 and is present in prerequisite commit 4c49205. +- Strict bidirectional native membership certification belongs to verify_index. + +Inferences: + +- Replacing the source DATA inline target with that DATA object's own offset + should admit an extra histogram record. This is a source prediction, not an + executed reproduction or an established new regression. + +Unknowns: + +- Actual baseline/current results, equivalent Go surfaces and live-publication + treatment must be established before selecting a repair. + +### Acceptance Criteria + +- Reproduce or reject the source prediction on baseline/current regular and + compact fixtures, including an actively growing captured-tail case. +- Establish the affected native-index contract in both languages and obtain + approval for any behavior/performance change before implementation. + +## Analysis + +Sources checked: + +- rust/src/crates/journal-index/src/file_indexer.rs: collect_source_field_info, + build_histogram and collect_data_entry_indices. +- rust/src/crates/journal-core/src/file/offset_array.rs: collect_offsets. +- .agents/sow/specs/product-scope.md: strict verification contract. +- SOW-0159: required-chain error propagation and documented value tolerance. + +Current state: + +- With the existing 14-row source fixture, source inspection predicts that + inline DATA target plus entries 2..14 enter the source map; global fallback + then adds entry 1, yielding 15 records. No traversal error is generated. +- Source postings beyond the captured tail are another condition to examine. + Rejecting them mechanically could alter live-file behavior. + +Risks: + +- Full row parsing, an extra membership container or repeated searches could + hurt indexing performance. Additional certification must not be introduced + casually into this native metadata path. + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- A native posting target may be accepted as a source-map key without proof + that the global ENTRY population contains it. Reproduction remains required. + +Evidence reviewed: + +- The source and inherited algorithm listed above; no runtime reproduction. + +Affected contracts and surfaces: + +- Optional Rust FileIndexer/engine histogram and cache results; investigate Go + aggregation equivalents. Core strict verification remains the certification API. + +Existing patterns to reuse: + +- Existing source timestamp fixtures, global offset pass, entry-offset map, + engine batch isolation and versioned cache keys. + +Risk and blast radius: + +- Membership policy, live snapshots, repeated values and indexing costs. + This lead does not change SOW-0159's decoder or bounded-chain repair. + +Sensitive data handling plan: + +- Synthetic journal identities and payloads only; generated evidence stays + ignored under .local/. Durable records contain no raw private data. + +Implementation plan: + +1. Reproduce baseline/current behavior without changing runtime code. +2. Inspect both language surfaces and native publication semantics. +3. Present behavior and cost alternatives if a repair is justified, then update + this gate with the approved plan before implementation. + +Validation plan: + +- Regular/compact, healthy/invalid targets, captured-tail growth and required + Go/Rust parity; representative cost evidence appropriate to the chosen change. + +Artifact impact plan: + +- AGENTS.md: existing parity/performance rules apply; no change identified. +- Runtime project skills: update if validation exposes a reusable missed pattern. +- Specs: record an approved membership and captured-tail contract if changed. +- End-user/operator docs: document any changed index/cache behavior. +- End-user/operator skills: none currently exist. +- SOW lifecycle: pending only; no new prerequisite for SDK then CLI delivery. +- SOW-status.md: add this investigation lead to both status indexes. + +Open-source reference evidence: + +- No external repository was needed for this local source-analysis lead. + +Open decisions: + +- No implementation choice is approved. First establish actual behavior and + scope; user design decisions follow concrete evidence. + +## Implications And Decisions + +1. SOW-0159 fixes an encountered absent-map facet panic on its existing cold + error branch. Source-map certification is a separate inherited question, + rather than another suppressed traversal error. Track it here without + introducing additional indexing passes into the current repair. + +## Plan + +1. Investigate only after explicit activation; retain the approved current order. + +## Implementation And Review Plan + +Implementation: + +- Not started. The project manager owns any subsequently approved changes. + +Reviewers: + +- No review has run for this pending investigation. Follow the repository's + standing review authorization and system-wide skill at a meaningful boundary. + +Repository boundary block for every external-reviewer prompt: + +```text +CRITICAL REPOSITORY BOUNDARY: +- Do not make changes outside this repository for any reason. +- Repository path: current repository root. +- You may inspect external references read-only when the task requires it. +- Write, edit, delete, move, reset, checkout, install, generate, cache, or format nothing outside this repository. +- The only write exception outside the repository is /tmp. +- Prefer .local/ inside this repository for scratch work, generated temporary files, cloned references, logs, and working notes. +``` + +Failure handling: + +- Preserve reproductions and qualify uncertainty; do not change live-file + behavior or allocate a verification framework without an approved plan. + +## Execution Log + +### 2026-10-17 + +- Recorded the source-analysis lead discovered during SOW-0159's consumer audit. + +## Validation + +- Source trace only. Runtime, same-failure and parity validation are not yet run. +- Artifact maintenance: this pending SOW and both status indexes are added; + product contracts, skills and runtime code remain unchanged by this record. +- Follow-up mapping: source-map membership and captured-tail investigation + belongs here; encountered native traversal errors remain SOW-0159. + +## Outcome + +Investigation is pending; no behavior change or successful repair is claimed. + +## Lessons Extracted + +Chain completion and cross-index membership are distinct guarantees. + +## Followup + +No additional follow-up is identified before investigation. + +## Regression Log + +No completed outcome has been reopened; origin and contract remain to establish. diff --git a/.agents/sow/specs/journalctl-v260-parity-matrix.md b/.agents/sow/specs/journalctl-v260-parity-matrix.md index 62ad60a2..03409b38 100644 --- a/.agents/sow/specs/journalctl-v260-parity-matrix.md +++ b/.agents/sow/specs/journalctl-v260-parity-matrix.md @@ -1,4 +1,4 @@ -# journalctl v260.1 Portable Parity Matrix +# journalctl v262 Portable Parity Matrix ## Scope @@ -7,19 +7,32 @@ SOW-0121. Baseline authority: -- `systemd/systemd @ c0a5a2516d28` (`v260.1`) -- `src/journal/journalctl.c:241` help surface -- `src/journal/journalctl.c:420` option table -- `src/journal/journalctl.c:520` parser switch -- `src/journal/journalctl.c:1085` parser interaction checks -- `src/journal/journalctl.h:7` action enum -- `src/shared/output-mode.c:26` output mode names -- `src/journal/journalctl-filter.c:17` file-backed filter construction -- `src/journal/journalctl-show.c:51` seek, cursor, follow, and output loop - -The portable Rust and Go commands must recognize 100% of the official v260.1 +- `systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492` (`v262`) +- `src/journal/journalctl.c`: OPTION/OPTION_FULL parser records, help and ordered setters +- `src/shared/options.h`: common option macros +- `src/journal/journalctl.h`: action enum +- `src/shared/output-mode.c`: output mode names +- `src/journal/journalctl-filter.c`: file-backed filter construction +- `src/journal/journalctl-show.c`: seek, cursor, follow, and output loop +- `src/shared/pcre2-util.c`: PCRE2-8 byte matching and automatic casing +- `src/basic/time-util.c`: timestamp/duration parsing + +The SDK journal-format baseline remains v260.1. The historical matrix and +manifest filenames are retained so existing links and validation commands work. +Both CLIs require PCRE2-8 >=10.42; Go enables CGO only in the CLI. SDK consumers +remain independent of PCRE2 and CGO. + +The portable Rust and Go commands must recognize 100% of the official v262 command-line option and action surface. Unknown-option errors for official -v260.1 options are compatibility failures. +v262 options are compatibility failures. + +Calendar rendering and bound parsing share native-compatible timezone +resolution, including POSIX TZ rules, TZDIR, TZif leap records and pre-transition +footer behavior. POSIX rules without daylight transition dates use the configured +`posixrules` file before the native US-date fallback. Header/boot/invocation tables retain local time with --utc, +matching stock; entry calendar output honors --utc while retaining leap records. +Short-unix preserves raw epoch values and the native ten-column seconds width. +Rendering timezone data is loaded lazily once per command process. ## Classification Values @@ -35,7 +48,7 @@ v260.1 options are compatibility failures. - `recognized-unsupported`: parse, validate arguments when practical, then fail intentionally with a portable unsupported message. Do not inspect live host journal state, invoke systemd services, or mutate host journal/catalog paths. -- `parser-required`: parser behavior that must match official v260.1 before any +- `parser-required`: parser behavior that must match official v262 before any action dispatch. ## Error Contract @@ -57,7 +70,7 @@ disk-image/rootfs-only, or unsafe without an explicit file/directory target. | `--system` | none | recognized-no-op | Recognize. With explicit file input it has no effect. With explicit directory input, filter system journal files only if repository directory metadata supports it; otherwise accept as no-op and record test evidence. Without file/directory input, return unsupported because default host journal discovery is not portable. | | `--user` | none | recognized-no-op | Recognize. With explicit file input it has no effect. With explicit directory input, filter current-user journal files only if repository directory metadata supports it; otherwise accept as no-op and record test evidence. Without file/directory input, return unsupported because current-user host journal discovery is not portable. | | `-M`, `--machine=` | string | recognized-unsupported | Requires local container or machine journal access. Never connect to a host or container. | -| `-m`, `--merge` | none | recognized-no-op | Recognize. Interleaving multiple explicit file/directory inputs is already the portable directory/file behavior. Preserve official conflicts with specific `--boot` selectors and `--list-boots`; allow `--boot=all` with `--merge` like systemd v260.1. | +| `-m`, `--merge` | none | recognized-no-op | Recognize. Interleaving multiple explicit file/directory inputs is already the portable directory/file behavior. Preserve official conflicts with specific `--boot` selectors and `--list-boots`; allow `--boot=all` with `--merge` like systemd v262. | | `-D`, `--directory=` | path | file-backed-required | Open journal files from the supplied directory, including supported repository `.journal.zst` directory files. Reject regular-file input passed through `--directory=` instead of silently treating it like `--file=`. | | `-i`, `--file=` | glob/path | file-backed-required | Open one or more explicit journal files. Multiple `--file=` occurrences and glob expansion must be supported. `--file=-` must be recognized; support only if a seekable stdin-backed implementation exists, otherwise return a specific unsupported message. | | `--root=` | path | recognized-unsupported | Requires alternate root filesystem discovery and catalog hierarchy access. Do not inspect host rootfs. | @@ -69,22 +82,22 @@ disk-image/rootfs-only, or unsafe without an explicit file/directory target. | Option | Args | Classification | Required portable behavior | | --- | --- | --- | --- | -| `-S`, `--since=` | timestamp | file-backed-required | Inclusive realtime lower bound. Match official timestamp grammar where practical, including stock-accepted ISO `T` forms and timezone offsets. | -| `-U`, `--until=` | timestamp | file-backed-required | Inclusive realtime upper bound. Reject `--since` later than `--until`. Match official timestamp grammar where practical, including stock-accepted ISO `T` forms and timezone offsets. | +| `-S`, `--since=` | timestamp | file-backed-required | Inclusive realtime lower bound. Parse native calendar/epoch/relative forms, checked compound durations, fractions, UTC/offset/IANA zones and command-local DST state once, in argument order. | +| `-U`, `--until=` | timestamp | file-backed-required | Inclusive realtime upper bound. Reject `--since` later than `--until`. Parse native calendar/epoch/relative forms, checked compound durations, fractions, UTC/offset/IANA zones and command-local DST state once, in argument order. | | `-c`, `--cursor=` | cursor | file-backed-required | Seek to the specified cursor and include that entry when it matches filters. Match stock cursor seek ordering, including ignoring the optional `x=` hash for locating candidate entries while preserving exact cursor matching where required. | | `--after-cursor=` | cursor | file-backed-required | Seek to the specified cursor and start after that entry when it is present. Match stock cursor seek ordering, including ignoring the optional `x=` hash for locating candidate entries while preserving exact cursor matching where required. | | `--cursor-file=` | path | file-backed-required | Read cursor from the caller-provided file if present, start after it, and atomically update the file with the final cursor. | | `-b`, `--boot[=ID]` | optional descriptor | file-backed-required | Support no argument, `all`, numeric offsets, boot IDs, and boot ID plus offset from explicit file/directory entries. Do not use the host boot ID. | -| `--this-boot` | none | file-backed-required | Deprecated alias for the current `--boot` selection. Resolve from explicit file/directory input, not host state. | -| `-u`, `--unit=` | unit/glob | file-backed-required | Add system unit match expansion using journal fields. Support exact names and globs through indexed unique field discovery. | -| `--user-unit=` | unit/glob | file-backed-required | Add user unit match expansion using journal fields. Support exact names and globs through indexed unique field discovery. | +| `--this-boot` | none | file-backed-required | Deprecated setter equivalent to `--boot=0`; last occurrence wins over other boot setters. Resolve from explicit file/directory input, not host state. | +| `-u`, `--unit=` | unit/glob | file-backed-required | Add system unit match expansion using journal fields. Support exact names and globs through indexed unique field discovery. If neither system nor user selectors add any unit, return the native no-data error. An exact absent name remains a valid zero-row selector. | +| `--user-unit=` | unit/glob | file-backed-required | Add user unit match expansion using journal fields. Support exact names and globs through indexed unique field discovery, with the same combined empty-expansion rule as `--unit`. | | `--invocation=` | ID/offset descriptor | file-backed-required | Match explicit invocation IDs and resolve offsets from explicit file/directory input when unit context is required. | -| `-I` | none | file-backed-required | Equivalent to `--invocation=0`; require the same unit-context validation as official v260.1. | +| `-I` | none | file-backed-required | Equivalent to `--invocation=0`; require the same unit-context validation as official v262. | | `-t`, `--identifier=` | string | file-backed-required | Add `SYSLOG_IDENTIFIER=` alternatives, with repeated values ORed. | -| `-T`, `--exclude-identifier=` | string | file-backed-required | Preserve v260.1 file-backed behavior. Short-family outputs (`short*` and `with-unit`) skip entries whose `SYSLOG_IDENTIFIER=` matches an excluded value; JSON, export, verbose, and cat outputs remain unchanged. | +| `-T`, `--exclude-identifier=` | string | file-backed-required | Preserve v262 file-backed behavior. Short-family outputs (`short*` and `with-unit`) skip entries whose `SYSLOG_IDENTIFIER=` matches an excluded value; JSON, export, verbose, and cat outputs remain unchanged. | | `-p`, `--priority=` | level/range | file-backed-required | Support numeric and named priorities plus `from..to` ranges, matching official inclusive expansion. | | `--facility=` | list | file-backed-required | Support comma-separated numeric/named syslog facilities and `help`. | -| `-g`, `--grep=` | pattern | file-backed-required | Filter `MESSAGE=` with compatible regular expression behavior. If `--lines` searches from tail and `--follow` is not set, preserve official reverse-search behavior. | +| `-g`, `--grep=` | pattern | file-backed-required | Match the first raw `MESSAGE=` value using native PCRE2-8 byte semantics, including lookarounds, backreferences, embedded NUL and invalid UTF-8 bytes. Compile the final pattern after parser interaction checks, propagate match errors, and exit 1 with empty stderr when the show action finds no grep-selected entry. If `--lines` searches from tail and `--follow` is not set, preserve official reverse-search behavior. | | `--case-sensitive[=BOOL]` | optional bool | file-backed-required | Affect `--grep=` only. No argument means case-sensitive. | | `-k`, `--dmesg` | none | file-backed-required | Add `_TRANSPORT=kernel`. Do not read the kernel ring buffer or host journal. | | path match argument | path | recognized-unsupported | Official path arguments inspect current filesystem metadata, executables, scripts, and devices. In portable mode, only `FIELD=VALUE` matches and `+` are supported; path matches must return a specific unsupported message. | @@ -95,8 +108,8 @@ disk-image/rootfs-only, or unsafe without an explicit file/directory target. | Option | Args | Classification | Required portable behavior | | --- | --- | --- | --- | -| `-o`, `--output=` | mode | file-backed-required | Support every official v260.1 output mode listed below. `--output=help` prints the official mode list. | -| `--output-fields=` | comma list | file-backed-required | Restrict verbose/export/json fields. | +| `-o`, `--output=` | mode | file-backed-required | Support every official v262 output mode listed below. `--output=help` prints the official mode list. | +| `--output-fields=` | comma list | file-backed-required | Union repeated comma-separated projections without trimming field-name bytes; an empty list retains default/prior projection. Cat uses the first raw value of each requested field and preserves embedded NUL/newlines; invalid cat field names fail only when rendering a row. | | `-n`, `--lines[=[+]N]` | optional int | file-backed-required | Support default, tail count, and `+N` oldest semantics. Preserve conflicts with reverse/follow. | | `-r`, `--reverse` | none | file-backed-required | Show newest matching entries first. Conflict with follow. | | `--show-cursor` | none | file-backed-required | Print final cursor after entries. | @@ -112,11 +125,11 @@ disk-image/rootfs-only, or unsafe without an explicit file/directory target. | `-q`, `--quiet` | none | file-backed-required | Suppress informational messages, boot separator output, and the stock empty-result `-- No entries --` line where applicable. | | `--synchronize-on-exit=` | bool | recognized-unsupported | Requires journald Varlink synchronization on signal exit. Parse and reject when true; false may be accepted as no-op. | | `--no-pager` | none | recognized-no-op | Portable commands do not spawn a pager. | -| `-e`, `--pager-end` | none | file-backed-required | Do not spawn a pager, but preserve official default-line side effects for file-backed output: when no explicit line limit exists, show the last 1000 matching entries. | +| `-e`, `--pager-end` | none | file-backed-required | Do not spawn a pager, but preserve official default-line side effects for file-backed output: when no explicit line limit exists, show the last 1000 matching entries, or 10 with follow. | ## Output Modes -Every official v260.1 output mode is `file-backed-required`: +Every official v262 output mode is `file-backed-required`: | Mode | Required portable behavior | | --- | --- | @@ -144,19 +157,20 @@ Every official v260.1 output mode is `file-backed-required`: | `--verify-key=` | key | file-backed-required | Verify sealed journal files when combined with `--verify` or implied verification. Invalid seed/key text must fail with official-style parse errors. | | `--interval=` | duration | recognized-no-op | Parse for `--setup-keys`; no portable behavior exists without `--setup-keys`, so it is accepted as a no-op elsewhere. | | `--force` | none | recognized-no-op | Parse for `--setup-keys`; no portable behavior exists without `--setup-keys`, so it is accepted as a no-op elsewhere. | -| `--setup-keys` | none | recognized-unsupported | Official v260.1 checks `/var/log/journal`, reads host machine and boot IDs, and writes `/var/log/journal//fss`; portable file-backed commands must reject it instead of mutating host journal state. Evidence: `systemd/systemd @ c0a5a2516d28`, `src/journal/journalctl-authenticate.c:80`, `src/journal/journalctl-authenticate.c:89`, `src/journal/journalctl-authenticate.c:93`, `src/journal/journalctl-authenticate.c:97`, `src/journal/journalctl-authenticate.c:139`. | +| `--setup-keys` | none | recognized-unsupported | Official v262 checks `/var/log/journal`, reads host machine and boot IDs, and writes `/var/log/journal//fss`; portable file-backed commands must reject it instead of mutating host journal state. Evidence: `systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492`, `src/journal/journalctl-authenticate.c:80`, `src/journal/journalctl-authenticate.c:89`, `src/journal/journalctl-authenticate.c:93`, `src/journal/journalctl-authenticate.c:97`, `src/journal/journalctl-authenticate.c:139`. | ## Commands And Actions | Command/action | Args | Classification | Required portable behavior | | --- | --- | --- | --- | -| `-h`, `--help` | none | portable-utility-required | Print help including the full official v260.1 option surface. | -| `--version` | none | portable-utility-required | Print portable command version and compatibility baseline. | +| `-h`, `--help` | none | portable-utility-required | Print help including the full official v262 option surface. | +| `--version` | none | portable-utility-required | Print portable command version, v262 CLI target and actual PCRE2 version. | +| `--introspect-cli` | none | portable-utility-required | Hidden terminal action. Print native option metadata with truthful project/version/features and requested JSON framing. | | `--new-id128` | none | portable-utility-required | Deprecated utility action; print a new ID128 without touching host state. | -| `-N`, `--fields` | none | file-backed-required | List field names from explicit file/directory input using FIELD indexes where safe. | -| `-F`, `--field=` | field | file-backed-required | List unique values for a field from explicit file/directory input using FIELD DATA chains where safe. | +| `-N`, `--fields` | none | file-backed-required | List field names from explicit file/directory input using FIELD indexes where safe; order is unspecified. Ignore line limits for names. Reject entry/time/cursor/grep/explicit boot or invocation restrictions with the final enumeration action. | +| `-F`, `--field=` | field | file-backed-required | List unique values for a field from explicit file/directory input using FIELD DATA chains where safe, applying value-count limits. Reject entry/time/cursor/grep/explicit boot or invocation restrictions with the final enumeration action. Bare `all` boot/invocation descriptors and implicit follow boot selection do not add a restriction. | | `--list-boots` | none | file-backed-required | List boots from explicit file/directory entries with official index, boot ID, and first/last timestamp table output. Preserve official `--lines` selection, `--reverse` ordering, and conflict with `--merge`. | -| `--list-invocations` | none | file-backed-required | List invocation IDs from explicit file/directory entries for the selected unit context. Correctness is required; SOW-0122 tracks remaining large-archive performance work for indexed invocation-range discovery. | +| `--list-invocations` | none | file-backed-required | Discover invocation IDs directionally through indexed unit/ID matches, retaining first-valid ID field precedence and boot-scoped global ID bounds. Oldest/newest limits select discovery direction. SOW-0122 tracks broader large-input envelope work. | | `--list-namespaces` | none | recognized-unsupported | Requires host journal namespace discovery. | | `--disk-usage` | none | file-backed-required | With explicit file/directory input, report disk usage for selected journal files. Without explicit input, return unsupported host journal discovery. | | `--vacuum-size=` | bytes | file-backed-maintenance-required | With explicit `--directory=`, scan that directory's direct regular files, delete recognized archived `.journal`/`.journal~` files until active plus archived journal allocated usage is under the requested size, and protect active/current, non-matching, unknown, and subdirectory files. Without explicit directory, return unsupported. | @@ -188,7 +202,7 @@ Every official v260.1 output mode is `file-backed-required`: | Explicit empty boot descriptor | parser-required | Treat bare `--boot` as the current boot, but reject explicit `--boot=` as a failed empty boot descriptor like stock. | | Explicit empty optional values | parser-required | Reject explicit empty values where stock rejects them, including `--lines=` and `--case-sensitive=`. Bare `--lines` and bare `--case-sensitive` remain valid. | | Short attached values and clusters | parser-required | Accept stock short-option attached values and clusters such as `-n2`, `-ball`, and `-rn2`; preserve stock failure behavior for forms such as `-n=2` and `-b=true`. | -| User plus unit rewrite | parser-required | With `--user --unit=`, treat the unit as `--user-unit=` before adding filters. | +| User plus unit rewrite | parser-required | With `--user --unit=` and no `--system`, treat the unit as `--user-unit=` before adding filters. | | Grep reverse implication | parser-required | With `--grep` and tail-searching `--lines`, set reverse search unless follow is set. | | Default host journal | parser-required | If no explicit `--file=` or `--directory=` is supplied, only portable utility commands may run. File/query actions must return unsupported default host journal behavior. | @@ -197,8 +211,58 @@ Every official v260.1 output mode is `file-backed-required`: - The parser should be table-driven or generated from a shared manifest where practical so Rust and Go cannot silently drift. - Parser tests must include every option in this matrix and must fail if Rust or - Go returns an unknown-option error for an official v260.1 option. + Go returns an unknown-option error for an official v262 option. - Behavior tests must use repository-local fixtures only. They must not read live host journals or write under host journal directories. - Unsupported behavior tests must verify message class, exit code, and absence of host journal or systemd service interaction. + +Ordered setters validate every occurrence; terminal help/version/introspection +returns immediately when encountered. Actions select the last requested action, +with the native rotate/vacuum combination. Superseded action parameters do not +force dispatch. Output JSON/export/cat makes quiet sticky; full/no-full follows +encounter order. Boot and invocation aliases share their underlying setter. +Cursor-relative forward limits begin at the cursor. Follow streams all history +selected by a resolved cursor even with a numeric line limit; an empty or missing +cursor file retains default tail behavior. Since-only follow and explicit all or +no-tail include the full existing matching backlog. Boot/invocation tables support +all four JSON framings with integer microsecond bounds. + +Unix input filenames retain native bytes through globbing, directory discovery +and the SDK facade. File patterns use platform BRACE/NOCHECK/NOSORT expansion +in the environment locale; literal paths bypass expansion. Windows uses +Unicode filenames, brace alternatives and Unicode wildcard matching. Header +path output and vacuum filename ordering preserve native Unix bytes. + +Unit names use native glob/suffix, byte-escape and pure path-conversion rules, +including hashed long paths. Unit wildcard matches retain raw FIELD/DATA values. +Unix uses fnmatch NOESCAPE, with a FreeBSD compatibility path for extended +brackets missing from libc; Windows uses the C-locale byte grammar. Follow keeps +the initial wildcard expansion and resolved invocation ID. Invocation discovery +uses base unit fields/groups, first-valid ID precedence, native directional +indexed traversal and boot-scoped global ID bounds. Explicit ID selection may +match secondary fields. Identifier/priority/facility/transport filters remain +active when invocation selection replaces unit/boot show matches. + +Rendering follows SYSTEMD_COLORS, NO_COLOR, TERM, COLORTERM and both stdout and +stderr terminal state. Fixed modes and explicit true override NO_COLOR; +automatic modes respect it. Include priority/audit colors, first-match grep +offsets, verbose fields/cursors, JSON primitive styles, boot separators and +table underlining. TERM=linux disables underlining. Embedded DOCUMENTATION and +CONFIG_FILE links follow SYSTEMD_URLIFY. Windows drive and UNC file links use +slash-separated, percent-encoded file URIs; unsupported device paths remain +plain text. Unix link formatting is unchanged. Host catalog lookup, including implicit +documentation-link lookup, remains outside portable mode. Preserve plain +rendering paths and avoid a second regex evaluation for highlighting. + +## Approved Vacuum Ordering Exception (2026-10-24) + +Both CLIs order each sequenced archive history by sequence number and filename, +then merge available history heads by adjusted realtime, sequenced-before- +unsequenced, sequence ID and filename. Each archive without sequence metadata +is an independent history. This replaces the nontransitive stock comparison +when clock rollback causes conflicting sequence/time order; incidental libc +survivor sets are not the contract for that case. Count/size/age stop conditions, +empty cleanup and protected files otherwise retain their existing semantics. +The history sort and lazy heap merge cost O(n log n + k log h), with O(h) +auxiliary ranges and no added journal payload reads or metadata syscalls. diff --git a/.agents/sow/specs/product-scope.md b/.agents/sow/specs/product-scope.md index 6865a38f..164ef839 100644 --- a/.agents/sow/specs/product-scope.md +++ b/.agents/sow/specs/product-scope.md @@ -100,7 +100,27 @@ does not claim that a particular version has been published. ## Core Contracts - Implementations must not link to system journal libraries. -- Go implementations must not use CGO. +- Go SDK implementations must not use CGO. Native PCRE2-8 >=10.42 is permitted + only in the standalone journalctl command, which requires CGO and a C compiler. + Rust also confines native PCRE2 and its timezone provider to that command. +- Both commands may use the platform C library for Unix file globbing, + temporary thread-local locales and terminal queries. Native Unix filename + bytes survive explicit files, directory discovery and facade opening. + File patterns use native BRACE/NOCHECK/NOSORT semantics in the environment + locale; literal paths bypass expansion. Windows filenames are Unicode, + with brace alternatives and Unicode wildcard matching. +- Rust facade constructors have additive native-path forms: SdJournalOpenPath, + SdJournalOpenFilePath, SdJournalOpenDirectoryPath and SdJournalOpenFilePaths, + including file/directory/list WithOptions forms. Existing string functions + retain their signatures and delegate to the same readers. Journal and + whole-file Zstandard suffix checks preserve native filename bytes. + Registry File::as_path supplies the original filesystem path. File::path + retains ordinary UTF8 text; newly supported non-UTF8 paths return a reserved + NUL-prefixed hexadecimal Unix key, which must not be used for filesystem I/O. + Metadata deserialization reconstructs those native paths. CACHE_VERSION stays + at 4: ordinary serialization and Hash inputs are unchanged, and new raw-path + keys cannot alias ordinary paths. Per-file metadata adds one optional pointer; + no per-row storage or index interpretation changes. - Core journal readers and writers are file-format implementations only. They must not execute external programs, probe host identity, read host identity files or registries, or enforce writer locks by default. @@ -485,8 +505,8 @@ Current Go writer feature slice: - uncompressed DATA objects by default; - optional zstd, xz, and lz4-compressed DATA object writing with configurable compression threshold using the shared systemd threshold policy, including - zstd frame content-size metadata required by stock systemd verification and - readback of large compressed payloads; + zstd frame content-size metadata retained for the v260.1 writer compatibility + target and readback of large compressed payloads; - keyed hash tables using the journal file ID; - byte-safe DATA field values through `Field.Value []byte`; - high-level directory writing with Netdata-compatible chain active naming by @@ -688,6 +708,49 @@ Current shared high-level directory writer API slice: Readers must support applicable historical journal files represented by the shared fixture suite, including corrupted fixture behavior where the expected result is a controlled error or partial recovery. +Traversed hash, FIELD DATA and ENTRY-array chains must terminate on cycles with +a controlled error. Ordinary indexed reads retain their native traversal paths +and accepted acyclic link orders; they do not run a full integrity scan at open. +A detected cycle must not trigger the field-catalog compatibility row fallback. +Counted ENTRY-array traversal rejects a required array revisit before consuming +duplicate slots, including a revisit after an acyclic backward link. Reader +links beyond the declared population are not traversed or validated. Writer +tail selection rejects required revisits before modifying a repeated array. + +Zstandard DATA and whole-file staging consume complete supported streams, +including unknown-size, concatenated and skippable frames. Invalid checksums, +truncated later frames and trailing garbage are errors, not successful decoded +prefixes. The 768 MiB DATA output limit applies to the aggregate logical DATA +payload. Whole-file staging is streaming and does not apply that per-DATA limit +to an entire journal. Known-size single-frame DATA retains its native fast path. + +Ordinary Rust/Go SDK file, directory and facade readers recover the intact entry +prefix by default after observed unavailable ENTRY/ENTRY-array tails. Iteration +ends with normal EOF; per-file copyable recovery status records the reason, +associated loss offset, declared count and surviving count. Explicit collection +diagnostics identify files and allocate on request. Raw header counters remain +declared. A zero status reports no observed loss, not an integrity certificate; +native catalog access does not preflight unrelated arrays. Recovery changes +in-memory reader state only; it never reconstructs source journal bytes. +Ordinary writer publication may make temporarily unavailable data readable. +Arbitrary external restoration of damaged entries is not a supported automatic +refresh contract and must not justify added runtime overhead. Synthetic +damage/restore tests establish fault-injection behavior, not journald parity. +Recovered reader EOF polling memoizes direction and declared entry count, so +unchanged terminal polls do not rediscover the prefix. Successful movement, +explicit positioning/filter changes and changed declared counts invalidate that +terminal observation. Go explicit Refresh remains an independent refresh request. +Rust applies this policy in JournalReader without changing the public +JournalCursor layout; the low-level cursor handles positioning and initial +error recovery, not automatic EOF refresh. + +Recovery retains native index/header access and does not expand DATA rows to +establish the prefix. Interior holes followed by readable entries, wrong types, +cycles, malformed nonzero sizes, decompression failures and general I/O errors +remain errors. Unknown secondary posting tails that could contain surviving +matches remain errors. Strict verification, excluded-writer IndexedSnapshot +capture, append-open and retention preconditions remain separate and strict. + Accepted reader API layers: - Rust and Go readers use bounded reader-memory access for @@ -702,6 +765,22 @@ Accepted reader API layers: file-path verification APIs are covered by the same bounded access contract: object-graph and sealed TAG/HMAC verification read through reader-backed byte sources rather than materializing the whole journal file in memory. + Internal indexed facet/histogram aggregation must also preserve DATA values + across nested posting and ENTRY reads, including mmap eviction and positioned + read scratch reuse. Aggregation map keys are owned. Go materializes map keys + for contributing values and reuses a histogram key across the DATA object's + matching entries without repeated decompression. Indexed results and counters + must match across access bounds. + Go ENTRY-index header validation uses fitting native cache views and a fixed + reader-owned buffer for cached crossing/non-power-of-two-window reads. Header + acquisition uses constant per-reader storage instead of a fresh header buffer + for each offset; cache-window metadata follows the existing bounded-accessor + contract. Header bytes are consumed before any subsequent accessor read. + Native Go journal input on Linux uses a blocking read-only descriptor with + atomic close-on-exec, interrupted-open retry, typed path errors and normal + symlink following. This avoids unnecessary regular-file poller setup while + preserving sized random access. Whole-file compressed sources and non-Linux + readers retain their existing platform opening paths. - Idiomatic file and directory readers expose language-native entry objects, binary field values, repeated field values, cursor/realtime metadata, field enumeration, unique value enumeration, and boot listing for the accepted file @@ -771,6 +850,11 @@ Accepted reader API layers: consumer-side semantic migrations without changing the physical cache directory. Setting the builder namespace does not rewrite keys constructed elsewhere with `FileIndexKey::new()`. + Unreleased source builds use generation 4 after native-array completeness + repairs. Generation-3 entries miss: valid files rebuild on demand once after + upgrade. Encountered required native-chain or posting failures fail indexing; + existing per-object value tolerance remains. Old storage follows normal + cache eviction. Cache-hit algorithms and consumer namespaces are unchanged. - Rust and Go also expose explicit Explorer execution strategy controls through their idiomatic APIs. `ExplorerStrategy::Traversal` is the default and remains the behavior of each product language's ordinary @@ -798,6 +882,20 @@ Accepted reader API layers: syntactically valid cursor is accepted as a seek location even when no current entry has that exact cursor. Invalid cursor syntax fails. `test_cursor()` remains the exact-current-position check. +- File and directory readers expose explicit eager directional cursor seeking: + Go `SeekCursorDirection`, Rust `seek_cursor_direction`, and facade extension + `SdJournalSeekCursorDirection`. Existing cursor seek methods remain unchanged. + Forward/backward selects the inclusive first/last eligible boundary using + native per-file sequence, boot/monotonic and realtime key precedence, with + DATA-index filtering and directory merge ordering. Getters access the selected + row immediately; no eligible row succeeds with no current entry. Exactness + remains a separate cursor test. IDs normalize to 32 hexadecimal digits. + This extension adds no persistent reader fields or changes to Netdata paths. +- Go direct `Reader.TestCursor` and `DirectoryReader.TestCursor` return + `false, ErrEndOfEntries` at EOF. The libsystemd-compatible facade returns + `false, nil` instead; other errors remain errors. The Go facade cursor getter + returns public `ErrNoEntry` when no current entry exists, matching Rust + facade `Error::NoEntry`; other getter failures retain their original errors. - `get_cursor()`, JSON output, export output, and file-backed `journalctl` cursor files emit official systemd cursor strings with `s=`, `i=`, `b=`, `m=`, `t=`, and `x=` fields. `seek_cursor()` and `test_cursor()` accept both @@ -882,8 +980,8 @@ Current Go reader feature slice: diagnostics, constrained-platform investigation, and controlled fallback evidence only. Access stats expose the selected backend, fallback reason, window budget, mapped/read-buffer bytes, row-arena peak, and refresh counters; -- whole-file zstd fixtures and zstd, xz, and lz4-compressed DATA objects - through pure-Go dependencies; +- complete whole-file zstd streams and zstd, xz, and lz4-compressed DATA objects + through pure-Go dependencies, including the stream variants above; - historical unkeyed-hash journal reading, including LZ4-compressed DATA objects, with core reader traversal exposing the file-format entry set rather than old systemd 239 same-file duplicate suppression behavior; @@ -896,7 +994,7 @@ Current Go reader feature slice: - systemd-compatible export output for binary fields using size-prefixed field values and blank-line entry separators; - systemd-compatible JSON output for duplicate fields and binary values; -- file-backed journalctl output modes cover the stock v260.1 family: +- file-backed journalctl output modes cover the stock v262 family: `short`, `short-full`, `short-iso`, `short-iso-precise`, `short-precise`, `short-monotonic`, `short-delta`, `short-unix`, `verbose`, `export`, `json`, `json-pretty`, `json-sse`, `json-seq`, @@ -954,8 +1052,8 @@ Current Rust writer feature slice: - uncompressed DATA objects by default; - optional zstd, xz, and lz4-compressed DATA object writing with configurable compression threshold, including Rust zstd frame content-size metadata - required by stock systemd verification, using the shared systemd threshold - policy; + retained for the v260.1 writer compatibility target, using the shared systemd + threshold policy; - keyed hash tables using the journal file ID; - deterministic file ID selection through `JournalFileOptions::with_file_id()` for reference fixture generation and conformance checks; @@ -1002,8 +1100,9 @@ Current Rust reader feature slice: - regular and compact journal files; - files named `.journal`, `.journal~`, `.journal.zst`, and `.journal~.zst`; -- whole-file zstd fixtures and zstd, lz4, and xz-compressed DATA objects through - pure-Rust dependencies; +- complete whole-file zstd streams and zstd, lz4, and xz-compressed DATA objects + through the existing compression dependencies, including the stream variants + above; - directory iteration across active and archived files with stock-compatible root plus one machine-id subdirectory traversal and interleaved multi-file ordering; @@ -1013,9 +1112,10 @@ Current Rust reader feature slice: systemd-compatible export/json/text formatting; - configurable reader bounds through `ReaderOptions`: default `Live` mode uses systemd-style cached mutable bounds and refreshes file size only when a read - would exceed the cached end of file, while `Snapshot` mode fixes the file - size at open time for polling/query use cases that do not need to observe - appends during the current scan; + would exceed the cached end of file, while `Snapshot` mode caps reads at the + original open extent for polling/query use cases that do not need to observe + appends during the current scan. Cold recovery can observe physical shrink + without increasing that original cap; - `ReaderOptions` exposes live/snapshot bounds and window size for public consumers. Default Rust live readers remain windowed with a 32 MiB window so indexed DATA-chain traversal does not remap per small object. Whole-file mmap @@ -1035,7 +1135,7 @@ Current Rust reader feature slice: - export byte output preserves non-UTF8 RAW field names; JSON output, field enumeration, unique queries, and `get_data` facade helpers remain UTF-8 field-name surfaces; -- file-backed journalctl output modes cover the stock v260.1 family: +- file-backed journalctl output modes cover the stock v262 family: `short`, `short-full`, `short-iso`, `short-iso-precise`, `short-precise`, `short-monotonic`, `short-delta`, `short-unix`, `verbose`, `export`, `json`, `json-pretty`, `json-sse`, `json-seq`, @@ -1055,6 +1155,31 @@ Current Rust reader feature slice: active; - libsystemd-style match tree behavior from `SdJournalAddMatch()`, `SdJournalAddDisjunction()`, and `SdJournalAddConjunction()`; +- unreleased exact SDK/facade filters resolve native DATA hash/posting indexes + in both languages, preserving same-field OR and different-field AND grouping + without expanding unrelated rows. Go `Step`/`StepBack` apply filters; its raw + `Next`/`Previous` movement helpers do not. File/directory/facade group operators return + errors, and an empty pending Go filter builder returns `ErrMalformedFilter`. + Required unknown secondary posting tails remain errors. Healthy installed + posting populations stay fixed until rebuilding; read-only recovery can clip + existing DATA identities to the intact prefix, while absent DATA stays + absent until rebuilding. Failed group/build operations retain pending matches + for retry. Recovery requalification preserves known DATA identities and retries + the whole Boolean view after partial failure. Clearing matches rewinds file, + directory and facade readers to the head; callers restore saved locations + explicitly when continuation is required. Filtered movement returns the + nearest preceding/following match relative to the last successful row, + including filter installation/replacement and direction changes after EOF. + Directory mutations preserve the committed key and discard prefetched + candidates. Proven sequential positions retain sequential traversal rather + than probing every file; reversals and merged/recovered states use key-based + positioning. Failed selected ENTRY loading retains + the previous successful position on failure. Explicit filtered + realtime seeks search native posting nodes and binary-search their captured + slots, selecting the first match at/after the requested time forward and the + last at/before it backward. Errors from visited slots/headers propagate; + skipped postings are not certified. Ordinary sequential posting traversal + retains its existing error-discovery behavior; - file-backed Rust journalctl source selection supports `--file`/`-i`, repeated `--file` inputs, stock-style glob expansion with no-match preservation, and `--directory`; `--file=-` is explicitly unsupported @@ -1096,6 +1221,115 @@ Current Rust reader limitations: Implement journalctl rewrites in Rust and Go for file-backed/query behavior. +The command contract targets `systemd/systemd` tag `v262`, commit +8cc40e0c5e9234bf45084751ac53b1fbfe70b492. SDK journal-format compatibility +continues to target v260.1. The complete command option/action classifications +are in `journalctl-v260-parity-matrix.md` (historical filename retained). + +Both commands parse setters in encounter order, preserve required values and +explicit-empty semantics, and stop at terminal help/version/introspection. +Opaque Unix match, identifier, exclusion and projection arguments preserve +raw bytes. Non-UTF8 projection names retain explicit projection presence and +rendering-time cat validation; they do not expand to unrestricted output. +The final action controls dispatch and conflicts. PCRE2 compiles the final raw +pattern once after global parser checks; matching uses the first raw MESSAGE, +native byte/case behavior and native errors. A show command with no grep-selected +entries exits 1 without a diagnostic. Rendering exclusions do not erase a +successful grep selection. + +The native CLI dependencies have an accepted small fixed startup cost, measured +on the tiny Linux control as Go about 3.0ms to 3.8ms and Rust 1.5ms to 1.8ms. +This allowance is confined to command startup; SDK and sustained query +performance retain their no-regression requirement. Static native builds are +not a separate supported packaging profile. + +With final `-F`/`-N`, entry/time/cursor/grep/explicit boot or invocation filters +are errors under v262. Unrestricted enumeration still uses native FIELD/DATA +indexes. `-F` honors value limits; field-name order is unspecified. Time bounds +are parsed once per occurrence, including checked duration/epoch arithmetic, +calendar rounding, local/UTC/offset/IANA zones, TZif leaps and cached DST +selection. Cursor-relative limits begin at the cursor. Follow streams resolved +cursor history regardless of numeric line caps; since-only follow, explicit all +and no-tail include the complete existing matching backlog. Repeated projections +form a union without trimming field-name bytes. Cat preserves raw first values; +boot/invocation tables support all four JSON framings. +Follow without a since bound defaults to ten existing rows, including with +pager-end; follow with since and pager-end defaults to one thousand. Explicit +line counts, all/no-tail and cursor positioning retain their precedence. + +System/user unit wildcard expansion uses indexed unique field values. If no +exact selector or wildcard expansion adds a unit, return the native no-data +error before row output, including with a zero line limit. Exact absent unit +names and mixed exact/wildcard selectors remain valid. An ordinary journal +field cannot turn an empty expansion into a match through a private sentinel. +Unit arguments retain native glob/suffix rules, byte escaping, pure path-to-unit +conversion and long-path hashing. Empty names are invalid. Unix wildcard +matching uses fnmatch NOESCAPE in the environment locale. FreeBSD uses a +shared CLI-private compatibility matcher for extended bracket syntax missing +from its libc, including named character classes and single-character +collating elements. Ordinary patterns keep native matching. The Windows fallback +matches unit payload bytes using the C-locale grammar. Current-user conversion +deduplicates moved system selectors. Follow retains the initial wildcard +expansion and invocation selection. + +Invocation discovery uses the three base system/user unit fields, excluding +show-only coredump and slice groups. It reads the first valid nonzero ID from +the ordered invocation fields and walks indexed matches in native direction. +ID bounds span every matching row in the selected boot, including rows without +unit metadata. Explicit IDs without offsets may match secondary ID fields; +relative offsets advance from the reference ID's boundary. Invocation selection +replaces boot/unit show matches while retaining identifier, transport, priority +and facility filters. Readers are reused and metadata getters avoid expanding +unrelated payloads. Broader input envelopes remain separate performance work. + +Native integer options preserve base-prefix/octal syntax and signed 32-bit +limits for line counts and boot/invocation offsets; vacuum file counts retain +the unsigned 64-bit range. Descriptor IDs use 32 hexadecimal digits, with an +explicit sign before an offset. Priority symbols are canonical and case +sensitive; numeric facilities range through 127. Invalid unique-field names +fail even with a zero limit. Empty requested cursor-file paths retain their +requested update semantics. + +Bounded show output stops before reading or matching a further row. Boot +transitions are observed before grep in both directions and follow polls; +initial follow-tail selection counts native rows before grep and poll resume +state includes nonmatching visits. Zero-limit boot/invocation tables exit one, +with quiet/JSON suppressing informational diagnostics. Quiet verification +retains errors while suppressing successful PASS messages. +Show-cursor and cursor-file checkpoints retain the final native position, +including grep misses, bound-crossing rows and zero-line positioning. Finite +selected counts still stop immediately without another regex evaluation. + +Short output selects the last repeated MESSAGE; grep and cat retain their +first-value rules. No-full output applies native per-line shortening, +prefix/suffix retention, three-line/300-byte thresholds and the v262 fixed +Unicode width table. COLUMNS is resolved once per shortened-output renderer; +valid values from 1 through 65535 override stdout's terminal width, followed by +an 80-column fallback. The default full-width path bypasses shortening. +Displayed realtime uses the last source-realtime C-string value when native +integer parsing yields a value strictly between zero and 2^55; otherwise it +uses the ENTRY realtime. This does not replace exported ENTRY metadata. +Before short rendering, native tab/ANSI/CR preparation strips completed color +and OSC sequences, expands tabs to eight spaces and drops trailing/pre-newline +CR. Show-all bypasses this step; cat/grep continue to use raw MESSAGE bytes. +JSON aggregation keeps a single binary byte array distinct from an outer array +of repeated values in all four framings. Vacuum-size parsing uses native +ordered uppercase compound suffixes and checked decimal/fraction arithmetic. + +Color output follows SYSTEMD_COLORS fixed/automatic/boolean modes, NO_COLOR, +stdout/stderr terminal state, TERM and COLORTERM. Enabled output includes native +priority/audit colors, first-match grep highlighting, verbose timestamps, +cursors and trusted fields, JSON primitive colors, boot separators and table +headers. Underlining is disabled on the Linux console. Grep captures offsets +from its existing PCRE2 match; only colored buffered output retains those +offsets. Plain output avoids color formatting. Verbose multiline fields indent +continuations; short-delta saturates negative differences to zero on reverse +output, following native timestamp subtraction. +Embedded DOCUMENTATION and CONFIG_FILE links follow SYSTEMD_URLIFY; optional +file links discover the command's hostname lazily outside the SDK. Host catalog +lookup, including implicit documentation-link lookup, remains outside portable +mode. Header path output and vacuum filename tie ordering preserve native bytes. + Matching semantics: - Different fields are ANDed. @@ -1116,7 +1350,8 @@ File-backed query semantics: daemon-backed Linux journal wait APIs on very large inputs. - `--pager-end` does not spawn a pager in portable mode, but it preserves stock line-selection semantics by applying an implicit 1000-entry tail when no - explicit `--lines`, `--head`, or `--tail` option is present. + explicit `--lines`, `--head`, or `--tail` option is present. Follow without + since retains its implicit ten-entry tail instead. - Bare `--lines` and `-n` select the stock default 10-entry tail. Explicit empty `--lines=` is a parse error, matching stock `journalctl`. - Empty default/verbose show results print `-- No entries --`; `--quiet` and @@ -1124,23 +1359,22 @@ File-backed query semantics: - `--cursor`, `--after-cursor`, `--cursor-file`, and `--show-cursor` use official systemd cursor strings for file-backed reads and cursor-file updates. -- `--list-invocations` is correctness-compatible for covered file-backed unit - contexts, but the current implementation derives invocation ranges from - matching entries. SOW-0122 tracks the remaining performance work before any - large-archive performance claim. +- `--list-invocations` uses native directional indexed discovery and global + boot-scoped ID bounds. SOW-0122 retains broader large-input envelope and + polling work, beyond the required correctness and performance repair. - Recognized options may appear before or after show-action match arguments, matching stock argument permutation. Short options accept stock attached values and clusters such as `-n2`, `-ball`, and `-rn2`, while stock-invalid forms such as `-n=2` and `-b=true` remain invalid. - `--identifier`, `--exclude-identifier`, `--priority`, `--facility`, `--grep`, `--case-sensitive`, `--dmesg`, `--unit`, and `--user-unit` are - file-backed filters. `--exclude-identifier` follows stock v260.1 output-mode + file-backed filters. `--exclude-identifier` follows stock v262 output-mode scope: short-family outputs filter matching `SYSLOG_IDENTIFIER=` entries, while JSON, export, verbose, and cat remain unchanged. `--grep` with tail-style `--lines=N` implies reverse traversal unless `--follow` is set. - Unit filters use the same systemd v260.1 unit-related fields, disjunction + Unit filters use the same systemd v262 unit-related fields, disjunction groups, unit suffix mangling, glob expansion over FIELD/DATA indexes, and - `--user --unit=` rewrite as stock file-backed `journalctl`. + `--user --unit=` rewrite (unless `--system` is also present) as stock file-backed `journalctl`. - `--disk-usage` reports allocated filesystem usage for explicit `--file`/`--directory` journal inputs. Without explicit input, it remains an unsupported host-journal discovery action. @@ -1150,15 +1384,24 @@ File-backed query semantics: non-matching, unknown, and subdirectory files. `--vacuum-files` uses the stock count condition: protected active/current files plus remaining archived candidates must be at or below the requested count. + Age is the earliest available filename, modification, access, change or + creation timestamp. Linux includes a valid legacy `user.crtime_usec` xattr + and attempts header reads without updating atime, retrying ordinary reads + when that open mode is unavailable. Metadata checks are CLI maintenance + operations; SDK and query traversal do not perform them. + The user accepted the creation-metadata cost for explicit age-based vacuum + maintenance: the measured warm control adds about 7 ms in Go and 3.8 ms in + Rust for 1000 recent archive names. This is a maintenance-only exception; + SDK and sustained query performance requirements remain unchanged. - `--new-id128` is a portable standalone utility action and prints a new v4 ID128 in the same string, UUID, `SD_ID128_MAKE()`, and Python constant forms - as stock systemd v260.1. + as stock systemd v262. - Positional `FIELD=value`, path, and `+` disjunction arguments belong to the show action. Non-show actions other than catalog listing/dumping reject extra positional arguments with the stock `Extraneous arguments starting with ''` parser error before portable unsupported dispatch. -- The long `--follow` option is supported. Existing short `-f` file aliases are - preserved in languages that already used `-f` for `--file`. +- `--follow` and `-f` follow explicit inputs. `--file` and `-i` select files; + the old conflicting `-f` file alias is not part of the v262 contract. Daemon-only commands are not implemented in this project. They must return documented unsupported behavior rather than silently pretending to perform daemon operations. diff --git a/.agents/sow/specs/rust-reader-performance.md b/.agents/sow/specs/rust-reader-performance.md index e57bb8b4..5121bd5c 100644 --- a/.agents/sow/specs/rust-reader-performance.md +++ b/.agents/sow/specs/rust-reader-performance.md @@ -44,6 +44,17 @@ scan large journal sets with predictable CPU, memory, and I/O behavior. Entry scans are acceptable only when the requested operation is inherently row-oriented or when an active SOW records evidence that the indexed path is unsafe or slower for that operation. +9. Explicit filtered realtime seeks must binary-search captured posting-node + slots rather than linearly visit preceding entries. Preserve lazy empty and + singleton paths, existing node/chain validation and ordinary sequential + traversal. Keep cursor direction state within existing layout padding where + possible, and restore failed ENTRY-load positions without cloning the whole + cursor per row. +10. Counted-array cycle history must preserve the four-word Node shape and + within-array movement without per-row sets or allocation. The chain head + replaces cached next; uncached array transitions read the current header. + Exceptional ordering checks only the visited native prefix. Cached reverse + node chains retain their existing reuse. ## Rolling Mmap Row Pins @@ -91,6 +102,12 @@ Rules: maximum. - The Rust `ruzstd` fallback remains required for compatibility with frames the native fast path rejects or cannot size safely. +- Decode the complete logical Zstandard stream, including concatenated, + skippable and unknown-size frames. Validate each frame's size/checksum when + present and reject truncation or trailing garbage. Enforce the 768 MiB DATA + output bound across all frames, and clear partial output on error. +- Whole-file `.journal.zst` staging streams input/output and decodes every + frame; the per-DATA output bound does not limit the size of a whole journal. - Decompressed bytes live in the current-row arena and are invalidated on row advance or explicit row reset. - A query-scoped reusable decompression cache is not part of the current diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 4af0e7fe..e0361b49 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -34,6 +34,10 @@ jobs: - name: Checkout uses: actions/checkout@v6 + - name: Install Go CLI build prerequisites + if: matrix.language == 'go' + run: sudo apt-get update && sudo apt-get install -y build-essential pkg-config libpcre2-dev + - name: Initialize CodeQL uses: github/codeql-action/init@v4 with: @@ -44,6 +48,8 @@ jobs: - name: Autobuild if: matrix.build-mode == 'autobuild' uses: github/codeql-action/autobuild@v4 + env: + CGO_ENABLED: '1' - name: Analyze uses: github/codeql-action/analyze@v4 diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index a5b53ade..f1a7852d 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -40,6 +40,9 @@ jobs: go-version-file: go/go.mod cache-dependency-path: go/go.sum + - name: Install CLI build prerequisites + run: sudo apt-get update && sudo apt-get install -y build-essential pkg-config libpcre2-dev + - name: Generate Go coverage run: tests/coverage/run_go_coverage.sh @@ -61,6 +64,12 @@ jobs: - name: Checkout uses: actions/checkout@v6 + - name: Setup Go for synthetic CLI fixtures + uses: actions/setup-go@v6 + with: + go-version-file: go/go.mod + cache-dependency-path: go/go.sum + - name: Setup Rust uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 with: @@ -71,6 +80,9 @@ jobs: with: tool: cargo-llvm-cov + - name: Install CLI build prerequisites + run: sudo apt-get update && sudo apt-get install -y pkg-config libpcre2-dev + - name: Generate Rust coverage run: tests/coverage/run_rust_coverage.sh diff --git a/.github/workflows/journalctl.yml b/.github/workflows/journalctl.yml new file mode 100644 index 00000000..da24facb --- /dev/null +++ b/.github/workflows/journalctl.yml @@ -0,0 +1,141 @@ +name: Portable journalctl + +on: + pull_request: + push: + branches: [master] + workflow_dispatch: + +permissions: + contents: read + +jobs: + native: + name: CLI on ${{ matrix.os }} + runs-on: ${{ matrix.os }} + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + include: + - os: ubuntu-24.04 + shell: bash + python: python3 + target: x86_64-unknown-linux-gnu + exe: '' + - os: macos-15 + shell: bash + python: python3 + target: aarch64-apple-darwin + exe: '' + - os: windows-2022 + shell: 'msys2 {0}' + python: python + target: x86_64-pc-windows-gnullvm + exe: .exe + # Windows cannot represent the upstream colon-named fuzz inputs. + sparse: | + /* + !/fixtures/systemd/test-data/corrupted/id:* + defaults: + run: + shell: ${{ matrix.shell }} + env: + CGO_ENABLED: '1' + GOTOOLCHAIN: local + PYTHONDONTWRITEBYTECODE: '1' + CARGO_HOME: ${{ github.workspace }}/.local/cargo-home + CARGO_TARGET_DIR: ${{ github.workspace }}/.local/cargo-target + CARGO_BUILD_TARGET: ${{ matrix.target }} + CARGO_TARGET_X86_64_PC_WINDOWS_GNULLVM_LINKER: clang + CC_x86_64_pc_windows_gnullvm: clang + AR_x86_64_pc_windows_gnullvm: llvm-ar + GOCACHE: ${{ github.workspace }}/.local/go-build + GOMODCACHE: ${{ github.workspace }}/.local/go-mod + GOPATH: ${{ github.workspace }}/.local/go-path + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + # Accept index paths on Windows; sparse rules keep them off disk. + env: + GIT_CONFIG_COUNT: ${{ runner.os == 'Windows' && '1' || '0' }} + GIT_CONFIG_KEY_0: core.protectNTFS + GIT_CONFIG_VALUE_0: 'false' + with: + persist-credentials: false + sparse-checkout: ${{ matrix.sparse }} + sparse-checkout-cone-mode: false + - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6 + with: + go-version-file: go/go.mod + cache: false + - name: Setup Windows C tools + if: runner.os == 'Windows' + uses: msys2/setup-msys2@ec48f7c5447b3140e2b088413ae3a55687bccb6e # v2.33.0 + with: + msystem: CLANG64 + path-type: inherit + install: mingw-w64-clang-x86_64-clang mingw-w64-clang-x86_64-pkgconf mingw-w64-clang-x86_64-pcre2 + - name: Install Linux CLI prerequisites + if: runner.os == 'Linux' + run: sudo apt-get update && sudo apt-get install -y build-essential pkg-config libpcre2-dev + - name: Install macOS CLI prerequisites + if: runner.os == 'macOS' + run: brew install pcre2 pkgconf + - name: Configure native libraries + run: | + if [ "$RUNNER_OS" = Windows ]; then + echo 'CC=clang' >> "$GITHUB_ENV" + echo 'PKG_CONFIG=pkg-config' >> "$GITHUB_ENV" + echo "PKG_CONFIG_PATH=$(cygpath -m "$MINGW_PREFIX/lib/pkgconfig")" >> "$GITHUB_ENV" + elif [ "$RUNNER_OS" = macOS ]; then + echo 'CC=clang' >> "$GITHUB_ENV" + echo "PKG_CONFIG_PATH=$(brew --prefix pcre2)/lib/pkgconfig" >> "$GITHUB_ENV" + fi + pkg-config --atleast-version=10.42 libpcre2-8 + pkg-config --modversion libpcre2-8 + - name: Read Rust compiler minimum + id: rust + run: ${{ matrix.python }} -c 'import tomllib; print("version=" + tomllib.load(open("rust/Cargo.toml", "rb"))["workspace"]["package"]["rust-version"])' >> "$GITHUB_OUTPUT" + - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 + with: + toolchain: ${{ steps.rust.outputs.version }} + targets: ${{ matrix.target }} + - name: Check pure Go SDK and native CLIs + run: | + go version + rustc --version + CGO_ENABLED=0 go -C go build ./journal + go -C go test -count=1 ./cmd/journalctl + cargo test --manifest-path rust/Cargo.toml --locked -p journalctl + go -C go build -o ../.local/go-journalctl${{ matrix.exe }} ./cmd/journalctl + cargo build --manifest-path rust/Cargo.toml --locked -p journalctl + ${{ matrix.python }} tests/interoperability/run_journalctl_portable_smoke.py --go .local/go-journalctl${{ matrix.exe }} --rust .local/cargo-target/${{ matrix.target }}/debug/journalctl${{ matrix.exe }} + + freebsd: + name: CLI on FreeBSD + runs-on: ubuntu-24.04 + timeout-minutes: 60 + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + with: + persist-credentials: false + - name: Build and exercise native CLIs + uses: vmactions/freebsd-vm@c46abacb49f09938ca4e1702d15d836285d694cc # v1.5.9 + with: + release: '14.4' + prepare: pkg install -y go rust pcre2 pkgconf python3 + run: | + set -eu + export CGO_ENABLED=1 GOTOOLCHAIN=local PYTHONDONTWRITEBYTECODE=1 CC=cc + export CARGO_HOME="$PWD/.local/cargo-home" CARGO_TARGET_DIR="$PWD/.local/cargo-target" + export GOCACHE="$PWD/.local/go-build" GOMODCACHE="$PWD/.local/go-mod" GOPATH="$PWD/.local/go-path" + pkg-config --atleast-version=10.42 libpcre2-8 + pkg-config --modversion libpcre2-8 + go version + rustc --version + CGO_ENABLED=0 go -C go build ./journal + go -C go test -count=1 ./cmd/journalctl + cargo test --manifest-path rust/Cargo.toml --locked -p journalctl + go -C go build -o ../.local/go-journalctl ./cmd/journalctl + cargo build --manifest-path rust/Cargo.toml --locked -p journalctl + python3 tests/interoperability/run_journalctl_portable_smoke.py --go .local/go-journalctl --rust .local/cargo-target/debug/journalctl diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ca4b4568..87984172 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -112,11 +112,18 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: python3 tests/release/release.py preflight --source "$RELEASE_SOURCE" --version "$RELEASE_VERSION" --commit "$RELEASE_COMMIT" + - name: Install CLI build prerequisites + run: sudo apt-get update && sudo apt-get install -y build-essential pkg-config libpcre2-dev - name: Test Rust run: cargo test --manifest-path "$RELEASE_SOURCE/rust/Cargo.toml" --workspace --locked - name: Test Go working-directory: .local/release/source/go - run: go test ./... + run: go test -count=1 ./... + - name: Test Go CLI with PCRE2 + working-directory: .local/release/source/go + env: + CGO_ENABLED: '1' + run: go test -count=1 ./cmd/journalctl publish: name: Publish and verify Rust crates diff --git a/AGENTS.md b/AGENTS.md index bb65d6b6..3e7be45d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,10 +77,15 @@ Mandatory rules: be treated as an explorer bug. - Treat repeated parsing of reusable DATA objects as a performance bug unless a SOW records evidence that the alternative is slower or unsafe. -- Benchmark and profile hot paths before accepting performance-sensitive code. - Compare benchmarks against systemd when practical and against the fastest - sibling implementation in this repository; record why if a comparison cannot - run. +- Choose performance evidence by tracing the actual changed work. Careful code + review and representative high-level tests can establish that a change does + not negatively affect performance. Use existing benchmarks and targeted + profiles for credible added costs or observed regressions; do not build or + expand a large measurement framework to prove every small detail. An + inconclusive timing interval alone is neither proof of equality nor evidence + of a regression. Preserve adverse observations and investigate them. + Compare equivalent benchmarks against systemd when practical and against the + fastest sibling implementation; explain unavailable comparisons. - Record any intentionally slower path in the SOW, docs, and tests, with the reason, measured cost, and the conditions where it is acceptable. @@ -461,6 +466,16 @@ Output/reference skills: ### Project-specific overrides - The baseline systemd compatibility target is `systemd/systemd` tag `v260.1`. +- The file-backed Rust and Go `journalctl` commands target `systemd/systemd` + tag `v262` (8cc40e0c5e9234bf45084751ac53b1fbfe70b492). This CLI target + does not advance the SDK journal-format target or compiler minimums. +- Native PCRE2-8 >=10.42 is allowed only in the two journalctl commands. + The Go command requires CGO and a C compiler; the Go SDK remains CGO-free. + The commands may also use the platform C library for Unix file globbing, + thread-local locale selection, and terminal queries. Windows filenames use + the Unicode path contract. These CLI exceptions do not extend to SDKs. + Keep CLI build/test steps explicitly CGO-enabled and preserve independent + pure SDK checks. System journal library linkage remains prohibited. - Netdata's consuming-module declarations are the authority for SDK compiler minimums. The SDK's Go and Rust minimums may match or remain below Netdata's corresponding declared minimums, but must never exceed them. Requiring a @@ -485,7 +500,7 @@ Output/reference skills: - The `+` separator is a systemd journalctl disjunction feature to replicate for file-backed journalctl behavior; it is not a new extension. - Implement journalctl rewrites for file-backed/query behavior in Rust and Go. - Do not implement daemon-only journalctl commands, including daemon sync, flush, rotate, and relinquish-var operations. -- Common compression-library dependencies are allowed after dependency review. Journal parsing/writing must not depend on systemd/libjournal; CGO and linking to system journal libraries remain disallowed unless the user explicitly changes those separate constraints. +- Common compression-library dependencies are allowed after dependency review. Journal parsing/writing must not depend on systemd/libjournal. CGO remains disallowed in SDKs; the journalctl-only PCRE2 exception above does not permit linking system journal libraries. - Before Netdata integration or stable release work advances, GitHub code scanning and Codacy analysis findings must be fixed or explicitly dispositioned under the active SOW policy. Raw SARIF, Codacy exports, and scanner logs must stay under `.local/`; durable artifacts may contain only sanitized aggregate summaries and evidence. - Rust and Go sources are frozen for the 2026-06-11 docs-and-parity program (SOW-0103 through SOW-0106). Problems found in Rust or Go become pending SOWs; they are not fixed inside the program SOWs. - Current review cadence: finish the complete active SOW locally first, including local validation and SOW evidence. diff --git a/RELEASING.md b/RELEASING.md index d6487be2..3eea45ef 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -84,7 +84,9 @@ PRs run the release-helper tests; they do not publish anything. alignment, published dependency pins, installation examples, Go module path, release environment, and any existing release tags or package versions. - Runs the full Rust and Go suites using the compiler minimums declared by the - selected source. Go runs with CGO disabled. + selected source. Go SDK tests run with CGO disabled; separate Go CLI tests + enable CGO. CLI-inclusive Rust/Go checks install PCRE2 development files and + pkg-config on the CI runner. Publishing and SDK consumer builds remain pure. - Dry-runs and publishes serially: **common → registry → core → host → log-writer → index → engine → public SDK**. It waits for each uploaded version to appear in both the registry API and index before continuing. diff --git a/SOW-status.md b/SOW-status.md index 01236e86..befb0597 100644 --- a/SOW-status.md +++ b/SOW-status.md @@ -1,16 +1,32 @@ # SOW Status -Last updated: 2026-10-11 +Last updated: 2026-10-26 This root file is a short convenience index. The canonical detailed SOW ledger is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. ## Current -- None. +- `SOW-0045-20260528-go-reader-alignment-optimization.md` - reopened and paused + for a separately reproduced healthy empty/EOF refresh positioning regression. + Unique-triggered refresh shares the failure; related pre-existing directory + seek/getter state and unfiltered time-probe error suppression are recorded. + SDK-then-journalctl execution remains unchanged; + no positioning implementation is mixed in. + +- `SOW-0071-20260530-runtime-purity-and-optional-platform-services.md` - + regression reopened and paused for static guard coverage. No SDK process + launch is demonstrated; scanner changes remain unapproved and do not alter + the active repair, SDK and journalctl sequence. ## Pending +- SOW-0161 - File Indexer Source Posting Membership: open source-analysis + lead, not activated. Investigate inherited source-map target membership and + captured-tail handling before selecting any behavior/performance change. +- SOW-0160 - Go XZ Reader Dictionary Allocation: open, not activated. Tracks + unchanged decoder allocation measured during SOW-0108; no redesign is + authorized and no dependency is added to the current execution chain. - SOW-0156 - Fixed-Header Consistency Audit: separate follow-up agreed in PR #9. Define invariants, audit both SDKs, then implement/test missing rules. Not activated by the current array-count repair. @@ -32,9 +48,10 @@ is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. votes from glm, minimax, kimi, mimo, deepseek, and qwen; Rust/Go implementation remains pending. - `SOW-0122-20260621-journalctl-performance-followups.md` - open. Tracks - non-blocking SOW-0121 performance follow-up for indexed + non-blocking SOW-0121 broader performance follow-up for `--list-invocations`, portable `--follow` scalability, and large-fixture - performance evidence. + performance evidence. It does not replace the mandatory no-regression gates + in the requested SDK and journalctl workstreams. - `SOW-0066-20260530-v1-release-and-registry-publication.md` - open. Publish the final stable Rust/Go `v1.0.0` SDK with Rust registry publication where applicable, Go module tags, and clean consumer install validation after compatibility, portability, real-corpus validation, parity closure, and release checks pass. - `SOW-0048-20260528-netdata-otel-writer-sdk-integration.md` - open. Integrate the SDK compact-default structured writer into Netdata OTEL logs ingestion after the inventory and writer gates are accepted. - `SOW-0049-20260528-netdata-reader-plugin-sdk-integration.md` - open. Integrate SDK reader/facade paths into Netdata OTEL signal viewer, no-libsystemd systemd-journal plugin mode, and static packaging after reader gates are accepted. @@ -44,6 +61,39 @@ is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. ## Done +- SOW-0121 - File-Backed Journalctl Full Parity And Ship Decision: completed. + Directional cursor integration and PR12 fixes pass three internal reviews, + 3,514 native comparisons and146 portable controls. Pushed-head native CI is + pending; historical qualification follows. + The vacuum regression is repaired under approved deterministic history-head + ordering. Both languages pass independent 10,000-set models; native CI passes + Linux/macOS/Windows/FreeBSD with 130 portable controls. Existing 78 utility + oracle observations pass. Final internal review passes the repair; lifecycle + impact review and normal final-commit CI complete publication. Upstream + Codacy configuration retry and coverage upload fail remotely; native tests, + coverage generation and CodeQL pass. These CI limitations stay explicit. + Broader scalability remains SOW-0122; PR #12 is draft, not merged/released. + +- `SOW-0159-20261014-rust-go-sdk-hardening-and-reader-compatibility.md` - completed; directional cursor APIs validated and reviewed in both SDKs; CLI integration is completed in SOW-0121. + Both SDKs provide default intact-prefix recovery/status, complete streams, + native filtering and approved position/time/error retry behavior. Go floor + amd64/386/race checks pass 1,995/1,996/1,995 events, excluding only the unchanged + live-stock stress test with a retained isolated pass and earlier deadline + failure. Rust floor workspace/allocative checks pass, including 200 SDK tests. + Final tools pass 552 shared observations and 63/34/42 v262 verifier/directory/ + mixed checks; 35 documentation examples and 16 pages pass. Performance is + accepted under representative evidence and careful review, with uncertain + intervals and adverse history retained. The fresh whole-SOW review has one + positive verdict, five coverage gaps and no verified unresolved blocker. + The complete-and-clean closing audit passes; signed commit 25accee keeps + implementation and completed lifecycle together. Journalctl SOW-0121 later reopened for CLI review findings; its vacuum repair remains accepted. +- `SOW-0108-20260614-cross-language-reader-window-accessor.md` - completed + again: Go indexed value lifetimes, bounded hybrid headers, native opener and + initial statistics are repaired. Full local checks and adjudicated review + pass; Qwen/Kimi coverage gaps and Mimo's disproven claim are retained. The + 118 timing cases have 79 improvements, 39 inconclusive results and no flag; + uncertainty remains explicit. All 1,176 measured allocation pairs avoid + increases. Signed commit `4c49205` keeps repair/lifecycle together. - `SOW-0158-20261011-v0-10-0-publication.md` - completed. Release CI published and verified eight Rust `0.10.0` crates; paired annotated tags target `288efe015c45`, and fresh exact Rust/Go consumers pass. @@ -103,11 +153,6 @@ is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. `4619776b74311338b1689b2ab1fb0936b7746e49`; both tags resolve to that commit, all eight Rust crates are published and non-yanked, and clean Go `1.26.2` and Rust `1.91.0` registry-consumer checks pass. -- `SOW-0121-20260621-file-backed-journalctl-full-parity.md` - completed again - after replacing the mixed-directory count-only boot assertion with complete - normalized stock-oracle row comparison and an oracle-completeness check. - Full-feature mixed validation passed 42/42; tagged-v260.1 directory and query - validation passed. - `SOW-0115-20260616-portable-writer-identity-helpers.md` - completed after a 2026-07-27 regression repair supplied deterministic synthetic identity to two missed internal Rust executable paths; all affected directory, @@ -183,7 +228,6 @@ is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. - `SOW-0109-20260614-python-node-netdata-edge-parity.md` - completed. Python and Node now match the Rust reference for the remaining scoped Netdata edge parity gaps discovered after SOW-0107: Python `data_only && delta` keeps analysis enabled, Python/Node expose and wire candidate-row callbacks, and Node no longer drops Explorer rows only because `commitRealtime` is zero. Focused tests, full Python/Node suites, Netdata function comparators, `git diff --check`, SOW audit, and all six reviewer votes passed. - `SOW-0105-20260611-node-explorer-netdata-parity.md` - completed after reconciliation. The Node Explorer API, Netdata function API, stdin wrapper, source selector labels, facade unique-values visitor, and TypeScript definitions landed in SOW-0105; the remaining review-discovered parity gaps were completed by SOW-0107 and SOW-0109 before closure. - `SOW-0107-20260613-python-node-explorer-sampling-engine.md` - completed. Python and Node now match the Rust Explorer/Netdata behavior for the scoped parity gaps: row-level sampling decision/estimation, Python Netdata FTS, Python `PRIORITY` facet sorting, Python/Node Index Compare validation, and O(1) indexed row collection. Local validation, four-peer high-row comparator evidence, `git diff --check`, and six production-grade reviewer votes passed; remaining reviewer-discovered edge parity items are tracked by SOW-0109. -- `SOW-0108-20260614-cross-language-reader-window-accessor.md` - completed after regression repair. Rust and Go public file-path verification APIs now use bounded reader-backed byte sources instead of materializing whole journal files; Go verifier source slices return owned buffers to avoid mmap/window aliasing hazards. Focused Rust/Go tiny-window sealed verifier tests, `go test ./...`, affected Rust package tests, the shared verify matrix, wiki docs validation, `git diff --check`, SOW audit, and all six reviewer votes passed. - `SOW-0104-20260611-python-explorer-netdata-parity.md` - completed. The Python SDK carries the full Rust feature surface (Explorer, Netdata logs function API with source selector labels and the complete tail/delta/sampling contract, stdin wrapper, `python/pyproject.toml` metadata, facade unique-values visitor). Parity proven by three-peer content comparison against the Rust wrapper and the installed Netdata plugin: 10/10 one-shot fixtures on the live journal and 5/5 stateful sequences on a frozen fixture; 5/5 `PRODUCTION GRADE: YES` reviewer verdicts in one round. - `SOW-0103-20260611-docs-api-perception-and-verified-examples.md` - completed. Consumer wiki restructured around API-perception decision paths with a new `Journalctl-CLI` page; all 31 Rust/Go wiki examples are machine-verified against synthetic fixtures by `tests/docs/verify_examples.py`, enforced by the extended wiki validator and the new `docs-examples.yml` workflow; new `project-docs-authoring` skill; reviewer round 3 returned 5/5 `PRODUCTION GRADE: YES`. - `SOW-0065-20260530-parallel-language-parity-closure.md` - closed without implementation on 2026-06-11, superseded by the docs-and-parity program SOW-0103 through SOW-0106 after all prerequisites completed and Go parity was already delivered by SOW-0095/SOW-0102; the user chose sequential SOWs with external implementer `llm-netdata-cloud/minimax-m3-coder` and the other `llm-netdata-cloud` pool models as read-only reviewers. @@ -200,7 +244,6 @@ is `.agents/sow/SOW-status.md`; if summaries differ, the canonical ledger wins. - `SOW-0055-20260529-rust-seek-cursor-systemd-parity.md` - `SOW-0026-20260526-netdata-sdk-integration.md` - `SOW-0063-20260530-cross-platform-portability.md` -- `SOW-0071-20260530-runtime-purity-and-optional-platform-services.md` - `SOW-0067-20260530-go-cross-platform-portability.md` - `SOW-0068-20260530-rust-cross-platform-portability.md` - `SOW-0069-20260530-python-cross-platform-portability.md` diff --git a/docs/Go-API.md b/docs/Go-API.md index bee6ea95..b47b4e34 100644 --- a/docs/Go-API.md +++ b/docs/Go-API.md @@ -38,6 +38,8 @@ The Go SDK is pure Go and does not use CGO. Use `OpenFile` for one journal file. +Recovery diagnostics in this example require an unreleased source build. + ```go r, err := journal.OpenFile("/var/log/journal/example/system.journal") @@ -51,9 +53,12 @@ r.SeekHead() for { ok, err := r.Step() - if err != nil || !ok { + if err != nil { return err } + if !ok { + break + } entry, err := r.GetEntry() if err != nil { return err @@ -62,11 +67,45 @@ for { fmt.Println(string(message)) } } + +if status := r.RecoveryStatus(); status.Incomplete() { + fmt.Printf("Observed tail loss at offset %d: recovered %d/%d entries\n", + status.FirstLostOffset, status.RecoveredEntries, status.DeclaredEntries) +} ``` `GetEntry()` materializes maps and owned payloads. It is convenient, but it is not the lowest-cost scan path. +Ordinary readers in that source build return an intact prefix and normal EOF +after observed tail loss. +`RecoveryStatus` describes that observation; it is not an integrity certificate. +Directory readers expose `RecoveryStatuses`, and facade callers use +`SdJournalRecoveryStatuses`. See [[Reader-APIs|Reader APIs]] for error and +strict-validation boundaries. + +In unreleased source builds, `Step` and facade iteration apply exact matches +through DATA indexes and posting arrays. Repeated values for one field are OR +alternatives; different fields are combined with AND. `Next` and `Previous` are +raw movement helpers; use `Step` and `StepBack` for filtered SDK iteration. +Filtered reverse movement returns the nearest preceding match, including after +forward EOF. A selected ENTRY load error preserves the last successful position; +it does not promise to follow external reconstruction of damaged entries. See +[[Reader-APIs|Reader APIs]] for explicit realtime-seek validation boundaries. + +`AddDisjunction` and `AddConjunction` now return errors on file and directory +readers and facade handle methods. Check these results as well as iteration +errors. Interfaces declaring these methods must include the error result; +standalone `SdJournalAddDisjunction`/`SdJournalAddConjunction` functions already +returned errors. A pending builder with no valid +`FIELD=value` match returns `ErrMalformedFilter` when built. See +[[Reader-APIs|Reader APIs]] for posting damage and live-match refresh behavior. + +`FlushMatches` clears the filter and rewinds to the head, matching Rust. To +continue from an existing location, save and explicitly restore that location +after clearing. Failed group/build operations retain the pending builder without +installing a partial filter; flush to discard the pending matches. + ## Scan Payloads With Minimal Work Use `VisitEntryPayloads` when the consumer can work with `FIELD=value` bytes. @@ -204,6 +243,9 @@ if err != nil { ``` Use `QueryUnique` only when the caller needs an owned slice of all values. +Query setup and each stateful enumeration step can fail on damaged chains or +compressed DATA. Handle both errors and discard incomplete aggregates; see +[[Reader-APIs|Reader APIs]] for the native-index and compressed-input contracts. Directory-level unique enumeration builds an exact 8-entry per-open-reader LRU cache from per-file FIELD/DATA chains and reuses it for repeated queries or facade stateful restarts while the already-open file set's header signatures diff --git a/docs/Indexed-Snapshots.md b/docs/Indexed-Snapshots.md index bf9fc0c7..892aed24 100644 --- a/docs/Indexed-Snapshots.md +++ b/docs/Indexed-Snapshots.md @@ -56,8 +56,10 @@ metadata during the callback. Independent queries use separate snapshots. Context/control cancellation is checked between objects, chunks and payloads. One syscall or decompression is cooperative, not interruptible. Regular and compact layouts, historical unkeyed files and supported DATA compression work. -Whole-file `.journal.zst` inputs use existing staging and incur full-file -decompression at open; ordinary `.journal` files use bounded reader windows. +Whole-file `.journal.zst` inputs use staging and incur complete-stream +decompression at open, including concatenated and unknown-size frames; ordinary +`.journal` files use bounded reader windows. See [[Reader-APIs|Reader APIs]] for +compressed-input limits and errors. ## Go @@ -129,6 +131,12 @@ Resources are released on drop. ## Integrity Before Recovery +Unreleased ordinary readers recover an observed intact prefix by default and +report tail loss separately from normal EOF. That policy also applies to +ordinary readers with snapshot bounds. `IndexedSnapshot` capture retains strict +validation and does not adopt the ordinary-reader recovery policy. See +[[Reader-APIs|Reader APIs]] for status inspection and its limits. + Snapshot capture checks local bounds and requested metadata; it does not certify every index edge. Its precondition is a valid native index graph. An interrupted append can leave postings ahead of the committed global count. diff --git a/docs/Journalctl-CLI.md b/docs/Journalctl-CLI.md index c6acd43e..e5f92a57 100644 --- a/docs/Journalctl-CLI.md +++ b/docs/Journalctl-CLI.md @@ -17,7 +17,7 @@ The rewrites cover file-backed query behavior only: `--directory `; repeated `--file` inputs and glob expansion match stock file-backed behavior, while `--file=-` is explicitly unsupported because portable stdin journals would need seekable mmap-capable descriptors; -- output: all stock v260.1 file-backed modes: `short`, `short-full`, +- output: all stock v262 file-backed modes: `short`, `short-full`, `short-iso`, `short-iso-precise`, `short-precise`, `short-monotonic`, `short-delta`, `short-unix`, `verbose`, `export`, `json`, `json-pretty`, `json-sse`, `json-seq`, `cat`, and `with-unit`; @@ -28,7 +28,7 @@ The rewrites cover file-backed query behavior only: - field projection: `--output-fields` for `verbose`, `export`, `json*`, and `cat`, with stock metadata retention for JSON/export modes; - output controls: full-width text output is the default; `--no-full`, `--all`, - and empty-result messages follow stock v260.1 behavior, including JSON large + and empty-result messages follow stock v262 behavior, including JSON large field `null` thresholding without `--all`; - metadata: `--list-boots`, `--list-invocations`, `--fields`, `--field ` (unique values), and `--header`; @@ -41,7 +41,7 @@ The rewrites cover file-backed query behavior only: `--user-unit`, `--invocation`, and `-I`; - unit filters: exact and glob system/user units use the same journal fields and disjunction groups as stock file-backed `journalctl`; `--user --unit=` - is treated as `--user-unit=` like stock systemd; + is treated as `--user-unit=` unless `--system` is also present; - time and boot windows: `--since`, `--until`, `--boot [ID]`; bare `--boot` selects the current boot, while explicit empty `--boot=` is rejected like stock `journalctl`; @@ -65,6 +65,135 @@ Host and daemon operations are out of scope by design. `--sync`, `--flush`, `--rotate`, `--relinquish-var`, and `--setup-keys` return a controlled error in every language. +## v262 Compatibility And Migration + +The command target is systemd v262. The SDK journal-format baseline remains +v260.1; this command update does not change journal bytes or compiler minimums. + +- `-F FIELD` and `-N` reject time bounds, entry matches, cursor restrictions, + grep and explicit boot/invocation restrictions. Earlier versions ignored + some of these combinations. Use unrestricted enumeration for the complete + file, or the SDK to enumerate values from filtered entries. +- Required option values remain literal, even when they look like flags. + Setters validate in encounter order; terminal help/version/introspection + stops parsing immediately. The last action controls dispatch. + Integer options follow native base-prefix, octal and range rules; priority + names use the canonical lowercase spellings. +- Grep uses PCRE2 byte semantics, including lookarounds and backreferences. + It matches the first raw MESSAGE value. No matches means exit 1 with empty + stderr; an invalid pattern or matching error includes a diagnostic. +- Unit wildcards use indexed field values. When all system/user unit selectors + are wildcards and none expands, the command exits 1 with a no-data error. + An exact unit name remains a valid selector even when it has no entries. + Native unit naming supports suffixless globs, escaped names, and logical + device/mount paths without inspecting the host filesystem. Empty names are + invalid. POSIX named character classes also work on FreeBSD. Follow keeps + the initial wildcard expansion. +- Forward cursor limits start at the cursor. Cursor-selected follow history, + explicit all/no-tail and since-only follow include the complete selected + backlog. `-f` means follow; use `-i` or `--file` to select a file. + Follow defaults to ten existing rows without since, including with `-e`. + With both since and `-e`, its default is one thousand existing rows. +- Cursor output and cursor files retain the final visited journal position, + including grep misses and a crossed time bound. A zero line limit still + positions the cursor, without evaluating MESSAGE patterns. +- Repeated output projections form a union. Field-name spaces are literal. + Cat writes first raw values, including embedded NULs and newlines. + Boot/invocation tables support JSON, pretty JSON, SSE and JSON sequences. + A zero table limit reports no data with exit 1. Quiet suppresses informational + diagnostics, including successful verification messages. +- On Unix, exact matches, identifiers and projection options preserve raw + argument bytes. A projection that names no field still restricts the output. +- Unix file and directory names preserve their original bytes. File patterns + use the platform's glob rules and environment locale, including brace + alternatives, brackets, escaping and explicit leading dots for hidden names. + Windows supports Unicode filenames, brace alternatives and Unicode wildcards. +- Colors follow `SYSTEMD_COLORS`, `NO_COLOR`, terminal detection, `TERM` and + `COLORTERM`. Priority text, grep matches, verbose fields, JSON values, boot + separators and table headers follow native styles. Fixed color modes and + explicit true override `NO_COLOR`; automatic modes respect it. Plain output + avoids color formatting and additional regex matching. + Embedded documentation and configuration-file links follow `SYSTEMD_URLIFY`. + Windows file links encode drive and UNC paths as file URIs, including spaces + and Unicode; unsupported device paths remain plain text. + Portable commands do not consult host catalog databases for implicit links. +- Short output uses the last repeated MESSAGE and header values. `--no-full` shortens + each line independently, retains the native prefix/suffix split and respects + line/byte thresholds and terminal character widths. Valid `COLUMNS` values + override stdout's terminal width, with 80 columns as the final fallback. +- Display timestamps use the last source-realtime value when native integer + parsing yields a value greater than zero and less than 2^55. Invalid values + fall back to the ENTRY timestamp; exported ENTRY metadata stays unchanged. +- Short output removes completed color/OSC sequences, expands tabs to eight + spaces and removes trailing or pre-newline carriage returns. `--all` keeps + raw text. JSON preserves repeated binary values as separate byte arrays. +- Vacuum sizes accept ordered uppercase compound units, such as `1G512M`; + lowercase and `KiB` suffixes are rejected. + Retention considers the oldest available filename or filesystem timestamp, + including access and creation times. Linux also recognizes the legacy + `user.crtime_usec` attribute and preserves atime on header reads when allowed. + Creation metadata adds filesystem work during age-based cleanup. The measured + warm control costs about 7 ms extra in Go and 3.8 ms in Rust for 1,000 recent + archive names. This accepted maintenance cost does not apply to ordinary + queries or SDK reads/writes; actual timings depend on the filesystem. +- Grep does not hide crossed boot separators. Initial follow-tail selection + counts journal rows before grep; subsequent polls resume after the last + visited row, including rows that grep did not select. +- Timestamp bounds parse once per occurrence. Calendar fractions round to + microseconds; duration/epoch fractions follow native checked arithmetic. + UTC, numeric offsets, IANA zones, DST folds/gaps and TZif leap records are + covered by the stock-oracle controls. Available named zones follow the + installed database, with a bundled fallback when it is absent. Calendar + output also honors POSIX `TZ` rules, custom `TZDIR` data and TZif leap records. + Rules that omit daylight-saving dates use the configured `posixrules` file + before the native fallback dates. + As in stock journalctl, `--utc` changes entry timestamps; header and + boot/invocation tables retain local time. + +## Archive Retention Ordering + +Vacuum preserves sequence-number order within each archived writer history. +Across histories it selects the oldest adjusted timestamp among the next +available files. Equal sequence numbers use filename order; equal head times +use sequenced files first, then sequence ID and filename. Archives without +sequence metadata are independent candidates. This makes Rust and Go agree +regardless of directory enumeration order. + +This is an intentional exception to stock behavior when clock corrections make +its sequence/time comparisons contradictory. Stock's survivor selection then +depends on its sorting implementation and enumeration order. These commands +use the deterministic rule above instead of reproducing that incidental order. +Age retention still stops when the next eligible file meets the age limit and +size/count limits are satisfied; it does not skip a file to remove a later +sequence from the same history. Empty-archive cleanup and protected active +files retain their existing behavior. + +## Build Prerequisites + +Both commands require PCRE2-8 development files (10.42 or later), `pkg-config` +and a C toolchain. The Go command additionally requires `CGO_ENABLED=1`. +SDK-only applications require neither PCRE2 nor CGO. + +| Platform | Native command build inputs | +|---|---| +| Linux | C compiler, `pkg-config`, `libpcre2-dev` or equivalent | +| macOS | Apple command-line tools, Homebrew `pcre2` and `pkgconf` | +| Windows | MSYS2 CLANG64 compiler, pkgconf and PCRE2; Rust GNU LLVM target | +| FreeBSD | Native compiler, `pcre2` and `pkgconf` packages | + +The maintained native build and smoke commands are in the `Portable journalctl` +workflow. Windows uses the CLANG64 PCRE2 import library and needs the matching +DLL on PATH. Other dynamically linked builds also need the matching runtime +library. Native CI checks complement Linux's complete v262 oracle matrix; +SDK cross-compilation alone does not validate these command dependencies. + +Native dependencies add a small fixed command startup cost. On the measured +tiny Linux command, Go changes from about 3.0ms to 3.8ms and Rust from 1.5ms +to 1.8ms. This CLI-only allowance does not relax SDK or sustained query +performance requirements. +These measurements describe that fixture and host, not a cross-platform timing +guarantee. + ## Portable Mode Limits The rewrites are file-backed tools. They never connect to journald, read the @@ -73,10 +202,11 @@ explicit files and directories by portable polling. Correctness is covered by the shared stock-oracle follow matrix, but very large inputs may cost more than daemon-backed Linux `journalctl --follow`. -`--list-invocations` is stock-compatible for covered file-backed unit contexts, -but it currently derives invocation ranges from matching entries. Treat it as a -correctness-compatible path, not a performance-certified large-archive index -path. Large-archive performance work remains tracked internally. +`--list-invocations` follows indexed unit and invocation matches. Its time range +includes every row carrying the selected ID within the boot context, including +rows without unit metadata. Invocation show queries retain identifier, +priority, facility and transport filters; follow keeps its initially selected +invocation. Broader large-archive performance work remains tracked internally. ## Running Each Rewrite @@ -87,10 +217,10 @@ cargo run --manifest-path rust/Cargo.toml --bin journalctl -- \ --file ./fixtures/system.journal --output json PRIORITY=3 # Go -go run ./go/cmd/journalctl --directory ./journal-dir --list-boots -go run ./go/cmd/journalctl --file ./fixtures/system.journal --disk-usage -go run ./go/cmd/journalctl --directory ./journal-dir --vacuum-time=30d -go run ./go/cmd/journalctl --new-id128 +CGO_ENABLED=1 go -C go run ./cmd/journalctl --directory ../journal-dir --list-boots +CGO_ENABLED=1 go -C go run ./cmd/journalctl --file ../fixtures/system.journal --disk-usage +CGO_ENABLED=1 go -C go run ./cmd/journalctl --directory ../journal-dir --vacuum-time=30d +CGO_ENABLED=1 go -C go run ./cmd/journalctl --new-id128 ``` @@ -109,6 +239,6 @@ bug and report it. | log explorer queries (facets, histogram, FTS) | Explorer API | | Netdata-shaped function output | Netdata function boundary | -The CLI is a consumer of the same SDK reader paths; it adds no private -capabilities. Anything the CLI can do, the library APIs can do with more -control. +The CLI consumes SDK reader paths and adds command parsing, rendering and +PCRE2 filtering. Applications use SDK APIs directly and choose their own +filtering and presentation dependencies. diff --git a/docs/Reader-APIs.md b/docs/Reader-APIs.md index bd746649..a42bf631 100644 --- a/docs/Reader-APIs.md +++ b/docs/Reader-APIs.md @@ -20,6 +20,20 @@ consumer problem and carries a different cost. | stock-like CLI behavior | journalctl rewrite | journalctl rewrite | operator/script use; see [[Journalctl-CLI]] | | integrity validation | `verify_file` | `VerifyFile` | diagnostics and corpus gates | +## Native File Paths + +Rust callers with native filesystem paths can use `SdJournalOpenPath`, +`SdJournalOpenFilePath`, `SdJournalOpenDirectoryPath` and `SdJournalOpenFilePaths`. +The file, directory and list forms also have `WithOptions` variants. These +constructors preserve non-UTF-8 Unix filenames; the existing string constructors +remain available. Go path strings already preserve Unix filename bytes. + +Low-level Rust registry consumers should use `File::as_path()` for filesystem +operations. `File::path()` remains a text accessor for ordinary UTF-8 paths; +newly supported non-UTF-8 paths return an encoded metadata key. That key is +serializable and distinct from replacement-character filenames, and cannot be +opened as a filesystem path. Windows readers use Unicode filenames. + ## Data Ownership The reader has three ownership modes: @@ -69,6 +83,88 @@ Performance rules: separate from query opening. Strict index certification has graph-sized time and memory costs; sealed TAG/HMAC authentication is a separate check. +## Observed Tail Recovery + +Unreleased source builds recover the intact entry prefix by default when an +interrupted append leaves a final ENTRY or ENTRY array unavailable. This applies +to ordinary file readers, directory readers and facades, including ordinary +readers with snapshot bounds. Iteration ends normally after that prefix: Go +`Step` returns `false, nil`, and Rust `next` returns `Ok(false)`. + +Inspect diagnostics after reading: + +| Surface | Rust | Go | +|---|---|---| +| One file | `recovery_status()` | `RecoveryStatus()` | +| File or directory collection | `recovery_statuses()` | `RecoveryStatuses()` | +| Facade | `SdJournal::recovery_statuses()` | `SdJournalRecoveryStatuses()` | + +The per-file status reports the reason, offset associated with the observed +loss, declared entry count and recovered entry count. Collection records include +the file ID. A status value is a copy; collections allocate only when explicitly +requested. Header counters retain their declared values. + +Detection is lazy where the native access path permits it. A zero/`None` status +means no loss has been observed; it does not certify integrity. A catalog query +does not validate unrelated entry arrays. + +Recovery is read-only: the SDK does not reconstruct or overwrite damaged source +journals. Journald abandons a corrupt file and writes to a new one. Normal live +publication can make temporarily unavailable data readable; this is distinct +from repairing corruption. Automatic refresh after arbitrary external rewriting +of damaged entries is not a supported recovery contract. + +After a recovered reader reaches EOF, repeated polling in the same direction +does not rediscover the prefix while the declared entry count is unchanged. +Repositioning, changing direction, or a changed count permits another check. +Go's explicit `Refresh` remains separate from automatic EOF polling. + +Tail recovery does not reclassify interior holes followed by readable entries, +wrong object types, cycles, malformed indexes, decompression failures or general +I/O failures as an interrupted append. Existing catalog compatibility fallbacks +remain available. A damaged secondary posting tail remains an error when its +unknown targets could include surviving entries. Discard incomplete aggregates +after such a failure. + +Exact matches use DATA hash lookup and posting arrays in both languages. They +do not expand every row to find matching values. The same-field alternatives +are OR groups; different field groups are combined with AND. Check errors from +group operators and iteration: a missing required posting tail can fail even +when an unfiltered read can return an intact prefix. + +Filtered movement selects the nearest match before or after the last successful +row, including after installing or replacing a filter on a directory reader. +Proven sequential directory traversal remains sequential after a filter change; +errors in a later file are discovered when traversal reaches that file. +Reversing after EOF also selects the nearest match. If loading a selected ENTRY fails, +the failed movement keeps the previous successful position. This preserves +reader state on failure; it does not promise to follow external file repair. + +Explicit filtered realtime seeks search native posting arrays: forward selects +the first matching entry at or after the timestamp, and backward selects the +last at or before it. These searches validate the slots and ENTRY headers they +visit. They do not inspect every skipped posting; use strict verification when +the whole index must be checked. Ordinary sequential filtering continues to +report malformed postings as it encounters them. + +Group/build failures retain the pending matches rather than installing a partial +filter; flush matches to discard the pending builder. +Flushing also rewinds the reader to the head in both languages, including +directory and facade readers. Explicitly reposition afterward when continuing +from a saved location. + +A built healthy filter retains the posting population it resolved. To include +newly published matches, flush and rebuild the filter. Read-only recovery can +restrict existing DATA posting lists to the observed intact prefix; a value +absent when the filter was built remains absent until rebuilding. Ordinary snapshot bounds cap +the byte extent; they do not create an immutable logical view. A new filter can +observe publications inside the preallocated original extent. Use an +`IndexedSnapshot` captured with writers excluded for a stable logical artifact. + +Strict verification, excluded-writer `IndexedSnapshot` capture and append-open +retain their rejection rules. Use [[Indexed-Snapshots|Indexed Snapshots]] when +deciding whether an uncertain file is suitable for writable reuse. + ## Directory Reader Use the directory reader when the caller needs stock-like ordering across active @@ -129,6 +225,12 @@ Field-name enumeration and unique-value queries must use journal indexes. Use list-return APIs only when the caller needs the full owned result set. Use visitor APIs when streaming is enough. +Unreleased source builds return an error for detected cycles in traversed hash, +FIELD DATA or ENTRY-array chains. This check uses the native indexes; opening a +file does not certify the +whole graph. Handle errors from query setup and from individual enumeration +steps, and discard incomplete aggregates after a failure. + ## Facade API The facade is for ports from libsystemd or Netdata `jf`-style code. @@ -147,6 +249,27 @@ Cursor metadata is emitted in the official systemd cursor shape. Cursor seek and test also accept the older SDK cursor shape for compatibility with existing callers. +For directional cursor positioning, use `SeekCursorDirection` in Go or +`seek_cursor_direction` in Rust on file or directory readers. The facade +extension is `SdJournalSeekCursorDirection`. These methods immediately select +an inclusive boundary: forward selects the first eligible row at or after the +cursor, backward the last eligible row at or before it. Read the selected row +before advancing; calling next/previous first would skip it. Existing cursor +seek APIs retain their previous behavior. + +Selection follows the native indexes and installed filters. A matching sequence +ID takes precedence over the cursor's time; otherwise boot/monotonic and realtime +components provide fallback locations. A cursor can therefore select a nearby +row rather than an exact match; use `TestCursor`/`test_cursor` when exactness +matters. No eligible row is a successful seek with no current entry; use the +normal getter/EOF contract. Supplied IDs must normalize to 32 hexadecimal digits. +The new operation adds no persistent reader state and does not change Explorer +or Netdata pagination. + +At EOF, Go's direct `Reader.TestCursor` and `DirectoryReader.TestCursor` +return `false, ErrEndOfEntries`. The libsystemd-compatible facade returns +`false, nil` for the same condition. + The facade should reuse the row-level borrowed-data contract. It should not copy every payload just because the call shape is libsystemd-like. @@ -174,3 +297,15 @@ Formatting APIs are output boundaries: Verification reads the object graph to prove integrity. It is not a query hot path. + +## Compressed Input + +With the unreleased reader hardening changes, both SDKs decode complete supported +Zstandard streams, including unknown-size, +concatenated and skippable frames. A bad checksum, truncated later frame or +trailing garbage returns an error. The DATA output limit is 768 MiB across all +frames belonging to that DATA object. + +Whole-file `.journal.zst` and `.journal~.zst` inputs are SDK extensions. They are +streamed into staging storage before reading, so opening incurs full-file +decompression. The per-DATA limit does not cap the size of a whole journal. diff --git a/docs/Rust-API.md b/docs/Rust-API.md index 8afa28dc..602b0c49 100644 --- a/docs/Rust-API.md +++ b/docs/Rust-API.md @@ -22,6 +22,18 @@ Upgrading from Rust `0.8.x`: `0.9.0` adds the `SdkError::Cancelled` and so code that matches every variant without a wildcard arm must handle the new variants. +Unreleased reader hardening adds `JournalError::CyclicObjectChain`. Exhaustive +matches must handle this variant. The lower-level `JournalObject::EXPECTED_TYPE` +hint has a default, so existing custom object implementations can omit it. + +The same unreleased update advances `journal-engine`'s persisted index cache +from generation 3 to 4. Old entries are invalidated, because earlier array +traversal could cache incomplete or duplicate rows from a damaged file. Valid +files rebuild on demand. Encountered required native-chain or posting failures +now fail indexing; existing value-tolerance policies remain. Allow for that +one-time rebuild cost for valid files after upgrade; cache-hit behavior and +consumer namespaces stay the same. + Use the lower-level packages only when the public package does not expose the type you need. For example, structured directory writes currently use `StructuredField` from `systemd-journal-sdk-log-writer`: @@ -48,6 +60,8 @@ journal_host = { package = "systemd-journal-sdk-host", version = "0.10.0" } Use `FileReader` when the caller owns ordering and reads one journal file. +Recovery diagnostics in this example require an unreleased source build. + ```rust use journal::FileReader; @@ -62,12 +76,31 @@ while reader.next()? { println!("{message}"); } } +let status = reader.recovery_status(); +if status.incomplete() { + println!("Observed tail loss at offset {}: recovered {}/{} entries", + status.first_lost_offset, status.recovered_entries, status.declared_entries); +} # Ok::<(), Box>(()) ``` `get_entry()` materializes maps and owned payloads. It is convenient, but it is not the lowest-cost scan path. +Ordinary readers in that source build return an intact prefix and normal EOF +after observed tail loss. +`RecoveryStatus` describes that observation; it is not an integrity certificate. +File readers, directory readers and `SdJournal` expose `recovery_statuses()` for +per-file diagnostics. See [[Reader-APIs|Reader APIs]] for error and +strict-validation boundaries. + +Filtered movement returns the nearest match in the requested direction, +including after reversing at EOF. A selected ENTRY load error preserves the +last successful position, including directory and facade iteration. This does +not promise to follow external reconstruction of damaged entries. +Explicit filtered realtime seeks validate the posting slots they +visit; see [[Reader-APIs|Reader APIs]] for the verification boundary. + ## Scan Payloads With Minimal Work Use `visit_entry_payloads()` when the consumer can work with `FIELD=value` @@ -184,7 +217,9 @@ For facade-style stateful enumeration, `FileReader` exposes state controls and `enumerate_unique_payload()` for multi-file de-duplicated payloads. These methods keep the same indexed FIELD/DATA-chain contract, so query setup and each enumeration step can fail independently if the journal -contains damaged or undecompressible DATA. Directory-level unique enumeration +contains damaged chains or undecompressible DATA. Discard incomplete aggregates +after a failure; see [[Reader-APIs|Reader APIs]] for native-index and compressed +input behavior. Directory-level unique enumeration builds an exact 8-entry per-open-reader LRU cache from per-file FIELD/DATA chains and reuses it for repeated queries or restarts while the already-open file set's header signatures stay unchanged. The entry count is bounded, but diff --git a/go/API.md b/go/API.md index fcbe99d6..d53067f9 100644 --- a/go/API.md +++ b/go/API.md @@ -86,6 +86,13 @@ current entry exactly matches a cursor. Cursor metadata is emitted in the official systemd cursor shape; seek and test also accept the older SDK cursor shape for compatibility. +`Reader.SeekCursorDirection`, `DirectoryReader.SeekCursorDirection` and the +`SdJournalSeekCursorDirection` facade extension select an inclusive forward or +backward cursor boundary immediately, using native indexes and installed +filters. Read the current entry before stepping. A missing boundary succeeds +with no current entry; use `TestCursor` to distinguish an exact match from a +nearby row. Existing `SeekCursor` APIs are unchanged. + Stateful data and unique enumeration return full `FIELD=value` payloads and are binary-safe. `SdJournalGetData` returns the first value for a repeated field; callers that need all repeated values must use the restart/enumerate data API. @@ -100,9 +107,27 @@ and Windows targets. `ReaderAccessReadAt` is retained only for tests, diagnostics, constrained-platform investigation, and controlled fallback evidence. It is not a production reader mode. If `ReaderAccessAuto` selects read-at in production, treat that as a deployment issue to investigate and -benchmark before accepting. `ReaderBoundsLive` refreshes visible entries when -active files grow; `ReaderBoundsSnapshot` fixes the visible file state at open -time. +benchmark before accepting. `ReaderBoundsLive` can observe active-file growth. +`ReaderBoundsSnapshot` caps visibility at the original byte extent; it does not +freeze a logical population. In unreleased source builds, a new exact-filter +builder can see publications inside that original, preallocated extent. +Installed healthy filters retain their captured DATA posting populations in +either bounds mode. Flush and rebuild to include new matches; flushing rewinds +to the head, so explicitly restore any saved location afterward. Group/build +errors preserve the pending builder for retry. File/directory/facade handle +`AddDisjunction` and `AddConjunction` methods now return errors; interfaces and +callers must handle that result. Use `OpenIndexedSnapshot` when a stable logical +artifact is required. The caller must exclude writers during capture; this API +does not acquire a writer lock. + +Filtered `Step`/`StepBack` select the nearest matching row in the requested +direction, including direction reversal after EOF. Selected ENTRY load errors +retain the last successful file/directory/facade selection on failure. +Recovery never rewrites the source journal. Following arbitrary external repair +of damaged entries is not a supported contract. +Explicit filtered realtime seeks binary-search native posting slots and report +errors from slots/headers they visit; skipped postings are not certified. +Ordinary sequential filtering still checks postings as it encounters them. `VisitEntryPayloads`, `EnumerateEntryPayload`, and `SdJournalEnumerateAvailableData` are zero-copy hot paths. Returned or callback diff --git a/go/README.md b/go/README.md index 1afc30f9..8912757b 100644 --- a/go/README.md +++ b/go/README.md @@ -91,7 +91,7 @@ Current reader scope: realtime/monotonic/seqnum/cursor metadata, and boot listing; - cursor metadata is emitted in the official systemd cursor shape; cursor seek and test also accept the older SDK cursor shape for compatibility; -- file-backed `journalctl` output covers the stock v260.1 short family, +- file-backed `journalctl` output covers the stock v262 short family, `verbose`, `with-unit`, `cat`, `export`, `json`, `json-pretty`, `json-sse`, and `json-seq` modes; `--output-fields` projects requested fields while JSON/export retain stock metadata fields; @@ -473,7 +473,18 @@ inputs. `--vacuum-size`, `--vacuum-files`, and `--vacuum-time` operate on explicit `--directory` inputs, deleting only stock-recognized archived `.journal` and `.journal~` files while protecting active/current and unknown files. -The CLI implements the stock v260.1 output-mode family, `--output-fields` +The CLI targets systemd v262. Building it requires PCRE2-8 development files +(version 10.42 or later), `pkg-config`, and a native C toolchain. Build/run the Go command with `CGO_ENABLED=1`; the SDK remains CGO-free. +The command links PCRE2 only; it never links system journal libraries. Native +library deployment must accompany dynamically linked command binaries. + +`-F`/`-N` with time, entry, cursor or grep restrictions now fail under v262; +remove the restriction for whole-file enumeration, or read filtered rows with +the SDK. Grep supports PCRE2 byte syntax and returns exit 1 without a diagnostic +when no entry matches. See [Journalctl CLI](../docs/Journalctl-CLI.md) for migration +and platform prerequisites. + +The CLI implements the stock v262 output-mode family, `--output-fields` projection for `verbose`, `export`, JSON modes, and `cat`, and stock output controls such as `--all`, `--full`, `--no-full`, empty-result messages, and `--pager-end` implicit 1000-line tail behavior. `--output=help` prints the diff --git a/go/adapter/complex_match.go b/go/adapter/complex_match.go index c01f49d5..95cb381b 100644 --- a/go/adapter/complex_match.go +++ b/go/adapter/complex_match.go @@ -18,7 +18,9 @@ func runComplexMatchTest(tc *TestCase) Result { return failResult(tc, tc.Expected.ResultFormat, err) } defer r.Close() - addSystemdComplexMatchExpression(r) + if err := addSystemdComplexMatchExpression(r); err != nil { + return failResult(tc, tc.Expected.ResultFormat, err) + } matched, err := collectMatchedEntries(r) if err != nil { @@ -135,9 +137,9 @@ func stringEntryFields(entry *journal.Entry) map[string]string { func addSystemdComplexMatchExpression(r interface { AddMatch([]byte) - AddDisjunction() - AddConjunction() -}) { + AddDisjunction() error + AddConjunction() error +}) error { r.AddMatch([]byte{'A', '=', 1, 2}) r.AddMatch([]byte{'B', '=', 'C', 0, 'D'}) r.AddMatch([]byte("HALLO=WALDO")) @@ -147,16 +149,23 @@ func addSystemdComplexMatchExpression(r interface { r.AddMatch([]byte("QUUX=xxxxx")) r.AddMatch([]byte("QUUX=yyyyy")) r.AddMatch([]byte("PIFF=paff")) - r.AddDisjunction() + if err := r.AddDisjunction(); err != nil { + return err + } r.AddMatch([]byte("ONE=one")) r.AddMatch([]byte("ONE=two")) r.AddMatch([]byte("TWO=two")) - r.AddConjunction() + if err := r.AddConjunction(); err != nil { + return err + } r.AddMatch([]byte("L4_1=yes")) r.AddMatch([]byte("L4_1=ok")) r.AddMatch([]byte("L4_2=yes")) r.AddMatch([]byte("L4_2=ok")) - r.AddDisjunction() + if err := r.AddDisjunction(); err != nil { + return err + } r.AddMatch([]byte("L3=yes")) r.AddMatch([]byte("L3=ok")) + return nil } diff --git a/go/adapter/main.go b/go/adapter/main.go index ea966c64..d5009bf7 100644 --- a/go/adapter/main.go +++ b/go/adapter/main.go @@ -34,6 +34,11 @@ func main() { listSupportedTests(os.Stdout) case "probe": probeAdapter(os.Stdout) + case "inspect-reader": + if err := runReaderProbe(os.Stdin, os.Stdout); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } default: fmt.Fprintf(os.Stderr, "Unknown subcommand: %s\n", os.Args[1]) os.Exit(1) diff --git a/go/adapter/reader_probe.go b/go/adapter/reader_probe.go new file mode 100644 index 00000000..d567f52e --- /dev/null +++ b/go/adapter/reader_probe.go @@ -0,0 +1,311 @@ +package main + +import ( + "encoding/hex" + "encoding/json" + "fmt" + "io" + "sort" + "strings" + + "github.com/netdata/systemd-journal-sdk/go/journal" +) + +type readerProbeRequest struct { + Path string `json:"path"` + Operation string `json:"operation"` + Field string `json:"field"` + MatchHex string `json:"match_hex"` + Realtime uint64 `json:"realtime"` + Snapshot bool `json:"snapshot"` + Directory bool `json:"directory"` +} + +type probeRecovery struct { + Reason journal.RecoveryReason `json:"reason"` + FirstLostOffset uint64 `json:"first_lost_offset"` + DeclaredEntries uint64 `json:"declared_entries"` + RecoveredEntries uint64 `json:"recovered_entries"` +} + +type probeFileRecovery struct { + FileID string `json:"file_id"` + Status probeRecovery `json:"status"` +} + +type readerProbeRow struct { + Realtime uint64 `json:"realtime"` + Payloads []string `json:"payloads"` +} + +type readerProbeResult struct { + Rows []readerProbeRow `json:"rows"` + Fields []string `json:"fields"` + Unique []string `json:"unique"` + RecoveryStatuses []probeFileRecovery `json:"recovery_statuses"` + Error bool `json:"error"` + ErrorMessage string `json:"error_message,omitempty"` +} + +type probeReader interface { + Close() error + Step() (bool, error) + StepBack() (bool, error) + SeekTail() error + SeekRealtimeUsec(uint64) error + GetRealtimeUsec() (uint64, error) + CollectEntryPayloads() ([][]byte, error) + EnumerateFields() (map[string]struct{}, error) + VisitUnique(string, func([]byte) error) error + QueryUniqueState(string) error + EnumerateUniquePayload() ([]byte, bool, error) + AddMatch([]byte) + FlushMatches() + RecoveryStatuses() []journal.FileRecoveryStatus +} + +// This probe exposes public SDK results for shared damaged-file subprocess tests. +func runReaderProbe(input io.Reader, output io.Writer) error { + var request readerProbeRequest + if err := json.NewDecoder(input).Decode(&request); err != nil { + return err + } + result := readerProbeResult{Rows: []readerProbeRow{}, Fields: []string{}, Unique: []string{}, RecoveryStatuses: []probeFileRecovery{}} + err := inspectReader(request, &result) + if err != nil { + result.Error = true + result.ErrorMessage = err.Error() + } + return json.NewEncoder(output).Encode(result) +} + +func inspectReader(request readerProbeRequest, result *readerProbeResult) error { + if request.Operation == "verify" { + return journal.VerifyFile(request.Path) + } + if strings.HasPrefix(request.Operation, "facade-") { + return inspectReaderFacade(request, result) + } + opts := journal.DefaultReaderOptions().WithSnapshot(request.Snapshot) + var reader probeReader + var err error + if request.Directory { + reader, err = journal.OpenDirectoryWithOptions(request.Path, opts) + } else { + reader, err = journal.OpenFileWithOptions(request.Path, opts) + } + if err != nil { + return err + } + defer reader.Close() + defer func() { + recordProbeRecovery(result, reader.RecoveryStatuses()) + }() + if request.MatchHex != "" { + value, err := hex.DecodeString(request.MatchHex) + if err != nil { + return err + } + reader.AddMatch(value) + } + switch request.Operation { + case "explorer", "explorer-filter", "explorer-facet", "explorer-histogram": + file, ok := reader.(*journal.Reader) + if !ok { + return fmt.Errorf("explorer probe needs one file") + } + query := journal.DefaultExplorerQuery().WithFilter([]byte(request.Field), []byte("absent")) + if request.Operation != "explorer" { + query = journal.DefaultExplorerQuery() + query.Limit = 0 + switch request.Operation { + case "explorer-filter": + query.Limit = 1 + query = query.WithFilter([]byte(request.Field), []byte("all")) + case "explorer-facet": + query = query.WithFacet([]byte(request.Field)) + case "explorer-histogram": + query = query.WithHistogram([]byte(request.Field)) + } + } + query.FieldMode = journal.ExplorerFieldModeAllValues + query.UseSourceRealtime = false + _, err := file.ExploreWithStrategy(query, journal.ExplorerStrategyIndex) + return err + case "fields": + fields, err := reader.EnumerateFields() + if err != nil { + return err + } + for field := range fields { + result.Fields = append(result.Fields, field) + } + sort.Strings(result.Fields) + return nil + case "unique": + return reader.VisitUnique(request.Field, func(value []byte) error { result.Unique = append(result.Unique, hex.EncodeToString(value)); return nil }) + case "unique-state": + if err := reader.QueryUniqueState(request.Field); err != nil { + return err + } + for { + value, ok, err := reader.EnumerateUniquePayload() + if err != nil { + return err + } + if !ok { + return nil + } + result.Unique = append(result.Unique, hex.EncodeToString(value)) + } + case "flush": + for i := 0; i < 2; i++ { + if ok, err := reader.Step(); err != nil || !ok { + if err != nil { + return err + } + return io.ErrUnexpectedEOF + } + } + reader.FlushMatches() + case "reverse": + if err := reader.SeekTail(); err != nil { + return err + } + case "seek-forward", "seek-reverse": + if err := reader.SeekRealtimeUsec(request.Realtime); err != nil { + return err + } + case "forward": + default: + return fmt.Errorf("unsupported probe operation %q", request.Operation) + } + for { + var ok bool + if request.Operation == "reverse" || request.Operation == "seek-reverse" { + ok, err = reader.StepBack() + } else { + ok, err = reader.Step() + } + if err != nil { + return err + } + if !ok { + return nil + } + time, err := reader.GetRealtimeUsec() + if err != nil { + return err + } + payloads, err := reader.CollectEntryPayloads() + if err != nil { + return err + } + row := readerProbeRow{Realtime: time, Payloads: []string{}} + for _, payload := range payloads { + row.Payloads = append(row.Payloads, hex.EncodeToString(payload)) + } + sort.Strings(row.Payloads) + result.Rows = append(result.Rows, row) + } +} + +func recordProbeRecovery(result *readerProbeResult, statuses []journal.FileRecoveryStatus) { + for _, status := range statuses { + s := status.Status + result.RecoveryStatuses = append(result.RecoveryStatuses, probeFileRecovery{status.FileID.String(), probeRecovery{s.Reason, s.FirstLostOffset, s.DeclaredEntries, s.RecoveredEntries}}) + } +} + +func inspectReaderFacade(request readerProbeRequest, result *readerProbeResult) error { + open := journal.SdJournalOpenFileWithOptions + if request.Directory { + open = journal.SdJournalOpenDirectoryWithOptions + } + reader, err := open(request.Path, 0, journal.DefaultReaderOptions().WithSnapshot(request.Snapshot)) + if err != nil { + return err + } + defer journal.SdJournalClose(reader) + defer func() { recordProbeRecovery(result, journal.SdJournalRecoveryStatuses(reader)) }() + if request.MatchHex != "" { + value, err := hex.DecodeString(request.MatchHex) + if err != nil { + return err + } + if err := journal.SdJournalAddMatch(reader, value); err != nil { + return err + } + } + operation := strings.TrimPrefix(request.Operation, "facade-") + switch operation { + case "fields": + fields, err := journal.SdJournalEnumerateFields(reader) + if err != nil { + return err + } + result.Fields = append(result.Fields, fields...) + sort.Strings(result.Fields) + return err + case "unique": + values, err := journal.SdJournalQueryUnique(reader, request.Field) + if err != nil { + return err + } + for _, value := range values { + result.Unique = append(result.Unique, hex.EncodeToString(value.Value)) + } + return nil + case "unique-state": + if err := journal.SdJournalQueryUniqueState(reader, request.Field); err != nil { + return err + } + for { + payload, ok, err := journal.SdJournalEnumerateAvailableUnique(reader) + if err != nil || !ok { + return err + } + result.Unique = append(result.Unique, hex.EncodeToString(payload)) + } + case "flush": + for i := 0; i < 2; i++ { + if count, err := journal.SdJournalNext(reader); err != nil || count != 1 { + if err != nil { + return err + } + return io.ErrUnexpectedEOF + } + } + if err := journal.SdJournalFlushMatches(reader); err != nil { + return err + } + case "forward", "reverse": + if operation == "reverse" { + if err := journal.SdJournalSeekTail(reader); err != nil { + return err + } + } + default: + return fmt.Errorf("unknown facade probe operation %q", operation) + } + for { + step := journal.SdJournalNext + if operation == "reverse" { + step = journal.SdJournalPrevious + } + count, err := step(reader) + if err != nil || count == 0 { + return err + } + entry, err := journal.SdJournalGetEntry(reader) + if err != nil { + return err + } + row := readerProbeRow{Realtime: entry.Realtime, Payloads: make([]string, 0, len(entry.Payloads))} + for _, payload := range entry.Payloads { + row.Payloads = append(row.Payloads, hex.EncodeToString(payload)) + } + sort.Strings(row.Payloads) + result.Rows = append(result.Rows, row) + } +} diff --git a/go/cmd/journalctl/colors.go b/go/cmd/journalctl/colors.go new file mode 100644 index 00000000..15bf296d --- /dev/null +++ b/go/cmd/journalctl/colors.go @@ -0,0 +1,162 @@ +//go:build cgo + +package main + +import ( + "os" + "strings" +) + +type matchRange struct{ start, end int } + +type outputColors struct { + level int + underline bool +} + +const ( + ansiNormal = "\x1b[0m" + ansiHighlight = "\x1b[0;1;39m" + ansiRed = "\x1b[0;1;31m" + ansiBlue = "\x1b[0;34m" + ansiGreen = "\x1b[0;32m" +) + +func detectOutputColors() outputColors { + mode := os.Getenv("SYSTEMD_COLORS") + level := 0 + switch mode { + case "off": + return outputColors{} + case "16": + level = 16 + case "256": + level = 256 + case "24bit": + level = 24 + default: + force, err := parseBoolOption("SYSTEMD_COLORS", mode) + if err == nil && !force { + return outputColors{} + } + if err != nil || !force { + _, noColor := os.LookupEnv("NO_COLOR") + term, termSet := os.LookupEnv("TERM") + if noColor || !termSet || term == "dumb" || os.Getpid() != 1 && !outputIsTerminal() { + return outputColors{} + } + } + switch mode { + case "auto-16": + level = 16 + case "auto-24bit": + level = 24 + case "auto-256": + level = 256 + default: + level = 256 + if value := os.Getenv("COLORTERM"); value == "truecolor" || value == "24bit" { + level = 24 + } + } + } + return outputColors{level: level, underline: level != 0 && os.Getenv("TERM") != "linux"} +} + +func (c outputColors) grey() string { + if c.level == 16 { + return "\x1b[0;90m" + } + if c.level == 24 { + return "\x1b[0;38:5:245m" + } + return "\x1b[0;90m\x1b[0;38:5:245m" +} + +func (c outputColors) log(priority int, audit bool) (on, off, highlight string) { + if c.level == 0 { + return "", "", "" + } + switch { + case priority <= 3: + on, highlight = ansiRed, ansiHighlight + case priority == 4: + on, highlight = "\x1b[0;1;33m", ansiHighlight + if c.level != 16 { + on = "\x1b[0;1;38:5:185m" + if c.level == 256 { + on = "\x1b[0;1;33m" + on + } + } + case priority == 5: + on, highlight = ansiHighlight, ansiRed + case priority >= 7: + on, highlight = c.grey(), ansiRed + } + if audit && on == "" { + on = ansiBlue + } + if on != "" { + off = ansiNormal + } + return +} + +func (c outputColors) boot(id string) string { + text := "-- Boot " + id + " --\n" + if c.level == 0 { + return text + } + return ansiHighlight + strings.TrimSuffix(text, "\n") + ansiNormal + "\n" +} + +func (c outputColors) tableHeader(id string) string { + if c.level == 0 || !c.underline { + return "IDX " + id + strings.Repeat(" ", 33-len(id)) + "FIRST ENTRY LAST ENTRY\n" + } + on := "" + if c.underline { + on = "\x1b[4m" + } + fields := []string{"IDX", id + strings.Repeat(" ", 32-len(id)), "FIRST ENTRY ", "LAST ENTRY "} + return on + strings.Join(fields, ansiNormal+on+" "+ansiNormal+on) + ansiNormal + "\n" +} + +// Color the existing JSON encoding only when requested; framing stays with its writer. +func colorJSON(data []byte) []byte { + var out strings.Builder + for i := 0; i < len(data); { + start := i + switch b := data[i]; { + case b == '"': + i++ + for i < len(data) && data[i] != '"' { + if data[i] == '\\' { + i++ + } + i++ + } + out.WriteByte('"') + out.WriteString(ansiGreen) + out.Write(data[start+1 : i]) + out.WriteString(ansiNormal) + out.WriteByte('"') + i++ + case b == '-' || b >= '0' && b <= '9' || b == 'n' || b == 't' || b == 'f': + on := "\x1b[0;1;34m" + if b == 'n' || b == 't' || b == 'f' { + on = ansiHighlight + } + for i < len(data) && !strings.ContainsRune(",]} \t\r\n", rune(data[i])) { + i++ + } + out.WriteString(on) + out.Write(data[start:i]) + out.WriteString(ansiNormal) + default: + out.WriteByte(b) + i++ + } + } + return []byte(out.String()) +} diff --git a/go/cmd/journalctl/disk_usage_other.go b/go/cmd/journalctl/disk_usage_other.go index fdba2ce7..3fd67761 100644 --- a/go/cmd/journalctl/disk_usage_other.go +++ b/go/cmd/journalctl/disk_usage_other.go @@ -1,4 +1,4 @@ -//go:build !unix +//go:build cgo && !unix package main diff --git a/go/cmd/journalctl/disk_usage_unix.go b/go/cmd/journalctl/disk_usage_unix.go index b7ba661b..c2e7e703 100644 --- a/go/cmd/journalctl/disk_usage_unix.go +++ b/go/cmd/journalctl/disk_usage_unix.go @@ -1,4 +1,4 @@ -//go:build unix +//go:build cgo && unix package main diff --git a/go/cmd/journalctl/file_glob_braces.go b/go/cmd/journalctl/file_glob_braces.go new file mode 100644 index 00000000..1db61f18 --- /dev/null +++ b/go/cmd/journalctl/file_glob_braces.go @@ -0,0 +1,40 @@ +//go:build cgo + +package main + +// Windows has no libc glob. Expand its brace alternatives before Unicode globbing. +func expandFileBraces(pattern string) []string { + for start := 0; start < len(pattern); start++ { + if pattern[start] != '{' { + continue + } + depth, part := 1, start+1 + var alternatives []string + for end := start + 1; end < len(pattern); end++ { + switch pattern[end] { + case '{': + depth++ + case ',': + if depth == 1 { + alternatives = append(alternatives, pattern[part:end]) + part = end + 1 + } + case '}': + depth-- + if depth != 0 { + continue + } + if len(alternatives) > 0 { + alternatives = append(alternatives, pattern[part:end]) + var result []string + for _, alternative := range alternatives { + result = append(result, expandFileBraces(pattern[:start]+alternative+pattern[end+1:])...) + } + return result + } + end = len(pattern) + } + } + } + return []string{pattern} +} diff --git a/go/cmd/journalctl/file_glob_braces_test.go b/go/cmd/journalctl/file_glob_braces_test.go new file mode 100644 index 00000000..a45fba7b --- /dev/null +++ b/go/cmd/journalctl/file_glob_braces_test.go @@ -0,0 +1,45 @@ +//go:build cgo + +package main + +import ( + "os" + "path/filepath" + "reflect" + "testing" +) + +func TestFileBraceAlternatives(t *testing.T) { + for pattern, expected := range map[string][]string{ + "{a,{b,c}}-é.journal": {"a-é.journal", "b-é.journal", "c-é.journal"}, + "{plain}.journal": {"{plain}.journal"}, + "{{a,b}}.journal": {"{a}.journal", "{b}.journal"}, + "{a,}?.journal": {"a?.journal", "?.journal"}, + } { + if actual := expandFileBraces(pattern); !reflect.DeepEqual(actual, expected) { + t.Fatalf("%q: got %q, want %q", pattern, actual, expected) + } + } +} + +func TestUnicodeFilePatterns(t *testing.T) { + directory := t.TempDir() + for _, name := range []string{"é.journal", ".hidden.journal", "literal-[x.journal"} { + if err := os.WriteFile(filepath.Join(directory, name), nil, 0o600); err != nil { + t.Fatal(err) + } + } + for pattern, names := range map[string][]string{ + "?.journal": {"é.journal"}, "[!a]*.journal": {"literal-[x.journal", "é.journal"}, + ".*.journal": {".hidden.journal"}, "literal-[x.journal": {"literal-[x.journal"}, + } { + var expected []string + for _, name := range names { + expected = append(expected, filepath.Join(directory, name)) + } + actual, err := globUnicodePaths(filepath.Join(directory, pattern)) + if err != nil || !reflect.DeepEqual(actual, expected) { + t.Fatalf("%q: got %q (%v), want %q", pattern, actual, err, expected) + } + } +} diff --git a/go/cmd/journalctl/file_glob_other.go b/go/cmd/journalctl/file_glob_other.go new file mode 100644 index 00000000..d47a024e --- /dev/null +++ b/go/cmd/journalctl/file_glob_other.go @@ -0,0 +1,27 @@ +//go:build cgo && !linux && !darwin && !freebsd + +package main + +import ( + "fmt" + "strings" + "unicode/utf8" +) + +func globFilePaths(pattern string) ([]string, error) { + if !utf8.ValidString(pattern) { + return nil, fmt.Errorf("file patterns require Unicode filenames on this platform") + } + if !strings.ContainsAny(pattern, "*?[]{}") { + return nil, nil + } + var paths []string + for _, alternative := range expandFileBraces(pattern) { + matches, err := globUnicodePaths(alternative) + if err != nil { + return nil, err + } + paths = append(paths, matches...) + } + return paths, nil +} diff --git a/go/cmd/journalctl/file_glob_unicode.go b/go/cmd/journalctl/file_glob_unicode.go new file mode 100644 index 00000000..f4aa7fca --- /dev/null +++ b/go/cmd/journalctl/file_glob_unicode.go @@ -0,0 +1,71 @@ +//go:build cgo + +package main + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "strings" +) + +// The fallback uses the same Unicode grammar as Rust, without libc glob. +func globUnicodePaths(pattern string) ([]string, error) { + volume := filepath.VolumeName(pattern) + rest := pattern[len(volume):] + prefix := volume + if len(rest) > 0 && os.IsPathSeparator(rest[0]) { + prefix += string(filepath.Separator) + } + prefixes := []string{prefix} + for _, part := range strings.FieldsFunc(rest, func(ch rune) bool { return ch < 128 && os.IsPathSeparator(uint8(ch)) }) { + if !isGlobPattern(part) { + for i, prefix := range prefixes { + prefixes[i] = joinGlobComponent(prefix, part, volume) + } + continue + } + matcher, err := regexp.Compile(globPatternToRegex(part)) + if err != nil { + return nil, err + } + var next []string + for _, prefix := range prefixes { + directory := prefix + if directory == "" { + directory = "." + } + entries, err := os.ReadDir(directory) + if err != nil { + continue + } + for _, entry := range entries { + name := entry.Name() + if strings.HasPrefix(name, ".") && !strings.HasPrefix(part, ".") { + continue + } + if matcher.MatchString(name) { + next = append(next, joinGlobComponent(prefix, name, volume)) + } + } + } + prefixes = next + } + paths := prefixes[:0] + for _, path := range prefixes { + if _, err := os.Stat(path); err == nil { + paths = append(paths, path) + } + } + sort.Strings(paths) + return paths, nil +} + +func joinGlobComponent(prefix, component, volume string) string { + // Preserve relative drive paths and parent components for OS resolution. + if prefix == "" || os.IsPathSeparator(prefix[len(prefix)-1]) || prefix == volume && strings.HasSuffix(volume, ":") { + return prefix + component + } + return prefix + string(filepath.Separator) + component +} diff --git a/go/cmd/journalctl/file_glob_unix.go b/go/cmd/journalctl/file_glob_unix.go new file mode 100644 index 00000000..51399261 --- /dev/null +++ b/go/cmd/journalctl/file_glob_unix.go @@ -0,0 +1,60 @@ +//go:build cgo && (linux || darwin || freebsd) + +package main + +/* +#define _GNU_SOURCE +#include +#include +#include + +static int journal_file_glob(const char *pattern, glob_t *paths) { + locale_t selected = newlocale(LC_ALL_MASK, "", (locale_t) 0); + if (!selected) + selected = newlocale(LC_ALL_MASK, "C", (locale_t) 0); + if (!selected) + return GLOB_NOSPACE; + locale_t previous = uselocale(selected); + if (!previous) { + freelocale(selected); + return GLOB_ABORTED; + } + int result = glob(pattern, GLOB_BRACE | GLOB_NOCHECK | GLOB_NOSORT, NULL, paths); + uselocale(previous); + freelocale(selected); + return result; +} +*/ +import "C" + +import ( + "fmt" + "strings" + "unsafe" +) + +func globFilePaths(pattern string) ([]string, error) { + // NOCHECK leaves an unescaped literal unchanged, including a missing path. + if !strings.ContainsAny(pattern, "*?[]{}\\") { + return nil, nil + } + if strings.IndexByte(pattern, 0) >= 0 { + return nil, fmt.Errorf("invalid NUL in file pattern") + } + value := C.CString(pattern) + defer C.free(unsafe.Pointer(value)) + var matches C.glob_t + status := C.journal_file_glob(value, &matches) + defer C.globfree(&matches) + if status != 0 { + return nil, fmt.Errorf("file pattern expansion failed (%d)", int(status)) + } + if uint64(matches.gl_pathc) > uint64(^uint(0)>>1) { + return nil, fmt.Errorf("too many file pattern matches") + } + paths := make([]string, 0, int(matches.gl_pathc)) + for _, path := range unsafe.Slice(matches.gl_pathv, int(matches.gl_pathc)) { + paths = append(paths, C.GoString(path)) + } + return paths, nil +} diff --git a/go/cmd/journalctl/introspection.go b/go/cmd/journalctl/introspection.go new file mode 100644 index 00000000..64182f14 --- /dev/null +++ b/go/cmd/journalctl/introspection.go @@ -0,0 +1,46 @@ +//go:build cgo + +package main + +import ( + _ "embed" + "encoding/json" + "io" +) + +//go:embed options.json +var cliOptionsJSON []byte + +func printCLIIntrospection(out io.Writer, mode string) error { + var options any + if err := json.Unmarshal(cliOptionsJSON, &options); err != nil { + return err + } + value := map[string]any{ + "mediaType": "application/vnd.io.systemd.cli-introspection-0", + "commands": []any{map[string]any{ + "names": []string{"journalctl"}, "project": "systemd-journal-sdk", + "version": "Go rewrite", "features": []string{"PCRE2 " + pcreVersion()}, + "abstract": []string{"Query file-backed journals."}, "options": options, + }}, + } + var data []byte + var err error + if mode == "json-pretty" { + data, err = json.MarshalIndent(value, "", "\t") + } else { + data, err = json.Marshal(value) + } + if err != nil { + return err + } + if mode == "json-sse" { + data = append(append([]byte("data: "), data...), '\n', '\n') + } else if mode == "json-seq" { + data = append(append([]byte{0x1e}, data...), '\n') + } else { + data = append(data, '\n') + } + _, err = out.Write(data) + return err +} diff --git a/go/cmd/journalctl/invocation.go b/go/cmd/journalctl/invocation.go new file mode 100644 index 00000000..9a272d6a --- /dev/null +++ b/go/cmd/journalctl/invocation.go @@ -0,0 +1,542 @@ +//go:build cgo + +package main + +import ( + "bytes" + "encoding/hex" + "errors" + "fmt" + "strings" + + "github.com/netdata/systemd-journal-sdk/go/journal" +) + +type invocationJournal struct { + cliJournal + data func(string) ([]byte, error) + realtime func() (uint64, error) + cursor func() (string, error) +} + +func (input cliInput) openInvocationJournal() (*invocationJournal, error) { + if input.directory != "" { + j, err := journal.SdJournalOpenDirectory(input.directory, 0) + if err != nil { + return nil, err + } + return &invocationJournal{j, func(field string) ([]byte, error) { return journal.SdJournalGetData(j, field) }, j.GetRealtimeUsec, func() (string, error) { return journal.SdJournalGetCursor(j) }}, nil + } + j, err := journal.SdJournalOpenFiles(input.files, 0) + if err != nil { + return nil, err + } + return &invocationJournal{j, func(field string) ([]byte, error) { return journal.SdJournalGetData(j, field) }, j.GetRealtimeUsec, func() (string, error) { return journal.SdJournalGetCursor(j) }}, nil +} + +func singleInvocationUnit(j cliJournal, flags *cliFlags, optionName string) ([]string, []string, error) { + raw := rawUnitSpecs(flags) + count := len(raw.system) + len(raw.user) + if count == 0 { + return nil, nil, fmt.Errorf("Using %s requires a unit. Please specify a unit name with -u/--unit=/--user-unit=.", optionName) + } + if count > 1 { + return nil, nil, fmt.Errorf("Using %s with multiple units is not supported.", optionName) + } + system, user, err := effectiveUnitSpecs(flags) + if err != nil { + return nil, nil, err + } + specs, fields := system, systemUnitFieldsFull[:3] + if len(user) > 0 { + specs, fields = user, userUnitFieldsFull[:3] + } + units, err := expandUnitSpecs(j, specs, fields) + if err != nil { + return nil, nil, err + } + if len(units) == 0 { + return nil, nil, fmt.Errorf("No matching unit found for '%s' in journal.", specs[0]) + } + if len(units) > 1 { + return nil, nil, fmt.Errorf("Multiple matching units found for '%s' in journal.", specs[0]) + } + if len(user) > 0 { + return nil, units, nil + } + return units, nil, nil +} + +func invocationBootID(j cliJournal, flags *cliFlags) (string, error) { + boot := effectiveBootFlag(flags) + if !boot.set || strings.TrimSpace(boot.value) == "all" { + return "", nil + } + return resolveBootID(j, strings.TrimSpace(boot.value)) +} + +func addInvocationBoot(j cliJournal, id string) error { + if id == "" { + return nil + } + return addFieldMatches(j, "_BOOT_ID", []string{id}) +} + +// Discovery reads only eligible indexed entries; a second reader supplies global +// ID bounds without changing the discovery reader's predicate or cursor state. +type invocationWalker struct { + scan, bounds *invocationJournal + bootID string + older bool + done bool + anchor string + previous string + units unitSelection + eligibility [][]byte + unrestricted bool +} + +func newInvocationWalker(input cliInput, flags *cliFlags, option string, older, show bool) (*invocationWalker, error) { + scan, err := input.openInvocationJournal() + if err != nil { + return nil, err + } + w := &invocationWalker{scan: scan, older: older} + ok := false + defer func() { + if !ok { + w.Close() + } + }() + // Show validates boot before unit; invocation tables acquire unit first. + if show { + w.bootID, err = invocationBootID(scan, flags) + if err != nil { + return nil, err + } + } + var system, user []string + if option != "" { + system, user, err = singleInvocationUnit(scan, flags, option) + if err != nil { + return nil, err + } + } + if !show { + w.bootID, err = invocationBootID(scan, flags) + if err != nil { + return nil, err + } + } + w.units = unitSelection{system, user} + if option != "" { + w.eligibility, err = invocationEligibility(scan) + if err != nil { + return nil, err + } + w.done = len(w.eligibility) == 0 + } + if w.done { + ok = true + return w, nil + } + if err = w.resetScan(false); err != nil { + return nil, err + } + if older { + err = scan.SeekTail() + } else { + err = scan.SeekHead() + } + if err != nil { + return nil, err + } + w.bounds, err = input.openInvocationJournal() + if err != nil { + return nil, err + } + ok = true + return w, nil +} + +func (w *invocationWalker) Close() { + if w.scan != nil { + _ = w.scan.Close() + } + if w.bounds != nil { + _ = w.bounds.Close() + } +} + +func parseInvocationIDBytes(value []byte) ([16]byte, bool) { + var id [16]byte + if len(value) > 37 { + return id, false + } + if end := bytes.IndexByte(value, 0); end >= 0 { + value = value[:end] + } + if len(value) == 36 { + if value[8] != '-' || value[13] != '-' || value[18] != '-' || value[23] != '-' { + return id, false + } + value = bytes.ReplaceAll(value, []byte{'-'}, nil) + } + if len(value) != 32 { + return id, false + } + if _, err := hex.Decode(id[:], value); err != nil || id == [16]byte{} { + return id, false + } + return id, true +} + +func parseInvocationIDValue(value []byte) string { + id, ok := parseInvocationIDBytes(value) + if !ok { + return "" + } + return hex.EncodeToString(id[:]) +} + +func invocationEligibility(j cliJournal) ([][]byte, error) { + var matches [][]byte + for _, field := range invocationIDFields { + if err := j.VisitUnique(field, func(value []byte) error { + if _, ok := parseInvocationIDBytes(value); ok { + matches = append(matches, append([]byte(field+"="), value...)) + } + return nil + }); err != nil { + return nil, err + } + } + return matches, nil +} + +func (w *invocationWalker) addUnitScope(j cliJournal) error { + for _, unit := range w.units.system { + if err := addSystemUnitMatchGroups(j, unit, false); err != nil { + return err + } + } + uid, uidOK := currentUIDString() + for _, unit := range w.units.user { + if err := addUserUnitMatchGroups(j, unit, uid, uidOK, false); err != nil { + return err + } + } + if len(w.units.system)+len(w.units.user) > 0 { + return j.AddConjunction() + } + return nil +} + +func (w *invocationWalker) resetScan(unrestricted bool) error { + j := w.scan + j.FlushMatches() + if !unrestricted { + if err := addInvocationBoot(j, w.bootID); err != nil { + return err + } + if err := w.addUnitScope(j); err != nil { + return err + } + } + for i, payload := range w.eligibility { + if i > 0 { + if err := j.AddDisjunction(); err != nil { + return err + } + } + j.AddMatch(payload) + } + if len(w.eligibility) > 0 { + if err := j.AddConjunction(); err != nil { + return err + } + } + w.unrestricted = unrestricted + return nil +} + +func primaryInvocationID(j *invocationJournal) (string, error) { + for _, field := range invocationIDFields { + payload, err := j.data(field) + if errors.Is(err, journal.ErrNoEntry) { + continue + } + if err != nil { + return "", err + } + if len(payload) > len(field) { + if id := parseInvocationIDValue(payload[len(field)+1:]); id != "" { + return id, nil + } + } + } + return "", nil +} + +func (w *invocationWalker) bound(id string) (invocationInfo, string, bool, error) { + j := w.bounds + if j == nil { + return invocationInfo{}, "", false, nil + } + j.FlushMatches() + if err := addInvocationBoot(j, w.bootID); err != nil { + return invocationInfo{}, "", false, err + } + var matchErr error + if id == "" { + matchErr = w.addUnitScope(j) + } else { + matchErr = addInvocationMatches(j, id) + } + if matchErr != nil { + return invocationInfo{}, "", false, matchErr + } + if err := j.SeekHead(); err != nil { + return invocationInfo{}, "", false, err + } + n, err := j.Next() + if err != nil || n == 0 { + return invocationInfo{}, "", false, err + } + first, err := j.realtime() + if err != nil { + return invocationInfo{}, "", false, err + } + anchor := "" + if w.older { + anchor, err = j.cursor() + if err != nil { + return invocationInfo{}, "", false, err + } + } + if err := j.SeekTail(); err != nil { + return invocationInfo{}, "", false, err + } + n, err = j.Previous() + if err != nil || n == 0 { + return invocationInfo{}, "", false, err + } + last, err := j.realtime() + if err != nil { + return invocationInfo{}, "", false, err + } + if !w.older { + anchor, err = j.cursor() + if err != nil { + return invocationInfo{}, "", false, err + } + } + return invocationInfo{id, first, last}, anchor, true, nil +} + +func (w *invocationWalker) step() (bool, error) { + if w.anchor != "" { + cursor := w.anchor + w.anchor = "" + direction := journal.DirectionForward + if w.older { + direction = journal.DirectionBackward + } + if err := w.scan.SeekCursorDirection(cursor, direction); err != nil { + return false, err + } + exact, err := w.scan.TestCursor(cursor) + if errors.Is(err, journal.ErrEndOfEntries) || errors.Is(err, journal.ErrNoEntry) { + return false, nil + } + if err != nil { + return false, err + } + if !exact { + // A cursor mismatch also represents EOF in the native facade. + _, err := w.scan.cursor() + if errors.Is(err, journal.ErrEndOfEntries) || errors.Is(err, journal.ErrNoEntry) { + return false, nil + } + return err == nil, err + } + } + var n int + var err error + if w.older { + n, err = w.scan.Previous() + } else { + n, err = w.scan.Next() + } + return n != 0, err +} + +func (w *invocationWalker) next() (invocationInfo, bool, error) { + if w.done { + return invocationInfo{}, false, nil + } + if w.unrestricted { + if err := w.resetScan(false); err != nil { + return invocationInfo{}, false, err + } + } + broken := make(map[string]bool) + for { + ok, err := w.step() + if err != nil || !ok { + w.done = true + return invocationInfo{}, false, err + } + id, err := primaryInvocationID(w.scan) + if err != nil { + return invocationInfo{}, false, err + } + if id == "" || id == w.previous || broken[id] { + continue + } + info, anchor, ok, err := w.bound(id) + if err != nil { + return invocationInfo{}, false, err + } + if !ok { + broken[id] = true + _, anchor, found, err := w.bound(w.previous) + if err != nil { + return invocationInfo{}, false, err + } + if !found { + return invocationInfo{}, false, fmt.Errorf("Cannot seek to the previous invocation: No data available") + } + w.anchor = anchor + // Native discovery resumes without unit/boot matches after a broken ID. + if err := w.resetScan(true); err != nil { + return invocationInfo{}, false, err + } + continue + } + w.anchor, w.previous = anchor, id + return info, true, nil + } +} + +func collectInvocationsFromInput(input cliInput, flags *cliFlags, unitOption string) ([]invocationInfo, error) { + older, max := false, int(^uint(0)>>1) + if flags.linesFlag.set { + limit, err := parseLinesLimitValue(flags.linesFlag.value) + if err != nil { + return nil, err + } + if !limit.all { + older, max = !limit.oldest, limit.count + } + } + if max == 0 { + j, err := input.openJournal() + if err != nil { + return nil, err + } + defer j.Close() + if unitOption != "" { + if _, _, err := singleInvocationUnit(j, flags, unitOption); err != nil { + return nil, err + } + } + _, err = invocationBootID(j, flags) + return nil, err + } + w, err := newInvocationWalker(input, flags, unitOption, older, false) + if err != nil { + return nil, err + } + defer w.Close() + var out []invocationInfo + seen := make(map[string]bool) + for len(out) < max { + info, ok, err := w.next() + if err != nil { + return nil, err + } + if !ok || seen[info.id] { + break + } + seen[info.id] = true + out = append(out, info) + } + if older { + for i, k := 0, len(out)-1; i < k; i, k = i+1, k-1 { + out[i], out[k] = out[k], out[i] + } + } + return out, nil +} + +func resolveInvocationFilter(input cliInput, flags *cliFlags) (string, bool, error) { + if !flags.invocationSet { + return "", false, nil + } + if !flags.invocationReady { + flags.invocationReady = true + flags.invocationID, flags.invocationActive, flags.invocationError = findInvocationFilter(input, flags) + } + return flags.invocationID, flags.invocationActive, flags.invocationError +} + +func findInvocationFilter(input cliInput, flags *cliFlags) (string, bool, error) { + descriptor, set := effectiveInvocationDescriptor(flags) + if !set { + return "", false, nil + } + id, offset, active, err := parseInvocationDescriptor(descriptor) + if err != nil || !active { + return "", false, err + } + option := "" + if id == "" || offset != 0 { + option = "-I/--invocation= with an offset" + } + w, err := newInvocationWalker(input, flags, option, offset <= 0, true) + if err != nil { + return "", false, err + } + defer w.Close() + notFound := func() (string, bool, error) { + return "", false, fmt.Errorf("No journal entry found for the invocation (%s%+d).", id, offset) + } + start := 1 + if offset <= 0 { + start = 0 + } + if id != "" { + _, anchor, ok, err := w.bound(id) + if err != nil { + return "", false, err + } + if !ok { + return notFound() + } + if offset == 0 { + return id, true, nil + } + w.anchor, w.previous = anchor, id + if offset < 0 { + start = -1 + } + } + for off := start; ; { + info, ok, err := w.next() + if err != nil { + return "", false, err + } + if !ok { + return notFound() + } + if off == offset { + return info.id, true, nil + } + if w.older { + off-- + } else { + off++ + } + } +} diff --git a/go/cmd/journalctl/main.go b/go/cmd/journalctl/main.go index b6e86be5..fad58a73 100644 --- a/go/cmd/journalctl/main.go +++ b/go/cmd/journalctl/main.go @@ -1,6 +1,9 @@ +//go:build cgo + package main import ( + "bytes" "crypto/rand" "encoding/binary" "encoding/hex" @@ -14,12 +17,21 @@ import ( "sort" "strconv" "strings" + "sync" "time" - "unicode" "github.com/netdata/systemd-journal-sdk/go/journal" ) +var errSilentNoData = errors.New("no matching entries") + +func noDataError(quiet bool, message string) error { + if quiet { + return errSilentNoData + } + return errors.New(message) +} + // HEADER_COMPATIBLE_SEALED from systemd journal-def.h const compatibleSealed = 1 const compatibleTailEntryBootID = 1 << 1 @@ -38,20 +50,11 @@ const coredumpMessageID = "fc2e22bc6ee647b6b90729ab34a250b1" var ( errStopIteration = errors.New("stop iteration") - signedDatePrefixRe = regexp.MustCompile(`^[+-]\d{4}-`) - epochTimestampRe = regexp.MustCompile(`^\d+(\.\d+)?$`) - durationTokenRe = regexp.MustCompile(`\s*(\d+(?:\.\d+)?)(?:\s*([A-Za-z]+))?`) - sizeTokenRe = regexp.MustCompile(`^\s*(\d+(?:\.\d+)?)\s*([A-Za-z]*)\s*$`) - - bootDescriptorRe = regexp.MustCompile(`^(([0-9A-Fa-f]{32})|([0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}))?([+-]?\d+)?$`) - bootDescriptorPatterns = []*regexp.Regexp{ - regexp.MustCompile(`^[+-]?\d+$`), - regexp.MustCompile(`^[0-9A-Fa-f]{32}([+-]\d+)?$`), - regexp.MustCompile(`^[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}([+-]\d+)?$`), - } - archivedJournalNameRe = regexp.MustCompile(`^.+@([0-9A-Fa-f]{32})-([0-9A-Fa-f]{16})-([0-9A-Fa-f]{16})\.journal$`) - corruptJournalNameRe = regexp.MustCompile(`^.+@([0-9A-Fa-f]{16})-([0-9A-Fa-f]{16})\.journal~$`) - unitSuffixes = []string{ + archivedJournalNameRe = sync.OnceValue(func() *regexp.Regexp { + return regexp.MustCompile(`^.+@([0-9A-Fa-f]{32})-([0-9A-Fa-f]{16})-([0-9A-Fa-f]{16})\.journal$`) + }) + corruptJournalNameRe = sync.OnceValue(func() *regexp.Regexp { return regexp.MustCompile(`^.+@([0-9A-Fa-f]{16})-([0-9A-Fa-f]{16})\.journal~$`) }) + unitSuffixes = []string{ ".automount", ".device", ".mount", @@ -86,7 +89,7 @@ var ( } ) -const officialOptionSurfaceHelp = `Official systemd v260.1 option surface: +const officialOptionSurfaceHelp = `Official systemd v262 option surface: --system --user --machine --merge --directory --file --root --image --image-policy --namespace --since --until --cursor --after-cursor --cursor-file --boot --this-boot --unit --user-unit --invocation --identifier --exclude-identifier --priority --facility --grep --case-sensitive --dmesg @@ -108,13 +111,13 @@ var invocationIDFields = []string{ type cliJournal interface { Close() error AddMatch([]byte) - AddDisjunction() - AddConjunction() + AddDisjunction() error + AddConjunction() error FlushMatches() SeekHead() error SeekTail() error SeekRealtimeUsec(uint64) error - SeekCursor(string) error + SeekCursorDirection(string, journal.Direction) error TestCursor(string) (bool, error) Next() (int, error) Previous() (int, error) @@ -132,28 +135,45 @@ type optionalStringFlag struct { } type cliFlags struct { - file multiStringFlag - directory *string - output *string - listBoots *bool - noTail *bool - follow *bool - boot optionalStringFlag - fields *bool - field *string - head *int - tail *int - since *string - until *string - sync *bool - flush *bool - rotate *bool - relinquish *bool - verify *bool - verifyOnly *bool - verifyKey *string - - // Parser-recognized v260.1 options. Each long option is registered + diagnostics io.Writer + unitsReady bool + units unitSelection + unitsError error + unitMatchesReady bool + unitMatches unitSelection + unitMatchesError error + invocationReady bool + invocationID string + invocationActive bool + invocationError error + action cliAction + parsed bool + seen map[string]bool + sinceUsec, untilUsec *uint64 + fullWidth bool + timestamps timestampParser + file multiStringFlag + directory *string + output *string + listBoots *bool + noTail *bool + follow *bool + boot optionalStringFlag + fields *bool + field *string + head *int + tail *int + since *string + until *string + sync *bool + flush *bool + rotate *bool + relinquish *bool + verify *bool + verifyOnly *bool + verifyKey *string + + // Parser-recognized v262 options. Each long option is registered // with `flag.Var` so the parser accepts it; the runtime dispatch // then either implements the requested behavior or returns a // portable-mode unsupported message. Source-of-truth for the option @@ -279,57 +299,40 @@ func (f *optionalStringFlag) IsBoolFlag() bool { func main() { if err := run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr); err != nil { - fmt.Fprintf(os.Stderr, "Error: %v\n", err) + if !errors.Is(err, errSilentNoData) { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + } os.Exit(1) } } func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error { fs, flags := newCLIFlagSet(stderr) - if argsContainHelp(args) { - printUsage(fs, stdout) - return nil - } - parseArgs := permuteFlagArgs(preprocessOptionalArgs(normalizeShortFlags(args, fs)), fs) - - if err := fs.Parse(parseArgs); err != nil { - if errors.Is(err, flag.ErrHelp) { - printUsage(fs, stdout) - return nil - } + operands, terminal, err := parseCLIOptions(args, fs, flags, stdout) + if err != nil || terminal { return err } - flags.invocationSet = flagWasSet(fs, "invocation") || *flags.invocationShortFlag - flags.grepSet = flagWasSet(fs, "grep") || flagWasSet(fs, "g") - if *flags.output == "help" { - printOutputModeHelp(stdout) - return nil - } - if err := validateOutputMode(*flags.output); err != nil { - return err + if *flags.noTail { + flags.linesFlag = optionalStringFlag{set: true, value: "all"} } - if err := flags.validateParserInteractions(); err != nil { return err } - if *flags.versionFlag { - fmt.Fprintln(stdout, "journalctl (systemd-journal-sdk Go rewrite)") - fmt.Fprintln(stdout, "baseline: systemd v260.1 (c0a5a2516d28)") - fmt.Fprintln(stdout, "portable file-backed mode") - return nil + hasVerifyKey := flags.seen["verify-key"] + fieldWasSet := flags.action == "field" + if err := flags.validateActionArguments(operands, hasVerifyKey, fieldWasSet); err != nil { + return err } - - if facilityHelpRequested(flags.facilityFlag.Values()) { - printFacilityHelp(stdout, *flags.quietFlag) - return nil + boot := effectiveBootFlag(flags) + if (boot.set && boot.value != "all" || flags.action == "list-boots") && *flags.mergeFlag { + return errors.New("Using --boot or --list-boots with --merge is not supported.") } - - hasVerifyKey := hasStringFlag(args, "verify-key") - fieldWasSet := flagWasSet(fs, "field") || flagWasSet(fs, "F") - if err := flags.validateActionArguments(fs.Args(), hasVerifyKey, fieldWasSet); err != nil { + postFilters, err := newCLIPostFilters(flags) + if err != nil { return err } + defer postFilters.grep.Close() if err := flags.validatePostAction(); err != nil { return err } @@ -340,7 +343,7 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error { if err != nil { return err } - if err := validatePathMatchArguments(fs.Args()); err != nil { + if err := validatePathMatchArguments(operands); err != nil { return err } if *flags.diskUsageFlag { @@ -355,8 +358,8 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error { if *flags.listInvocationsFlag { return runListInvocations(input, flags, stdout) } - if *flags.verify || *flags.verifyOnly || hasVerifyKey { - return runVerify(input, *flags.verifyKey, hasVerifyKey, stdout, stderr) + if flags.action == "verify" || flags.action == "verify-only" { + return runVerify(input, *flags.verifyKey, hasVerifyKey, flags.effectiveQuiet(), stdout, stderr) } if *flags.listBoots { return runListBoots(input, flags, stdout) @@ -366,41 +369,50 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error { if err != nil { return err } - postFilters, err := newCLIPostFilters(flags) - if err != nil { - return err - } cursorControl, err := newCursorControl(flags) if err != nil { return err } - outputOptions := newOutputOptions(flags, flagWasSet(fs, "output-fields"), resolveFullWidth(args)) + outputOptions := newOutputOptions(flags, *flags.outputFieldsFlag != "", flags.fullWidth) + postFilters.captureHighlight = (outputOptions.colors.level != 0 || outputOptions.urlify) && (outputOptions.mode == "cat" || outputSkipsMissingMessage(outputOptions.mode)) - if *flags.follow { - tail := 10 + if *flags.follow && flags.action == actionShow { + tail, all := 0, true + if !flags.seen["since"] { + tail, all = 10, false + } else if *flags.pagerEndFlag { + tail, all = 1000, false + } if flagWasSet(fs, "tail") { - tail = *flags.tail + tail, all = *flags.tail, false } - if flagWasSet(fs, "lines") { + if flags.linesFlag.set { limit, err := parseLinesLimitValue(flags.linesFlag.value) if err != nil { return err } - if limit.set && !limit.all { - tail = limit.count + if limit.all { + tail, all = 0, true + } else { + tail, all = limit.count, false } } - return runFollow(input, fs.Args(), flags, *flags.output, sinceUsec, untilUsec, tail, *flags.noTail, stdout, postFilters, cursorControl, outputOptions) + all = all || *flags.noTail + return runFollow(input, operands, flags, *flags.output, sinceUsec, untilUsec, tail, all, stdout, postFilters, cursorControl, outputOptions) } - j, err := openFilteredJournal(input, fs.Args(), flags, *flags.output, optionalStringFlag{}) + j, err := openFilteredJournal(input, operands, flags, *flags.output, optionalStringFlag{}) if err != nil { return err } defer j.Close() - return flags.dispatch(j, sinceUsec, untilUsec, stdout, postFilters, cursorControl, outputOptions) + err = flags.dispatch(j, sinceUsec, untilUsec, stdout, postFilters, cursorControl, outputOptions) + if err == nil && flags.action == actionShow && flags.grepSet && !postFilters.shown { + return errSilentNoData + } + return err } func resolveFullWidth(args []string) bool { @@ -441,8 +453,8 @@ func parseLinesLimitValue(value string) (linesLimit, error) { } oldest := strings.HasPrefix(value, "+") stripped := strings.TrimPrefix(value, "+") - n, err := strconv.Atoi(stripped) - if err != nil || n < 0 { + n, err := parseNativeInt32(stripped) + if err != nil || n < 0 || uint64(n) > 2147483647 { return linesLimit{}, fmt.Errorf("Failed to parse --lines='%s'.", value) } return linesLimit{set: true, oldest: oldest, count: n}, nil @@ -455,30 +467,30 @@ type cursorSeek struct { type cursorControl struct { seek *cursorSeek - updateFile string + updateFile *string } func newCursorControl(f *cliFlags) (cursorControl, error) { - if *f.cursorFlag != "" { + if *f.cursorFlag != "" || f.seen["cursor"] { return cursorControl{seek: &cursorSeek{cursor: *f.cursorFlag}}, nil } - if *f.afterCursorFlag != "" { + if *f.afterCursorFlag != "" || f.seen["after-cursor"] { return cursorControl{seek: &cursorSeek{cursor: *f.afterCursorFlag, after: true}}, nil } - if *f.cursorFileFlag == "" { + if *f.cursorFileFlag == "" && !f.seen["cursor-file"] { return cursorControl{}, nil } content, err := os.ReadFile(*f.cursorFileFlag) // nosec G304 - caller explicitly supplies --cursor-file path. if err != nil { if errors.Is(err, os.ErrNotExist) { - return cursorControl{updateFile: *f.cursorFileFlag}, nil + return cursorControl{updateFile: f.cursorFileFlag}, nil } return cursorControl{}, fmt.Errorf("Failed to read cursor file %s: %w", *f.cursorFileFlag, err) } cursor := strings.SplitN(string(content), "\n", 2)[0] cursor = strings.TrimSuffix(cursor, "\r") - control := cursorControl{updateFile: *f.cursorFileFlag} + control := cursorControl{updateFile: f.cursorFileFlag} if cursor != "" { control.seek = &cursorSeek{cursor: cursor, after: true} } @@ -489,24 +501,25 @@ func newCLIFlagSet(stderr io.Writer) (*flag.FlagSet, *cliFlags) { fs := flag.NewFlagSet("journalctl", flag.ContinueOnError) fs.SetOutput(stderr) flags := &cliFlags{ - directory: fs.String("directory", "", "journal directory"), - output: fs.String("output", "short", "output mode: short, short-full, short-iso, short-iso-precise, short-precise, short-monotonic, short-delta, short-unix, verbose, export, json, json-pretty, json-sse, json-seq, cat, with-unit"), - listBoots: fs.Bool("list-boots", false, "list boots"), - noTail: fs.Bool("no-tail", false, "show all entries, start from the beginning"), - follow: fs.Bool("follow", false, "follow appended entries"), - fields: fs.Bool("fields", false, "show field names"), - field: fs.String("field", "", "show values for a field"), - head: fs.Int("head", 0, "show first N entries"), - tail: fs.Int("tail", 0, "show last N entries"), - since: fs.String("since", "", "show entries since timestamp"), - until: fs.String("until", "", "show entries until timestamp"), - sync: fs.Bool("sync", false, "sync journal (portable mode does not support this)"), - flush: fs.Bool("flush", false, "flush journal (portable mode does not support this)"), - rotate: fs.Bool("rotate", false, "rotate journal (portable mode does not support this)"), - relinquish: fs.Bool("relinquish-var", false, "relinquish var (portable mode does not support this)"), - verify: fs.Bool("verify", false, "verify journal file"), - verifyOnly: fs.Bool("verify-only", false, "verify only"), - verifyKey: fs.String("verify-key", "", "FSS verification key"), + diagnostics: stderr, + directory: fs.String("directory", "", "journal directory"), + output: fs.String("output", "short", "output mode: short, short-full, short-iso, short-iso-precise, short-precise, short-monotonic, short-delta, short-unix, verbose, export, json, json-pretty, json-sse, json-seq, cat, with-unit"), + listBoots: fs.Bool("list-boots", false, "list boots"), + noTail: fs.Bool("no-tail", false, "show all entries, start from the beginning"), + follow: fs.Bool("follow", false, "follow appended entries"), + fields: fs.Bool("fields", false, "show field names"), + field: fs.String("field", "", "show values for a field"), + head: fs.Int("head", 0, "show first N entries"), + tail: fs.Int("tail", 0, "show last N entries"), + since: fs.String("since", "", "show entries since timestamp"), + until: fs.String("until", "", "show entries until timestamp"), + sync: fs.Bool("sync", false, "sync journal (portable mode does not support this)"), + flush: fs.Bool("flush", false, "flush journal (portable mode does not support this)"), + rotate: fs.Bool("rotate", false, "rotate journal (portable mode does not support this)"), + relinquish: fs.Bool("relinquish-var", false, "relinquish var (portable mode does not support this)"), + verify: fs.Bool("verify", false, "verify journal file"), + verifyOnly: fs.Bool("verify-only", false, "verify only"), + verifyKey: fs.String("verify-key", "", "FSS verification key"), systemFlag: fs.Bool("system", false, "show the system journal (portable: no-op)"), userFlag: fs.Bool("user", false, "show the user journal (portable: no-op)"), @@ -609,25 +622,12 @@ func printUsage(fs *flag.FlagSet, out io.Writer) { fs.SetOutput(out) defer fs.SetOutput(previous) fmt.Fprintf(out, "Usage: %s [options]\n", fs.Name()) - fmt.Fprintf(out, "Pure-Go systemd journal reader (portable mode, systemd v260.1 baseline)\n") + fmt.Fprintf(out, "Go systemd journal reader (portable mode, systemd v262 CLI; native PCRE2)\n") fmt.Fprintf(out, "\nOptions:\n") fs.PrintDefaults() fmt.Fprintf(out, "\n%s", officialOptionSurfaceHelp) } -func argsContainHelp(args []string) bool { - for _, arg := range args { - if arg == "--" { - return false - } - switch arg { - case "--help", "-help", "-h": - return true - } - } - return false -} - func (f *cliFlags) validate() error { if err := f.validateParserInteractions(); err != nil { return err @@ -642,10 +642,10 @@ func (f *cliFlags) validateParserInteractions() error { if len(f.file.values) > 0 { sources++ } - if *f.directory != "" { + if *f.directory != "" || f.seen["directory"] { sources++ } - if *f.machineFlag != "" { + if *f.machineFlag != "" || f.seen["machine"] { sources++ } if *f.rootFlag != "" { @@ -659,11 +659,7 @@ func (f *cliFlags) validateParserInteractions() error { } // Since/until order. - sinceUsec, err := parseOptionalTimestampUsec(*f.since) - if err != nil { - return err - } - untilUsec, err := parseOptionalTimestampUsec(*f.until) + sinceUsec, untilUsec, err := f.timeBounds() if err != nil { return err } @@ -673,16 +669,16 @@ func (f *cliFlags) validateParserInteractions() error { // Cursor source exclusivity. cursorSources := 0 - if *f.cursorFlag != "" { + if *f.cursorFlag != "" || f.seen["cursor"] { cursorSources++ } - if *f.afterCursorFlag != "" { + if *f.afterCursorFlag != "" || f.seen["after-cursor"] { cursorSources++ } - if *f.cursorFileFlag != "" { + if *f.cursorFileFlag != "" || f.seen["cursor-file"] { cursorSources++ } - if *f.since != "" { + if *f.since != "" || f.seen["since"] { cursorSources++ } if cursorSources > 1 { @@ -722,6 +718,9 @@ func (f *cliFlags) validateParserInteractions() error { } func (f *cliFlags) isShowAction() bool { + if f.parsed { + return f.action == actionShow + } return !*f.versionFlag && !*f.newID128Flag && !*f.fields && @@ -747,6 +746,9 @@ func (f *cliFlags) isShowAction() bool { } func (f *cliFlags) validateActionArguments(args []string, hasVerifyKey, fieldWasSet bool) error { + if len(args) == 0 && f.parsed && (f.action == "field" || f.action == "fields") && f.fieldListHasScope() { + return errors.New("-F/--field= and -N/--fields cannot be combined with options that limit the journal.") + } if len(args) == 0 || !f.actionRejectsArguments(hasVerifyKey, fieldWasSet) { return nil } @@ -754,6 +756,9 @@ func (f *cliFlags) validateActionArguments(args []string, hasVerifyKey, fieldWas } func (f *cliFlags) actionRejectsArguments(hasVerifyKey, fieldWasSet bool) bool { + if f.parsed { + return f.action != actionShow && f.action != "list-catalog" && f.action != "dump-catalog" + } return *f.newID128Flag || *f.setupKeysFlag || *f.updateCatalogFlag || @@ -814,7 +819,7 @@ func (f *cliFlags) validatePostAction() error { if *f.flush { return portableUnsupported("--flush", "daemon-only runtime-to-persistent flush; no journald in portable mode") } - if *f.rotate && hasVacuumFlags(f) { + if f.action == actionRotateVacuum || !f.parsed && *f.rotate && hasVacuumFlags(f) { return portableUnsupported("--rotate with --vacuum-*", "official rotate-and-vacuum action requires journald rotation; portable mode can only vacuum explicit directories without rotation") } if *f.rotate { @@ -844,7 +849,7 @@ func (f *cliFlags) validatePostAction() error { if *f.diskUsageFlag && len(f.file.values) == 0 && *f.directory == "" { return portableUnsupported("--disk-usage", "requires host journal directory; pass --file or --directory to compute disk usage for explicit input") } - if *f.vacuumSizeFlag != "" || *f.vacuumFilesFlag != "" || *f.vacuumTimeFlag != "" { + if hasVacuumFlags(f) { if *f.directory == "" { return portableUnsupported("--vacuum-*", "vacuum actions require explicit --directory= input") } @@ -868,11 +873,45 @@ type cliInput struct { files []string } +type cursorJournal struct { + cliJournal + cursor func() (string, error) +} + +func currentJournalCursor(j cliJournal) (string, error) { + if source, ok := j.(*cursorJournal); ok { + return source.cursor() + } + entry, err := j.GetEntry() + if err != nil { + return "", err + } + return entry.Cursor, nil +} + func (i cliInput) openJournal() (cliJournal, error) { + return i.openJournalWithCursor(false) +} + +func (i cliInput) openJournalWithCursor(capture bool) (cliJournal, error) { if i.directory != "" { - return journal.SdJournalOpenDirectory(i.directory, 0) + j, err := journal.SdJournalOpenDirectory(i.directory, 0) + if err != nil { + return nil, err + } + if !capture { + return j, nil + } + return &cursorJournal{j, func() (string, error) { return journal.SdJournalGetCursor(j) }}, nil + } + j, err := journal.SdJournalOpenFiles(i.files, 0) + if err != nil { + return nil, err } - return journal.SdJournalOpenFiles(i.files, 0) + if !capture { + return j, nil + } + return &cursorJournal{j, func() (string, error) { return journal.SdJournalGetCursor(j) }}, nil } func (i cliInput) journalFiles(context string) ([]string, bool, error) { @@ -913,7 +952,7 @@ func resolveFileInputs(values []string) ([]string, error) { "stdin-backed journals require seekable mmap-capable file descriptors and are not supported in portable mode", ) } - matches, err := filepath.Glob(value) + matches, err := globFilePaths(value) if err != nil { return nil, fmt.Errorf("failed to add paths: %w", err) } @@ -949,6 +988,12 @@ func canonicalFileKey(path string) string { } func (f *cliFlags) timeBounds() (*uint64, *uint64, error) { + if f.parsed { + if f.sinceUsec != nil && f.untilUsec != nil && *f.sinceUsec > *f.untilUsec { + return nil, nil, errors.New("--since= must be before --until=.") + } + return f.sinceUsec, f.untilUsec, nil + } sinceUsec, err := parseOptionalTimestampUsec(*f.since) if err != nil { return nil, nil, err @@ -964,6 +1009,9 @@ func (f *cliFlags) timeBounds() (*uint64, *uint64, error) { } func validatePathMatchArguments(matches []string) error { + if err := validateDisjunctions(matches); err != nil { + return err + } for _, item := range matches { if item != "+" && !strings.Contains(item, "=") { return portableUnsupported( @@ -989,11 +1037,13 @@ func effectiveBootFlag(flags *cliFlags) optionalStringFlag { } func applyCLIMatches(j cliJournal, flags *cliFlags) error { - systemUnits, userUnits := effectiveUnitSpecs(flags) - if err := addJournalctlUnitMatches(j, systemUnits, userUnits); err != nil { + if err := addJournalctlUnitMatches(j, flags); err != nil { return err } + return applyNonUnitMatches(j, flags) +} +func applyNonUnitMatches(j cliJournal, flags *cliFlags) error { if *flags.dmesgFlag { if err := addFieldMatches(j, "_TRANSPORT", []string{"kernel"}); err != nil { return err @@ -1033,16 +1083,6 @@ func applyCLIMatches(j cliJournal, flags *cliFlags) error { return nil } -func effectiveUnitSpecs(flags *cliFlags) ([]string, []string) { - systemUnits := flags.unitFlag.Values() - userUnits := flags.userUnitFlag.Values() - if *flags.userFlag && len(systemUnits) > 0 { - userUnits = append(userUnits, systemUnits...) - systemUnits = nil - } - return systemUnits, userUnits -} - func addFieldMatches(j cliJournal, field string, values []string) error { if len(values) == 0 { return nil @@ -1054,8 +1094,7 @@ func addFieldMatches(j cliJournal, field string, values []string) error { } j.AddMatch(match) } - j.AddConjunction() - return nil + return j.AddConjunction() } func addInvocationMatches(j cliJournal, id string) error { @@ -1067,14 +1106,15 @@ func addInvocationMatches(j cliJournal, id string) error { } for i, field := range fields { if i > 0 { - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } } if err := addMatchPair(j, field, id); err != nil { return err } } - j.AddConjunction() - return nil + return j.AddConjunction() } func addMatchPair(j cliJournal, field, value string) error { @@ -1106,8 +1146,7 @@ func applyBootMatch(j cliJournal, flags *cliFlags, bootOverride optionalStringFl return err } j.AddMatch(match) - j.AddConjunction() - return nil + return j.AddConjunction() } func addMatchGroup(j cliJournal, pairs [][2]string) error { @@ -1119,110 +1158,129 @@ func addMatchGroup(j cliJournal, pairs [][2]string) error { return nil } -func addImpossibleMatch(j cliJournal, reason string) error { - if err := addMatchPair(j, "__JOURNALCTL_NEVER_MATCH", reason); err != nil { - return err +func addJournalctlUnitMatches(j cliJournal, flags *cliFlags) error { + if len(flags.unitFlag.values) == 0 && len(flags.userUnitFlag.values) == 0 { + return nil } - j.AddConjunction() - return nil -} - -func addJournalctlUnitMatches(j cliJournal, systemUnits, userUnits []string) error { + // Follow retains the initial wildcard expansion, like native journalctl. + if !flags.unitMatchesReady { + flags.unitMatchesReady = true + system, user, err := effectiveUnitSpecs(flags) + if err == nil { + flags.unitMatches.system, err = expandUnitSpecs(j, system, systemUnitFieldsFull) + } + if err == nil { + flags.unitMatches.user, err = expandUnitSpecs(j, user, userUnitFieldsFull) + } + if err == nil && len(system)+len(user) > 0 && len(flags.unitMatches.system)+len(flags.unitMatches.user) == 0 { + err = errors.New("Failed to add filter for units: No data available") + } + flags.unitMatchesError = err + } + if flags.unitMatchesError != nil { + return flags.unitMatchesError + } + systemUnits, userUnits := flags.unitMatches.system, flags.unitMatches.user if len(systemUnits) == 0 && len(userUnits) == 0 { return nil } - added := false - expandedSystemUnits, err := expandUnitSpecs(j, systemUnits, systemUnitFieldsFull) - if err != nil { - return err - } - for _, unit := range expandedSystemUnits { - if err := addSystemUnitMatchGroups(j, unit); err != nil { + for _, unit := range systemUnits { + if err := addSystemUnitMatchGroups(j, unit, true); err != nil { return err } - added = true } - expandedUserUnits, err := expandUnitSpecs(j, userUnits, userUnitFieldsFull) - if err != nil { - return err - } uid, uidOK := currentUIDString() - for _, unit := range expandedUserUnits { - if err := addUserUnitMatchGroups(j, unit, uid, uidOK); err != nil { + for _, unit := range userUnits { + if err := addUserUnitMatchGroups(j, unit, uid, uidOK, true); err != nil { return err } - added = true } - if !added { - return addImpossibleMatch(j, "unit-glob") - } - j.AddConjunction() - return nil + return j.AddConjunction() } -func addSystemUnitMatchGroups(j cliJournal, unit string) error { +func addSystemUnitMatchGroups(j cliJournal, unit string, full bool) error { if err := addMatchGroup(j, [][2]string{{"_SYSTEMD_UNIT", unit}}); err != nil { return err } - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } if err := addMatchGroup(j, [][2]string{{"_SYSTEMD_CGROUP", "/init.scope"}, {"UNIT", unit}}); err != nil { return err } - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } if err := addMatchGroup(j, [][2]string{{"_UID", "0"}, {"OBJECT_SYSTEMD_UNIT", unit}}); err != nil { return err } - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } + if !full { + return nil + } if err := addMatchGroup(j, [][2]string{{"MESSAGE_ID", coredumpMessageID}, {"COREDUMP_UNIT", unit}}); err != nil { return err } if strings.HasSuffix(unit, ".slice") { - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } if err := addMatchGroup(j, [][2]string{{"_SYSTEMD_SLICE", unit}}); err != nil { return err } } - j.AddDisjunction() - return nil + return j.AddDisjunction() } -func addUserUnitMatchGroups(j cliJournal, unit, uid string, uidOK bool) error { +func addUserUnitMatchGroups(j cliJournal, unit, uid string, uidOK, full bool) error { if err := addUserUnitMatchGroup(j, [][2]string{{"_SYSTEMD_USER_UNIT", unit}}, uid, uidOK, false); err != nil { return err } - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } if err := addUserUnitMatchGroup(j, [][2]string{{"USER_UNIT", unit}}, uid, uidOK, false); err != nil { return err } - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } if err := addUserUnitMatchGroup(j, [][2]string{{"OBJECT_SYSTEMD_USER_UNIT", unit}}, uid, uidOK, true); err != nil { return err } - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } + if !full { + return nil + } if err := addUserUnitMatchGroup(j, [][2]string{{"COREDUMP_USER_UNIT", unit}}, uid, uidOK, true); err != nil { return err } if strings.HasSuffix(unit, ".slice") { - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return err + } if err := addUserUnitMatchGroup(j, [][2]string{{"_SYSTEMD_USER_SLICE", unit}}, uid, uidOK, false); err != nil { return err } } - j.AddDisjunction() - return nil + return j.AddDisjunction() } func addUserUnitMatchGroup(j cliJournal, pairs [][2]string, uid string, uidOK, includeRootUID bool) error { @@ -1248,7 +1306,7 @@ func expandUnitSpecs(j cliJournal, specs []string, fields []string) ([]string, e var patterns []string for _, spec := range specs { - unit := mangleUnitName(spec) + unit := spec if isGlobPattern(unit) { patterns = append(patterns, unit) continue @@ -1262,13 +1320,21 @@ func expandUnitSpecs(j cliJournal, specs []string, fields []string) ([]string, e if len(patterns) == 0 { return out, nil } + matcher, err := newUnitPatternMatcher(patterns) + if err != nil { + return nil, err + } + defer matcher.Close() for _, field := range fields { err := j.VisitUnique(field, func(value []byte) error { - unit := string(value) - if !matchesAnyGlob(patterns, unit) { + if end := bytes.IndexByte(value, 0); end >= 0 { + value = value[:end] + } + if !matcher.Matches(value) { return nil } + unit := string(value) if _, ok := seen[unit]; ok { return nil } @@ -1284,82 +1350,106 @@ func expandUnitSpecs(j cliJournal, specs []string, fields []string) ([]string, e return out, nil } -func mangleUnitName(value string) string { - value = strings.TrimSpace(value) - for _, suffix := range unitSuffixes { - if strings.HasSuffix(value, suffix) { - return value - } - } - return value + ".service" -} - func isGlobPattern(value string) bool { return strings.ContainsAny(value, "*?[") } -func matchesAnyGlob(patterns []string, value string) bool { - for _, pattern := range patterns { - if globPatternMatches(pattern, value) { - return true - } - } - return false -} - -func globPatternMatches(pattern, value string) bool { - re, err := regexp.Compile(globPatternToRegex(pattern)) - return err == nil && re.MatchString(value) -} - func globPatternToRegex(pattern string) string { - var b strings.Builder - b.WriteString("^") - runes := []rune(pattern) - for i := 0; i < len(runes); i++ { - switch runes[i] { + var out strings.Builder + out.WriteString("(?s)^") + chars := []rune(pattern) + for i := 0; i < len(chars); i++ { + switch chars[i] { case '*': - b.WriteString(".*") + out.WriteString(".*") case '?': - b.WriteByte('.') + out.WriteByte('.') case '[': - class := []rune{'['} - closed := false - if i+1 < len(runes) && (runes[i+1] == '!' || runes[i+1] == '^') { - i++ - class = append(class, '^') + end := i + 1 + if end < len(chars) && (chars[end] == '!' || chars[end] == '^') { + end++ } - for i+1 < len(runes) { - i++ - class = append(class, runes[i]) - if runes[i] == ']' { - closed = true - break + if end < len(chars) && chars[end] == ']' { + end++ + } + for end < len(chars) && chars[end] != ']' { + if chars[end] == '[' && end+1 < len(chars) && (chars[end+1] == ':' || chars[end+1] == '.') { + k := end + 2 + for k+1 < len(chars) && !(chars[k] == chars[end+1] && chars[k+1] == ']') { + k++ + } + if k+1 < len(chars) { + end = k + 2 + continue + } } + end++ } - if closed { - b.WriteString(string(class)) - } else { - b.WriteString(`\[`) - b.WriteString(regexp.QuoteMeta(string(class[1:]))) + if end == len(chars) { + out.WriteString(`\[`) + continue + } + out.WriteByte('[') + start := i + 1 + if chars[start] == '!' || chars[start] == '^' { + out.WriteByte('^') + start++ + } + for k := start; k < end; k++ { + if chars[k] == '[' && k+1 < end && (chars[k+1] == ':' || chars[k+1] == '.') { + marker := chars[k+1] + n := k + 2 + for n+1 < end && !(chars[n] == marker && chars[n+1] == ']') { + n++ + } + if n+1 < end { + if marker == ':' { + out.WriteString(string(chars[k : n+2])) + } else { + if n != k+3 { + return "[" + } + if strings.ContainsRune(`\[]^-`, chars[k+2]) { + out.WriteByte('\\') + } + out.WriteRune(chars[k+2]) + } + k = n + 1 + continue + } + } + if chars[k] == '\\' || chars[k] == ']' || chars[k] == '[' { + out.WriteByte('\\') + } + out.WriteRune(chars[k]) } + out.WriteByte(']') + i = end default: - b.WriteString(regexp.QuoteMeta(string(runes[i]))) + out.WriteString(regexp.QuoteMeta(string(chars[i]))) } } - b.WriteString("$") - return b.String() + out.WriteString("$") + return out.String() } type cliPostFilters struct { - grep *regexp.Regexp + grep *pcrePattern + shown bool excludeIdentifiers map[string]struct{} + captureHighlight bool + highlight matchRange + bufferedHighlights []matchRange } func newCLIPostFilters(flags *cliFlags) (*cliPostFilters, error) { - grep, err := compileGrepFilter(*flags.grepFlag, flags.caseSensitiveFlag) - if err != nil { - return nil, err + var grep *pcrePattern + if flags.grepSet { + var err error + grep, err = compileGrepFilter(*flags.grepFlag, flags.caseSensitiveFlag) + if err != nil { + return nil, err + } } filters := &cliPostFilters{grep: grep} if outputUsesExcludeIdentifier(*flags.output) { @@ -1374,35 +1464,50 @@ func newCLIPostFilters(flags *cliFlags) (*cliPostFilters, error) { return filters, nil } -func (f *cliPostFilters) matches(entry *journal.Entry) bool { - if f == nil { - return true +func (f *cliPostFilters) matches(entry *journal.Entry) (bool, error) { + if f == nil || f.grep == nil { + return true, nil } - if f.grep != nil { - matched := false - for _, value := range entryValues(entry, "MESSAGE") { - if f.grep.MatchString(string(value)) { - matched = true - break - } - } - if !matched { - return false - } + values := entryValues(entry, "MESSAGE") + if len(values) == 0 { + return false, nil } - return true + if f.captureHighlight { + return f.grep.match(values[0], &f.highlight) + } + return f.grep.Match(values[0]) +} + +func (f *cliPostFilters) bufferHighlight() { + if f != nil && f.captureHighlight && f.grep != nil { + f.bufferedHighlights = append(f.bufferedHighlights, f.highlight) + } +} + +func (f *cliPostFilters) restoreHighlight() { + if f != nil && len(f.bufferedHighlights) > 0 { + f.highlight = f.bufferedHighlights[0] + f.bufferedHighlights = f.bufferedHighlights[1:] + } +} + +func entryMatches(entry *journal.Entry, since, until *uint64, filters *cliPostFilters) (bool, error) { + if !entryInTimeRange(entry, since, until) { + return false, nil + } + return filters.matches(entry) } func (f *cliPostFilters) excludesEntry(entry *journal.Entry) bool { if f == nil || len(f.excludeIdentifiers) == 0 { return false } - for _, value := range entryValues(entry, "SYSLOG_IDENTIFIER") { - if _, found := f.excludeIdentifiers[string(value)]; found { - return true - } + value, ok := lastEntryValue(entry, "SYSLOG_IDENTIFIER") + if !ok { + return false } - return false + _, found := f.excludeIdentifiers[string(cStringBytes(value))] + return found } func outputUsesExcludeIdentifier(mode string) bool { @@ -1428,46 +1533,68 @@ func entryValues(entry *journal.Entry, field string) [][]byte { return nil } -func compileGrepFilter(pattern string, caseSensitive optionalStringFlag) (*regexp.Regexp, error) { - if pattern == "" { - return nil, nil +func parseBoolOption(option, value string) (bool, error) { + switch strings.ToLower(value) { + case "1", "true", "yes", "y", "t", "on": + return true, nil + case "0", "false", "no", "n", "f", "off": + return false, nil + default: + return false, fmt.Errorf("Bad %s= argument %q", option, value) } - sensitive := hasUppercase(pattern) - if caseSensitive.set { - parsed, err := parseBoolOption("--case-sensitive", caseSensitive.value) - if err != nil { - return nil, err - } - sensitive = parsed +} + +// Match journalctl's integer grammar, including its fixed 32-bit int range. +func parseNativeNumber(value string) (uint64, bool, error) { + value = strings.TrimLeft(value, " \t\n\r\v\f") + base := 0 + if strings.HasPrefix(value, "0b") || strings.HasPrefix(value, "0B") { + base = 2 + } else if strings.HasPrefix(value, "0o") || strings.HasPrefix(value, "0O") { + base = 8 } - if !sensitive { - pattern = "(?i)" + pattern + if base != 0 { + value = strings.TrimLeft(value[2:], " \t\n\r\v\f") } - re, err := regexp.Compile(pattern) - if err != nil { - return nil, fmt.Errorf("Bad pattern %q: %w", pattern, err) + negative := strings.HasPrefix(value, "-") + if strings.HasPrefix(value, "+") || negative { + value = value[1:] } - return re, nil + if base == 0 { + base = 10 + if strings.HasPrefix(value, "0x") || strings.HasPrefix(value, "0X") { + base, value = 16, value[2:] + } else if strings.HasPrefix(value, "0") { + base = 8 + } + } + number, err := strconv.ParseUint(value, base, 64) + return number, negative, err } -func hasUppercase(value string) bool { - for _, ch := range value { - if unicode.IsUpper(ch) { - return true - } +func parseNativeUint64(value string) (uint64, error) { + number, negative, err := parseNativeNumber(value) + if err == nil && negative && number != 0 { + err = strconv.ErrRange } - return false + return number, err } -func parseBoolOption(option, value string) (bool, error) { - switch strings.ToLower(strings.TrimSpace(value)) { - case "1", "true", "yes", "y", "t", "on": - return true, nil - case "0", "false", "no", "n", "f", "off": - return false, nil - default: - return false, fmt.Errorf("Bad %s= argument %q", option, value) +func parseNativeInt32(value string) (int, error) { + number, negative, err := parseNativeNumber(value) + if err != nil { + return 0, strconv.ErrRange + } + if negative { + if number > 1<<31 { + return 0, strconv.ErrRange + } + return int(-int64(number)), nil + } + if number > 1<<31-1 { + return 0, strconv.ErrRange } + return int(number), nil } func parsePriorityFilter(values []string) ([]uint8, error) { @@ -1506,16 +1633,16 @@ func parsePriorityFilter(values []string) ([]uint8, error) { } func parsePriorityLevel(value string) (uint8, error) { - switch strings.ToLower(strings.TrimSpace(value)) { - case "emerg", "panic": + switch value { + case "emerg": return 0, nil case "alert": return 1, nil - case "crit", "critical": + case "crit": return 2, nil - case "err", "error": + case "err": return 3, nil - case "warning", "warn": + case "warning": return 4, nil case "notice": return 5, nil @@ -1524,7 +1651,7 @@ func parsePriorityLevel(value string) (uint8, error) { case "debug": return 7, nil } - number, err := strconv.Atoi(value) + number, err := parseNativeInt32(value) if err == nil && number >= 0 && number <= 7 { return uint8(number), nil } @@ -1545,17 +1672,6 @@ var facilityHelpNames = []string{ "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7", } -func facilityHelpRequested(values []string) bool { - for _, value := range values { - for _, item := range strings.Split(value, ",") { - if strings.TrimSpace(item) == "help" { - return true - } - } - } - return false -} - func printFacilityHelp(stdout io.Writer, quiet bool) { if !quiet { fmt.Fprintln(stdout, "Available facilities:") @@ -1590,7 +1706,7 @@ func parseFacilityFilter(values []string) ([]uint8, error) { } func parseFacility(value string) (uint8, error) { - if number, err := strconv.Atoi(value); err == nil && number >= 0 && number <= 23 { + if number, err := parseNativeInt32(value); err == nil && number >= 0 && number <= 127 { return uint8(number), nil } if facility, ok := facilityNames[value]; ok { @@ -1611,49 +1727,94 @@ func validateOutputMode(mode string) error { } func (f *cliFlags) dispatch(j cliJournal, sinceUsec, untilUsec *uint64, stdout io.Writer, postFilters *cliPostFilters, cursorControl cursorControl, outputOptions outputOptions) error { - grepTailReverse := f.grepTailImpliesReverse() - if f.linesFlag.set { - limit, err := parseLinesLimitValue(f.linesFlag.value) - if err != nil { - return err - } - if limit.set && !limit.all { - if limit.oldest { - return showForward(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) - } - if limit.count == 0 { - return showTail(j, 0, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) - } - if grepTailReverse || *f.reverseFlag { - return showReverse(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) - } - return showTail(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) - } - } - - switch { - case *f.fields: + if *f.fields { fields, err := j.EnumerateFields() if err != nil { return fmt.Errorf("enumerate fields: %w", err) } - sort.Strings(fields) - for _, f := range fields { - fmt.Fprintln(stdout, f) + for _, field := range fields { + if _, err := fmt.Fprintln(stdout, field); err != nil { + return err + } } return nil - - case *f.field != "": - return j.VisitUnique(*f.field, func(value []byte) error { - if _, err := stdout.Write(value); err != nil { + } + if f.action == "field" || !f.parsed && *f.field != "" { + if !validFieldName(*f.field) { + return fmt.Errorf("Invalid field name: %s", *f.field) + } + limit := -1 + if f.linesFlag.set { + parsed, err := parseLinesLimitValue(f.linesFlag.value) + if err != nil { return err } - if _, err := fmt.Fprintln(stdout); err != nil { - return err + if !parsed.all { + limit = parsed.count } + } + if limit == 0 { return nil - }) + } + count := 0 + err := j.VisitUnique(*f.field, func(value []byte) error { + if count == limit { + return errStopIteration + } + if i := strings.IndexByte(string(value), 0); i >= 0 { + value = value[:i] + } + if _, err := stdout.Write(value); err != nil { + return err + } + if _, err := fmt.Fprintln(stdout); err != nil { + return err + } + count++ + return nil + }) + if errors.Is(err, errStopIteration) { + return nil + } + return err + } + grepTailReverse := f.grepTailImpliesReverse() + if *f.noTail { + if *f.reverseFlag { + return showReverse(j, 0, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + return showForward(j, 0, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + if f.linesFlag.set && !*f.noTail { + limit, err := parseLinesLimitValue(f.linesFlag.value) + if err != nil { + return err + } + if limit.set && !limit.all { + if limit.count == 0 { + return showZero(j, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), cursorControl, !limit.oldest || *f.reverseFlag, *f.reverseFlag || grepTailReverse) + } + if limit.oldest { + return showForward(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + if cursorControl.seek != nil && !*f.reverseFlag && !grepTailReverse { + return showForward(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + if grepTailReverse || *f.reverseFlag { + return showReverse(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + return showTail(j, limit.count, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + if limit.all { + if *f.reverseFlag { + return showReverse(j, 0, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + return showForward(j, 0, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) + } + } + + switch { case *f.head > 0: return showForward(j, *f.head, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) @@ -1665,7 +1826,7 @@ func (f *cliFlags) dispatch(j cliJournal, sinceUsec, untilUsec *uint64, stdout i default: if *f.pagerEndFlag { - if *f.reverseFlag { + if *f.reverseFlag || grepTailReverse { return showReverse(j, 1000, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) } return showTail(j, 1000, sinceUsec, untilUsec, stdout, *f.showCursorFlag, f.effectiveQuiet(), postFilters, cursorControl, outputOptions) @@ -1678,9 +1839,12 @@ func (f *cliFlags) dispatch(j cliJournal, sinceUsec, untilUsec *uint64, stdout i } func (f *cliFlags) grepTailImpliesReverse() bool { - if !f.grepSet || *f.follow || !f.linesFlag.set { + if !f.grepSet || *f.follow || *f.noTail { return false } + if !f.linesFlag.set { + return *f.pagerEndFlag + } limit, err := parseLinesLimitValue(f.linesFlag.value) if err != nil { return false @@ -1701,7 +1865,7 @@ func (f *cliFlags) effectiveQuiet() bool { } func openFilteredJournal(input cliInput, matches []string, flags *cliFlags, outputMode string, bootOverride optionalStringFlag) (cliJournal, error) { - j, err := input.openJournal() + j, err := input.openJournalWithCursor(*flags.showCursorFlag || *flags.cursorFileFlag != "" || flags.seen["cursor-file"] || *flags.cursorFlag != "" || *flags.afterCursorFlag != "" || flags.seen["cursor"] || flags.seen["after-cursor"]) if err != nil { return nil, fmt.Errorf("open journal: %w", err) } @@ -1720,6 +1884,9 @@ func openFilteredJournal(input cliInput, matches []string, flags *cliFlags, outp if err := addInvocationMatches(j, invocationID); err != nil { return nil, err } + if err := applyNonUnitMatches(j, flags); err != nil { + return nil, err + } } else { if err := applyBootMatch(j, flags, bootOverride); err != nil { return nil, err @@ -1731,7 +1898,9 @@ func openFilteredJournal(input cliInput, matches []string, flags *cliFlags, outp for _, arg := range matches { if arg == "+" { - j.AddDisjunction() + if err := j.AddDisjunction(); err != nil { + return nil, err + } continue } if strings.Contains(arg, "=") { @@ -1748,154 +1917,6 @@ func openFilteredJournal(input cliInput, matches []string, flags *cliFlags, outp return j, nil } -func preprocessOptionalArgs(args []string) []string { - out := make([]string, 0, len(args)) - for i := 0; i < len(args); i++ { - arg := args[i] - if arg == "--boot" || arg == "-b" { - if i+1 < len(args) && looksLikeBootDescriptor(args[i+1]) { - out = append(out, arg+"="+args[i+1]) - i++ - } else { - out = append(out, arg+"=0") - } - continue - } - if arg == "--lines" || arg == "-n" { - if i+1 < len(args) && looksLikeLinesDescriptor(args[i+1]) { - out = append(out, arg+"="+args[i+1]) - i++ - } else { - out = append(out, arg+"=10") - } - continue - } - out = append(out, arg) - } - return out -} - -func normalizeShortFlags(args []string, fs *flag.FlagSet) []string { - out := make([]string, 0, len(args)) - for i := 0; i < len(args); i++ { - arg := args[i] - if arg == "--" { - out = append(out, args[i:]...) - break - } - if arg == "-" || strings.HasPrefix(arg, "--") || !strings.HasPrefix(arg, "-") { - out = append(out, arg) - continue - } - body := strings.TrimPrefix(arg, "-") - for len(body) > 0 { - name := body[:1] - flg := fs.Lookup(name) - if flg == nil { - out = append(out, "-"+body) - break - } - body = body[1:] - if flagIsBool(flg) && !flagIsOptionalString(flg) { - out = append(out, "-"+name) - continue - } - if body == "" { - out = append(out, "-"+name) - } else { - out = append(out, "-"+name+"="+body) - } - break - } - } - return out -} - -func permuteFlagArgs(args []string, fs *flag.FlagSet) []string { - options := make([]string, 0, len(args)) - operands := make([]string, 0, len(args)) - for i := 0; i < len(args); i++ { - arg := args[i] - if arg == "--" { - operands = append(operands, args[i+1:]...) - break - } - name, inlineValue, isFlag := splitFlagArg(arg) - if !isFlag { - operands = append(operands, arg) - continue - } - flg := fs.Lookup(name) - if flg == nil { - options = append(options, arg) - continue - } - options = append(options, arg) - if inlineValue || flagIsBool(flg) { - continue - } - if i+1 < len(args) { - i++ - options = append(options, args[i]) - } - } - return append(options, operands...) -} - -func splitFlagArg(arg string) (string, bool, bool) { - if arg == "-" || !strings.HasPrefix(arg, "-") { - return "", false, false - } - if strings.HasPrefix(arg, "--") { - body := strings.TrimPrefix(arg, "--") - if body == "" { - return "", false, false - } - name, _, found := strings.Cut(body, "=") - return name, found, true - } - body := strings.TrimPrefix(arg, "-") - if body == "" { - return "", false, false - } - name, _, found := strings.Cut(body, "=") - return name, found, true -} - -func flagIsBool(flg *flag.Flag) bool { - boolFlag, ok := flg.Value.(interface{ IsBoolFlag() bool }) - return ok && boolFlag.IsBoolFlag() -} - -func flagIsOptionalString(flg *flag.Flag) bool { - _, ok := flg.Value.(*optionalStringFlag) - return ok -} - -func looksLikeBootDescriptor(value string) bool { - if value == "all" { - return true - } - for _, pattern := range bootDescriptorPatterns { - if pattern.MatchString(value) { - return true - } - } - return false -} - -func looksLikeLinesDescriptor(value string) bool { - if value == "all" { - return true - } - value = strings.TrimPrefix(value, "+") - if value == "" { - return false - } - n, err := strconv.Atoi(value) - return err == nil && n >= 0 -} - func parseOptionalTimestampUsec(value string) (*uint64, error) { if strings.TrimSpace(value) == "" { return nil, nil @@ -1908,131 +1929,7 @@ func parseOptionalTimestampUsec(value string) (*uint64, error) { } func parseTimestampUsec(value string) (uint64, error) { - value = strings.TrimSpace(value) - if usec, ok := parseRelativeTimestampUsec(value); ok { - return usec, nil - } - if strings.HasPrefix(value, "@") { - return parseEpochTimestampUsec(strings.TrimPrefix(value, "@")) - } - if usec, ok, err := parseAgoTimestampUsec(value); ok || err != nil { - return usec, err - } - if usec, ok, err := parseSignedDurationTimestampUsec(value); ok || err != nil { - return usec, err - } - - now := time.Now() - if usec, ok := parseDateTimestampUsec(value); ok { - return usec, nil - } - if usec, ok := parseTimeOfDayTimestampUsec(value, now); ok { - return usec, nil - } - return 0, fmt.Errorf("failed to parse timestamp: %s", value) -} - -func parseRelativeTimestampUsec(value string) (uint64, bool) { - switch value { - case "now": - return uint64(time.Now().UnixMicro()), true - case "today", "yesterday", "tomorrow": - now := time.Now() - day := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.Local) - if value == "yesterday" { - day = day.AddDate(0, 0, -1) - } else if value == "tomorrow" { - day = day.AddDate(0, 0, 1) - } - return uint64(day.UnixMicro()), true - default: - return 0, false - } -} - -func parseAgoTimestampUsec(value string) (uint64, bool, error) { - duration, ok := strings.CutSuffix(value, " ago") - if !ok { - return 0, false, nil - } - delta, err := parseDurationUsec(strings.TrimSpace(duration)) - if err != nil { - return 0, true, err - } - now := time.Now().UnixMicro() - then := now - int64(delta) - if then < 0 { - then = 0 - } - return uint64(then), true, nil -} - -func parseSignedDurationTimestampUsec(value string) (uint64, bool, error) { - if len(value) <= 1 || (value[0] != '+' && value[0] != '-') || signedDatePrefixRe.MatchString(value) { - return 0, false, nil - } - delta, err := parseDurationUsec(value[1:]) - if err != nil { - return 0, true, err - } - now := time.Now().UnixMicro() - if value[0] == '+' { - return uint64(now + int64(delta)), true, nil - } - return uint64(now - int64(delta)), true, nil -} - -func parseDateTimestampUsec(value string) (uint64, bool) { - for _, layout := range []string{ - "2006-01-02T15:04:05.999999Z07:00", - "2006-01-02T15:04:05Z07:00", - "2006-01-02T15:04Z07:00", - } { - if t, err := time.Parse(layout, value); err == nil { - return uint64(t.UnixMicro()), true - } - } - for _, layout := range []string{ - "2006-01-02T15:04:05.999999", - "2006-01-02T15:04:05", - "2006-01-02T15:04", - "2006-01-02 15:04:05.999999", - "2006-01-02 15:04:05", - "2006-01-02 15:04", - "2006-01-02", - } { - if t, err := time.ParseInLocation(layout, value, time.Local); err == nil { - return uint64(t.UnixMicro()), true - } - } - return 0, false -} - -func parseTimeOfDayTimestampUsec(value string, now time.Time) (uint64, bool) { - for _, layout := range []string{"15:04:05.999999", "15:04:05", "15:04"} { - if t, err := time.ParseInLocation(layout, value, time.Local); err == nil { - t = time.Date(now.Year(), now.Month(), now.Day(), t.Hour(), t.Minute(), t.Second(), t.Nanosecond(), time.Local) - return uint64(t.UnixMicro()), true - } - } - return 0, false -} - -func parseEpochTimestampUsec(value string) (uint64, error) { - if !epochTimestampRe.MatchString(value) { - return 0, fmt.Errorf("failed to parse timestamp: @%s", value) - } - whole, frac, _ := strings.Cut(value, ".") - seconds, err := strconv.ParseUint(whole, 10, 64) - if err != nil { - return 0, err - } - frac = (frac + "000000")[:6] - usec, err := strconv.ParseUint(frac, 10, 64) - if err != nil { - return 0, err - } - return seconds*1_000_000 + usec, nil + return new(timestampParser).parse(value) } func parseDurationUsec(value string) (uint64, error) { @@ -2044,49 +1941,14 @@ func parseDurationUsecAllowZero(value string) (uint64, error) { } func parseDurationUsecMode(value string, allowZero bool) (uint64, error) { - units := map[string]float64{ - "us": 1, "usec": 1, "usecs": 1, - "ms": 1_000, "msec": 1_000, "msecs": 1_000, - "s": 1_000_000, "sec": 1_000_000, "secs": 1_000_000, "second": 1_000_000, "seconds": 1_000_000, - "m": 60_000_000, "min": 60_000_000, "mins": 60_000_000, "minute": 60_000_000, "minutes": 60_000_000, - "h": 3_600_000_000, "hr": 3_600_000_000, "hour": 3_600_000_000, "hours": 3_600_000_000, - "d": 86_400_000_000, "day": 86_400_000_000, "days": 86_400_000_000, - "w": 604_800_000_000, "week": 604_800_000_000, "weeks": 604_800_000_000, - "month": 2_629_800_000_000, "months": 2_629_800_000_000, - "y": 31_557_600_000_000, "year": 31_557_600_000_000, "years": 31_557_600_000_000, - } - var total float64 - pos := 0 - seen := false - for _, match := range durationTokenRe.FindAllStringSubmatchIndex(value, -1) { - if match[0] != pos { - return 0, fmt.Errorf("failed to parse duration: %s", value) - } - seen = true - number, err := strconv.ParseFloat(value[match[2]:match[3]], 64) - if err != nil { - return 0, err - } - unit := "s" - if match[4] >= 0 { - unit = value[match[4]:match[5]] - } - if unit == "M" { - total += number * 2_629_800_000_000 - pos = match[1] - continue - } - multiplier, ok := units[strings.ToLower(unit)] - if !ok { - return 0, fmt.Errorf("failed to parse duration: %s", value) - } - total += number * multiplier - pos = match[1] + result, err := durationValue(value) + if err != nil { + return 0, err } - if pos != len(value) || !seen || (!allowZero && total == 0) { + if !allowZero && result == 0 { return 0, fmt.Errorf("failed to parse duration: %s", value) } - return uint64(total), nil + return result, nil } type bootInfo struct { @@ -2106,15 +1968,22 @@ func runListBoots(input cliInput, flags *cliFlags, stdout io.Writer) error { if err != nil { return fmt.Errorf("list boots: %w", err) } - if len(boots) == 0 { - return errors.New("No boot found.") - } display, err := selectBootRows(boots, flags) if err != nil { return err } + if len(display) == 0 { + return noDataError(flags.effectiveQuiet(), "No boot found.") + } + if jsonTableMode(*flags.output) { + rows := make([]map[string]any, 0, len(display)) + for _, row := range display { + rows = append(rows, map[string]any{"index": row.index, "boot_id": row.bootID, "first_entry": tableTimestamp(row.firstEntry), "last_entry": tableTimestamp(row.lastEntry)}) + } + return writeTableJSON(stdout, *flags.output, rows) + } if !*flags.quietFlag { - fmt.Fprintln(stdout, "IDX BOOT ID FIRST ENTRY LAST ENTRY") + fmt.Fprint(stdout, detectOutputColors().tableHeader("BOOT ID")) } idxWidth := 1 if *flags.quietFlag { @@ -2209,7 +2078,9 @@ func collectBootsIndexed(j cliJournal) ([]bootInfo, error) { if err := addMatchPair(j, "_BOOT_ID", bootID); err != nil { return nil, err } - j.AddConjunction() + if err := j.AddConjunction(); err != nil { + return nil, err + } if err := j.SeekHead(); err != nil { return nil, err @@ -2364,18 +2235,24 @@ func parseBootDescriptor(descriptor string) (string, int, error) { if descriptor == "" { return "", 0, errors.New("failed to parse boot descriptor: ") } - m := bootDescriptorRe.FindStringSubmatch(descriptor) - if m == nil { - return "", 0, fmt.Errorf("failed to parse boot descriptor: %s", descriptor) - } - bootID := strings.ToLower(strings.ReplaceAll(m[1], "-", "")) - offset := 0 - if m[4] != "" { - var err error - offset, err = strconv.Atoi(m[4]) - if err != nil { - return "", 0, err + bootID, rest := "", descriptor + if len(descriptor) >= 32 { + if id, err := journal.ParseUUID(descriptor[:32]); err == nil { + if id != (journal.UUID{}) { + bootID = id.String() + } + rest = descriptor[32:] } + if rest != "" && rest[0] != '+' && rest[0] != '-' { + return "", 0, fmt.Errorf("failed to parse boot descriptor: %s", descriptor) + } + } + if rest == "" { + return bootID, 0, nil + } + offset, err := parseNativeInt32(rest) + if err != nil { + return "", 0, fmt.Errorf("failed to parse boot descriptor: %s", descriptor) } return bootID, offset, nil } @@ -2412,221 +2289,6 @@ func parseInvocationDescriptor(descriptor string) (string, int, bool, error) { return id, offset, true, nil } -func resolveInvocationFilter(input cliInput, flags *cliFlags) (string, bool, error) { - descriptor, set := effectiveInvocationDescriptor(flags) - if !set { - return "", false, nil - } - id, offset, active, err := parseInvocationDescriptor(descriptor) - if err != nil { - return "", false, err - } - if !active { - return "", false, nil - } - - invocationUnitOption := "" - if id == "" || offset != 0 { - invocationUnitOption = "-I/--invocation= with an offset" - } - - infos, err := collectInvocationsFromInput(input, flags, invocationUnitOption) - if err != nil { - return "", false, err - } - target := -1 - if id != "" { - for i, info := range infos { - if info.id == id { - target = i + offset - break - } - } - } else if offset > 0 { - target = offset - 1 - } else { - target = len(infos) - 1 + offset - } - if target < 0 || target >= len(infos) { - return "", false, fmt.Errorf("No journal entry found for the invocation (%s%+d).", id, offset) - } - return infos[target].id, true, nil -} - -func applySingleInvocationUnit(j cliJournal, flags *cliFlags, optionName string) error { - systemUnits, userUnits, err := singleInvocationUnit(j, flags, optionName) - if err != nil { - return err - } - return addJournalctlUnitMatches(j, systemUnits, userUnits) -} - -func singleInvocationUnit(j cliJournal, flags *cliFlags, optionName string) ([]string, []string, error) { - systemSpecs, userSpecs := effectiveUnitSpecs(flags) - count := len(systemSpecs) + len(userSpecs) - if count == 0 { - return nil, nil, fmt.Errorf("Using %s requires a unit. Please specify a unit name with -u/--unit=/--user-unit=.", optionName) - } - if count > 1 { - return nil, nil, fmt.Errorf("Using %s with multiple units is not supported.", optionName) - } - if len(systemSpecs) == 1 { - units, err := expandUnitSpecs(j, systemSpecs, systemUnitFieldsFull) - if err != nil { - return nil, nil, err - } - if len(units) == 0 { - return nil, nil, fmt.Errorf("No matching unit found for '%s' in journal.", mangleUnitName(systemSpecs[0])) - } - if len(units) > 1 { - return nil, nil, fmt.Errorf("Multiple matching units found for '%s' in journal.", mangleUnitName(systemSpecs[0])) - } - return units, nil, nil - } - units, err := expandUnitSpecs(j, userSpecs, userUnitFieldsFull) - if err != nil { - return nil, nil, err - } - if len(units) == 0 { - return nil, nil, fmt.Errorf("No matching unit found for '%s' in journal.", mangleUnitName(userSpecs[0])) - } - if len(units) > 1 { - return nil, nil, fmt.Errorf("Multiple matching units found for '%s' in journal.", mangleUnitName(userSpecs[0])) - } - return nil, units, nil -} - -func collectInvocationsFromInput(input cliInput, flags *cliFlags, unitOption string) ([]invocationInfo, error) { - indexedJournal, err := openInvocationScopeJournal(input, flags, unitOption) - if err != nil { - return nil, err - } - candidateIDs, err := collectInvocationCandidateIDs(indexedJournal) - closeErr := indexedJournal.Close() - if closeErr != nil && err == nil { - return nil, closeErr - } - if err != nil { - return nil, err - } - - out := make([]invocationInfo, 0, len(candidateIDs)) - for _, id := range candidateIDs { - info, ok, err := invocationBoundsForID(input, flags, unitOption, id) - if err != nil { - return nil, err - } - if ok { - out = append(out, info) - } - } - sort.Slice(out, func(i, k int) bool { - if out[i].firstUsec != out[k].firstUsec { - return out[i].firstUsec < out[k].firstUsec - } - return out[i].id < out[k].id - }) - return out, nil -} - -func openInvocationScopeJournal(input cliInput, flags *cliFlags, unitOption string) (cliJournal, error) { - j, err := input.openJournal() - if err != nil { - return nil, fmt.Errorf("open journal: %w", err) - } - if err := applyBootMatch(j, flags, optionalStringFlag{}); err != nil { - j.Close() - return nil, err - } - if unitOption != "" { - if err := applySingleInvocationUnit(j, flags, unitOption); err != nil { - j.Close() - return nil, err - } - } - return j, nil -} - -func collectInvocationCandidateIDs(j cliJournal) ([]string, error) { - seen := make(map[string]struct{}) - var out []string - for _, field := range invocationIDFields { - if err := j.VisitUnique(field, func(value []byte) error { - id, err := journal.ParseUUID(string(value)) - if err != nil { - return nil - } - text := id.String() - if strings.Trim(text, "0") == "" { - return nil - } - if _, exists := seen[text]; exists { - return nil - } - seen[text] = struct{}{} - out = append(out, text) - return nil - }); err != nil { - return nil, fmt.Errorf("visit unique invocation values for %s: %w", field, err) - } - } - return out, nil -} - -func invocationBoundsForID(input cliInput, flags *cliFlags, unitOption, id string) (invocationInfo, bool, error) { - firstJournal, err := openInvocationScopeJournal(input, flags, unitOption) - if err != nil { - return invocationInfo{}, false, err - } - defer firstJournal.Close() - if err := addInvocationMatches(firstJournal, id); err != nil { - return invocationInfo{}, false, err - } - if err := firstJournal.SeekHead(); err != nil { - return invocationInfo{}, false, err - } - ok, err := firstJournal.Next() - if err != nil { - return invocationInfo{}, false, err - } - if ok == 0 { - return invocationInfo{}, false, nil - } - firstEntry, err := firstJournal.GetEntry() - if err != nil { - return invocationInfo{}, false, err - } - - lastJournal, err := openInvocationScopeJournal(input, flags, unitOption) - if err != nil { - return invocationInfo{}, false, err - } - defer lastJournal.Close() - if err := addInvocationMatches(lastJournal, id); err != nil { - return invocationInfo{}, false, err - } - if err := lastJournal.SeekTail(); err != nil { - return invocationInfo{}, false, err - } - ok, err = lastJournal.Previous() - if err != nil { - return invocationInfo{}, false, err - } - if ok == 0 { - return invocationInfo{}, false, nil - } - lastEntry, err := lastJournal.GetEntry() - if err != nil { - return invocationInfo{}, false, err - } - - return invocationInfo{ - id: id, - firstUsec: firstEntry.Realtime, - lastUsec: lastEntry.Realtime, - }, true, nil -} - func entryInTimeRange(entry *journal.Entry, sinceUsec, untilUsec *uint64) bool { if sinceUsec != nil && entry.Realtime < *sinceUsec { return false @@ -2637,16 +2299,19 @@ func entryInTimeRange(entry *journal.Entry, sinceUsec, untilUsec *uint64) bool { return true } -func nextMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFilters *cliPostFilters, cursorSeek *cursorSeek, fn func(*journal.Entry) error) error { +func nextEntries(j cliJournal, sinceUsec, untilUsec *uint64, cursorSeek *cursorSeek, checkpoint *string, fn func(*journal.Entry) error) error { if cursorSeek != nil { - entry, ok, err := seekCursorStart(j, cursorSeek, false) + entry, ok, err := seekCursorStartTracked(j, cursorSeek, false, checkpoint) if err != nil { return err } if !ok { return nil } - if entryInTimeRange(entry, sinceUsec, untilUsec) && postFilters.matches(entry) { + if checkpoint != nil { + *checkpoint = entry.Cursor + } + if entryInTimeRange(entry, sinceUsec, untilUsec) { if err := fn(entry); err != nil { return err } @@ -2670,10 +2335,13 @@ func nextMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFilters if err != nil { return err } + if checkpoint != nil { + *checkpoint = entry.Cursor + } if untilUsec != nil && entry.Realtime > *untilUsec { return nil } - if entryInTimeRange(entry, sinceUsec, untilUsec) && postFilters.matches(entry) { + if entryInTimeRange(entry, sinceUsec, untilUsec) { if err := fn(entry); err != nil { return err } @@ -2684,17 +2352,24 @@ func nextMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFilters func showForward(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout io.Writer, showCursor bool, quiet bool, postFilters *cliPostFilters, cursorControl cursorControl, outputOptions outputOptions) error { count := 0 var lastCursor string + var checkpoint *string + if showCursor || cursorControl.updateFile != nil { + checkpoint = &lastCursor + } renderer := newOutputRenderer(outputOptions) bootSeparators := newBootSeparatorState(outputOptions.mode, quiet, outputOptions.merge) - err := nextMatchingEntries(j, sinceUsec, untilUsec, postFilters, cursorControl.seek, func(entry *journal.Entry) error { - if limit > 0 && count >= limit { - return errStopIteration - } - if err := renderEntry(stdout, renderer, &bootSeparators, postFilters, entry); err != nil { + err := nextEntries(j, sinceUsec, untilUsec, cursorControl.seek, checkpoint, func(entry *journal.Entry) error { + matched, err := renderMatchingEntry(stdout, renderer, &bootSeparators, postFilters, entry) + if err != nil { return err } - lastCursor = entry.Cursor + if !matched { + return nil + } count++ + if limit > 0 && count >= limit { + return errStopIteration + } return nil }) if errors.Is(err, errStopIteration) { @@ -2707,24 +2382,72 @@ func showForward(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout i return finishCursorOutput(stdout, showCursor, cursorControl.updateFile, lastCursor) } +func initialCursor(j cliJournal, sinceUsec, untilUsec *uint64, seek *cursorSeek, fromTail, reverse bool) (string, error) { + var cursor string + if seek != nil { + entry, ok, err := seekCursorStartTracked(j, seek, reverse, &cursor) + if ok { + cursor = entry.Cursor + } + return cursor, err + } + var err error + if fromTail { + if untilUsec != nil { + err = j.SeekRealtimeUsec(*untilUsec) + } else { + err = j.SeekTail() + } + } else if sinceUsec != nil { + err = j.SeekRealtimeUsec(*sinceUsec) + } else { + err = j.SeekHead() + } + if err != nil { + return "", err + } + var n int + if fromTail { + n, err = j.Previous() + } else { + n, err = j.Next() + } + if err != nil || n == 0 { + return "", err + } + return currentJournalCursor(j) +} + +func showZero(j cliJournal, sinceUsec, untilUsec *uint64, stdout io.Writer, showCursor, quiet bool, control cursorControl, fromTail, reverse bool) error { + var cursor string + if showCursor || control.updateFile != nil || control.seek != nil { + var err error + cursor, err = initialCursor(j, sinceUsec, untilUsec, control.seek, fromTail, reverse) + if err != nil { + return err + } + } + printNoEntries(stdout, 0, quiet) + return finishCursorOutput(stdout, showCursor, control.updateFile, cursor) +} + func showTail(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout io.Writer, showCursor bool, quiet bool, postFilters *cliPostFilters, cursorControl cursorControl, outputOptions outputOptions) error { if limit == 0 { - if cursorControl.seek != nil { - if _, _, err := seekCursorStart(j, cursorControl.seek, false); err != nil { - return err - } - } - printNoEntries(stdout, 0, quiet) - return finishCursorOutput(stdout, showCursor, cursorControl.updateFile, "") + return showZero(j, sinceUsec, untilUsec, stdout, showCursor, quiet, cursorControl, true, false) + } + var lastCursor string + var checkpoint *string + if showCursor || cursorControl.updateFile != nil { + checkpoint = &lastCursor } var entries []*journal.Entry start := 0 if cursorControl.seek == nil { - err := previousMatchingEntries(j, sinceUsec, untilUsec, postFilters, nil, func(entry *journal.Entry) error { + err := previousEntries(j, sinceUsec, untilUsec, nil, func(entry *journal.Entry) error { + entries = append(entries, entry) if len(entries) >= limit { return errStopIteration } - entries = append(entries, entry) return nil }) if errors.Is(err, errStopIteration) { @@ -2734,7 +2457,7 @@ func showTail(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout io.W return err } } else { - if err := nextMatchingEntries(j, sinceUsec, untilUsec, postFilters, cursorControl.seek, func(entry *journal.Entry) error { + if err := nextEntries(j, sinceUsec, untilUsec, cursorControl.seek, checkpoint, func(entry *journal.Entry) error { if len(entries) < limit { entries = append(entries, entry) return nil @@ -2746,16 +2469,20 @@ func showTail(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout io.W return err } } - var lastCursor string renderer := newOutputRenderer(outputOptions) bootSeparators := newBootSeparatorState(outputOptions.mode, quiet, outputOptions.merge) + count := 0 if cursorControl.seek == nil { for i := len(entries) - 1; i >= 0; i-- { entry := entries[i] - if err := renderEntry(stdout, renderer, &bootSeparators, postFilters, entry); err != nil { + matched, err := renderMatchingEntry(stdout, renderer, &bootSeparators, postFilters, entry) + if err != nil { return err } lastCursor = entry.Cursor + if matched { + count++ + } } } else { for i := 0; i < len(entries); i++ { @@ -2764,13 +2491,33 @@ func showTail(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout io.W idx = (start + i) % limit } entry := entries[idx] - if err := renderEntry(stdout, renderer, &bootSeparators, postFilters, entry); err != nil { + matched, err := renderMatchingEntry(stdout, renderer, &bootSeparators, postFilters, entry) + if err != nil { return err } - lastCursor = entry.Cursor + if matched { + count++ + } } } - printNoEntries(stdout, len(entries), quiet) + if cursorControl.seek == nil && untilUsec != nil && len(entries) < limit && (showCursor || cursorControl.updateFile != nil) { + if len(entries) > 0 { + if err := j.SeekCursorDirection(lastCursor, journal.DirectionForward); err != nil { + return err + } + } else if err := j.SeekRealtimeUsec(*untilUsec); err != nil { + return err + } + if n, err := j.Next(); err != nil { + return err + } else if n > 0 { + lastCursor, err = currentJournalCursor(j) + if err != nil { + return err + } + } + } + printNoEntries(stdout, count, quiet) return finishCursorOutput(stdout, showCursor, cursorControl.updateFile, lastCursor) } @@ -2779,19 +2526,19 @@ func showReverse(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout i return streamPreviousOutputEvents(j, sinceUsec, untilUsec, postFilters, cursorControl.seek, stdout, showCursor, quiet, cursorControl, outputOptions) } renderer := newOutputRenderer(outputOptions) - events, count, lastCursor, err := previousOutputEvents(j, sinceUsec, untilUsec, postFilters, cursorControl.seek, limit, outputOptions, quiet) + events, count, lastCursor, err := previousOutputEvents(j, sinceUsec, untilUsec, postFilters, cursorControl.seek, limit, outputOptions, quiet, showCursor || cursorControl.updateFile != nil) if err != nil { return err } - disabledBootSeparators := bootSeparatorState{} for _, event := range events { if event.bootID != "" { - if _, err := fmt.Fprintf(stdout, "-- Boot %s --\n", event.bootID); err != nil { + if _, err := fmt.Fprint(stdout, outputOptions.colors.boot(event.bootID)); err != nil { return err } continue } - if err := renderEntry(stdout, renderer, &disabledBootSeparators, postFilters, event.entry); err != nil { + postFilters.restoreHighlight() + if err := renderEntry(stdout, renderer, postFilters, event.entry); err != nil { return err } } @@ -2801,31 +2548,31 @@ func showReverse(j cliJournal, limit int, sinceUsec, untilUsec *uint64, stdout i func streamPreviousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, postFilters *cliPostFilters, cursorSeek *cursorSeek, stdout io.Writer, showCursor bool, quiet bool, cursorControl cursorControl, outputOptions outputOptions) error { renderer := newOutputRenderer(outputOptions) - disabledBootSeparators := bootSeparatorState{} bootContext := newBootSeparatorState(outputOptions.mode, quiet, outputOptions.merge) - var pendingBoot string var count int var lastCursor string visit := func(entry *journal.Entry) (bool, error) { + lastCursor = entry.Cursor if sinceUsec != nil && entry.Realtime < *sinceUsec { return false, nil } if bootContext.enabled { bootID := entry.BootID.String() if bootContext.previousBoot != "" && bootContext.previousBoot != bootID { - pendingBoot = bootID - } - bootContext.previousBoot = bootID - } - if entryInTimeRange(entry, sinceUsec, untilUsec) && postFilters.matches(entry) { - if pendingBoot != "" { - if _, err := fmt.Fprintf(stdout, "-- Boot %s --\n", pendingBoot); err != nil { + if _, err := fmt.Fprint(stdout, outputOptions.colors.boot(bootID)); err != nil { return false, err } - pendingBoot = "" } - if err := renderEntry(stdout, renderer, &disabledBootSeparators, postFilters, entry); err != nil { + bootContext.previousBoot = bootID + } + matched, matchErr := entryMatches(entry, sinceUsec, untilUsec, postFilters) + if matchErr != nil { + err := matchErr + return false, err + } + if matched { + if err := renderEntry(stdout, renderer, postFilters, entry); err != nil { return false, err } lastCursor = entry.Cursor @@ -2835,7 +2582,11 @@ func streamPreviousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, post } if cursorSeek != nil { - entry, ok, err := seekCursorStart(j, cursorSeek, true) + var checkpoint *string + if showCursor || cursorControl.updateFile != nil { + checkpoint = &lastCursor + } + entry, ok, err := seekCursorStartTracked(j, cursorSeek, true, checkpoint) if err != nil { return err } @@ -2879,11 +2630,6 @@ func streamPreviousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, post break } } - if pendingBoot != "" && count > 0 { - if _, err := fmt.Fprintf(stdout, "-- Boot %s --\n", pendingBoot); err != nil { - return err - } - } printNoEntries(stdout, count, quiet) return finishCursorOutput(stdout, showCursor, cursorControl.updateFile, lastCursor) } @@ -2893,30 +2639,32 @@ type reverseOutputEvent struct { entry *journal.Entry } -func previousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, postFilters *cliPostFilters, cursorSeek *cursorSeek, limit int, outputOptions outputOptions, quiet bool) ([]reverseOutputEvent, int, string, error) { +func previousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, postFilters *cliPostFilters, cursorSeek *cursorSeek, limit int, outputOptions outputOptions, quiet, captureCursor bool) ([]reverseOutputEvent, int, string, error) { bootContext := newBootSeparatorState(outputOptions.mode, quiet, outputOptions.merge) var events []reverseOutputEvent - var pendingBoot string var count int var lastCursor string visit := func(entry *journal.Entry) (bool, error) { + lastCursor = entry.Cursor if sinceUsec != nil && entry.Realtime < *sinceUsec { return false, nil } if bootContext.enabled { bootID := entry.BootID.String() if bootContext.previousBoot != "" && bootContext.previousBoot != bootID { - pendingBoot = bootID + events = append(events, reverseOutputEvent{bootID: bootID}) } bootContext.previousBoot = bootID } - if entryInTimeRange(entry, sinceUsec, untilUsec) && postFilters.matches(entry) { - if pendingBoot != "" { - events = append(events, reverseOutputEvent{bootID: pendingBoot}) - pendingBoot = "" - } + matched, matchErr := entryMatches(entry, sinceUsec, untilUsec, postFilters) + if matchErr != nil { + err := matchErr + return false, err + } + if matched { events = append(events, reverseOutputEvent{entry: entry}) + postFilters.bufferHighlight() lastCursor = entry.Cursor count++ if limit > 0 && count >= limit { @@ -2927,7 +2675,11 @@ func previousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, postFilter } if cursorSeek != nil { - entry, ok, err := seekCursorStart(j, cursorSeek, true) + var checkpoint *string + if captureCursor { + checkpoint = &lastCursor + } + entry, ok, err := seekCursorStartTracked(j, cursorSeek, true, checkpoint) if err != nil || !ok { return events, count, lastCursor, err } @@ -2963,9 +2715,6 @@ func previousOutputEvents(j cliJournal, sinceUsec, untilUsec *uint64, postFilter return events, count, lastCursor, nil } } - if pendingBoot != "" && count > 0 { - events = append(events, reverseOutputEvent{bootID: pendingBoot}) - } return events, count, lastCursor, nil } @@ -2975,7 +2724,7 @@ func printNoEntries(stdout io.Writer, count int, quiet bool) { } } -func previousMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFilters *cliPostFilters, cursorSeek *cursorSeek, fn func(*journal.Entry) error) error { +func previousEntries(j cliJournal, sinceUsec, untilUsec *uint64, cursorSeek *cursorSeek, fn func(*journal.Entry) error) error { if cursorSeek != nil { entry, ok, err := seekCursorStart(j, cursorSeek, true) if err != nil { @@ -2984,7 +2733,7 @@ func previousMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFil if !ok { return nil } - if entryInTimeRange(entry, sinceUsec, untilUsec) && postFilters.matches(entry) { + if entryInTimeRange(entry, sinceUsec, untilUsec) { if err := fn(entry); err != nil { return err } @@ -3011,7 +2760,7 @@ func previousMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFil if sinceUsec != nil && entry.Realtime < *sinceUsec { return nil } - if entryInTimeRange(entry, sinceUsec, untilUsec) && postFilters.matches(entry) { + if entryInTimeRange(entry, sinceUsec, untilUsec) { if err := fn(entry); err != nil { return err } @@ -3020,7 +2769,15 @@ func previousMatchingEntries(j cliJournal, sinceUsec, untilUsec *uint64, postFil } func seekCursorStart(j cliJournal, seek *cursorSeek, reverse bool) (*journal.Entry, bool, error) { - if err := j.SeekCursor(seek.cursor); err != nil { + return seekCursorStartTracked(j, seek, reverse, nil) +} + +func seekCursorStartTracked(j cliJournal, seek *cursorSeek, reverse bool, checkpoint *string) (*journal.Entry, bool, error) { + direction := journal.DirectionForward + if reverse { + direction = journal.DirectionBackward + } + if err := j.SeekCursorDirection(seek.cursor, direction); err != nil { return nil, false, fmt.Errorf("seek cursor: %w", err) } if seek.after { @@ -3032,6 +2789,15 @@ func seekCursorStart(j cliJournal, seek *cursorSeek, reverse bool) (*journal.Ent return nil, false, fmt.Errorf("test cursor: %w", err) } if match { + if checkpoint != nil { + cursor, err := currentJournalCursor(j) + if err == nil { + *checkpoint = cursor + } else if !errors.Is(err, journal.ErrNoEntry) && !errors.Is(err, journal.ErrEndOfEntries) { + return nil, false, err + } + } + var n int var err error if reverse { @@ -3053,12 +2819,15 @@ func seekCursorStart(j cliJournal, seek *cursorSeek, reverse bool) (*journal.Ent if errors.Is(err, journal.ErrNoEntry) || errors.Is(err, journal.ErrEndOfEntries) { return nil, false, nil } + if _, positionErr := currentJournalCursor(j); errors.Is(positionErr, journal.ErrNoEntry) || errors.Is(positionErr, journal.ErrEndOfEntries) { + return nil, false, nil + } return nil, false, err } return entry, true, nil } -func finishCursorOutput(stdout io.Writer, showCursor bool, cursorFile, cursor string) error { +func finishCursorOutput(stdout io.Writer, showCursor bool, cursorFile *string, cursor string) error { if cursor == "" { return nil } @@ -3067,17 +2836,17 @@ func finishCursorOutput(stdout io.Writer, showCursor bool, cursorFile, cursor st return err } } - if cursorFile != "" { - return writeCursorFileAtomic(cursorFile, cursor) + if cursorFile != nil { + return writeCursorFileAtomic(*cursorFile, cursor) } return nil } func updateCursorFile(control cursorControl, cursor string) error { - if control.updateFile == "" || cursor == "" { + if control.updateFile == nil || cursor == "" { return nil } - return writeCursorFileAtomic(control.updateFile, cursor) + return writeCursorFileAtomic(*control.updateFile, cursor) } func writeCursorFileAtomic(path, cursor string) error { @@ -3116,6 +2885,7 @@ func writeCursorFileAtomic(path, cursor string) error { type bootSeparatorState struct { enabled bool previousBoot string + colors outputColors } func newBootSeparatorState(mode string, quiet bool, suppressed bool) bootSeparatorState { @@ -3132,27 +2902,55 @@ func outputEmitsBootSeparators(mode string) bool { } } -func renderEntry(stdout io.Writer, renderer *outputRenderer, bootSeparators *bootSeparatorState, postFilters *cliPostFilters, entry *journal.Entry) error { +func (bootSeparators *bootSeparatorState) beforeEntry(stdout io.Writer, entry *journal.Entry) error { if bootSeparators.enabled { bootID := entry.BootID.String() if bootSeparators.previousBoot != "" && bootSeparators.previousBoot != bootID { - if _, err := fmt.Fprintf(stdout, "-- Boot %s --\n", bootID); err != nil { + if _, err := fmt.Fprint(stdout, bootSeparators.colors.boot(bootID)); err != nil { return err } } bootSeparators.previousBoot = bootID } + return nil +} + +func renderMatchingEntry(stdout io.Writer, renderer *outputRenderer, boots *bootSeparatorState, postFilters *cliPostFilters, entry *journal.Entry) (bool, error) { + boots.colors = renderer.options.colors + if err := boots.beforeEntry(stdout, entry); err != nil { + return false, err + } + matched, err := postFilters.matches(entry) + if err != nil || !matched { + return false, err + } + return true, renderEntry(stdout, renderer, postFilters, entry) +} + +func renderEntry(stdout io.Writer, renderer *outputRenderer, postFilters *cliPostFilters, entry *journal.Entry) error { if postFilters.excludesEntry(entry) { + if postFilters != nil { + postFilters.shown = true + } return nil } if outputSkipsMissingMessage(renderer.options.mode) && len(entryValues(entry, "MESSAGE")) == 0 { + if postFilters != nil { + postFilters.shown = true + } return nil } + if postFilters != nil && postFilters.captureHighlight { + renderer.highlight = postFilters.highlight + } out, err := renderer.render(entry) if err != nil { return err } _, err = fmt.Fprint(stdout, out) + if err == nil && postFilters != nil { + postFilters.shown = true + } return err } @@ -3176,28 +2974,34 @@ func resolveFollowBootFlag(input cliInput, flags *cliFlags) (optionalStringFlag, return optionalStringFlag{set: true, value: bootID}, nil } -func forEachFollowEntry(input cliInput, matches []string, flags *cliFlags, outputMode string, bootOverride optionalStringFlag, sinceUsec, untilUsec *uint64, postFilters *cliPostFilters, cursorSeek *cursorSeek, fn func(*journal.Entry) error) error { +func forEachFollowEntry(input cliInput, matches []string, flags *cliFlags, outputMode string, bootOverride optionalStringFlag, sinceUsec, untilUsec *uint64, cursorSeek *cursorSeek, fn func(*journal.Entry) error) error { j, err := openFilteredJournal(input, matches, flags, outputMode, bootOverride) if err != nil { return err } defer j.Close() - return nextMatchingEntries(j, sinceUsec, untilUsec, postFilters, cursorSeek, fn) + return nextEntries(j, sinceUsec, untilUsec, cursorSeek, nil, fn) } func runFollow(input cliInput, matches []string, flags *cliFlags, outputMode string, sinceUsec, untilUsec *uint64, tail int, noTail bool, stdout io.Writer, postFilters *cliPostFilters, cursorControl cursorControl, outputOptions outputOptions) error { var lastSeenCursor string + var lastVisitedCursor string renderer := newOutputRenderer(outputOptions) bootSeparators := newBootSeparatorState(outputOptions.mode, flags.effectiveQuiet(), outputOptions.merge) bootOverride, err := resolveFollowBootFlag(input, flags) if err != nil { return err } - if noTail { - if err := forEachFollowEntry(input, matches, flags, outputMode, bootOverride, sinceUsec, untilUsec, postFilters, cursorControl.seek, func(entry *journal.Entry) error { - if err := renderEntry(stdout, renderer, &bootSeparators, postFilters, entry); err != nil { + if noTail || cursorControl.seek != nil { + if err := forEachFollowEntry(input, matches, flags, outputMode, bootOverride, sinceUsec, untilUsec, cursorControl.seek, func(entry *journal.Entry) error { + lastVisitedCursor = entry.Cursor + matched, err := renderMatchingEntry(stdout, renderer, &bootSeparators, postFilters, entry) + if err != nil { return err } + if !matched { + return nil + } if entry.Cursor != "" { lastSeenCursor = entry.Cursor return updateCursorFile(cursorControl, lastSeenCursor) @@ -3209,9 +3013,9 @@ func runFollow(input cliInput, matches []string, flags *cliFlags, outputMode str } else { var selected []*journal.Entry start := 0 - if err := forEachFollowEntry(input, matches, flags, outputMode, bootOverride, sinceUsec, untilUsec, postFilters, cursorControl.seek, func(entry *journal.Entry) error { + if err := forEachFollowEntry(input, matches, flags, outputMode, bootOverride, sinceUsec, untilUsec, cursorControl.seek, func(entry *journal.Entry) error { if entry.Cursor != "" { - lastSeenCursor = entry.Cursor + lastVisitedCursor = entry.Cursor } if tail == 0 { return nil @@ -3231,27 +3035,42 @@ func runFollow(input cliInput, matches []string, flags *cliFlags, outputMode str if len(selected) == tail { idx = (start + i) % tail } - if err := renderEntry(stdout, renderer, &bootSeparators, postFilters, selected[idx]); err != nil { + matched, err := renderMatchingEntry(stdout, renderer, &bootSeparators, postFilters, selected[idx]) + if err != nil { return err } + if matched { + lastSeenCursor = selected[idx].Cursor + } } if err := updateCursorFile(cursorControl, lastSeenCursor); err != nil { return err } } + if cursorControl.updateFile != nil && lastVisitedCursor != "" && lastVisitedCursor != lastSeenCursor { + if err := updateCursorFile(cursorControl, lastVisitedCursor); err != nil { + return err + } + lastSeenCursor = lastVisitedCursor + } pollInterval := 100 * time.Millisecond const maxPollInterval = time.Second for { time.Sleep(pollInterval) seek := cursorControl.seek - if lastSeenCursor != "" { - seek = &cursorSeek{cursor: lastSeenCursor, after: true} + if lastVisitedCursor != "" { + seek = &cursorSeek{cursor: lastVisitedCursor, after: true} } emitted := 0 - if err := forEachFollowEntry(input, matches, flags, outputMode, bootOverride, sinceUsec, untilUsec, postFilters, seek, func(entry *journal.Entry) error { - if err := renderEntry(stdout, renderer, &bootSeparators, postFilters, entry); err != nil { + if err := forEachFollowEntry(input, matches, flags, outputMode, bootOverride, sinceUsec, untilUsec, seek, func(entry *journal.Entry) error { + lastVisitedCursor = entry.Cursor + matched, err := renderMatchingEntry(stdout, renderer, &bootSeparators, postFilters, entry) + if err != nil { return err } + if !matched { + return nil + } emitted++ if entry.Cursor != "" { lastSeenCursor = entry.Cursor @@ -3261,6 +3080,12 @@ func runFollow(input cliInput, matches []string, flags *cliFlags, outputMode str }); err != nil { return err } + if cursorControl.updateFile != nil && lastVisitedCursor != "" && lastVisitedCursor != lastSeenCursor { + if err := updateCursorFile(cursorControl, lastVisitedCursor); err != nil { + return err + } + lastSeenCursor = lastVisitedCursor + } if emitted > 0 { pollInterval = 100 * time.Millisecond } else { @@ -3272,7 +3097,7 @@ func runFollow(input cliInput, matches []string, flags *cliFlags, outputMode str } } -func runVerify(input cliInput, verifyKey string, hasVerifyKey bool, stdout, stderr io.Writer) error { +func runVerify(input cliInput, verifyKey string, hasVerifyKey, quiet bool, stdout, stderr io.Writer) error { if err := validateVerificationKeyOption(verifyKey, hasVerifyKey, stderr); err != nil { return err } @@ -3290,7 +3115,7 @@ func runVerify(input cliInput, verifyKey string, hasVerifyKey bool, stdout, stde var firstErr error for _, path := range files { - if err := verifyOneFile(path, verifyKey, hasVerifyKey, directoryInput, stderr); err != nil { + if err := verifyOneFile(path, verifyKey, hasVerifyKey, directoryInput, quiet, stderr); err != nil { if firstErr == nil { firstErr = err } @@ -3368,6 +3193,9 @@ type vacuumCandidate struct { } func hasVacuumFlags(f *cliFlags) bool { + if f.parsed { + return f.action == actionVacuum || f.action == actionRotateVacuum + } return *f.vacuumSizeFlag != "" || *f.vacuumFilesFlag != "" || *f.vacuumTimeFlag != "" } @@ -3399,7 +3227,7 @@ func parseVacuumOptions(flags *cliFlags) (vacuumOptions, error) { opts.maxUse = value } if *flags.vacuumFilesFlag != "" { - value, err := strconv.ParseUint(strings.TrimSpace(*flags.vacuumFilesFlag), 10, 64) + value, err := parseNativeUint64(*flags.vacuumFilesFlag) if err != nil { return vacuumOptions{}, fmt.Errorf("failed to parse --vacuum-files value: %s", *flags.vacuumFilesFlag) } @@ -3417,32 +3245,85 @@ func parseVacuumOptions(flags *cliFlags) (vacuumOptions, error) { } func parseVacuumSize(value string) (uint64, error) { - match := sizeTokenRe.FindStringSubmatch(value) - if match == nil { - return 0, fmt.Errorf("failed to parse --vacuum-size value: %s", value) - } - number, err := strconv.ParseFloat(match[1], 64) - if err != nil { - return 0, err - } - unit := strings.ToLower(match[2]) - multipliers := map[string]uint64{ - "": 1, "b": 1, "byte": 1, "bytes": 1, - "k": 1024, "kb": 1024, "kib": 1024, - "m": 1024 * 1024, "mb": 1024 * 1024, "mib": 1024 * 1024, - "g": 1024 * 1024 * 1024, "gb": 1024 * 1024 * 1024, "gib": 1024 * 1024 * 1024, - "t": 1024 * 1024 * 1024 * 1024, "tb": 1024 * 1024 * 1024 * 1024, "tib": 1024 * 1024 * 1024 * 1024, - "p": 1024 * 1024 * 1024 * 1024 * 1024, "pb": 1024 * 1024 * 1024 * 1024 * 1024, "pib": 1024 * 1024 * 1024 * 1024 * 1024, - "e": 1024 * 1024 * 1024 * 1024 * 1024 * 1024, "eb": 1024 * 1024 * 1024 * 1024 * 1024 * 1024, "eib": 1024 * 1024 * 1024 * 1024 * 1024 * 1024, - } - multiplier, ok := multipliers[unit] - if !ok { - return 0, fmt.Errorf("failed to parse --vacuum-size value: %s", value) - } - if number < 0 || number > float64(^uint64(0))/float64(multiplier) { - return 0, fmt.Errorf("failed to parse --vacuum-size value: %s", value) + invalid := func() (uint64, error) { return 0, fmt.Errorf("failed to parse --vacuum-size value: %s", value) } + units := [...]string{"E", "P", "T", "G", "M", "K", "B", ""} + factors := [...]uint64{1 << 60, 1 << 50, 1 << 40, 1 << 30, 1 << 20, 1 << 10, 1, 1} + remaining, firstUnit := value, 0 + var total uint64 + for { + remaining = strings.TrimLeft(remaining, " \t\n\r") + if strings.HasPrefix(remaining, "-") { + return invalid() + } + number := strings.TrimLeft(remaining, " \t\n\r\v\f") + negative := strings.HasPrefix(number, "-") + if strings.HasPrefix(number, "+") || negative { + number = number[1:] + } + end := 0 + for end < len(number) && number[end] >= '0' && number[end] <= '9' { + end++ + } + if end == 0 { + return invalid() + } + whole, err := strconv.ParseUint(number[:end], 10, 64) + if err != nil { + return invalid() + } + if negative { + whole = -whole + } + remaining = number[end:] + fraction := float64(0) + if strings.HasPrefix(remaining, ".") { + remaining = remaining[1:] + end = 0 + for end < len(remaining) && remaining[end] >= '0' && remaining[end] <= '9' { + end++ + } + if end > 0 { + part, err := strconv.ParseUint(remaining[:end], 10, 64) + if err != nil { + return invalid() + } + fraction = float64(part) + for range end { + fraction /= 10 + } + remaining = remaining[end:] + } + } + remaining = strings.TrimLeft(remaining, " \t\n\r") + unit := firstUnit + for unit < len(units) && !strings.HasPrefix(remaining, units[unit]) { + unit++ + } + if unit == len(units) { + return invalid() + } + factor := factors[unit] + ceiling := whole + if fraction > 0 { + if ceiling == ^uint64(0) { + return invalid() + } + ceiling++ + } + if ceiling > ^uint64(0)/factor { + return invalid() + } + part := whole*factor + uint64(fraction*float64(factor)) + if part > ^uint64(0)-total { + return invalid() + } + total += part + remaining = remaining[len(units[unit]):] + if remaining == "" { + return total, nil + } + firstUnit = unit + 1 } - return uint64(number * float64(multiplier)), nil } func vacuumDirectory(dir string, opts vacuumOptions, quiet bool, stderr io.Writer) error { @@ -3493,10 +3374,6 @@ func vacuumDirectory(dir string, opts vacuumOptions, quiet bool, stderr io.Write sum = saturatingAdd(sum, usage) } - sort.Slice(candidates, func(i, j int) bool { - return vacuumCandidateLess(candidates[i], candidates[j]) - }) - retentionLimit := uint64(0) if opts.maxRetentionUsec > 0 { now := currentRealtimeUsec() @@ -3504,8 +3381,11 @@ func vacuumDirectory(dir string, opts vacuumOptions, quiet bool, stderr io.Write retentionLimit = now - opts.maxRetentionUsec } } + patchVacuumCreationTimes(candidates, opts, retentionLimit, sum, activeFiles) + order := newVacuumOrder(candidates) - for i, candidate := range candidates { + for i := 0; i < len(candidates); i++ { + candidate := order.next() left := activeFiles + uint64(len(candidates)-i) if (opts.maxRetentionUsec == 0 || candidate.realtime >= retentionLimit) && (opts.maxUse == 0 || sum <= opts.maxUse) && @@ -3536,7 +3416,7 @@ func vacuumDirectory(dir string, opts vacuumOptions, quiet bool, stderr io.Write func parseVacuumCandidate(path, name string, usage uint64) (*vacuumCandidate, bool) { if strings.HasSuffix(name, ".journal") { - match := archivedJournalNameRe.FindStringSubmatch(name) + match := archivedJournalNameRe().FindStringSubmatch(name) if match == nil { return nil, true } @@ -3556,7 +3436,7 @@ func parseVacuumCandidate(path, name string, usage uint64) (*vacuumCandidate, bo }, false } if strings.HasSuffix(name, ".journal~") { - match := corruptJournalNameRe.FindStringSubmatch(name) + match := corruptJournalNameRe().FindStringSubmatch(name) if match == nil { return nil, true } @@ -3574,19 +3454,6 @@ func parseVacuumCandidate(path, name string, usage uint64) (*vacuumCandidate, bo return nil, false } -func vacuumCandidateLess(a, b vacuumCandidate) bool { - if a.haveSeqno && b.haveSeqno && a.seqnumID == b.seqnumID { - return a.seqnum < b.seqnum - } - if a.realtime != b.realtime { - return a.realtime < b.realtime - } - if a.haveSeqno && b.haveSeqno && a.seqnumID != b.seqnumID { - return a.seqnumID < b.seqnumID - } - return a.name < b.name -} - func vacuumJournalFileEmpty(path string, info os.FileInfo) (bool, error) { if info.Size() < journalHeaderSize { return true, nil @@ -3611,10 +3478,37 @@ func vacuumJournalFileEmpty(path string, info os.FileInfo) (bool, error) { } func patchVacuumRealtime(candidate *vacuumCandidate, info os.FileInfo) { - usec := timeToUsec(info.ModTime()) - if usec > 0 && usec < candidate.realtime { - candidate.realtime = usec + realtime := earlierVacuumUsec(candidate.realtime, timeToUsec(info.ModTime())) + candidate.realtime = vacuumMetadataRealtime(info, realtime) +} + +func patchVacuumCreationTimes(candidates []vacuumCandidate, opts vacuumOptions, retentionLimit, usage, activeFiles uint64) { + if len(candidates) == 0 || retentionLimit == 0 && + (opts.maxUse == 0 || usage <= opts.maxUse) && + (opts.maxFiles == 0 || activeFiles+uint64(len(candidates)) <= opts.maxFiles) { + return } + sequenceOnly := candidates[0].haveSeqno + for _, candidate := range candidates[1:] { + if !candidate.haveSeqno || candidate.seqnumID != candidates[0].seqnumID { + sequenceOnly = false + break + } + } + for i := range candidates { + // A shared sequence ID fixes order; only unresolved age can need I/O. + if sequenceOnly && (retentionLimit == 0 || candidates[i].realtime < retentionLimit) { + continue + } + candidates[i].realtime = vacuumCreationRealtime(candidates[i].path, candidates[i].realtime) + } +} + +func earlierVacuumUsec(current, candidate uint64) uint64 { + if candidate > 0 && candidate < current { + return candidate + } + return current } func currentRealtimeUsec() uint64 { @@ -3738,15 +3632,22 @@ func runListInvocations(input cliInput, flags *cliFlags, stdout io.Writer) error if err != nil { return err } - if len(infos) == 0 { - return errors.New("No invocation ID found.") - } display, err := selectInvocationRows(infos, flags) if err != nil { return err } + if len(display) == 0 { + return noDataError(flags.effectiveQuiet(), "No invocation ID found.") + } + if jsonTableMode(*flags.output) { + rows := make([]map[string]any, 0, len(display)) + for _, row := range display { + rows = append(rows, map[string]any{"index": row.index, "invocation_id": row.info.id, "first_entry": tableTimestamp(row.info.firstUsec), "last_entry": tableTimestamp(row.info.lastUsec)}) + } + return writeTableJSON(stdout, *flags.output, rows) + } if !*flags.quietFlag { - fmt.Fprintln(stdout, "IDX INVOCATION ID FIRST ENTRY LAST ENTRY") + fmt.Fprint(stdout, detectOutputColors().tableHeader("INVOCATION ID")) } idxWidth := 1 if *flags.quietFlag { @@ -3886,10 +3787,7 @@ func formatHeaderTimestamp(usec uint64) string { if usec == 0 { return "n/a" } - return time.Unix( - int64(usec/1_000_000), - int64(usec%1_000_000)*1_000, - ).Local().Format("Mon 2006-01-02 15:04:05 MST") + return formatCalendarTimestamp(usec, false, "Mon 2006-01-02 15:04:05 MST", true, false) } func formatHeaderTimespan(usec uint64) string { @@ -4000,7 +3898,7 @@ func validateVerificationKeyOption(verifyKey string, hasVerifyKey bool, stderr i return errors.New("failed to parse seed") } -func verifyOneFile(path, verifyKey string, hasVerifyKey, directoryInput bool, stderr io.Writer) error { +func verifyOneFile(path, verifyKey string, hasVerifyKey, directoryInput, quiet bool, stderr io.Writer) error { sealed, err := isFileSealed(path) if err != nil { return reportVerifyOpenError(path, err, directoryInput, stderr) @@ -4012,7 +3910,9 @@ func verifyOneFile(path, verifyKey string, hasVerifyKey, directoryInput bool, st fmt.Fprintf(stderr, "FAIL: %s (%v)\n", path, err) return err } - fmt.Fprintf(stderr, "PASS: %s\n", path) + if !quiet { + fmt.Fprintf(stderr, "PASS: %s\n", path) + } return nil } diff --git a/go/cmd/journalctl/main_test.go b/go/cmd/journalctl/main_test.go index 062e56a7..a08b3419 100644 --- a/go/cmd/journalctl/main_test.go +++ b/go/cmd/journalctl/main_test.go @@ -1,3 +1,5 @@ +//go:build cgo + package main import ( @@ -72,8 +74,9 @@ func TestShowTailLimitZeroDoesNotScan(t *testing.T) { } } -func TestFormatTimestampPreservesFullUint64SourceRealtime(t *testing.T) { +func TestFormatTimestampFallsBackForInvalidSourceRealtime(t *testing.T) { entry := &journal.Entry{ + Realtime: 1_700_000_000_000_000, Fields: map[string][]byte{ "_SOURCE_REALTIME_TIMESTAMP": []byte("18446744073709551615"), }, @@ -84,7 +87,7 @@ func TestFormatTimestampPreservesFullUint64SourceRealtime(t *testing.T) { if err != nil { t.Fatalf("formatTimestamp error: %v", err) } - if want := "586524-01-19T08:01:49.551615+00:00"; got != want { + if want := "2023-11-14T22:13:20.000000+00:00"; got != want { t.Fatalf("formatTimestamp = %q, want %q", got, want) } } @@ -234,11 +237,11 @@ type trackingCLIJournal struct { getEntryCalls int } -func (j *trackingCLIJournal) Close() error { return nil } -func (j *trackingCLIJournal) AddMatch([]byte) {} -func (j *trackingCLIJournal) AddDisjunction() {} -func (j *trackingCLIJournal) AddConjunction() {} -func (j *trackingCLIJournal) FlushMatches() {} +func (j *trackingCLIJournal) Close() error { return nil } +func (j *trackingCLIJournal) AddMatch([]byte) {} +func (j *trackingCLIJournal) AddDisjunction() error { return nil } +func (j *trackingCLIJournal) AddConjunction() error { return nil } +func (j *trackingCLIJournal) FlushMatches() {} func (j *trackingCLIJournal) SeekHead() error { j.seekHeadCalls++ return nil @@ -247,9 +250,9 @@ func (j *trackingCLIJournal) SeekTail() error { j.seekTailCalls++ return nil } -func (j *trackingCLIJournal) SeekRealtimeUsec(uint64) error { return nil } -func (j *trackingCLIJournal) SeekCursor(string) error { return nil } -func (j *trackingCLIJournal) TestCursor(string) (bool, error) { return false, nil } +func (j *trackingCLIJournal) SeekRealtimeUsec(uint64) error { return nil } +func (j *trackingCLIJournal) SeekCursorDirection(string, journal.Direction) error { return nil } +func (j *trackingCLIJournal) TestCursor(string) (bool, error) { return false, nil } func (j *trackingCLIJournal) Next() (int, error) { j.nextCalls++ return 0, nil @@ -538,7 +541,7 @@ func patchCompatibleFlags(t *testing.T, path string, flagsToSet uint32) { // -- SOW-0121 parser parity tests ---------------------------------- // -// Every official systemd v260.1 long option must be recognized by the +// Every official systemd v262 long option must be recognized by the // parser. The set is enumerated by the shared manifest at // tests/parser-parity/v260-manifest.json and is duplicated here so the // parser contract is enforced by Go unit tests in addition to the @@ -567,18 +570,29 @@ var officialOutputModes = []string{ } func TestEveryOfficialLongOptionIsParsed(t *testing.T) { - for _, opt := range officialLongOptions { - // boolean-style options get a `=true` placeholder so the flag - // package does not try to consume the next positional. - argv := []string{"--" + opt, "true"} - err := parseOnlyForTest(argv) - if err != nil { - // flag.ErrHelp is a normal parser exit for --help; the - // parser still recognized the option. - if errors.Is(err, flag.ErrHelp) { - continue + samples := map[string]string{"output": "short", "since": "@0", "until": "@0", "priority": "3", "facility": "3", "lines": "1", "boot": "all", "invocation": "all", "interval": "1s", "vacuum-time": "1s", "vacuum-size": "1M", "vacuum-files": "1", "synchronize-on-exit": "false"} + for _, opt := range append(officialLongOptions, "introspect-cli") { + fs, _ := newCLIFlagSet(io.Discard) + flg := fs.Lookup(opt) + argv := []string{"--" + opt} + optional := opt == "boot" || opt == "lines" || opt == "case-sensitive" + takesValue := optional + if flg != nil { + b, ok := flg.Value.(interface{ IsBoolFlag() bool }) + takesValue = takesValue || !ok || !b.IsBoolFlag() + } + if takesValue { + value, ok := samples[opt] + if !ok { + value = "placeholder" } - t.Errorf("parser rejected official option --%s: %v", opt, err) + if opt == "case-sensitive" { + value = "true" + } + argv = append(argv, value) + } + if err := parseOnlyForTest(argv); err != nil { + t.Errorf("parser rejected %s: %v", opt, err) } } } @@ -625,61 +639,31 @@ func TestParseLinesLimitValuePreservesSystemdDirection(t *testing.T) { } } -func TestOptionalArgumentPreprocessingMatchesSystemd(t *testing.T) { - fs, _ := newCLIFlagSet(io.Discard) - tests := []struct { - name string - args []string - want []string +func TestOrderedOptionalArgumentsAndRequiredValues(t *testing.T) { + for _, tc := range []struct { + args []string + boot, lines, grep string + wantError bool }{ - { - name: "bare boot", - args: []string{"--boot", "TEST_ID=value"}, - want: []string{"--boot=0", "TEST_ID=value"}, - }, - { - name: "explicit empty boot", - args: []string{"--boot="}, - want: []string{"--boot="}, - }, - { - name: "short attached boot", - args: []string{"-ball"}, - want: []string{"-b=all"}, - }, - { - name: "short equals boot keeps equals in value", - args: []string{"-b=true"}, - want: []string{"-b==true"}, - }, - { - name: "bare lines", - args: []string{"--lines", "TEST_ID=value"}, - want: []string{"--lines=10", "TEST_ID=value"}, - }, - { - name: "short attached lines", - args: []string{"-n5"}, - want: []string{"-n=5"}, - }, - { - name: "short equals lines keeps equals in value", - args: []string{"-n=5"}, - want: []string{"-n==5"}, - }, - { - name: "short cluster with value option", - args: []string{"-qrn2"}, - want: []string{"-q", "-r", "-n=2"}, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - got := preprocessOptionalArgs(normalizeShortFlags(tt.args, fs)) - if fmt.Sprint(got) != fmt.Sprint(tt.want) { - t.Fatalf("preprocess = %#v, want %#v", got, tt.want) - } - }) + {args: []string{"--boot", "TEST_ID=value"}, boot: "0"}, + {args: []string{"-ball", "-n5"}, boot: "all", lines: "5"}, + {args: []string{"--boot="}, wantError: true}, + {args: []string{"-b=true"}, wantError: true}, + {args: []string{"-n=5"}, wantError: true}, + {args: []string{"--grep", "--help", "-rn2"}, grep: "--help", lines: "2"}, + {args: []string{"--since=bad", "--help"}, wantError: true}, + } { + fs, f := newCLIFlagSet(io.Discard) + _, terminal, err := parseCLIOptions(tc.args, fs, f, io.Discard) + if (err != nil) != tc.wantError { + t.Fatalf("%q: error=%v", tc.args, err) + } + if err != nil { + continue + } + if terminal || f.boot.value != tc.boot || f.linesFlag.value != tc.lines || *f.grepFlag != tc.grep { + t.Fatalf("%q: boot=%q lines=%q grep=%q terminal=%v", tc.args, f.boot.value, f.linesFlag.value, *f.grepFlag, terminal) + } } } @@ -691,7 +675,7 @@ func TestPortableUnsupportedMessageFormat(t *testing.T) { } for _, opt := range unsupported { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{opt}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{opt}); err != nil { t.Fatalf("parse error for %s: %v", opt, err) } err := flags.validate() @@ -709,7 +693,7 @@ func TestPortableUnsupportedForSourceOptions(t *testing.T) { unsupported := []string{"--machine", "--root", "--image", "--namespace"} for _, opt := range unsupported { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{opt, "/dev/null"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{opt, "/dev/null"}); err != nil { t.Fatalf("parse error for %s: %v", opt, err) } err := flags.validate() @@ -725,7 +709,7 @@ func TestPortableUnsupportedForSourceOptions(t *testing.T) { func TestSourceExclusivityEnforced(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--directory=/tmp", "--file=/tmp/x.journal"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--directory=/tmp", "--file=/tmp/x.journal"}); err != nil { t.Fatalf("parse error: %v", err) } err := flags.validate() @@ -756,7 +740,7 @@ func TestResolveFileInputsDeduplicatesRepeatedPaths(t *testing.T) { func TestSinceUntilOrderEnforced(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{ + if err := parseFlagsForTest(fs, flags, []string{ "--file=/tmp/x.journal", "--since=2020-01-02", "--until=2020-01-01", @@ -774,7 +758,7 @@ func TestSinceUntilOrderEnforced(t *testing.T) { func TestFollowReverseConflictEnforced(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--file=/tmp/x.journal", "--follow", "--reverse"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--file=/tmp/x.journal", "--follow", "--reverse"}); err != nil { t.Fatalf("parse error: %v", err) } err := flags.validate() @@ -788,7 +772,7 @@ func TestFollowReverseConflictEnforced(t *testing.T) { func TestOldestLinesConflictAppliesOnlyToShowAction(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--file=/tmp/x.journal", "--lines=+1", "--reverse"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--file=/tmp/x.journal", "--lines=+1", "--reverse"}); err != nil { t.Fatalf("parse error: %v", err) } if err := flags.validateParserInteractions(); err == nil { @@ -796,7 +780,7 @@ func TestOldestLinesConflictAppliesOnlyToShowAction(t *testing.T) { } fs, flags = newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--file=/tmp/x.journal", "--list-boots", "--lines=+1", "--reverse"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--file=/tmp/x.journal", "--list-boots", "--lines=+1", "--reverse"}); err != nil { t.Fatalf("parse error: %v", err) } if err := flags.validateParserInteractions(); err != nil { @@ -832,25 +816,28 @@ func TestTimestampParserAcceptsStockISOTForms(t *testing.T) { } func TestDirectoryInputDoesNotOpenRegularFileAsFile(t *testing.T) { - dir := t.TempDir() - path := filepath.Join(dir, "regular.journal") - if err := os.WriteFile(path, []byte("not-a-real-journal"), 0o644); err != nil { - t.Fatalf("write fixture: %v", err) - } - + path := writeCLIJournal(t, []cliEntry{{message: "regular-file", priority: "6"}}) var stdout, stderr bytes.Buffer + if err := run([]string{"--file", path, "--output=cat"}, strings.NewReader(""), &stdout, &stderr); err != nil { + t.Fatalf("read regular-file control: %v", err) + } + if stdout.String() != "regular-file\n" { + t.Fatalf("unexpected regular-file control: %q", stdout.String()) + } + stdout.Reset() + stderr.Reset() err := run([]string{"--directory", path, "--output=cat"}, strings.NewReader(""), &stdout, &stderr) if err == nil { t.Fatalf("expected --directory regular file to fail") } - if !strings.Contains(strings.ToLower(err.Error()), "not a directory") { - t.Fatalf("expected not-a-directory error, got: %v", err) + if stdout.Len() != 0 { + t.Fatalf("--directory rendered a regular file: %q", stdout.String()) } } func TestSynchronizeOnExitTrueIsRejected(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--synchronize-on-exit=true"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--synchronize-on-exit=true"}); err != nil { t.Fatalf("parse error: %v", err) } err := flags.validate() @@ -864,7 +851,7 @@ func TestSynchronizeOnExitTrueIsRejected(t *testing.T) { func TestSynchronizeOnExitFalseIsAccepted(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--synchronize-on-exit=false"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--synchronize-on-exit=false"}); err != nil { t.Fatalf("parse error: %v", err) } if err := flags.validate(); err != nil { @@ -874,7 +861,7 @@ func TestSynchronizeOnExitFalseIsAccepted(t *testing.T) { func TestVacuumWithoutDirectoryIsRejected(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--vacuum-size=1G"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--vacuum-size=1G"}); err != nil { t.Fatalf("parse error: %v", err) } err := flags.validate() @@ -888,7 +875,7 @@ func TestVacuumWithoutDirectoryIsRejected(t *testing.T) { func TestRotateWithVacuumIsDistinctUnsupportedAction(t *testing.T) { fs, flags := newCLIFlagSet(io.Discard) - if err := fs.Parse([]string{"--rotate", "--vacuum-files=2"}); err != nil { + if err := parseFlagsForTest(fs, flags, []string{"--rotate", "--vacuum-files=2"}); err != nil { t.Fatalf("parse error: %v", err) } err := flags.validate() @@ -954,7 +941,7 @@ func TestVersionPrintsBaselineMetadata(t *testing.T) { t.Fatalf("run --version error: %v; stderr=%s", err, stderr.String()) } banner := stdout.String() - if !strings.Contains(banner, "v260.1") || !strings.Contains(banner, "baseline") { + if !strings.Contains(banner, "v262") || !strings.Contains(banner, "CLI compatibility") { t.Fatalf("expected version banner, got: %q", banner) } } @@ -982,23 +969,24 @@ func TestBootAllMergeIsNotRejectedAsConflict(t *testing.T) { } func TestUnrecognizedOptionIsRejected(t *testing.T) { - fs, _ := newCLIFlagSet(io.Discard) - err := fs.Parse([]string{"--not-an-official-option"}) + fs, flags := newCLIFlagSet(io.Discard) + err := parseFlagsForTest(fs, flags, []string{"--not-an-official-option"}) if err == nil { t.Fatalf("expected parse error for unknown option") } - if !strings.Contains(err.Error(), "flag provided but not defined") { + if !strings.Contains(err.Error(), "unrecognized option") { t.Fatalf("expected unknown flag error, got: %v", err) } } -// parseOnlyForTest parses argv through the flag set used by `run`. -// It returns nil if the parser accepted every token. It does NOT run -// validation or dispatch so unknown-class options can be tested in -// isolation. +func parseFlagsForTest(fs *flag.FlagSet, f *cliFlags, argv []string) error { + _, _, err := parseCLIOptions(argv, fs, f, io.Discard) + return err +} + func parseOnlyForTest(argv []string) error { - fs, _ := newCLIFlagSet(io.Discard) - return fs.Parse(permuteFlagArgs(preprocessOptionalArgs(normalizeShortFlags(argv, fs)), fs)) + fs, f := newCLIFlagSet(io.Discard) + return parseFlagsForTest(fs, f, argv) } func archivedJournalNameForTest(seq uint64, realtime uint64) string { diff --git a/go/cmd/journalctl/options.json b/go/cmd/journalctl/options.json new file mode 100644 index 00000000..767b3295 --- /dev/null +++ b/go/cmd/journalctl/options.json @@ -0,0 +1,787 @@ +[ + { + "names": [ + "--system" + ], + "argument": "no_argument", + "help": "Show the system journal", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "--user" + ], + "argument": "no_argument", + "help": "Show the user journal for the current user", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "-M", + "--machine" + ], + "argument": "required_argument", + "metavar": "CONTAINER", + "help": "Operate on local container (unsupported in portable mode)", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "-m", + "--merge" + ], + "argument": "no_argument", + "help": "Show entries from all available journals", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "-D", + "--directory" + ], + "argument": "required_argument", + "metavar": "PATH", + "help": "Show journal files from directory", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "-i", + "--file" + ], + "argument": "required_argument", + "metavar": "PATH", + "help": "Show journal file", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "--root" + ], + "argument": "required_argument", + "metavar": "PATH", + "help": "Operate on an alternate filesystem root (unsupported in portable mode)", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "--image" + ], + "argument": "required_argument", + "metavar": "PATH", + "help": "Operate on disk image as filesystem root (unsupported in portable mode)", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "--image-policy" + ], + "argument": "required_argument", + "metavar": "POLICY", + "help": "Specify disk image dissection policy (unsupported in portable mode)", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "--namespace" + ], + "argument": "required_argument", + "metavar": "NAMESPACE", + "help": "Show journal data from specified journal namespace (unsupported in portable mode)", + "group": [ + "Source Options" + ] + }, + { + "names": [ + "-S", + "--since" + ], + "argument": "required_argument", + "metavar": "DATE", + "help": "Show entries not older than the specified date", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-U", + "--until" + ], + "argument": "required_argument", + "metavar": "DATE", + "help": "Show entries not newer than the specified date", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-c", + "--cursor" + ], + "argument": "required_argument", + "metavar": "CURSOR", + "help": "Show entries starting at the specified cursor", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--after-cursor" + ], + "argument": "required_argument", + "metavar": "CURSOR", + "help": "Show entries after the specified cursor", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--cursor-file" + ], + "argument": "required_argument", + "metavar": "FILE", + "help": "Show entries after cursor in FILE and update FILE", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--this-boot" + ], + "argument": "no_argument", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-b", + "--boot" + ], + "argument": "optional_argument", + "metavar": "ID", + "help": "Show current boot or the specified boot", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-u", + "--unit" + ], + "argument": "required_argument", + "metavar": "UNIT", + "help": "Show logs from the specified unit", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--user-unit" + ], + "argument": "required_argument", + "metavar": "UNIT", + "help": "Show logs from the specified user unit", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--invocation" + ], + "argument": "required_argument", + "metavar": "ID", + "help": "Show logs from the matching invocation ID", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-I" + ], + "argument": "no_argument", + "help": "Show logs from the latest invocation of unit", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-t", + "--identifier" + ], + "argument": "required_argument", + "metavar": "ID", + "help": "Show entries with the specified syslog identifier", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-T", + "--exclude-identifier" + ], + "argument": "required_argument", + "metavar": "ID", + "help": "Hide entries with the specified syslog identifier", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-p", + "--priority" + ], + "argument": "required_argument", + "metavar": "RANGE", + "help": "Show entries within the specified priority range", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--facility" + ], + "argument": "required_argument", + "metavar": "FACILITY\u2026", + "help": "Show entries with the specified facilities", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-g", + "--grep" + ], + "argument": "required_argument", + "metavar": "PATTERN", + "help": "Show entries with MESSAGE matching PATTERN", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "--case-sensitive" + ], + "argument": "optional_argument", + "metavar": "BOOL", + "help": "Force case sensitive or insensitive matching", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-k", + "--dmesg" + ], + "argument": "no_argument", + "help": "Show kernel message log from the current boot", + "group": [ + "Filtering Options" + ] + }, + { + "names": [ + "-o", + "--output" + ], + "argument": "required_argument", + "metavar": "STRING", + "help": "Change journal output mode (short, short-precise, short-iso, short-iso-precise, short-full, short-monotonic, short-unix, verbose, export, json, json-pretty, json-sse, json-seq, cat, with-unit)", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--output-fields" + ], + "argument": "required_argument", + "metavar": "LIST", + "help": "Select fields to print in verbose/export/json modes", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-n", + "--lines" + ], + "argument": "optional_argument", + "metavar": "[+]INTEGER", + "help": "Number of journal entries to show", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-r", + "--reverse" + ], + "argument": "no_argument", + "help": "Show the newest entries first", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--show-cursor" + ], + "argument": "no_argument", + "help": "Print the cursor after all the entries", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--utc" + ], + "argument": "no_argument", + "help": "Express time in Coordinated Universal Time (UTC)", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-x", + "--catalog" + ], + "argument": "no_argument", + "help": "Add message explanations where available", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-W", + "--no-hostname" + ], + "argument": "no_argument", + "help": "Suppress output of hostname field", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-l", + "--full" + ], + "argument": "no_argument", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--no-full" + ], + "argument": "no_argument", + "help": "Ellipsize fields", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-a", + "--all" + ], + "argument": "no_argument", + "help": "Show all fields, including long and unprintable", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-f", + "--follow" + ], + "argument": "no_argument", + "help": "Follow the journal", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--no-tail" + ], + "argument": "no_argument", + "help": "Show all lines, even in follow mode", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--truncate-newline" + ], + "argument": "no_argument", + "help": "Truncate entries by first newline character", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "-q", + "--quiet" + ], + "argument": "no_argument", + "help": "Do not show info messages and privilege warning", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--synchronize-on-exit" + ], + "argument": "required_argument", + "metavar": "BOOL", + "help": "Wait for Journal synchronization before exiting (unsupported in portable mode)", + "group": [ + "Output Control Options" + ] + }, + { + "names": [ + "--no-pager" + ], + "argument": "no_argument", + "help": "Do not start a pager", + "group": [ + "Pager Control Options" + ] + }, + { + "names": [ + "-e", + "--pager-end" + ], + "argument": "no_argument", + "help": "Immediately jump to the end in the pager", + "group": [ + "Pager Control Options" + ] + }, + { + "names": [ + "--interval" + ], + "argument": "required_argument", + "metavar": "TIME", + "help": "Time interval for changing the FSS sealing key", + "group": [ + "Forward Secure Sealing (FSS) Options" + ] + }, + { + "names": [ + "--verify-key" + ], + "argument": "required_argument", + "metavar": "KEY", + "help": "Specify FSS verification key", + "group": [ + "Forward Secure Sealing (FSS) Options" + ] + }, + { + "names": [ + "--force" + ], + "argument": "no_argument", + "help": "Override of the FSS key pair with --setup-keys", + "group": [ + "Forward Secure Sealing (FSS) Options" + ] + }, + { + "names": [ + "-h", + "--help" + ], + "argument": "no_argument", + "help": "Show this help", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--version" + ], + "argument": "no_argument", + "help": "Show package version", + "group": [ + "Commands" + ] + }, + { + "names": [ + "-N", + "--fields" + ], + "argument": "no_argument", + "help": "List all field names currently used", + "group": [ + "Commands" + ] + }, + { + "names": [ + "-F", + "--field" + ], + "argument": "required_argument", + "metavar": "FIELD", + "help": "List all values that a specified field takes", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--list-boots" + ], + "argument": "no_argument", + "help": "Show terse information about recorded boots", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--list-invocations" + ], + "argument": "no_argument", + "help": "Show invocation IDs of specified unit", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--list-namespaces" + ], + "argument": "no_argument", + "help": "Show list of journal namespaces (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--disk-usage" + ], + "argument": "no_argument", + "help": "Show total disk usage of all journal files", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--vacuum-size" + ], + "argument": "required_argument", + "metavar": "BYTES", + "help": "Reduce disk usage below specified size", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--vacuum-files" + ], + "argument": "required_argument", + "metavar": "INT", + "help": "Leave only the specified number of journal files", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--vacuum-time" + ], + "argument": "required_argument", + "metavar": "TIME", + "help": "Remove journal files older than specified time", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--verify" + ], + "argument": "no_argument", + "help": "Verify journal file consistency", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--sync" + ], + "argument": "no_argument", + "help": "Synchronize unwritten journal messages to disk (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--relinquish-var" + ], + "argument": "no_argument", + "help": "Stop logging to disk, log to temporary file system (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--smart-relinquish-var" + ], + "argument": "no_argument", + "help": "Similar, but NOP if log directory is on root mount (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--flush" + ], + "argument": "no_argument", + "help": "Flush all journal data from /run into /var (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--rotate" + ], + "argument": "no_argument", + "help": "Request immediate rotation of the journal files (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--header" + ], + "argument": "no_argument", + "help": "Show journal header information", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--list-catalog" + ], + "argument": "no_argument", + "help": "Show all message IDs in the catalog (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--dump-catalog" + ], + "argument": "no_argument", + "help": "Show entries in the message catalog (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--update-catalog" + ], + "argument": "no_argument", + "help": "Update the message catalog database (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--setup-keys" + ], + "argument": "no_argument", + "help": "Generate a new FSS key pair (unsupported in portable mode)", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--new-id128" + ], + "argument": "no_argument", + "group": [ + "Commands" + ] + }, + { + "names": [ + "--introspect-cli" + ], + "argument": "no_argument", + "group": [ + "Commands" + ] + } +] diff --git a/go/cmd/journalctl/output.go b/go/cmd/journalctl/output.go index 4f4dabed..afa63792 100644 --- a/go/cmd/journalctl/output.go +++ b/go/cmd/journalctl/output.go @@ -1,3 +1,5 @@ +//go:build cgo + package main import ( @@ -5,10 +7,9 @@ import ( "encoding/binary" "encoding/json" "fmt" + "os" "sort" - "strconv" "strings" - "time" "unicode/utf8" "github.com/netdata/systemd-journal-sdk/go/journal" @@ -24,6 +25,8 @@ type outputOptions struct { showAll bool truncateNewline bool merge bool + colors outputColors + urlify bool } const ( @@ -33,23 +36,25 @@ const ( ) func newOutputOptions(flags *cliFlags, outputFieldsSet bool, fullWidth bool) outputOptions { + colors := detectOutputColors() return outputOptions{ mode: *flags.output, outputFields: parseOutputFields(*flags.outputFieldsFlag), - outputFieldsSet: outputFieldsSet, + outputFieldsSet: outputFieldsSet && len(parseOutputFields(*flags.outputFieldsFlag)) > 0, utc: *flags.utcFlag, noHostname: *flags.noHostnameFlag, fullWidth: fullWidth, showAll: *flags.allFlag, truncateNewline: *flags.truncateNewlineFlag, merge: *flags.mergeFlag, + colors: colors, + urlify: urlifyEnabled(colors), } } func parseOutputFields(value string) []string { var fields []string for _, item := range strings.Split(value, ",") { - item = strings.TrimSpace(item) if item != "" { fields = append(fields, item) } @@ -58,8 +63,11 @@ func parseOutputFields(value string) []string { } type outputRenderer struct { - options outputOptions - previousDelta *deltaState + options outputOptions + previousDelta *deltaState + catFieldsValidated bool + columns int + highlight matchRange } type deltaState struct { @@ -69,7 +77,15 @@ type deltaState struct { } func newOutputRenderer(options outputOptions) *outputRenderer { - return &outputRenderer{options: options} + columns := defaultColumns + if !options.fullWidth && !options.showAll && outputSkipsMissingMessage(options.mode) { + if n, err := parseNativeUint64(os.Getenv("COLUMNS")); err == nil && n > 0 && n <= 65535 { + columns = int(n) + } else if n, ok := stdoutColumns(); ok { + columns = n + } + } + return &outputRenderer{options: options, catFieldsValidated: !options.outputFieldsSet, columns: columns} } func (r *outputRenderer) render(entry *journal.Entry) (string, error) { @@ -87,7 +103,7 @@ func (r *outputRenderer) render(entry *journal.Entry) (string, error) { case "verbose": return r.renderVerbose(entry) case "cat": - return r.renderCat(entry), nil + return r.renderCat(entry) case "with-unit": return r.renderWithUnit(entry) case "short-full": @@ -129,8 +145,9 @@ func (r *outputRenderer) renderShort(entry *journal.Entry, mode timestampMode) ( if err != nil { return "", err } - prefix := fmt.Sprintf("%s %s: ", timestamp, entryLabel(entry, r.options)) - return prefix + displayMessage(entry, r.options, len(prefix)) + "\n", nil + header, columns := shortHeader(entry, r.options, false) + prefix := timestamp + header + ": " + return prefix + displayMessage(entry, r.options, len(prefix)-len(header)+columns, r.columns, r.highlight) + "\n", nil } func (r *outputRenderer) renderWithUnit(entry *journal.Entry) (string, error) { @@ -138,13 +155,9 @@ func (r *outputRenderer) renderWithUnit(entry *journal.Entry) (string, error) { if err != nil { return "", err } - label := unitLabel(entry) - if label == "" { - label = baseEntryLabel(entry) - } - label = formatEntryLabel(entry, label, r.options) - prefix := fmt.Sprintf("%s %s: ", timestamp, label) - return prefix + displayMessage(entry, r.options, len(prefix)) + "\n", nil + header, columns := shortHeader(entry, r.options, true) + prefix := timestamp + header + ": " + return prefix + displayMessage(entry, r.options, len(prefix)-len(header)+columns, r.columns, r.highlight) + "\n", nil } func (r *outputRenderer) renderShortDelta(entry *journal.Entry) (string, error) { @@ -159,40 +172,75 @@ func (r *outputRenderer) renderShortDelta(entry *journal.Entry) (string, error) } var diff uint64 if marker == "*" { - diff = absDiff(currentRealtime, r.previousDelta.realtime) + diff = displayTimeDifference(currentRealtime, r.previousDelta.realtime) } else { - diff = absDiff(currentMonotonic, r.previousDelta.monotonic) + diff = displayTimeDifference(currentMonotonic, r.previousDelta.monotonic) } delta = fmt.Sprintf(" <%s%s>", formatMonotonic(diff), marker) } r.previousDelta = &deltaState{realtime: currentRealtime, monotonic: currentMonotonic, bootID: entry.BootID} - prefix := fmt.Sprintf("[%s%s] %s: ", monotonic, delta, entryLabel(entry, r.options)) - return prefix + displayMessage(entry, r.options, len(prefix)) + "\n", nil + header, columns := shortHeader(entry, r.options, false) + prefix := fmt.Sprintf("[%s%s]%s: ", monotonic, delta, header) + return prefix + displayMessage(entry, r.options, len(prefix)-len(header)+columns, r.columns, r.highlight) + "\n", nil } -func absDiff(a, b uint64) uint64 { - if a > b { - return a - b +func displayTimeDifference(current, previous uint64) uint64 { + if current < previous { + return 0 } - return b - a + return current - previous } -func (r *outputRenderer) renderCat(entry *journal.Entry) string { +func (r *outputRenderer) renderCat(entry *journal.Entry) (string, error) { var b strings.Builder - if r.options.outputFieldsSet { - for _, name := range r.options.outputFields { - for _, value := range entryValues(entry, name) { - b.WriteString(displayValue(value, r.options.truncateNewline)) - b.WriteByte('\n') + fields := r.options.outputFields + if !r.options.outputFieldsSet { + fields = []string{"MESSAGE"} + } + if !r.catFieldsValidated { + for _, name := range fields { + if !validFieldName(name) { + return "", fmt.Errorf("Invalid field name: %s", name) + } + } + r.catFieldsValidated = true + } + for _, name := range fields { + values := entryValues(entry, name) + if len(values) > 0 { + if r.options.colors.level == 0 { + b.Write(values[0]) + } else { + on, off, highlight := r.options.colors.log(entryPriority(entry, true), false) + b.WriteString(on) + if name == "MESSAGE" { + span := r.highlight + b.Write(values[0][:span.start]) + b.WriteString(highlight) + b.Write(values[0][span.start:span.end]) + b.WriteString(on) + b.Write(values[0][span.end:]) + } else { + b.Write(values[0]) + } + b.WriteString(off) } + b.WriteByte('\n') } - return b.String() } - for _, value := range entryValues(entry, "MESSAGE") { - b.WriteString(displayValue(value, r.options.truncateNewline)) - b.WriteByte('\n') + return b.String(), nil +} + +func validFieldName(name string) bool { + if name == "" || strings.HasPrefix(name, "__") { + return false + } + for _, c := range []byte(name) { + if !(c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '_') { + return false + } } - return b.String() + return true } func (r *outputRenderer) renderVerbose(entry *journal.Entry) (string, error) { @@ -201,9 +249,42 @@ func (r *outputRenderer) renderVerbose(entry *journal.Entry) (string, error) { return "", err } var b strings.Builder - fmt.Fprintf(&b, "%s [%s]\n", timestamp, entry.Cursor) + colors := r.options.colors + if colors.level == 0 { + fmt.Fprintf(&b, "%s [%s]\n", timestamp, entry.Cursor) + } else { + on := "" + if colors.underline { + on = "\x1b[0;4m" + } + fmt.Fprintf(&b, "%s%s%s %s[%s]%s\n", on, timestamp, ansiNormal, colors.grey(), entry.Cursor, colors.grey()) + } for _, field := range verboseFields(entry, r.options) { - fmt.Fprintf(&b, " %s=%s\n", field.name, displayVerboseValue(field.name, field.value, r.options)) + if colors.level == 0 { + fmt.Fprintf(&b, " %s=%s\n", field.name, displayVerboseValue(field.name, field.value, r.options)) + continue + } + on, off := "", "" + if colors.level != 0 { + if field.name == "MESSAGE" { + on, off = ansiHighlight, ansiNormal + } else if strings.HasPrefix(field.name, "_") { + on, off = ansiGreen, ansiNormal + } + } + value := displayVerboseValue(field.name, field.value, r.options) + // Native resets after a printable multiline field's terminating newline. + printable := r.options.showAll || (r.options.fullWidth || len(field.name)+1+len(field.value) < printCharThreshold) && journalTextPrintable(field.value) + if field.name == "CONFIG_FILE" && printable { + if linked := fileLink(field.value, r.options.urlify); linked != nil { + value = string(fullMessageWithNUL(linked, 4+len(field.name)+1)) + } + } + if printable { + fmt.Fprintf(&b, " %s%s=%s\n%s", on, field.name, value, off) + } else { + fmt.Fprintf(&b, " %s%s=%s%s\n", on, field.name, value, off) + } } return b.String(), nil } @@ -241,6 +322,9 @@ func (r *outputRenderer) renderJSON(entry *journal.Entry, frame jsonFrame) (stri if err != nil { return "", err } + if r.options.colors.level != 0 { + encoded = colorJSON(encoded) + } switch frame { case jsonFrameSSE: return "data: " + string(encoded) + "\n\n", nil @@ -257,30 +341,21 @@ func (r *outputRenderer) formatTimestamp(entry *journal.Entry, mode timestampMod } realtime := displayRealtimeUsec(entry) if mode == timestampShortUnix { - return fmt.Sprintf("%d.%06d", realtime/1_000_000, realtime%1_000_000), nil - } - t := time.Unix( - int64(realtime/1_000_000), - int64(realtime%1_000_000)*1_000, - ) - if r.options.utc { - t = t.UTC() - } else { - t = t.Local() + return fmt.Sprintf("%10d.%06d", realtime/1_000_000, realtime%1_000_000), nil } switch mode { case timestampShort: - return t.Format("Jan 02 15:04:05"), nil + return formatCalendarTimestamp(realtime, r.options.utc, "Jan 02 15:04:05", false, false), nil case timestampShortFull: - return t.Format("Mon 2006-01-02 15:04:05 MST"), nil + return formatCalendarTimestamp(realtime, r.options.utc, "Mon 2006-01-02 15:04:05 MST", true, false), nil case timestampShortISO: - return t.Format("2006-01-02T15:04:05-07:00"), nil + return formatCalendarTimestamp(realtime, r.options.utc, "2006-01-02T15:04:05-07:00", false, true), nil case timestampShortISOPrecise: - return t.Format("2006-01-02T15:04:05.000000-07:00"), nil + return formatCalendarTimestamp(realtime, r.options.utc, "2006-01-02T15:04:05.000000-07:00", false, true), nil case timestampShortPrecise: - return t.Format("Jan 02 15:04:05.000000"), nil + return formatCalendarTimestamp(realtime, r.options.utc, "Jan 02 15:04:05.000000", false, false), nil case timestampVerbose: - return t.Format("Mon 2006-01-02 15:04:05.000000 MST"), nil + return formatCalendarTimestamp(realtime, r.options.utc, "Mon 2006-01-02 15:04:05.000000 MST", true, false), nil default: return "", fmt.Errorf("unknown timestamp mode") } @@ -311,12 +386,15 @@ func displayMonotonicUsec(entry *journal.Entry) uint64 { } func sourceRealtimeUsec(entry *journal.Entry) (uint64, bool) { - values := entryValues(entry, "_SOURCE_REALTIME_TIMESTAMP") - if len(values) == 0 { + if entry == nil { + return 0, false + } + raw, ok := lastEntryValue(entry, "_SOURCE_REALTIME_TIMESTAMP") + if !ok { return 0, false } - value, err := strconv.ParseUint(string(values[0]), 10, 64) - if err != nil || value == 0 { + value, err := parseNativeUint64(string(cStringBytes(raw))) + if err != nil || value == 0 || value >= 1<<55 { return 0, false } return value, true @@ -341,66 +419,248 @@ func formatMonotonic(usec uint64) string { return fmt.Sprintf("%5d.%06d", usec/1_000_000, usec%1_000_000) } -func entryLabel(entry *journal.Entry, options outputOptions) string { - return formatEntryLabel(entry, baseEntryLabel(entry), options) +func lastEntryValue(entry *journal.Entry, name string) ([]byte, bool) { + if values := entry.FieldValues[name]; len(values) > 0 { + return values[len(values)-1], true + } + value, ok := entry.Fields[name] + return value, ok } -func baseEntryLabel(entry *journal.Entry) string { - for _, name := range []string{"SYSLOG_IDENTIFIER", "_COMM", "_EXE"} { - if value := firstString(entry, name); value != "" { - return value - } - } - return "unknown" +func shortField(entry *journal.Entry, name string, options outputOptions) ([]byte, bool) { + value, ok := lastEntryValue(entry, name) + return value, ok && (options.showAll || len(value) < printCharThreshold && journalTextPrintable(value)) } -func formatEntryLabel(entry *journal.Entry, label string, options outputOptions) string { - var parts []string +func shortHeader(entry *journal.Entry, options outputOptions, withUnit bool) (string, int) { + var out strings.Builder + columns := 0 if !options.noHostname { - if hostname := firstString(entry, "_HOSTNAME"); hostname != "" { - parts = append(parts, hostname) + if value, ok := shortField(entry, "_HOSTNAME", options); ok { + out.WriteByte(' ') + out.Write(cStringBytes(value)) + columns += len(value) + 1 } } - if label == "" { - label = "unknown" - } - if pid := firstString(entry, "_PID"); pid != "" { - label += "[" + pid + "]" - } else if pid := firstString(entry, "SYSLOG_PID"); pid != "" { - label += "[" + pid + "]" + unitShown := false + if withUnit { + _, unitOK := shortField(entry, "_SYSTEMD_UNIT", options) + _, userOK := shortField(entry, "_SYSTEMD_USER_UNIT", options) + if unitOK || userOK { + unit, unitOK := lastEntryValue(entry, "_SYSTEMD_UNIT") + userUnit, userOK := lastEntryValue(entry, "_SYSTEMD_USER_UNIT") + if unitOK { + out.WriteByte(' ') + out.Write(cStringBytes(unit)) + columns += len(unit) + 1 + } + if userOK { + if unitOK { + out.WriteByte('/') + } else { + out.WriteByte(' ') + } + out.Write(cStringBytes(userUnit)) + // v262 accounts the system-unit length here as well. + columns += len(unit) + 1 + } + unitShown = true + } } - parts = append(parts, label) - return strings.Join(parts, " ") -} - -func unitLabel(entry *journal.Entry) string { - for _, name := range []string{"_SYSTEMD_UNIT", "_SYSTEMD_USER_UNIT", "UNIT", "USER_UNIT", "OBJECT_SYSTEMD_UNIT", "OBJECT_SYSTEMD_USER_UNIT"} { - if value := firstString(entry, name); value != "" { - return value + if !unitShown { + value, ok := shortField(entry, "SYSLOG_IDENTIFIER", options) + if !ok { + value, ok = shortField(entry, "_COMM", options) + } + if ok { + out.WriteByte(' ') + out.Write(cStringBytes(value)) + columns += len(value) + 1 + } else { + out.WriteString(" unknown") } } - return "" + pid, ok := shortField(entry, "_PID", options) + if !ok { + pid, ok = shortField(entry, "SYSLOG_PID", options) + } + if ok { + out.WriteByte('[') + out.Write(cStringBytes(pid)) + out.WriteByte(']') + columns += len(pid) + 2 + } + return out.String(), columns } -func displayMessage(entry *journal.Entry, options outputOptions, prefixColumns int) string { +func displayMessage(entry *journal.Entry, options outputOptions, prefixColumns, columns int, highlight matchRange) string { + if options.colors.level != 0 || options.urlify { + return displayColoredMessage(entry, options, prefixColumns, columns, highlight) + } values := entryValues(entry, "MESSAGE") if len(values) == 0 { return "" } - value := values[0] - if options.truncateNewline { + value := values[len(values)-1] + if !options.showAll { + var printable bool + value, printable = prepareShortMessage(value, options.truncateNewline) + if !printable { + return blobData(len(value)) + } + } else if options.truncateNewline { value = truncateAtNewline(value) } - if !options.showAll && !journalTextPrintable(value) { - return blobData(len(value)) + if options.showAll && bytes.IndexByte(value, 0) >= 0 { + return string(fullMessageWithNUL(value, prefixColumns)) } - out := indentContinuationLines(cStringBytes(value), prefixColumns) if !options.showAll && !options.fullWidth { - out = ellipsizeLine(out, prefixColumns) + return string(shortenedMessage(value, prefixColumns, columns)) + } + out := indentContinuationLines(value, prefixColumns) + if len(out) > 0 && out[len(out)-1] == '\n' { + out = out[:len(out)-1] } return string(out) } +func prepareShortMessage(value []byte, truncate bool) ([]byte, bool) { + printable := true + for i, ch := range string(value) { + if ch == '\x1b' || ch == '\t' || ch == '\r' { + value = stripShortControls(value, i) + if truncate { + value = truncateAtNewline(value) + } + return value, journalTextPrintable(value) + } + if truncate && (ch == '\n' || ch == 0) { + return value[:i], printable + } + if ch < 0x20 && ch != '\n' || ch >= 0x7f && ch <= 0x9f { + printable = false + } else if ch == utf8.RuneError { + _, size := utf8.DecodeRune(value[i:]) + printable = printable && size != 1 + } + } + return value, printable +} + +func stripShortControls(value []byte, start int) []byte { + return stripShortControlsWithHighlight(value, start, nil) +} + +func stripShortControlsWithHighlight(value []byte, start int, highlight *matchRange) []byte { + var original, shift matchRange + if highlight != nil { + original = *highlight + } + out := make([]byte, 0, len(value)) + out = append(out, value[:start]...) + for i := start; i < len(value); { + switch value[i] { + case '\t': + out = append(out, " "...) + advanceHighlight(original, &shift, i, 7) + i++ + case '\r': + end := i + 1 + for end < len(value) && value[end] == '\r' { + end++ + } + if end < len(value) && value[end] != '\n' { + out = append(out, value[i:end]...) + } + i = end + case '\x1b': + if i+1 == len(value) { + out = append(out, value[i]) + i++ + continue + } + end := i + 2 + complete := false + invalidAt, size := i+1, 1 + switch value[i+1] { + case '[': + size = 2 + for end < len(value) { + b := value[end] + end++ + if b == 'm' { + complete = true + break + } + if b != 0 && b != ';' && b != ':' && (b < '0' || b > '9') { + invalidAt = end - 1 + break + } + invalidAt = end + } + case ']': + size = 2 + for end < len(value) { + b := value[end] + end++ + if b == '\x07' { + complete = true + break + } + if b == '\x1b' { + invalidAt = end + if end < len(value) && value[end] == '\\' { + end++ + complete = true + } + break + } + if b < 32 || b >= 127 { + invalidAt = end - 1 + break + } + invalidAt = end + } + } + if complete { + i = end + } else { + // Native preserves malformed sequences and processes their body again. + advanceHighlight(original, &shift, invalidAt, size) + out = append(out, value[i:i+2]...) + i += 2 + } + default: + out = append(out, value[i]) + i++ + } + } + if highlight != nil { + highlight.start += shift.start + highlight.end += shift.end + } + return out +} + +func fullMessageWithNUL(value []byte, prefixColumns int) []byte { + var out []byte + for start := 0; start < len(value); { + end := bytes.IndexByte(value[start:], '\n') + if end < 0 { + end = len(value) + } else { + end += start + } + if start > 0 { + out = append(out, '\n') + out = append(out, bytes.Repeat([]byte{' '}, prefixColumns)...) + } + out = append(out, cStringBytes(value[start:end])...) + start = end + 1 + } + return out +} + func indentContinuationLines(value []byte, prefixColumns int) []byte { if !bytes.Contains(value, []byte{'\n'}) { return value @@ -416,14 +676,6 @@ func indentContinuationLines(value []byte, prefixColumns int) []byte { return out } -func firstString(entry *journal.Entry, name string) string { - values := entryValues(entry, name) - if len(values) == 0 { - return "" - } - return string(values[0]) -} - func displayValue(value []byte, truncateNewline bool) string { if truncateNewline { value = truncateAtNewline(value) @@ -432,17 +684,17 @@ func displayValue(value []byte, truncateNewline bool) string { } func displayVerboseValue(name string, value []byte, options outputOptions) string { - if options.showAll { - return string(cStringBytes(value)) - } - if !journalTextPrintable(value) || (!options.fullWidth && len(name)+1+len(value) >= printCharThreshold) { + if !options.showAll && (!journalTextPrintable(value) || (!options.fullWidth && len(name)+1+len(value) >= printCharThreshold)) { return blobData(len(value)) } + if bytes.ContainsAny(value, "\n\x00") { + return string(fullMessageWithNUL(value, 4+len(name)+1)) + } return string(value) } func truncateAtNewline(value []byte) []byte { - if idx := bytes.IndexByte(value, '\n'); idx >= 0 { + if idx := bytes.IndexAny(value, "\x00\r\n"); idx >= 0 { return value[:idx] } return value @@ -455,21 +707,169 @@ func cStringBytes(value []byte) []byte { return value } -func ellipsizeLine(value []byte, prefixColumns int) []byte { - limit := defaultColumns - prefixColumns - 1 - if limit < 0 { - limit = 0 +var ellipsisMarker = func() string { + locale := os.Getenv("LC_ALL") + if locale == "" { + locale = os.Getenv("LC_CTYPE") + } + if locale == "" { + locale = os.Getenv("LANG") } - if len(value) <= limit { + if strings.Contains(strings.ToLower(locale), "utf") { + return "…" + } + return "..." +}() + +func shortenedMessage(value []byte, prefixColumns, columns int) []byte { + var out []byte + for start, line := 0, 0; start < len(value); line++ { + end := bytes.IndexByte(value[start:], '\n') + if end < 0 { + end = len(value) + } else { + end += start + } + part := value[start:end] + tail := line == 2 || end+1 >= printCharThreshold + shorten := prefixColumns+len(part)+1 >= columns || tail + if line > 0 { + out = append(out, '\n') + } + if shorten && columns-prefixColumns < 3 { + out = append(out, "..."...) + } else { + if line > 0 { + out = append(out, bytes.Repeat([]byte{' '}, prefixColumns)...) + } + if !shorten { + out = append(out, part...) + } else if columns-prefixColumns > len(part)+3 { + out = append(out, part...) + out = append(out, "..."...) + } else { + percent := 90 + if tail { + percent = 100 + } + out = append(out, ellipsizeLine(part, columns-prefixColumns, percent)...) + } + } + if tail { + break + } + start = end + 1 + } + return out +} + +func ellipsizeLine(value []byte, columns, percent int) []byte { + if isASCII(value) { + if len(value) <= columns { + return value + } + markerWidth := len(ellipsisMarker) + if ellipsisMarker == "…" { + markerWidth = 1 + } + left := ((columns-markerWidth)*percent + 50) / 100 + right := columns - markerWidth - left + out := append([]byte{}, value[:left]...) + out = append(out, ellipsisMarker...) + return append(out, value[len(value)-right:]...) + } + left, right, used := 0, len(value), 0 + budget := (columns - 1) * percent / 100 + for left < len(value) { + ch, size := utf8.DecodeRune(value[left:]) + width := journalRuneWidth(ch) + if used+width > budget { + break + } + used += width + left += size + } + for right > left && used < columns { + ch, size := utf8.DecodeLastRune(value[left:right]) + width := journalRuneWidth(ch) + if used+width > columns { + break + } + used += width + right -= size + } + if left >= right { return value } - for limit > 0 && !utf8.RuneStart(value[limit]) { - limit-- + if used >= columns { + if right < len(value) { + _, size := utf8.DecodeRune(value[right:]) + right += size + } else if left > 0 { + _, size := utf8.DecodeLastRune(value[:left]) + left -= size + } } - out := make([]byte, 0, limit+len("…")) - out = append(out, value[:limit]...) + out := append([]byte{}, value[:left]...) out = append(out, "…"...) - return out + return append(out, value[right:]...) +} + +func isASCII(value []byte) bool { + for _, b := range value { + if b >= utf8.RuneSelf { + return false + } + } + return true +} + +// v262 fixes character width to these intervals. +var journalWideRunes = [][2]rune{ + {0x1100, 0x115F}, + {0x2329, 0x232A}, + {0x2E80, 0x2E99}, + {0x2E9B, 0x2EF3}, + {0x2F00, 0x2FD5}, + {0x2FF0, 0x2FFB}, + {0x3000, 0x303E}, + {0x3041, 0x3096}, + {0x3099, 0x30FF}, + {0x3105, 0x312D}, + {0x3131, 0x318E}, + {0x3190, 0x31BA}, + {0x31C0, 0x31E3}, + {0x31F0, 0x321E}, + {0x3220, 0x3247}, + {0x3250, 0x32FE}, + {0x3300, 0x4DBF}, + {0x4E00, 0xA48C}, + {0xA490, 0xA4C6}, + {0xA960, 0xA97C}, + {0xAC00, 0xD7A3}, + {0xF900, 0xFAFF}, + {0xFE10, 0xFE19}, + {0xFE30, 0xFE52}, + {0xFE54, 0xFE66}, + {0xFE68, 0xFE6B}, + {0xFF01, 0xFF60}, + {0xFFE0, 0xFFE6}, + {0x1B000, 0x1B001}, + {0x1F200, 0x1F202}, + {0x1F210, 0x1F23A}, + {0x1F240, 0x1F248}, + {0x1F250, 0x1F251}, + {0x1F300, 0x1F567}, + {0x20000, 0x2FFFD}, + {0x30000, 0x3FFFD}, +} + +func journalRuneWidth(ch rune) int { + i := sort.Search(len(journalWideRunes), func(i int) bool { return journalWideRunes[i][1] >= ch }) + if i < len(journalWideRunes) && journalWideRunes[i][0] <= ch { + return 2 + } + return 1 } func journalTextPrintable(value []byte) bool { @@ -577,9 +977,12 @@ func jsonObject(entry *journal.Entry, options outputOptions) map[string]any { func addJSONValue(object map[string]any, name string, value []byte, showAll bool) { encoded := jsonValueForBytes(name, value, showAll) if existing, ok := object[name]; ok { - if values, ok := existing.([]any); ok { - object[name] = append(values, encoded) - return + if values, ok := existing.([]any); ok && len(values) > 0 { + // A scalar binary value is a number array, not repeated field values. + if _, binary := values[0].(int); !binary { + object[name] = append(values, encoded) + return + } } object[name] = []any{existing, encoded} return diff --git a/go/cmd/journalctl/output_color.go b/go/cmd/journalctl/output_color.go new file mode 100644 index 00000000..209f1b2a --- /dev/null +++ b/go/cmd/journalctl/output_color.go @@ -0,0 +1,117 @@ +//go:build cgo + +package main + +import ( + "bytes" + "strings" + + "github.com/netdata/systemd-journal-sdk/go/journal" +) + +func entryPriority(entry *journal.Entry, first bool) int { + value, _ := lastEntryValue(entry, "PRIORITY") + if first { + if values := entryValues(entry, "PRIORITY"); len(values) > 0 { + value = values[0] + } + } + if len(value) == 1 && value[0] >= '0' && value[0] <= '7' { + return int(value[0] - '0') + } + return 6 +} + +func advanceHighlight(original matchRange, shift *matchRange, at, size int) { + if at < original.start { + shift.start += size + } + if at < original.end { + shift.end += size + } +} + +func displayColoredMessage(entry *journal.Entry, options outputOptions, prefix, columns int, span matchRange) string { + value, _ := lastEntryValue(entry, "MESSAGE") + if !options.showAll { + if start := bytes.IndexAny(value, "\t\r\x1b"); start >= 0 { + value = stripShortControlsWithHighlight(value, start, &span) + } + } + if options.truncateNewline { + value = truncateAtNewline(value) + } + documentation := "" + if options.urlify { + if url, ok := shortField(entry, "DOCUMENTATION", options); ok { + documentation = documentationLink(url) + } + } + if !options.showAll && !journalTextPrintable(value) { + return documentation + blobData(len(value)) + } + if config, ok := lastEntryValue(entry, "CONFIG_FILE"); ok && bytes.HasPrefix(value, config) && + (len(value) == len(config) || value[len(config)] == ':' || value[len(config)] == ' ') && + (span.start == 0 || span.start > len(config)) { + if linked := fileLink(config, options.urlify); linked != nil && len(linked) >= len(config) { + shift := len(linked) - len(config) + value = append(linked, value[len(config):]...) + span.start += shift + span.end += shift + } + } + transport, _ := lastEntryValue(entry, "_TRANSPORT") + on, off, highlight := options.colors.log(entryPriority(entry, false), string(cStringBytes(transport)) == "audit") + var out strings.Builder + out.WriteString(documentation) + for start, line := 0, 0; start < len(value); line++ { + end := bytes.IndexByte(value[start:], '\n') + if end < 0 { + end = len(value) + } else { + end += start + } + part := value[start:end] + tail := line == 2 || end+1 >= printCharThreshold + shorten := !options.showAll && !options.fullWidth && (prefix+len(part)+1 >= columns || tail) + if line > 0 { + out.WriteByte('\n') + } + if shorten && columns-prefix < 3 { + out.WriteString("...") + } else { + if line > 0 { + out.WriteString(strings.Repeat(" ", prefix)) + } + out.WriteString(on) + if !shorten { + if start <= span.start && span.start < len(part) { + out.Write(cStringBytes(part[:span.start])) + out.WriteString(highlight) + out.Write(cStringBytes(part[span.start:min(span.end, len(part))])) + if len(part) > span.end { + out.WriteString(on) + out.Write(cStringBytes(part[span.end:])) + } + } else { + out.Write(cStringBytes(part)) + } + } else if columns-prefix > len(part)+3 { + out.Write(cStringBytes(part)) + out.WriteString("...") + } else { + percent := 90 + if tail { + percent = 100 + } + out.Write(cStringBytes(ellipsizeLine(part, columns-prefix, percent))) + } + out.WriteString(off) + } + if shorten && tail { + break + } + start = end + 1 + } + return out.String() +} diff --git a/go/cmd/journalctl/output_links.go b/go/cmd/journalctl/output_links.go new file mode 100644 index 00000000..f5f0ae95 --- /dev/null +++ b/go/cmd/journalctl/output_links.go @@ -0,0 +1,142 @@ +//go:build cgo + +package main + +import ( + "os" + "path/filepath" + "runtime" + "strings" + "sync" +) + +func urlifyEnabled(colors outputColors) bool { + if value, ok := os.LookupEnv("SYSTEMD_URLIFY"); ok { + if enabled, err := parseBoolOption("SYSTEMD_URLIFY", value); err == nil { + return enabled + } + } + return colors.level != 0 +} + +func terminalLink(url, text string) string { + if len(url) > 2000 { + return text + } + for i := range len(url) { + if url[i] < 32 || url[i] >= 127 { + return text + } + } + return "\x1b]8;;" + url + "\x1b\\" + text + "\x1b]8;;\x1b\\" +} + +var outputHostname = sync.OnceValue(func() string { + hostname, _ := os.Hostname() + return hostname +}) + +func fileLink(value []byte, enabled bool) []byte { + text := string(cStringBytes(value)) + if text == "" { + return nil + } + if !enabled { + return []byte(text) + } + if runtime.GOOS == "windows" { + path, err := filepath.Abs(text) + if err != nil { + return []byte(text) + } + url := windowsFileURL(path, outputHostname()) + if url == "" { + return []byte(text) + } + return []byte(terminalLink(url, text)) + } + path := text + if !filepath.IsAbs(path) { + cwd, err := os.Getwd() + if err != nil { + return nil + } + if !os.IsPathSeparator(cwd[len(cwd)-1]) { + cwd += string(filepath.Separator) + } + path = cwd + path + } + hostname := outputHostname() + if hostname == "" { + return nil + } + return []byte(terminalLink("file://"+hostname+path, text)) +} + +func documentationLink(value []byte) string { + url := string(cStringBytes(value)) + valid := false + for _, prefix := range []string{"http://", "https://", "file:/", "info:", "man:"} { + if strings.HasPrefix(url, prefix) && len(url) > len(prefix) { + valid = true + break + } + } + if !valid || !isASCII([]byte(url)) { + return "" + } + glyph := "[LNK]" + unicode := ellipsisMarker == "…" + if value, ok := os.LookupEnv("SYSTEMD_UTF8"); ok { + if flag, err := parseBoolOption("SYSTEMD_UTF8", value); err == nil { + unicode = flag + } + } + if unicode { + glyph = "[🡕]" + } + return terminalLink(url, glyph) + " " +} + +// Windows paths need a URI authority and slash-separated, escaped path bytes. +func windowsFileURL(path, hostname string) string { + path = strings.ReplaceAll(path, `\`, "/") + if strings.HasPrefix(path, "//?/") || strings.HasPrefix(path, "//./") { + return "" + } + authority := hostname + if strings.HasPrefix(path, "//") { + rest := path[2:] + slash := strings.IndexByte(rest, '/') + if slash <= 0 || slash == len(rest)-1 { + return "" + } + authority, path = rest[:slash], rest[slash:] + } else { + if len(path) < 3 || path[1] != ':' || path[2] != '/' || !((path[0] >= 'A' && path[0] <= 'Z') || (path[0] >= 'a' && path[0] <= 'z')) { + return "" + } + path = "/" + path + } + for i := range len(authority) { + c := authority[i] + if !((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '.' || c == '-') { + return "" + } + } + const hex = "0123456789ABCDEF" + var url strings.Builder + url.WriteString("file://") + url.WriteString(authority) + for i := range len(path) { + c := path[i] + if (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || strings.ContainsRune("/-._~:", rune(c)) { + url.WriteByte(c) + } else { + url.WriteByte('%') + url.WriteByte(hex[c>>4]) + url.WriteByte(hex[c&15]) + } + } + return url.String() +} diff --git a/go/cmd/journalctl/output_links_test.go b/go/cmd/journalctl/output_links_test.go new file mode 100644 index 00000000..d1e7a3d0 --- /dev/null +++ b/go/cmd/journalctl/output_links_test.go @@ -0,0 +1,40 @@ +//go:build cgo + +package main + +import ( + "path/filepath" + "runtime" + "testing" +) + +func TestWindowsFileURL(t *testing.T) { + for _, tc := range []struct{ path, want string }{ + {`C:\logs\example.conf`, "file://host/C:/logs/example.conf"}, + {`\\server\share\é #%.conf`, "file://server/share/%C3%A9%20%23%25.conf"}, + {"C:/logs/a?b", "file://host/C:/logs/a%3Fb"}, + {`\\?\C:\logs`, ""}, {`\\.\pipe\name`, ""}, {"relative", ""}, {"//server", ""}, + } { + if got := windowsFileURL(tc.path, "host"); got != tc.want { + t.Errorf("%q: got %q want %q", tc.path, got, tc.want) + } + } +} + +func TestRelativeWindowsFileLink(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("native Windows path resolution") + } + path := "relative #.conf" + absolute, err := filepath.Abs(path) + if err != nil { + t.Fatal(err) + } + want := terminalLink(windowsFileURL(absolute, outputHostname()), path) + if got := string(fileLink([]byte(path), true)); got != want { + t.Fatalf("got %q want %q", got, want) + } + if got := string(fileLink([]byte(path), false)); got != path { + t.Fatalf("plain got %q", got) + } +} diff --git a/go/cmd/journalctl/parser.go b/go/cmd/journalctl/parser.go new file mode 100644 index 00000000..1db92389 --- /dev/null +++ b/go/cmd/journalctl/parser.go @@ -0,0 +1,331 @@ +//go:build cgo + +package main + +import ( + "errors" + "flag" + "fmt" + "io" + "strconv" + "strings" +) + +type cliAction string + +const ( + actionShow cliAction = "show" + actionVacuum cliAction = "vacuum" + actionRotateVacuum cliAction = "rotate-and-vacuum" +) + +var shortOptions = map[byte]string{ + 'h': "help", 'e': "pager-end", 'f': "follow", 'o': "output", 'a': "all", 'l': "full", + 'n': "lines", 'q': "quiet", 'm': "merge", 'b': "boot", 'k': "dmesg", 'D': "directory", + 'p': "priority", 'g': "grep", 'c': "cursor", 'S': "since", 'U': "until", 't': "identifier", + 'T': "exclude-identifier", 'u': "unit", 'I': "I", 'N': "fields", 'F': "field", 'x': "catalog", + 'r': "reverse", 'M': "machine", 'i': "file", 'W': "no-hostname", +} + +func (f *cliFlags) selectAction(action cliAction) { + if action == "rotate" && f.action == actionVacuum || action == actionVacuum && f.action == "rotate" { + action = actionRotateVacuum + } + f.action = action + for name, dest := range map[cliAction]*bool{ + "list-boots": f.listBoots, "fields": f.fields, "sync": f.sync, "flush": f.flush, + "rotate": f.rotate, "relinquish-var": f.relinquish, "verify": f.verify, + "verify-only": f.verifyOnly, "new-id128": f.newID128Flag, + "list-invocations": f.listInvocationsFlag, "list-namespaces": f.listNamespacesFlag, + "disk-usage": f.diskUsageFlag, "header": f.headerFlag, "list-catalog": f.listCatalogFlag, + "dump-catalog": f.dumpCatalogFlag, "update-catalog": f.updateCatalogFlag, + "setup-keys": f.setupKeysFlag, "smart-relinquish-var": f.smartRelinquishVarFlag, + } { + *dest = name == action + } + if action != "field" { + *f.field = "" + } +} + +// parseCLIOptions preserves argv order and never rewrites a required option value. +func parseCLIOptions(argv []string, fs *flag.FlagSet, f *cliFlags, stdout io.Writer) ([]string, bool, error) { + f.fullWidth = true + f.parsed = true + f.action = actionShow + f.seen = make(map[string]bool) + var operands []string + for i := 0; i < len(argv); i++ { + arg := argv[i] + if arg == "--" { + operands = append(operands, argv[i+1:]...) + break + } + if arg == "-" || !strings.HasPrefix(arg, "-") { + operands = append(operands, arg) + continue + } + long := strings.HasPrefix(arg, "--") + token := arg[1:] + if long { + token = arg[2:] + } + for token != "" { + var name, value string + var attached bool + if long { + name, value, attached = strings.Cut(token, "=") + var err error + name, err = resolveLongOption(name, fs) + if err != nil { + return nil, false, err + } + token = "" + } else { + var ok bool + name, ok = shortOptions[token[0]] + if !ok { + return nil, false, fmt.Errorf("invalid option -- '%c'", token[0]) + } + token = token[1:] + } + optional := name == "boot" || name == "lines" || name == "case-sensitive" + noArg := name == "help" || name == "introspect-cli" + if opt := fs.Lookup(name); opt != nil { + if b, ok := opt.Value.(interface{ IsBoolFlag() bool }); ok { + noArg = b.IsBoolFlag() && !optional + } + } + if noArg { + if attached { + return nil, false, fmt.Errorf("option '--%s' doesn't allow an argument", name) + } + value = "true" + } else { + if !long && token != "" { + value, attached, token = token, true, "" + } + if !attached { + if optional { + switch name { + case "case-sensitive": + value = "true" + case "boot": + value = "0" + case "lines": + value = "10" + } + if i+1 < len(argv) && name != "case-sensitive" { + candidate := argv[i+1] + valid := false + if name == "boot" { + _, _, err := parseBootDescriptor(candidate) + valid = err == nil || candidate == "all" + } + if name == "lines" { + _, err := parseLinesLimitValue(candidate) + valid = err == nil + } + if valid { + i++ + value = candidate + } + } + } else { + if i+1 >= len(argv) { + return nil, false, fmt.Errorf("option '--%s' requires an argument", name) + } + i++ + value = argv[i] + } + } + } + terminal, err := applyCLIOption(name, value, fs, f, stdout) + if err != nil || terminal { + return nil, terminal, err + } + } + } + return operands, false, nil +} + +func resolveLongOption(name string, fs *flag.FlagSet) (string, error) { + if name == "help" || name == "introspect-cli" || fs.Lookup(name) != nil && len(name) > 1 { + return name, nil + } + var found string + ambiguous := false + visit := func(candidate string) { + if len(candidate) > 1 && strings.HasPrefix(candidate, name) { + if found != "" { + ambiguous = true + } + found = candidate + } + } + fs.VisitAll(func(f *flag.Flag) { visit(f.Name) }) + visit("help") + visit("introspect-cli") + if ambiguous { + return "", fmt.Errorf("option '--%s' is ambiguous", name) + } + if found == "" { + return "", fmt.Errorf("unrecognized option '--%s'", name) + } + return found, nil +} + +func applyCLIOption(name, value string, fs *flag.FlagSet, f *cliFlags, stdout io.Writer) (bool, error) { + switch name { + case "help": + printUsage(fs, stdout) + return true, nil + case "version": + fmt.Fprintln(stdout, "journalctl (systemd-journal-sdk Go rewrite)") + fmt.Fprintln(stdout, "CLI compatibility: systemd v262 (8cc40e0c5e92)") + fmt.Fprintln(stdout, "portable file-backed mode; PCRE2 "+pcreVersion()) + return true, nil + case "introspect-cli": + return true, printCLIIntrospection(stdout, *f.output) + case "output": + if value == "help" { + printOutputModeHelp(stdout) + return true, nil + } + if err := validateOutputMode(value); err != nil { + return false, err + } + case "facility": + for _, part := range strings.Split(value, ",") { + if part == "" { + continue + } + if part == "help" { + printFacilityHelp(stdout, *f.quietFlag) + return true, nil + } + if _, err := parseFacility(part); err != nil { + return false, err + } + } + case "priority": + if _, err := parsePriorityFilter([]string{value}); err != nil { + return false, err + } + case "lines": + if _, err := parseLinesLimitValue(value); err != nil { + return false, err + } + case "boot": + if value != "all" { + if _, _, err := parseBootDescriptor(value); err != nil { + return false, err + } + } + *f.thisBootFlag = false + case "this-boot": + f.boot = optionalStringFlag{set: true, value: "0"} + case "invocation": + if _, _, _, err := parseInvocationDescriptor(value); err != nil { + return false, err + } + *f.invocationShortFlag = false + f.invocationSet = true + case "I": + *f.invocationFlag = "0" + f.invocationSet = true + case "case-sensitive", "synchronize-on-exit": + if _, err := parseBoolOption("--"+name, value); err != nil { + return false, err + } + case "since", "until": + parsed, err := f.timestamps.parse(value) + if err != nil { + return false, fmt.Errorf("Failed to parse timestamp: %s", value) + } + if name == "since" { + f.sinceUsec = &parsed + } else { + f.untilUsec = &parsed + } + case "vacuum-size": + if _, err := parseVacuumSize(value); err != nil { + return false, err + } + case "vacuum-files": + if _, err := parseNativeUint64(value); err != nil { + return false, fmt.Errorf("Failed to parse vacuum files: %s", value) + } + case "vacuum-time", "interval": + parsed, err := parseDurationUsecAllowZero(value) + if err != nil || name == "interval" && parsed == 0 { + return false, fmt.Errorf("Failed to parse %s: %s", name, value) + } + case "head", "tail": + if n, err := strconv.Atoi(value); err != nil || n < 0 { + return false, fmt.Errorf("--%s must be a non-negative integer", name) + } + } + if name == "output-fields" { + fields := parseOutputFields(*f.outputFieldsFlag) + seen := make(map[string]bool, len(fields)) + for _, field := range fields { + seen[field] = true + } + for _, field := range parseOutputFields(value) { + if !seen[field] { + fields = append(fields, field) + seen[field] = true + } + } + value = strings.Join(fields, ",") + } + switch name { + case "field", "fields", "list-boots", "list-invocations", "list-namespaces", "disk-usage", "header", + "list-catalog", "dump-catalog", "update-catalog", "setup-keys", "new-id128", "verify", "verify-only", + "sync", "relinquish-var", "smart-relinquish-var", "flush", "rotate": + f.selectAction(cliAction(name)) + case "vacuum-size", "vacuum-files", "vacuum-time": + f.selectAction(actionVacuum) + case "verify-key": + f.selectAction("verify") + *f.mergeFlag = false + } + if err := fs.Set(name, value); err != nil { + return false, err + } + f.seen[name] = true + if name == "grep" { + f.grepSet = true + } + if name == "full" { + *f.noFullFlag = false + f.fullWidth = true + } + if name == "no-full" { + *f.fullFlag = false + f.fullWidth = false + } + if name == "output" && (*f.output == "cat" || *f.output == "export" || strings.HasPrefix(*f.output, "json")) { + *f.quietFlag = true + } + return false, nil +} + +func (f *cliFlags) fieldListHasScope() bool { + boot := f.boot.set && f.boot.value != "all" + invocation := f.invocationSet && (*f.invocationShortFlag || *f.invocationFlag != "all") + return boot || invocation || *f.dmesgFlag || len(f.unitFlag.values) > 0 || len(f.userUnitFlag.values) > 0 || + len(f.identifierFlag.values) > 0 || len(f.excludeIdentifierFlag.values) > 0 || len(f.priorityFlag.values) > 0 || + len(f.facilityFlag.values) > 0 || f.seen["since"] || f.seen["until"] || f.seen["cursor"] || + f.seen["after-cursor"] || f.seen["cursor-file"] || f.grepSet +} + +func validateDisjunctions(args []string) error { + for i, arg := range args { + if arg == "+" && (i == 0 || i == len(args)-1 || args[i-1] == "+") { + return errors.New("\"+\" can only be used between terms.") + } + } + return nil +} diff --git a/go/cmd/journalctl/pcre.go b/go/cmd/journalctl/pcre.go new file mode 100644 index 00000000..f770ee73 --- /dev/null +++ b/go/cmd/journalctl/pcre.go @@ -0,0 +1,142 @@ +//go:build cgo + +package main + +/* +#cgo pkg-config: libpcre2-8 +#define PCRE2_CODE_UNIT_WIDTH 8 +#include +#if PCRE2_MAJOR < 10 || (PCRE2_MAJOR == 10 && PCRE2_MINOR < 42) +#error journalctl requires PCRE2 10.42 or later +#endif + +// Match data must die before returning: PCRE2 retains a borrowed subject pointer. +static int journal_pattern_match(pcre2_code *code, const unsigned char *subject, size_t size, size_t *range) { + static const unsigned char empty = 0; + pcre2_match_data *md = pcre2_match_data_create(1, NULL); + if (!md) return PCRE2_ERROR_NOMEMORY; + int rc = pcre2_match(code, subject ? subject : &empty, size, 0, 0, md, NULL); + if (rc >= 0 && range) { + PCRE2_SIZE *offsets = pcre2_get_ovector_pointer(md); + range[0] = offsets[0]; + range[1] = offsets[1]; + } + pcre2_match_data_free(md); + return rc; +} + +static pcre2_code *journal_pattern_compile(const unsigned char *pattern, size_t size, uint32_t flags, + int *error, size_t *offset) { + static const unsigned char empty = 0; + return pcre2_compile(pattern ? pattern : &empty, size, flags, error, offset, NULL); +} +*/ +import "C" + +import ( + "fmt" + "runtime" + "unsafe" +) + +type pcrePattern struct{ code *C.pcre2_code } + +func pcreVersion() string { + var buf [128]byte + if C.pcre2_config(C.PCRE2_CONFIG_VERSION, unsafe.Pointer(&buf[0])) < 0 { + return "unknown" + } + for i, b := range buf { + if b == 0 { + return string(buf[:i]) + } + } + return string(buf[:]) +} + +func pcreError(code C.int) string { + var buf [512]byte + n := C.pcre2_get_error_message(code, (*C.uchar)(unsafe.Pointer(&buf[0])), C.size_t(len(buf))) + if n < 0 { + return fmt.Sprintf("PCRE2 error %d", int(code)) + } + return string(buf[:int(n)]) +} + +func compilePCRE(pattern []byte, sensitive bool) (*pcrePattern, error) { + var ptr *C.uchar + if len(pattern) > 0 { + ptr = (*C.uchar)(unsafe.Pointer(&pattern[0])) + } + var code C.int + var offset C.size_t + var options C.uint32_t + if !sensitive { + options = C.PCRE2_CASELESS + } + compiled := C.journal_pattern_compile(ptr, C.size_t(len(pattern)), options, &code, &offset) + runtime.KeepAlive(pattern) + if compiled == nil { + return nil, fmt.Errorf("Bad pattern: %s (offset %d)", pcreError(code), uint64(offset)) + } + return &pcrePattern{code: compiled}, nil +} + +func (p *pcrePattern) Close() { + if p != nil && p.code != nil { + C.pcre2_code_free(p.code) + p.code = nil + } +} + +func (p *pcrePattern) Match(subject []byte) (bool, error) { + return p.match(subject, nil) +} + +func (p *pcrePattern) match(subject []byte, highlight *matchRange) (bool, error) { + var ptr *C.uchar + if len(subject) > 0 { + ptr = (*C.uchar)(unsafe.Pointer(&subject[0])) + } + var rc C.int + if highlight == nil { + rc = C.journal_pattern_match(p.code, ptr, C.size_t(len(subject)), nil) + } else { + var offsets [2]C.size_t + rc = C.journal_pattern_match(p.code, ptr, C.size_t(len(subject)), &offsets[0]) + if rc >= 0 { + *highlight = matchRange{start: int(offsets[0]), end: int(offsets[1])} + } + } + runtime.KeepAlive(subject) + runtime.KeepAlive(p) + if rc == C.PCRE2_ERROR_NOMATCH { + return false, nil + } + if rc < 0 { + return false, fmt.Errorf("Pattern matching failed: %s", pcreError(rc)) + } + return true, nil +} + +func compileGrepFilter(pattern string, caseSensitive optionalStringFlag) (*pcrePattern, error) { + var sensitive bool + if caseSensitive.set { + parsed, err := parseBoolOption("--case-sensitive", caseSensitive.value) + if err != nil { + return nil, err + } + sensitive = parsed + } else { + upper, err := compilePCRE([]byte("[[:upper:]]"), true) + if err != nil { + return nil, err + } + sensitive, err = upper.Match([]byte(pattern)) + upper.Close() + if err != nil { + return nil, err + } + } + return compilePCRE([]byte(pattern), sensitive) +} diff --git a/go/cmd/journalctl/table_json.go b/go/cmd/journalctl/table_json.go new file mode 100644 index 00000000..3c89fe13 --- /dev/null +++ b/go/cmd/journalctl/table_json.go @@ -0,0 +1,64 @@ +//go:build cgo + +package main + +import ( + "bytes" + "encoding/json" + "io" + "strings" +) + +func writeTableJSON(out io.Writer, mode string, rows []map[string]any) error { + var data []byte + var err error + if mode == "json-pretty" { + var buf bytes.Buffer + if rows == nil { + buf.WriteString("null") + } else { + buf.WriteByte('[') + if len(rows) > 0 { + buf.WriteByte('\n') + } + for i, row := range rows { + buf.WriteByte('\t') + if err := writeSystemdPrettyJSONValue(&buf, row, 1); err != nil { + return err + } + if i+1 < len(rows) { + buf.WriteByte(',') + } + buf.WriteByte('\n') + } + buf.WriteByte(']') + } + data = buf.Bytes() + } else { + data, err = json.Marshal(rows) + } + if err != nil { + return err + } + if detectOutputColors().level != 0 { + data = colorJSON(data) + } + if mode == "json-sse" { + data = append(append([]byte("data: "), data...), '\n', '\n') + } else if mode == "json-seq" { + data = append(append([]byte{0x1e}, data...), '\n') + } else { + data = append(data, '\n') + } + _, err = out.Write(data) + return err +} + +func jsonTableMode(mode string) bool { return strings.HasPrefix(mode, "json") } + +func tableTimestamp(value uint64) any { + if value == ^uint64(0) { + return nil + } + return value +} diff --git a/go/cmd/journalctl/table_json_test.go b/go/cmd/journalctl/table_json_test.go new file mode 100644 index 00000000..1689ef35 --- /dev/null +++ b/go/cmd/journalctl/table_json_test.go @@ -0,0 +1,20 @@ +//go:build cgo + +package main + +import ( + "bytes" + "testing" +) + +func TestTablePrettyJSONNativeSpacing(t *testing.T) { + t.Setenv("SYSTEMD_COLORS", "0") + var out bytes.Buffer + if err := writeTableJSON(&out, "json-pretty", []map[string]any{{"index": 0, "label": "a:b"}}); err != nil { + t.Fatal(err) + } + want := "[\n\t{\n\t\t\"index\" : 0,\n\t\t\"label\" : \"a:b\"\n\t}\n]\n" + if out.String() != want { + t.Fatalf("got %q, want %q", out.String(), want) + } +} diff --git a/go/cmd/journalctl/terminal_other.go b/go/cmd/journalctl/terminal_other.go new file mode 100644 index 00000000..41c38c49 --- /dev/null +++ b/go/cmd/journalctl/terminal_other.go @@ -0,0 +1,9 @@ +//go:build cgo && !unix && !windows + +package main + +func outputIsTerminal() bool { return false } + +func stdoutColumns() (int, bool) { + return 0, false +} diff --git a/go/cmd/journalctl/terminal_unix.go b/go/cmd/journalctl/terminal_unix.go new file mode 100644 index 00000000..8aec1fb4 --- /dev/null +++ b/go/cmd/journalctl/terminal_unix.go @@ -0,0 +1,25 @@ +//go:build cgo && unix + +package main + +/* +#include +static int journal_output_isatty(void) { return isatty(STDOUT_FILENO) && isatty(STDERR_FILENO); } +*/ +import "C" + +import ( + "os" + + "golang.org/x/sys/unix" +) + +func outputIsTerminal() bool { return C.journal_output_isatty() == 1 } + +func stdoutColumns() (int, bool) { + size, err := unix.IoctlGetWinsize(int(os.Stdout.Fd()), unix.TIOCGWINSZ) + if err != nil || size.Col == 0 { + return 0, false + } + return int(size.Col), true +} diff --git a/go/cmd/journalctl/terminal_windows.go b/go/cmd/journalctl/terminal_windows.go new file mode 100644 index 00000000..ce3edafc --- /dev/null +++ b/go/cmd/journalctl/terminal_windows.go @@ -0,0 +1,24 @@ +//go:build cgo && windows + +package main + +import ( + "os" + + "golang.org/x/sys/windows" +) + +func outputIsTerminal() bool { + var mode uint32 + return windows.GetConsoleMode(windows.Handle(os.Stdout.Fd()), &mode) == nil && + windows.GetConsoleMode(windows.Handle(os.Stderr.Fd()), &mode) == nil +} + +func stdoutColumns() (int, bool) { + var info windows.ConsoleScreenBufferInfo + if err := windows.GetConsoleScreenBufferInfo(windows.Handle(os.Stdout.Fd()), &info); err != nil { + return 0, false + } + width := int(info.Window.Right) - int(info.Window.Left) + 1 + return width, width > 0 +} diff --git a/go/cmd/journalctl/timestamp.go b/go/cmd/journalctl/timestamp.go new file mode 100644 index 00000000..cd04f795 --- /dev/null +++ b/go/cmd/journalctl/timestamp.go @@ -0,0 +1,492 @@ +//go:build cgo + +package main + +import ( + "errors" + "fmt" + "math" + "strconv" + "strings" + "time" + _ "time/tzdata" +) + +const timestampMax = uint64(253402214399000000) + +type timestampParser struct { + localGuess, utcGuess int64 + local *timestampZone +} + +func durationValue(value string) (uint64, error) { + original := value + value = strings.TrimLeft(value, " \t\r\n\v\f") + if strings.TrimRight(value, " \t\r\n\v\f") == "infinity" { + return math.MaxUint64, nil + } + units := []struct { + name string + multiplier uint64 + }{ + {"seconds", 1000000}, {"second", 1000000}, {"sec", 1000000}, {"s", 1000000}, + {"minutes", 60000000}, {"minute", 60000000}, {"min", 60000000}, + {"months", 2629800000000}, {"month", 2629800000000}, {"M", 2629800000000}, + {"msec", 1000}, {"ms", 1000}, {"m", 60000000}, + {"hours", 3600000000}, {"hour", 3600000000}, {"hr", 3600000000}, {"h", 3600000000}, + {"days", 86400000000}, {"day", 86400000000}, {"d", 86400000000}, + {"weeks", 604800000000}, {"week", 604800000000}, {"w", 604800000000}, + {"years", 31557600000000}, {"year", 31557600000000}, {"y", 31557600000000}, + {"usec", 1}, {"us", 1}, {"μs", 1}, {"µs", 1}, + } + var total uint64 + seen := false + bad := func() (uint64, error) { return 0, fmt.Errorf("failed to parse duration: %s", original) } + for { + value = strings.TrimLeft(value, " \t\r\n\v\f") + if value == "" { + if !seen { + return bad() + } + return total, nil + } + if value[0] == '-' { + return bad() + } + n := 0 + if value[0] == '+' { + n++ + } + digits := n + for n < len(value) && value[n] >= '0' && value[n] <= '9' { + n++ + } + var integer uint64 + if n > digits { + parsed, err := strconv.ParseUint(value[digits:n], 10, 63) + if err != nil { + return bad() + } + integer = parsed + } else if n > 0 || n >= len(value) || value[n] != '.' { + return bad() + } + fraction := "" + if n < len(value) && value[n] == '.' { + n++ + start := n + for n < len(value) && value[n] >= '0' && value[n] <= '9' { + n++ + } + if n == start { + return bad() + } + fraction = value[start:n] + } + afterNumber := value[n:] + suffix := strings.TrimLeft(afterNumber, " \t\r\n\v\f") + multiplier := uint64(1000000) + matched := false + for _, unit := range units { + if strings.HasPrefix(suffix, unit.name) { + multiplier = unit.multiplier + suffix = suffix[len(unit.name):] + matched = true + break + } + } + if !matched && len(afterNumber) == len(suffix) && suffix != "" { + return bad() + } + if integer >= math.MaxUint64/multiplier { + return bad() + } + amount := integer * multiplier + if amount >= math.MaxUint64-total { + return bad() + } + total += amount + weight := multiplier / 10 + for _, digit := range []byte(fraction) { + amount = uint64(digit-'0') * weight + if amount >= math.MaxUint64-total { + return bad() + } + total += amount + weight /= 10 + } + value = suffix + seen = true + } +} + +func parseUTCOffset(value string) (int64, bool) { + if value == "Z" { + return 0, true + } + if len(value) != 3 && len(value) != 5 && len(value) != 6 { + return 0, false + } + if value[0] != '+' && value[0] != '-' { + return 0, false + } + parse2 := func(s string) (int64, bool) { + if len(s) != 2 || s[0] < '0' || s[0] > '9' || s[1] < '0' || s[1] > '9' { + return 0, false + } + return int64(s[0]-'0')*10 + int64(s[1]-'0'), true + } + h, ok := parse2(value[1:3]) + if !ok { + return 0, false + } + var m int64 + if len(value) > 3 { + if len(value) == 6 { + if value[3] != ':' { + return 0, false + } + m, ok = parse2(value[4:]) + } else { + m, ok = parse2(value[3:]) + } + if !ok || m >= 60 { + return 0, false + } + } + offset := h*3600 + m*60 + if offset > 86400 { + return 0, false + } + if value[0] == '-' { + offset = -offset + } + return offset, true +} + +func (p *timestampParser) parse(value string) (uint64, error) { + if strings.HasPrefix(value, "@") { + return durationValue(value[1:]) + } + if p.local == nil { + p.local = localTimestampZone() + } + if len(value) > 2 && strings.HasSuffix(value, "Z") { + if result, err := p.parseInZone(value, value[:len(value)-1], p.local, true, -1, 0, true); err == nil { + return result, nil + } + } + if len(value) > 7 && value[len(value)-7] != ' ' && (value[len(value)-6] == '+' || value[len(value)-6] == '-') { + if offset, ok := parseUTCOffset(value[len(value)-6:]); ok { + return p.parseInZone(value, value[:len(value)-6], p.local, true, -1, offset, true) + } + } + if n := strings.LastIndexByte(value, ' '); n >= 0 { + prefix, suffix := value[:n], value[n+1:] + if suffix == "UTC" { + return p.parseInZone(value, prefix, p.local, true, -1, 0, true) + } + if offset, ok := parseUTCOffset(suffix); ok { + return p.parseInZone(value, prefix, p.local, true, -1, offset, true) + } + if dst, ok := p.local.abbreviation(suffix); ok { + return p.parseInZone(value, prefix, p.local, false, dst, 0, true) + } + if zone, ok := namedTimestampZone(suffix); ok { + result, err := p.parseInZone(value, prefix, zone, false, -1, 0, true) + p.local.restoreAfterNamed(zone) + return result, err + } + } + return p.parseInZone(value, value, p.local, false, -1, 0, false) +} + +func (p *timestampParser) parseInZone(value, text string, zone *timestampZone, utc bool, isdst int, offset int64, withZone bool) (uint64, error) { + bad := func() (uint64, error) { return 0, fmt.Errorf("failed to parse timestamp: %s", value) } + now := time.Now() + if !withZone { + var delta uint64 + var err error + relative := false + negative := false + switch { + case value == "now": + return uint64(now.UnixMicro()), nil + case strings.HasPrefix(value, "+"): + delta, err = durationValue(value[1:]) + relative = true + case strings.HasPrefix(value, "-"): + delta, err = durationValue(value[1:]) + relative = true + negative = true + case strings.HasSuffix(value, " ago"): + delta, err = durationValue(value[:len(value)-4]) + relative = true + negative = true + case strings.HasSuffix(value, " left"): + delta, err = durationValue(value[:len(value)-5]) + relative = true + } + if relative { + if err != nil { + return bad() + } + base := uint64(now.UnixMicro()) + if negative { + if delta > base { + return bad() + } + base -= delta + } else { + if delta > timestampMax-base { + return bad() + } + base += delta + } + if base > timestampMax { + return bad() + } + return base, nil + } + } + seed, _, _ := zone.civil(now.Unix(), utc) + var civil time.Time + var second int + var fraction uint64 + var weekday = -1 + switch text { + case "today", "yesterday", "tomorrow": + d := seed.Day() + if text == "yesterday" { + d-- + } + if text == "tomorrow" { + d++ + } + civil = time.Date(seed.Year(), seed.Month(), d, 0, 0, 0, 0, time.UTC) + default: + for i, name := range []string{"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"} { + for _, prefix := range []string{name, name[:3]} { + if len(text) > len(prefix) && strings.EqualFold(text[:len(prefix)], prefix) && text[len(prefix)] == ' ' { + weekday = i + text = text[len(prefix)+1:] + break + } + } + if weekday >= 0 { + break + } + } + var ok bool + civil, second, fraction, ok = parseCalendar(text, seed) + if !ok { + return bad() + } + } + // strptime checks the civil date's weekday before mktime normalizes it. + if weekday >= 0 && int(civil.Weekday()) != weekday { + return bad() + } + guess := &p.localGuess + if utc { + guess = &p.utcGuess + } + seconds, err := inverseCivil(civil, zone, utc, isdst, guess) + if err != nil { + return bad() + } + seconds += int64(second) - int64(civil.Second()) + seconds -= offset + if seconds < 0 { + return bad() + } + usec := uint64(seconds)*1000000 + fraction + if usec > timestampMax { + return bad() + } + return usec, nil +} + +func parseCalendar(value string, seed time.Time) (time.Time, int, uint64, bool) { + formats := []string{"%y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%b %d %H:%M:%S", "%y-%m-%d %H:%M", "%Y-%m-%d %H:%M", "%Y-%m-%dT%H:%M", "%y-%m-%d", "%Y-%m-%d", "%H:%M:%S", "%H:%M"} + for _, format := range formats { + year, month, day := seed.Year(), int(seed.Month()), seed.Day() + hour, minute, second := seed.Hour(), seed.Minute(), seed.Second() + s := value + valid := true + hasTime := strings.Contains(format, "%H") + hasSeconds := strings.Contains(format, "%S") + if !hasTime { + hour, minute, second = 0, 0, 0 + } else if !hasSeconds { + second = 0 + } + for i := 0; i < len(format) && valid; i++ { + c := format[i] + if c == ' ' { + s = strings.TrimLeft(s, " \t\r\n\v\f") + continue + } + if c != '%' { + if len(s) == 0 || s[0] != c { + valid = false + } else { + s = s[1:] + } + continue + } + i++ + c = format[i] + if c == 'b' { + found := false + for m, name := range []string{"January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"} { + for _, prefix := range []string{name, name[:3]} { + if len(s) >= len(prefix) && strings.EqualFold(s[:len(prefix)], prefix) { + month = m + 1 + s = s[len(prefix):] + found = true + break + } + } + if found { + break + } + } + valid = found + continue + } + s = strings.TrimLeft(s, " \t\r\n\v\f") + width := 2 + if c == 'Y' { + width = 4 + } + n := 0 + for n < len(s) && n < width && s[n] >= '0' && s[n] <= '9' { + n++ + } + if n == 0 { + valid = false + break + } + number, _ := strconv.Atoi(s[:n]) + s = s[n:] + switch c { + case 'y': + if number >= 69 { + year = 1900 + number + } else { + year = 2000 + number + } + case 'Y': + year = number + case 'm': + month = number + valid = number >= 1 && number <= 12 + case 'd': + day = number + valid = number >= 1 && number <= 31 + case 'H': + hour = number + valid = number <= 23 + case 'M': + minute = number + valid = number <= 59 + case 'S': + second = number + valid = number <= 61 + } + } + var fractional uint64 + if valid && hasSeconds && strings.HasPrefix(s, ".") { + s = s[1:] + if s == "" { + valid = false + } + for i, c := range []byte(s) { + if c < '0' || c > '9' { + valid = false + break + } + if i < 6 { + fractional = fractional*10 + uint64(c-'0') + } else if i == 6 && c >= '5' { + fractional++ + } + } + if len(s) < 6 { + for i := len(s); i < 6; i++ { + fractional *= 10 + } + } + s = "" + } + if valid && s == "" { + clamped := second + if clamped > 59 { + clamped = 59 + } + return time.Date(year, time.Month(month), day, hour, minute, clamped, 0, time.UTC), second, fractional, true + } + } + return time.Time{}, 0, 0, false +} + +// Invert UTC-to-local probes with the cached-offset and oscillation policy used +// by glibc mktime. A fixed earlier/later DST preference cannot reproduce it. +func inverseCivil(civil time.Time, zone *timestampZone, utc bool, isdst int, guess *int64) (int64, error) { + target := civil.Unix() + t := target + *guess + t1, t2 := t, t + previousDST := -1 + matched, oscillated := false, false + for probe := 0; probe < 6; probe++ { + local, dst, _ := zone.civil(t, utc) + actual := local.Unix() + diff := target - actual + if diff == 0 { + matched = true + break + } + if t == t1 && t != t2 && (dst < 0 || isdst < 0 && previousDST <= dst || isdst >= 0 && dst != isdst) { + matched = true + oscillated = true + break + } + t1, t2 = t2, t + previousDST = dst + t += diff + } + if !matched { + return 0, errors.New("timestamp conversion did not converge") + } + _, dst, _ := zone.civil(t, utc) + if isdst >= 0 && dst != isdst && !oscillated { + found := false + for delta := int64(601200); delta < 457243209/2+601200 && !found; delta += 601200 { + for _, direction := range []int64{-1, 1} { + candidate := t + direction*delta + probe, otherDST, _ := zone.civil(candidate, utc) + if otherDST != isdst { + continue + } + t = candidate + target - probe.Unix() + found = true + break + } + } + if !found { + t += int64(dst-isdst) * 3600 + } + zone.civil(t, utc) + } + *guess = t - target + // An inserted second can falsely match the following minute's zero second. + if civil.Second() == 0 && zone.leapHit(t) { + actual, _, _ := zone.civil(t, utc) + if actual.Second() == 0 { + t++ + zone.civil(t, utc) + } + } + return t, nil +} diff --git a/go/cmd/journalctl/timestamp_zone.go b/go/cmd/journalctl/timestamp_zone.go new file mode 100644 index 00000000..d0e400fc --- /dev/null +++ b/go/cmd/journalctl/timestamp_zone.go @@ -0,0 +1,699 @@ +//go:build cgo + +package main + +import ( + "encoding/binary" + "io" + "math" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +type timestampZoneType struct { + offset int64 + dst int + name string + standard, universal bool +} +type timestampTransition struct { + at int64 + kind int +} +type timestampLeap struct{ at, correction int64 } +type timestampTZif struct { + types []timestampZoneType + transitions []timestampTransition + leaps []timestampLeap + firstName string + footer [2]string + footerRule string +} +type timestampZone struct { + location *time.Location + data *timestampTZif + names [2]string + initialNames [2]string + fileInfo os.FileInfo + renderLocations []*time.Location +} + +func timestampZoneDirs() []string { + if dir := os.Getenv("TZDIR"); dir != "" { + return []string{dir} + } + return []string{"/usr/share/zoneinfo", "/usr/share/lib/zoneinfo", "/usr/lib/locale/TZ"} +} + +func readTimestampZone(path string) (*timestampZone, bool) { + file, err := os.Open(path) + if err != nil { + return nil, false + } + defer file.Close() + info, err := file.Stat() + if err != nil || !info.Mode().IsRegular() { + return nil, false + } + data, err := io.ReadAll(io.LimitReader(file, (10<<20)+1)) + if err != nil || len(data) > 10<<20 { + return nil, false + } + location, err := time.LoadLocationFromTZData(path, data) + if err != nil { + return nil, false + } + meta := parseTimestampTZif(data) + if meta == nil { + return nil, false + } + z := ×tampZone{location: location, data: meta, fileInfo: info} + for i := len(meta.transitions) - 1; i >= 0; i-- { + kind := meta.types[meta.transitions[i].kind] + if z.names[kind.dst] == "" { + z.names[kind.dst] = kind.name + } + } + if z.names[0] == "" { + z.names[0] = meta.firstName + } + if z.names[1] == "" { + z.names[1] = z.names[0] + } + z.initialNames = z.names + return z, true +} + +func localTimestampZone() *timestampZone { + if value, set := os.LookupEnv("TZ"); set { + if value == "" { + return ×tampZone{location: time.FixedZone("Universal", 0)} + } + value = strings.TrimPrefix(value, ":") + if value == "" { + return ×tampZone{location: time.UTC, names: [2]string{"UTC", "UTC"}} + } + if filepath.IsAbs(value) { + if zone, ok := readTimestampZone(value); ok { + return zone + } + } + if value != "" { + for _, dir := range timestampZoneDirs() { + if zone, ok := readTimestampZone(filepath.Join(dir, value)); ok { + return zone + } + } + } + if zone, ok := namedTimestampZone(value); ok { + return zone + } + if zone := defaultTimestampZone(value); zone != nil { + return zone + } + if zone := posixTimestampZone(value); zone != nil { + return zone + } + return ×tampZone{location: time.FixedZone(fallbackTimestampName(value), 0)} + } else if zone, ok := readTimestampZone("/etc/localtime"); ok { + return zone + } + return ×tampZone{location: time.Local} +} + +func posixTimestampZone(rule string) *timestampZone { + rule = normalizeTimestampRule(rule) + // A TZif footer lets the standard library parse POSIX rules without private APIs. + names := timestampFooterNames(rule) + if len(names[0]) < 3 { + return nil + } + header := make([]byte, 44) + copy(header, "TZif2") + binary.BigEndian.PutUint32(header[32:], 1) + binary.BigEndian.PutUint32(header[36:], 1) + binary.BigEndian.PutUint32(header[40:], 4) + block := func(width int) []byte { + var b []byte + if width == 4 { + b = binary.BigEndian.AppendUint32(b, 1<<31) + } else { + b = binary.BigEndian.AppendUint64(b, 1<<63) + } + return append(b, 0, 0, 0, 0, 0, 0, 0, 'U', 'T', 'C', 0) + } + data := append(append(append(append([]byte{}, header...), block(4)...), header...), block(8)...) + data = append(data, '\n') + data = append(data, rule...) + data = append(data, '\n') + location, err := time.LoadLocationFromTZData("local", data) + if err != nil { + return nil + } + name, _ := time.Now().In(location).Zone() + if name != names[0] && name != names[1] { + return nil + } + return ×tampZone{location: location, names: names, initialNames: names} +} + +func namedTimestampZone(name string) (*timestampZone, bool) { + if name == "" || len(name) >= 4096 || strings.HasPrefix(name, "/") || strings.HasSuffix(name, "/") || strings.Contains(name, "//") { + return nil, false + } + for _, c := range []byte(name) { + if !(c >= 'A' && c <= 'Z' || c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || strings.ContainsRune("-_+/", rune(c))) { + return nil, false + } + } + installed := false + for _, dir := range timestampZoneDirs() { + if info, err := os.Stat(dir); err == nil && info.IsDir() { + installed = true + } + if zone, ok := readTimestampZone(filepath.Join(dir, name)); ok { + return zone, true + } + } + if installed { + return nil, false + } + location, err := time.LoadLocation(name) + if err != nil { + return nil, false + } + return ×tampZone{location: location}, true +} + +func (z *timestampZone) restoreAfterNamed(named *timestampZone) { + if z.data == nil { + return + } + if z.fileInfo != nil && named.fileInfo != nil && os.SameFile(z.fileInfo, named.fileInfo) && z.fileInfo.ModTime().Unix() == named.fileInfo.ModTime().Unix() { + z.names = named.names + } else { + z.names = z.initialNames + } +} + +func (z *timestampZone) abbreviation(name string) (int, bool) { + if z.names[0] != "" { + for dst, n := range z.names { + if n == name { + return dst, true + } + } + return 0, false + } + now := time.Now() + for _, month := range []time.Month{time.January, time.July} { + t := time.Date(now.Year(), month, 1, 12, 0, 0, 0, z.location) + abbr, _ := t.Zone() + if abbr == name { + if t.IsDST() { + return 1, true + } + return 0, true + } + } + return 0, false +} + +// Providers discard leap records and may use a footer before any transition. +// Keep those native calendar semantics in the CLI's cold timestamp path. +func (z *timestampZone) civil(seconds int64, utc bool) (time.Time, int, int64) { + t := time.Unix(seconds, 0).In(z.location) + _, offsetInt := t.Zone() + offset := int64(offsetInt) + dst := 0 + if t.IsDST() { + dst = 1 + } + if utc { + offset = 0 + dst = 0 + } + if z.data != nil { + meta := z.data + if !utc { + i := sort.Search(len(meta.transitions), func(i int) bool { return meta.transitions[i].at > seconds }) + if i > 0 && i == len(meta.transitions) && meta.footer[0] != "" { + z.names = meta.footer + } else { + z.names = [2]string{} + kind := 0 + if i == 0 { + for j, k := range meta.types { + if k.dst == 0 { + kind = j + break + } + } + z.names[0] = meta.types[kind].name + for _, k := range meta.types { + if k.dst == 1 { + z.names[1] = k.name + break + } + } + } else { + kind = meta.transitions[i-1].kind + k := meta.types[kind] + z.names[k.dst] = k.name + for _, transition := range meta.transitions[i:] { + k := meta.types[transition.kind] + if z.names[k.dst] == "" { + z.names[k.dst] = k.name + } + } + } + if z.names[0] == "" { + z.names[0] = z.names[1] + } + if z.names[1] == "" { + z.names[1] = z.names[0] + } + offset = meta.types[kind].offset + dst = meta.types[kind].dst + } + } + i := sort.Search(len(meta.leaps), func(i int) bool { return meta.leaps[i].at > seconds }) + if i > 0 { + leap := meta.leaps[i-1] + offset -= leap.correction + if leap.at == seconds && (i == 1 && leap.correction > 0 || i > 1 && leap.correction > meta.leaps[i-2].correction) { + offset++ + for j := i - 1; j > 0 && meta.leaps[j].at == meta.leaps[j-1].at+1 && meta.leaps[j].correction == meta.leaps[j-1].correction+1; j-- { + offset++ + } + } + } + } + return time.Unix(seconds+offset, 0).UTC(), dst, offset +} + +func (z *timestampZone) leapHit(seconds int64) bool { + if z.data == nil { + return false + } + leaps := z.data.leaps + i := sort.Search(len(leaps), func(i int) bool { return leaps[i].at >= seconds }) + return i < len(leaps) && leaps[i].at == seconds && (i == 0 && leaps[i].correction > 0 || i > 0 && leaps[i].correction > leaps[i-1].correction) +} + +func parseTimestampTZif(data []byte) *timestampTZif { + width := 4 + header := func(b []byte) ([6]uint64, bool) { + var counts [6]uint64 + if len(b) < 44 || string(b[:4]) != "TZif" { + return counts, false + } + for i := range counts { + counts[i] = uint64(binary.BigEndian.Uint32(b[20+i*4:])) + } + return counts, true + } + size := func(n [6]uint64, w int) uint64 { + return n[3]*uint64(w+1) + n[4]*6 + n[5] + n[2]*uint64(w+4) + n[0] + n[1] + } + n, ok := header(data) + if !ok { + return nil + } + if data[4] != 0 { + skip := 44 + size(n, 4) + if skip > uint64(len(data)) { + return nil + } + data = data[skip:] + n, ok = header(data) + if !ok { + return nil + } + width = 8 + } + blockSize := size(n, width) + if blockSize > uint64(len(data)-44) || n[4] == 0 || n[4] > 256 || n[0] > n[4] || n[1] > n[4] { + return nil + } + block := data[44 : 44+blockSize] + at := 0 + signed := func(b []byte) int64 { + if width == 4 { + return int64(int32(binary.BigEndian.Uint32(b))) + } + return int64(binary.BigEndian.Uint64(b)) + } + meta := ×tampTZif{} + for i := uint64(0); i < n[3]; i++ { + epoch := signed(block[at:]) + at += width + if i > 0 && epoch <= meta.transitions[i-1].at { + return nil + } + meta.transitions = append(meta.transitions, timestampTransition{at: epoch}) + } + for i := range meta.transitions { + kind := int(block[at]) + at++ + if uint64(kind) >= n[4] { + return nil + } + meta.transitions[i].kind = kind + } + types := block[at : at+int(n[4])*6] + at += len(types) + names := block[at : at+int(n[5])] + at += len(names) + nameAt := func(i int) (string, bool) { + if i >= len(names) { + return "", false + } + s := string(names[i:]) + end := strings.IndexByte(s, 0) + if end < 0 { + return "", false + } + return s[:end], true + } + meta.firstName, ok = nameAt(0) + if !ok { + return nil + } + for i := 0; i < len(types); i += 6 { + name, valid := nameAt(int(types[i+5])) + if !valid || types[i+4] > 1 { + return nil + } + meta.types = append(meta.types, timestampZoneType{offset: int64(int32(binary.BigEndian.Uint32(types[i:]))), dst: int(types[i+4]), name: name}) + } + for i := uint64(0); i < n[2]; i++ { + epoch := signed(block[at:]) + at += width + correction := int64(int32(binary.BigEndian.Uint32(block[at:]))) + at += 4 + if i > 0 && epoch <= meta.leaps[i-1].at { + return nil + } + meta.leaps = append(meta.leaps, timestampLeap{epoch, correction}) + } + for i := 0; i < int(n[1]); i++ { + meta.types[i].standard = block[at+i] != 0 + } + at += int(n[1]) + for i := 0; i < int(n[0]); i++ { + meta.types[i].universal = block[at+i] != 0 + } + if width == 8 { + footer := strings.Trim(string(data[44+blockSize:]), "\n") + meta.footerRule = footer + meta.footer = timestampFooterNames(footer) + } + return meta +} + +func timestampFooterNames(text string) [2]string { + name := func() string { + if strings.HasPrefix(text, "<") { + if i := strings.IndexByte(text, '>'); i > 1 { + n := text[1:i] + text = text[i+1:] + return n + } + return "" + } + i := 0 + for i < len(text) && ((text[i] >= 'a' && text[i] <= 'z') || (text[i] >= 'A' && text[i] <= 'Z')) { + i++ + } + n := text[:i] + text = text[i:] + return n + } + standard := name() + i := 0 + for i < len(text) && strings.ContainsRune("+-0123456789:", rune(text[i])) { + i++ + } + text = text[i:] + daylight := name() + if daylight == "" { + daylight = standard + } + return [2]string{standard, daylight} +} + +// Rendering has no parser abbreviation state and reads zone data only once. +var renderingTimestampZone = sync.OnceValue(func() *timestampZone { + z := localTimestampZone() + if z.data != nil { + for _, kind := range z.data.types { + z.renderLocations = append(z.renderLocations, time.FixedZone(kind.name, int(kind.offset))) + } + } + return z +}) + +func (z *timestampZone) displayLocation(seconds int64, utc bool) *time.Location { + if utc { + return time.UTC + } + if meta := z.data; meta != nil { + i := sort.Search(len(meta.transitions), func(i int) bool { return meta.transitions[i].at > seconds }) + if i == 0 { + for kind, value := range meta.types { + if value.dst == 0 { + return z.renderLocations[kind] + } + } + return z.renderLocations[0] + } + if i < len(meta.transitions) || meta.footer[0] == "" { + return z.renderLocations[meta.transitions[i-1].kind] + } + } + return z.location +} + +func (z *timestampZone) display(seconds int64, utc bool) (time.Time, string, int64, bool) { + location := z.displayLocation(seconds, utc) + t := time.Unix(seconds, 0).In(location) + name, nativeOffset := t.Zone() + + offset := int64(nativeOffset) + correction, leap := int64(0), false + if z.data != nil { + leaps := z.data.leaps + i := sort.Search(len(leaps), func(i int) bool { return leaps[i].at > seconds }) + if i > 0 { + correction = leaps[i-1].correction + leap = leaps[i-1].at == seconds && (i == 1 && correction > 0 || i > 1 && correction > leaps[i-2].correction) + } + } + return time.Unix(seconds+offset-correction, 0).UTC(), name, offset, leap +} + +func formatCalendarTimestamp(usec uint64, utc bool, layout string, named, numeric bool) string { + zone := renderingTimestampZone() + if zone.data == nil || len(zone.data.leaps) == 0 { + seconds := int64(usec / 1_000_000) + location := zone.displayLocation(seconds, utc) + t := time.Unix(seconds, int64(usec%1_000_000)*1000).In(location) + if named && location.String() == "" { + layout = strings.TrimSuffix(layout, " MST") + } + return t.Format(layout) + } + if named { + layout = strings.TrimSuffix(layout, " MST") + } + if numeric { + layout = strings.TrimSuffix(layout, "-07:00") + } + t, name, offset, leap := zone.display(int64(usec/1_000_000), utc) + t = t.Add(time.Duration(usec%1_000_000) * time.Microsecond) + var buffer [128]byte + text := buffer[:0] + if leap { + // Go time.Time cannot represent second 60; retain the TZif leap label. + before, after, _ := strings.Cut(layout, "05") + text = t.AppendFormat(text, before) + text = strconv.AppendInt(text, int64(t.Second()+1), 10) + text = t.AppendFormat(text, after) + } else { + text = t.AppendFormat(text, layout) + } + if named && name != "" { + text = append(text, ' ') + text = append(text, name...) + } + if numeric { + sign := byte('+') + if offset < 0 { + sign, offset = '-', -offset + } + text = append(text, sign) + if offset/3600 < 10 { + text = append(text, '0') + } + text = strconv.AppendInt(text, offset/3600, 10) + text = append(text, ':', byte('0'+offset%3600/600), byte('0'+offset%600/60)) + } + return string(text) +} + +// libc retains a valid standard name even when the remaining TZ rule is invalid. +func fallbackTimestampName(value string) string { + valid := func(b byte) bool { return b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' } + if strings.HasPrefix(value, "<") { + if end := strings.IndexByte(value, '>'); end >= 4 { + for _, b := range []byte(value[1:end]) { + if !valid(b) && !(b >= '0' && b <= '9') && b != '+' && b != '-' { + return "" + } + } + return value[1:end] + } + return "" + } + end := 0 + for end < len(value) && valid(value[end]) { + end++ + } + if end < 3 { + return "" + } + return value[:end] +} + +// Native TZ rules clamp offset components; providers otherwise accept wider hours. +func normalizeTimestampRule(rule string) string { + var result strings.Builder + for count := 0; count < 2; count++ { + name := fallbackTimestampName(rule) + if name == "" { + break + } + end := len(name) + if strings.HasPrefix(rule, "<") { + end += 2 + } + result.WriteString(rule[:end]) + rule = rule[end:] + if len(rule) > 0 && (rule[0] == '+' || rule[0] == '-') { + result.WriteByte(rule[0]) + rule = rule[1:] + } + for part := 0; part < 3; part++ { + end = 0 + for end < len(rule) && rule[end] >= '0' && rule[end] <= '9' { + end++ + } + if end == 0 { + break + } + value, err := strconv.ParseUint(rule[:end], 10, 64) + if err != nil { + value = ^uint64(0) + } + // Native parsing wraps through unsigned short before clamping. + value &= 0xffff + limit := uint64(59) + if part == 0 { + limit = 24 + } + if value > limit { + value = limit + } + result.WriteString(strconv.FormatUint(value, 10)) + rule = rule[end:] + if part == 2 || !strings.HasPrefix(rule, ":") { + break + } + result.WriteByte(':') + rule = rule[1:] + } + } + result.WriteString(rule) + return result.String() +} + +// Omitted DST dates use the native default rule file before built-in US dates. +func defaultTimestampZone(rule string) *timestampZone { + rule = strings.TrimSuffix(normalizeTimestampRule(rule), ",") + if strings.Contains(rule, ",") { + return nil + } + names := timestampFooterNames(rule) + standard := fallbackTimestampName(rule) + if standard == "" { + return nil + } + end := len(standard) + if strings.HasPrefix(rule, "<") { + end += 2 + } + rest := strings.TrimLeft(rule[end:], "+-0123456789:") + if fallbackTimestampName(rest) == "" { + return nil + } + base := posixTimestampZone(rule + ",M3.2.0,M11.1.0") + if base == nil { + return nil + } + var offsets [2]int64 + for i, seconds := range []int64{1705276800, 1721001600} { + _, offset := time.Unix(seconds, 0).In(base.location).Zone() + offsets[i] = int64(offset) + } + for _, dir := range timestampZoneDirs() { + z, ok := readTimestampZone(filepath.Join(dir, "posixrules")) + if !ok || len(z.data.types) < 2 { + continue + } + meta := z.data + // Native default-file conversion starts with zero remembered DST offset. + oldOffsets := [2]int64{meta.types[0].offset, 0} + for i := len(meta.transitions) - 1; i >= 0; i-- { + kind := meta.types[meta.transitions[i].kind] + if kind.dst == 0 { + oldOffsets[0] = kind.offset + break + } + } + previousDST := 0 + for i := range meta.transitions { + transition := &meta.transitions[i] + kind := meta.types[transition.kind] + shiftKind := 0 + if previousDST != 0 && !kind.standard { + shiftKind = 1 + } + if !kind.universal { + shift := offsets[shiftKind] - oldOffsets[shiftKind] + if shift > 0 && transition.at > math.MaxInt64-shift || shift < 0 && transition.at < math.MinInt64-shift { + return base + } + transition.at += shift + } + transition.kind, previousDST = kind.dst, kind.dst + } + meta.types = []timestampZoneType{{offset: offsets[0], name: names[0]}, {offset: offsets[1], dst: 1, name: names[1]}} + meta.firstName = names[0] + if meta.footer[0] != "" { + meta.footer = names + if footer := posixTimestampZone(meta.footerRule); footer != nil { + z.location = footer.location + } + } + z.names, z.initialNames = names, names + z.fileInfo = nil + return z + } + return base +} diff --git a/go/cmd/journalctl/uid_other.go b/go/cmd/journalctl/uid_other.go index 403411f4..d8b15682 100644 --- a/go/cmd/journalctl/uid_other.go +++ b/go/cmd/journalctl/uid_other.go @@ -1,4 +1,4 @@ -//go:build !unix +//go:build cgo && !unix package main diff --git a/go/cmd/journalctl/uid_unix.go b/go/cmd/journalctl/uid_unix.go index fc2a235d..07973858 100644 --- a/go/cmd/journalctl/uid_unix.go +++ b/go/cmd/journalctl/uid_unix.go @@ -1,4 +1,4 @@ -//go:build unix +//go:build cgo && unix package main diff --git a/go/cmd/journalctl/unit_name.go b/go/cmd/journalctl/unit_name.go new file mode 100644 index 00000000..9c9d99c7 --- /dev/null +++ b/go/cmd/journalctl/unit_name.go @@ -0,0 +1,213 @@ +//go:build cgo + +package main + +import ( + "encoding/binary" + "encoding/hex" + "fmt" + "math/bits" + "strings" +) + +const unitNameChars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789:-_.\\@" + +type unitSelection struct{ system, user []string } + +func rawUnitSpecs(flags *cliFlags) unitSelection { + s := unitSelection{flags.unitFlag.Values(), flags.userUnitFlag.Values()} + if *flags.userFlag && !*flags.systemFlag { + for _, unit := range s.system { + if !containsString(s.user, unit) { + s.user = append(s.user, unit) + } + } + s.system = nil + } + return s +} + +func effectiveUnitSpecs(flags *cliFlags) ([]string, []string, error) { + if !flags.unitsReady { + flags.unitsReady = true + raw := rawUnitSpecs(flags) + for i, values := range [][]string{raw.system, raw.user} { + var normalized []string + for _, value := range values { + name, escaped, err := normalizeUnitName(value) + if err != nil { + flags.unitsError = err + break + } + if escaped != "" && !flags.effectiveQuiet() && flags.diagnostics != nil { + fmt.Fprintf(flags.diagnostics, "Invalid unit name \"%s\" escaped as \"%s\" (maybe you should use systemd-escape?).\n", value, escaped) + } + normalized = append(normalized, name) + } + if i == 0 { + flags.units.system = normalized + } else { + flags.units.user = normalized + } + if flags.unitsError != nil { + break + } + } + } + return flags.units.system, flags.units.user, flags.unitsError +} + +func unitHasSuffix(name string) bool { + for _, suffix := range unitSuffixes { + if strings.HasSuffix(name, suffix) { + return true + } + } + return false +} + +func validUnitName(name string) bool { + if len(name) == 0 || len(name) >= 256 || strings.LastIndexByte(name, '.') <= 0 || name[0] == '@' || !unitHasSuffix(name) { + return false + } + for i := 0; i < len(name); i++ { + if strings.IndexByte(unitNameChars, name[i]) < 0 { + return false + } + } + return true +} + +func normalizeUnitName(value string) (name, escaped string, err error) { + if value == "" || strings.IndexByte(value, 0) >= 0 { + return "", "", fmt.Errorf("Failed to mangle unit name '%s': Invalid argument", value) + } + valid := func(chars string) bool { + for i := 0; i < len(value); i++ { + if !strings.ContainsRune(chars, rune(value[i])) { + return false + } + } + return true + } + if validUnitName(value) || isGlobPattern(value) && valid(unitNameChars+"[]!*?") { + return value, "", nil + } + if strings.HasPrefix(value, "/") { + var components []string + for _, part := range strings.Split(value, "/") { + if part == "" || part == "." || part == ".." && len(components) == 0 { + continue + } + components = append(components, part) + } + path := strings.Join(components, "/") + if !containsString(components, "..") && validUnitPathComponents(components) && len(path) < 4095 { + suffix := ".mount" + if strings.HasPrefix(path, "dev/") || strings.HasPrefix(path, "sys/") { + suffix = ".device" + } + var b strings.Builder + if path == "" { + b.WriteByte('-') + } + for i := 0; i < len(path); i++ { + c := path[i] + switch { + case c == '/': + b.WriteByte('-') + case c == '-' || c == '\\' || c == '.' && i == 0 || !strings.ContainsRune(strings.TrimSuffix(unitNameChars, "@"), rune(c)): + fmt.Fprintf(&b, `\x%02x`, c) + default: + b.WriteByte(c) + } + } + name = b.String() + suffix + if len(name) >= 256 { + var hash [8]byte + binary.LittleEndian.PutUint64(hash[:], unitNameHash(append([]byte(name), 0))) + name = name[:256-1-len(suffix)-16-1] + "_" + hex.EncodeToString(hash[:]) + suffix + } + return name, "", nil + } + } + var b strings.Builder + for i := 0; i < len(value); i++ { + c := value[i] + switch { + case c == '/': + b.WriteByte('-') + case !strings.ContainsRune(unitNameChars+"[]!*?", rune(c)): + fmt.Fprintf(&b, `\x%02x`, c) + default: + b.WriteByte(c) + } + } + name = b.String() + if name != value { + escaped = name + } + if !isGlobPattern(name) && !validUnitName(name) { + name += ".service" + } + return name, escaped, nil +} + +func containsString(values []string, value string) bool { + for _, candidate := range values { + if candidate == value { + return true + } + } + return false +} + +// The journalctl path-name hash is independent from a journal file's DATA hash. +func unitNameHash(msg []byte) uint64 { + key := [16]byte{0xec, 0xf2, 0x37, 0xfb, 0x58, 0x32, 0x4a, 0x32, 0x84, 0x9f, 0x06, 0x9b, 0x0d, 0x21, 0xeb, 0x9a} + k0, k1 := binary.LittleEndian.Uint64(key[:8]), binary.LittleEndian.Uint64(key[8:]) + v0, v1, v2, v3 := uint64(0x736f6d6570736575)^k0, uint64(0x646f72616e646f6d)^k1, uint64(0x6c7967656e657261)^k0, uint64(0x7465646279746573)^k1 + round := func() { + v0 += v1 + v1 = bits.RotateLeft64(v1, 13) ^ v0 + v0 = bits.RotateLeft64(v0, 32) + v2 += v3 + v3 = bits.RotateLeft64(v3, 16) ^ v2 + v0 += v3 + v3 = bits.RotateLeft64(v3, 21) ^ v0 + v2 += v1 + v1 = bits.RotateLeft64(v1, 17) ^ v2 + v2 = bits.RotateLeft64(v2, 32) + } + rest := msg + for len(rest) >= 8 { + m := binary.LittleEndian.Uint64(rest[:8]) + v3 ^= m + round() + round() + v0 ^= m + rest = rest[8:] + } + b := uint64(len(msg)) << 56 + for i, c := range rest { + b |= uint64(c) << (8 * uint(i)) + } + v3 ^= b + round() + round() + v0 ^= b + v2 ^= 0xff + for range 4 { + round() + } + return v0 ^ v1 ^ v2 ^ v3 +} + +func validUnitPathComponents(parts []string) bool { + for _, part := range parts { + if len(part) > 255 { + return false + } + } + return true +} diff --git a/go/cmd/journalctl/unit_pattern_compat.h b/go/cmd/journalctl/unit_pattern_compat.h new file mode 100644 index 00000000..18679b38 --- /dev/null +++ b/go/cmd/journalctl/unit_pattern_compat.h @@ -0,0 +1,235 @@ +#ifndef JOURNALCTL_UNIT_PATTERN_COMPAT_H +#define JOURNALCTL_UNIT_PATTERN_COMPAT_H + +#include +#include +#include +#include +#ifdef __APPLE__ +/* Darwin needs xlocale for the thread-local MB_CUR_MAX definition. */ +#include +#endif +#include +#include +#include +#include + +/* CLI-only FreeBSD fallback for brackets missing from its native fnmatch. */ +struct journal_unit_pattern { + wchar_t *bytes, *wide, *value; + size_t capacity; + int posix; +}; + +static int journal_unit_pattern_needed(const char *p) { + return strstr(p, "[:") || strstr(p, "[.") || strstr(p, "[="); +} + +/* A successful member still needs a closing bracket, but not valid later members. */ +static const wchar_t *journal_unit_bracket_end(const wchar_t *p, int *invalid) { + while (*p && *p != ']') { + if (p[0] == '[' && (p[1] == ':' || p[1] == '.' || p[1] == '=')) { + wchar_t marker = p[1]; + const wchar_t *end = p + 2; + while (*end && !(end[0] == marker && end[1] == ']')) end++; + if (!*end && marker != ':') { *invalid = 1; return NULL; } + if (*end) { + if (marker == '=' && end != p + 3) { *invalid = 1; return NULL; } + if (marker == ':') { + const wchar_t *name = p + 2; + while (name < end && *name >= 'a' && *name < 'z') name++; + if (name != end) { p++; continue; } + if (end - p >= 257) { *invalid = 1; return NULL; } + } + p = end + 2; continue; + } + } + p++; + } + return *p ? p + 1 : NULL; +} + +/* FreeBSD and glibc wide collation keys separate weight levels with 1. */ +static int journal_unit_equivalent(wchar_t left, wchar_t right, int bytes) { + if (left == right) return 1; + if (bytes) { + wint_t a = btowc((unsigned char)left), b = btowc((unsigned char)right); + if (a == WEOF || b == WEOF) return 0; + left = (wchar_t)a; right = (wchar_t)b; + } + wchar_t a[] = {left, 0}, b[] = {right, 0}; + wchar_t local_a[32], local_b[32], *key_a = local_a, *key_b = local_b; + size_t na = wcsxfrm(local_a, a, 32), nb = wcsxfrm(local_b, b, 32); + if (na == SIZE_MAX || nb == SIZE_MAX || na >= SIZE_MAX / sizeof(wchar_t) || nb >= SIZE_MAX / sizeof(wchar_t)) return 0; + if (na >= 32) { + key_a = malloc((na + 1) * sizeof(wchar_t)); + if (!key_a) return 0; + if (wcsxfrm(key_a, a, na + 1) > na) { free(key_a); return 0; } + } + if (nb >= 32) { + key_b = malloc((nb + 1) * sizeof(wchar_t)); + if (!key_b) { if (key_a != local_a) free(key_a); return 0; } + if (wcsxfrm(key_b, b, nb + 1) > nb) { + if (key_a != local_a) free(key_a); + free(key_b); return 0; + } + } + size_t i = 0; + while (key_a[i] > 1 && key_b[i] > 1 && key_a[i] == key_b[i]) i++; + int equal = key_a[i] <= 1 && key_b[i] <= 1; + if (key_a != local_a) free(key_a); + if (key_b != local_b) free(key_b); + return equal; +} + +/* Keep range ordering consistent with the platform's ordinary matcher. */ +static int journal_unit_range(wchar_t low, wchar_t high, wchar_t value, int bytes) { + wchar_t pattern[] = {'[', '\\', low, '-', '\\', high, ']', 0}; + wchar_t candidate[] = {value, 0}; + char encoded[7 * MB_LEN_MAX + 1], text[MB_LEN_MAX + 1]; + if (bytes) { + for (size_t i = 0; i < sizeof(pattern) / sizeof(*pattern); i++) encoded[i] = (char)pattern[i]; + text[0] = (char)value; text[1] = 0; + } else { + mbstate_t state = {0}; const wchar_t *source = pattern; + if (wcsrtombs(encoded, &source, sizeof(encoded), &state) == (size_t)-1 || source) return 0; + memset(&state, 0, sizeof(state)); source = candidate; + if (wcsrtombs(text, &source, sizeof(text), &state) == (size_t)-1 || source) return 0; + } + return fnmatch(encoded, text, 0) == 0; +} + +static int journal_unit_bracket(const wchar_t **cursor, const wchar_t *candidate, int bytes, int posix) { + wchar_t value = *candidate; + const wchar_t *start = *cursor, *p = start + 1; + int negate = *p == '!' || (!posix && *p == '^'); + if (negate) p++; + int first = 1; + while (*p && (first || *p != ']')) { + first = 0; + wchar_t low = *p++; + int member = 0, character = 1; + if (low == '[' && *p == ':') { + const wchar_t *end = p + 1; + char name[256]; size_t n = 0; + while (*end >= 'a' && *end < 'z' && n < sizeof(name) - 1) name[n++] = (char)*end++; + if (n == sizeof(name) - 1) return 0; + if (*end == ':' && end[1] == ']') { + name[n] = 0; + wctype_t type = wctype(name); + if (!type) return 0; + member = iswctype(bytes ? btowc((unsigned char)value) : (wint_t)value, type) != 0; + p = end + 2; + character = 0; + } + } else if (low == '[' && (*p == '.' || *p == '=')) { + wchar_t marker = *p; + const wchar_t *end = p + 1; + while (*end && !(end[0] == marker && end[1] == ']')) end++; + if (*end && end == p + 2) { + low = p[1]; p = end + 2; + if (marker == '=') { + if (bytes && MB_CUR_MAX > 1) { + char raw[MB_LEN_MAX]; size_t n = 0; + while (n < sizeof(raw) && candidate[n]) { raw[n] = (char)candidate[n]; n++; } + mbstate_t state = {0}; wchar_t decoded; + size_t used = mbrtowc(&decoded, raw, n, &state); + wint_t first = btowc((unsigned char)low); + member = first != WEOF && used != (size_t)-1 && used != (size_t)-2 && + journal_unit_equivalent((wchar_t)first, decoded, 0); + } else member = journal_unit_equivalent(low, value, bytes); + character = 0; + } + } else if (marker == '.') { + return 0; + } + } + if (character) { + if (*p == '-' && !p[1] && low != value) return 0; + if (*p == '-' && p[1] && p[1] != ']') { + p++; + wchar_t high = *p++; + if (high == '[' && *p == '.') { + if (p[1] && p[2] == '.' && p[3] == ']') { high = p[1]; p += 4; } + else return 0; + } + member = journal_unit_range(low, high, value, bytes); + } else member = low == value; + } + if (member) { + int invalid = 0; + const wchar_t *end = journal_unit_bracket_end(p, &invalid); + if (invalid) return 0; + if (!end) { *cursor = start + 1; return value == '['; } + *cursor = end; + return !negate; + } + } + if (!*p) { *cursor = start + 1; return value == '['; } + *cursor = p + 1; + return negate; +} + +static int journal_unit_wildcard(const wchar_t *pattern, const wchar_t *value, int bytes, int posix) { + const wchar_t *p = pattern, *v = value, *star = NULL, *retry = NULL; + for (;;) { + if (*p == '*') { + while (*p == '*') p++; + if (!*p) return 1; + star = p; retry = v; + continue; + } + if (!*p && !*v) return 1; + const wchar_t *next = p + (*p != 0); + int matches = *v && (*p == '?' || *p == *v); + if (*p == '[' && *v) { + next = p; + matches = journal_unit_bracket(&next, v, bytes, posix); + } + if (matches) { p = next; v++; continue; } + if (!star || !*retry) return 0; + v = ++retry; p = star; + } +} + +static void journal_unit_pattern_free(struct journal_unit_pattern *p) { + if (!p) return; + free(p->bytes); free(p->wide); free(p->value); free(p); +} + +static struct journal_unit_pattern *journal_unit_pattern_new(const char *pattern) { + size_t n = strlen(pattern) + 1; /* Flawfinder: ignore -- Both CLI callers use terminated CString storage. */ + if (n > SIZE_MAX / sizeof(wchar_t)) return NULL; + struct journal_unit_pattern *p = calloc(1, sizeof(*p)); + if (!p) return NULL; + p->posix = getenv("POSIXLY_CORRECT") != NULL; + p->bytes = malloc(n * sizeof(wchar_t)); + if (!p->bytes) { journal_unit_pattern_free(p); return NULL; } + for (size_t i = 0; i < n; i++) p->bytes[i] = (unsigned char)pattern[i]; + if (MB_CUR_MAX > 1) { + p->wide = malloc(n * sizeof(wchar_t)); + if (!p->wide) { journal_unit_pattern_free(p); return NULL; } + mbstate_t state = {0}; const char *source = pattern; + if (mbsrtowcs(p->wide, &source, n, &state) == (size_t)-1) { free(p->wide); p->wide = NULL; } + } + return p; +} + +static int journal_unit_pattern_match(struct journal_unit_pattern *p, const char *value) { + size_t n = strlen(value) + 1; /* Flawfinder: ignore -- Both CLI callers append a NUL to the reused buffer. */ + if (n > p->capacity) { + if (n > SIZE_MAX / sizeof(wchar_t)) return 0; + wchar_t *buffer = realloc(p->value, n * sizeof(wchar_t)); + if (!buffer) return 0; + p->value = buffer; p->capacity = n; + } + if (p->wide) { + mbstate_t state = {0}; const char *source = value; + if (mbsrtowcs(p->value, &source, n, &state) != (size_t)-1 && + journal_unit_wildcard(p->wide, p->value, 0, p->posix)) return 1; + } + /* glibc retries byte matching after an unsuccessful multibyte match. */ + for (size_t i = 0; i < n; i++) p->value[i] = (unsigned char)value[i]; + return journal_unit_wildcard(p->bytes, p->value, 1, p->posix); +} +#endif diff --git a/go/cmd/journalctl/unit_pattern_compat_test.go b/go/cmd/journalctl/unit_pattern_compat_test.go new file mode 100644 index 00000000..33163f41 --- /dev/null +++ b/go/cmd/journalctl/unit_pattern_compat_test.go @@ -0,0 +1,76 @@ +//go:build cgo && (linux || darwin || freebsd) + +package main + +import "testing" + +func TestUnitPatternCompatibility(t *testing.T) { + for _, locale := range []string{"C", "en_US.UTF-8"} { + t.Run(locale, func(t *testing.T) { + t.Setenv("LC_ALL", locale) + for _, tc := range []struct { + pattern, value string + want bool + }{ + {"[[:alpha:]]", "é", locale != "C"}, + {"[[=a=]]", "à", locale != "C"}, + {"[[=a=]]", "A", locale != "C"}, + {"[[=a=]]?", "à", locale != "C"}, + {"[[=a=]-z]", "-", true}, + {"[[:alpha:]]lpha.*", "alpha.service", true}, + {"[[:alpha:]]lpha.*", "1lpha.service", false}, + {"[![:alpha:]]*", "1.service", true}, + {"[![:alpha:]]*", "alpha.service", false}, + {"[[.a.]]lpha.*", "alpha.service", true}, + {"[[=a=]]lpha.*", "alpha.service", true}, + {"[[:alpha:]]*", "a\xff.service", true}, + {"[![:alpha:]]?", "é", true}, + {"*[[:alpha:]]*", "\xffa\xff", true}, + {"[[:alpha:]\\]", "\\", true}, + {"[[:alpha:]]alpha[", "aalpha[", true}, + {"[[:alpha:]]$", "a$", true}, + {"[[:alpha:]]?", "a\n", true}, + {"[[:alpha:]]", "a\n", false}, + {"[[:unknown:]]*", "a", false}, + {"[[:<:]]*", "a", false}, + {"[[.period.]]*", ".", false}, + {"[[:alpha:]", "a", false}, + {"alpha.*", "alpha.service", true}, + {"[[[:bogus:]]", "[", true}, + {"[a-[:alpha:]]", "a]", true}, + {"[[:alpha:]z-a]", "a", true}, + {"[[[:<:]]", "[", false}, + {"[[:-", "[[:-", false}, + {"[[[.a]", "[", false}, + {"[[[=a", "[[[=a", false}, + } { + m, err := newUnitPatternMatcherWithCompatibility([]string{tc.pattern}, true) + if err != nil { + t.Fatal(err) + } + got := m.Matches([]byte(tc.value)) + m.Close() + if got != tc.want { + t.Errorf("pattern %q value %q: got %v, want %v", tc.pattern, tc.value, got, tc.want) + } + } + }) + } + t.Setenv("LC_ALL", "en_US.UTF-8") + native, err := newUnitPatternMatcherWithCompatibility([]string{"[a-z]"}, false) + if err != nil { + t.Fatal(err) + } + defer native.Close() + compat, err := newUnitPatternMatcherWithCompatibility([]string{"[[:digit:]a-z]"}, true) + if err != nil { + t.Fatal(err) + } + defer compat.Close() + for _, value := range []string{"a", "A", "é", "É", "à", "z", "\xff"} { + if compat.Matches([]byte(value)) != native.Matches([]byte(value)) { + t.Errorf("range differs from native matching for %q", value) + } + } + +} diff --git a/go/cmd/journalctl/unit_pattern_fallback.go b/go/cmd/journalctl/unit_pattern_fallback.go new file mode 100644 index 00000000..fb1f1a07 --- /dev/null +++ b/go/cmd/journalctl/unit_pattern_fallback.go @@ -0,0 +1,9 @@ +//go:build cgo && !(linux || darwin || freebsd) + +package main + +type unitPatternMatcher = byteUnitMatcher + +func newUnitPatternMatcher(patterns []string) (*unitPatternMatcher, error) { + return newByteUnitMatcher(patterns) +} diff --git a/go/cmd/journalctl/unit_pattern_test.go b/go/cmd/journalctl/unit_pattern_test.go new file mode 100644 index 00000000..b1842b14 --- /dev/null +++ b/go/cmd/journalctl/unit_pattern_test.go @@ -0,0 +1,33 @@ +//go:build cgo + +package main + +import "testing" + +func TestUnitPatternFallbackByteGrammar(t *testing.T) { + for _, test := range []struct { + pattern, value string + want bool + }{ + {"[[.a.]]lpha.*", "alpha.service", true}, + {"[[]", "[", true}, + {"[[:alpha:]]lpha.*", "alpha.service", true}, + {"[!b]lpha.*", "blpha.service", false}, + {"[]a]*", "]name", true}, + {"alpha[", "alpha[", true}, + {`alpha\x20*`, `alpha\x20name.service`, true}, + {"bad-?.service", "bad-\xff.service", true}, + {"bad-?.service", "bad-é.service", false}, + {"bad-??.service", "bad-é.service", true}, + } { + matcher, err := newByteUnitMatcher([]string{test.pattern}) + if err != nil { + t.Fatal(err) + } + got := matcher.Matches([]byte(test.value)) + matcher.Close() + if got != test.want { + t.Errorf("%q matches %q: got %v, want %v", test.pattern, test.value, got, test.want) + } + } +} diff --git a/go/cmd/journalctl/unit_pattern_unicode.go b/go/cmd/journalctl/unit_pattern_unicode.go new file mode 100644 index 00000000..1bbaba2e --- /dev/null +++ b/go/cmd/journalctl/unit_pattern_unicode.go @@ -0,0 +1,49 @@ +//go:build cgo + +package main + +import ( + "regexp" + "unicode/utf8" +) + +type byteUnitMatcher struct { + patterns []*regexp.Regexp + buffer []byte +} + +func newByteUnitMatcher(patterns []string) (*byteUnitMatcher, error) { + m := &byteUnitMatcher{} + for _, pattern := range patterns { + re, err := regexp.Compile(globPatternToRegex(pattern)) + if err != nil { + continue + } // An invalid bracket expression cannot match. + m.patterns = append(m.patterns, re) + } + return m, nil +} + +func (m *byteUnitMatcher) Matches(value []byte) bool { + for _, c := range value { + if c < 128 { + continue + } + // Unit patterns are mangled to ASCII. Map each payload byte to one + // rune so Go's UTF-8 regexp preserves native C-locale byte matching. + m.buffer = m.buffer[:0] + for _, b := range value { + m.buffer = utf8.AppendRune(m.buffer, rune(b)) + } + value = m.buffer + break + } + for _, pattern := range m.patterns { + if pattern.Match(value) { + return true + } + } + return false +} + +func (*byteUnitMatcher) Close() {} diff --git a/go/cmd/journalctl/unit_pattern_unix.go b/go/cmd/journalctl/unit_pattern_unix.go new file mode 100644 index 00000000..2ff08dea --- /dev/null +++ b/go/cmd/journalctl/unit_pattern_unix.go @@ -0,0 +1,143 @@ +//go:build cgo && (linux || darwin || freebsd) + +package main + +/* +#define _GNU_SOURCE +#include +#include +#include +#include "unit_pattern_compat.h" + +static struct journal_unit_pattern *unit_pattern_prepare(locale_t selected, const char *pattern) { + locale_t previous = uselocale(selected); + if (!previous) return NULL; + struct journal_unit_pattern *p = journal_unit_pattern_new(pattern); + uselocale(previous); + return p; +} +static int unit_pattern_compat_matches(locale_t selected, char **patterns, struct journal_unit_pattern **compat, size_t count, const char *value) { + locale_t previous = uselocale(selected); + if (!previous) return 0; + int matched = 0; + for (size_t i = 0; i < count; i++) + if (compat[i] ? journal_unit_pattern_match(compat[i], value) : fnmatch(patterns[i], value, FNM_NOESCAPE) == 0) { + matched = 1; + break; + } + uselocale(previous); + return matched; +} + +static locale_t unit_pattern_locale(void) { + locale_t selected = newlocale(LC_ALL_MASK, "", (locale_t) 0); + return selected ? selected : newlocale(LC_ALL_MASK, "C", (locale_t) 0); +} +static int unit_pattern_matches(locale_t selected, char **patterns, size_t count, const char *value) { + locale_t previous = uselocale(selected); + if (!previous) return 0; + int matched = 0; + for (size_t i = 0; i < count; i++) + if (fnmatch(patterns[i], value, FNM_NOESCAPE) == 0) { + matched = 1; + break; + } + uselocale(previous); + return matched; +} +*/ +import "C" + +import ( + "fmt" + "runtime" + "strings" + "unsafe" +) + +type unitPatternMatcher struct { + locale C.locale_t + patterns **C.char + count int + buffer []byte + compat **C.struct_journal_unit_pattern +} + +func newUnitPatternMatcher(patterns []string) (*unitPatternMatcher, error) { + return newUnitPatternMatcherWithCompatibility(patterns, runtime.GOOS == "freebsd") +} + +func newUnitPatternMatcherWithCompatibility(patterns []string, compatibility bool) (*unitPatternMatcher, error) { + m := &unitPatternMatcher{locale: C.unit_pattern_locale(), count: len(patterns)} + if m.locale == nil { + return nil, fmt.Errorf("failed to initialize unit pattern locale") + } + m.patterns = (**C.char)(C.calloc(C.size_t(len(patterns)), C.size_t(unsafe.Sizeof((*C.char)(nil))))) + if m.patterns == nil { + m.Close() + return nil, fmt.Errorf("failed to allocate unit patterns") + } + if compatibility { + compatibility = false + for _, pattern := range patterns { + if strings.Contains(pattern, "[:") || strings.Contains(pattern, "[.") || strings.Contains(pattern, "[=") { + compatibility = true + break + } + } + } + if compatibility { + m.compat = (**C.struct_journal_unit_pattern)(C.calloc(C.size_t(len(patterns)), C.size_t(unsafe.Sizeof((*C.struct_journal_unit_pattern)(nil))))) + if m.compat == nil { + m.Close() + return nil, fmt.Errorf("failed to allocate unit pattern compatibility state") + } + } + for i, pattern := range patterns { + p := C.CString(pattern) + if p == nil { + m.Close() + return nil, fmt.Errorf("failed to allocate unit pattern") + } + unsafe.Slice(m.patterns, m.count)[i] = p + if compatibility && C.journal_unit_pattern_needed(p) != 0 { + c := C.unit_pattern_prepare(m.locale, p) + if c == nil { + m.Close() + return nil, fmt.Errorf("failed to prepare unit pattern") + } + unsafe.Slice(m.compat, m.count)[i] = c + } + } + return m, nil +} + +func (m *unitPatternMatcher) Matches(value []byte) bool { + m.buffer = append(m.buffer[:0], value...) + m.buffer = append(m.buffer, 0) + if m.compat != nil { + return C.unit_pattern_compat_matches(m.locale, m.patterns, m.compat, C.size_t(m.count), (*C.char)(unsafe.Pointer(&m.buffer[0]))) != 0 + } + return C.unit_pattern_matches(m.locale, m.patterns, C.size_t(m.count), (*C.char)(unsafe.Pointer(&m.buffer[0]))) != 0 +} + +func (m *unitPatternMatcher) Close() { + if m.compat != nil { + for _, p := range unsafe.Slice(m.compat, m.count) { + C.journal_unit_pattern_free(p) + } + C.free(unsafe.Pointer(m.compat)) + m.compat = nil + } + if m.patterns != nil { + for _, pattern := range unsafe.Slice(m.patterns, m.count) { + C.free(unsafe.Pointer(pattern)) + } + C.free(unsafe.Pointer(m.patterns)) + m.patterns = nil + } + if m.locale != nil { + C.freelocale(m.locale) + m.locale = nil + } +} diff --git a/go/cmd/journalctl/vacuum_metadata_bsd.go b/go/cmd/journalctl/vacuum_metadata_bsd.go new file mode 100644 index 00000000..8ae630ec --- /dev/null +++ b/go/cmd/journalctl/vacuum_metadata_bsd.go @@ -0,0 +1,24 @@ +//go:build cgo && (darwin || freebsd) + +package main + +import ( + "os" + "syscall" + "time" +) + +const vacuumNoAtime = 0 + +func vacuumMetadataRealtime(info os.FileInfo, realtime uint64) uint64 { + if stat, ok := info.Sys().(*syscall.Stat_t); ok { + for _, stamp := range [...]syscall.Timespec{stat.Atimespec, stat.Ctimespec, stat.Birthtimespec} { + realtime = earlierVacuumUsec(realtime, timeToUsec(time.Unix(stamp.Sec, stamp.Nsec))) + } + } + return realtime +} + +func vacuumCreationRealtime(_ string, realtime uint64) uint64 { + return realtime +} diff --git a/go/cmd/journalctl/vacuum_metadata_linux.go b/go/cmd/journalctl/vacuum_metadata_linux.go new file mode 100644 index 00000000..4f4639e3 --- /dev/null +++ b/go/cmd/journalctl/vacuum_metadata_linux.go @@ -0,0 +1,35 @@ +//go:build cgo && linux + +package main + +import ( + "encoding/binary" + "os" + "syscall" + "time" + + "golang.org/x/sys/unix" +) + +const vacuumNoAtime = unix.O_NOATIME + +func vacuumMetadataRealtime(info os.FileInfo, realtime uint64) uint64 { + if stat, ok := info.Sys().(*syscall.Stat_t); ok { + realtime = earlierVacuumUsec(realtime, timeToUsec(time.Unix(int64(stat.Atim.Sec), int64(stat.Atim.Nsec)))) + realtime = earlierVacuumUsec(realtime, timeToUsec(time.Unix(int64(stat.Ctim.Sec), int64(stat.Ctim.Nsec)))) + } + return realtime +} + +func vacuumCreationRealtime(path string, realtime uint64) uint64 { + var stat unix.Statx_t + if err := unix.Statx(unix.AT_FDCWD, path, unix.AT_STATX_DONT_SYNC, unix.STATX_BTIME, &stat); err == nil && stat.Mask&unix.STATX_BTIME != 0 && stat.Btime.Sec != 0 { + realtime = earlierVacuumUsec(realtime, timeToUsec(time.Unix(stat.Btime.Sec, int64(stat.Btime.Nsec)))) + } + // Older journals may retain their original creation time in this xattr. + var data [8]byte + if n, err := unix.Getxattr(path, "user.crtime_usec", data[:]); err == nil && n == len(data) { + realtime = earlierVacuumUsec(realtime, binary.LittleEndian.Uint64(data[:])) + } + return realtime +} diff --git a/go/cmd/journalctl/vacuum_metadata_other.go b/go/cmd/journalctl/vacuum_metadata_other.go new file mode 100644 index 00000000..f7392033 --- /dev/null +++ b/go/cmd/journalctl/vacuum_metadata_other.go @@ -0,0 +1,15 @@ +//go:build cgo && !linux && !darwin && !freebsd && !windows + +package main + +import "os" + +const vacuumNoAtime = 0 + +func vacuumMetadataRealtime(_ os.FileInfo, realtime uint64) uint64 { + return realtime +} + +func vacuumCreationRealtime(_ string, realtime uint64) uint64 { + return realtime +} diff --git a/go/cmd/journalctl/vacuum_metadata_windows.go b/go/cmd/journalctl/vacuum_metadata_windows.go new file mode 100644 index 00000000..5c496c6c --- /dev/null +++ b/go/cmd/journalctl/vacuum_metadata_windows.go @@ -0,0 +1,22 @@ +//go:build cgo && windows + +package main + +import ( + "os" + "syscall" + "time" +) + +func vacuumMetadataRealtime(info os.FileInfo, realtime uint64) uint64 { + if stat, ok := info.Sys().(*syscall.Win32FileAttributeData); ok { + for _, stamp := range [...]syscall.Filetime{stat.LastAccessTime, stat.CreationTime} { + realtime = earlierVacuumUsec(realtime, timeToUsec(time.Unix(0, stamp.Nanoseconds()))) + } + } + return realtime +} + +func vacuumCreationRealtime(_ string, realtime uint64) uint64 { + return realtime +} diff --git a/go/cmd/journalctl/vacuum_open_other.go b/go/cmd/journalctl/vacuum_open_other.go index 47279a3c..93c8a4df 100644 --- a/go/cmd/journalctl/vacuum_open_other.go +++ b/go/cmd/journalctl/vacuum_open_other.go @@ -1,4 +1,4 @@ -//go:build !unix +//go:build cgo && !unix package main diff --git a/go/cmd/journalctl/vacuum_open_unix.go b/go/cmd/journalctl/vacuum_open_unix.go index 67bffe01..65da509f 100644 --- a/go/cmd/journalctl/vacuum_open_unix.go +++ b/go/cmd/journalctl/vacuum_open_unix.go @@ -1,4 +1,4 @@ -//go:build unix +//go:build cgo && unix package main @@ -9,7 +9,12 @@ import ( ) func openVacuumCandidate(path string) (*os.File, error) { - fd, err := unix.Open(path, unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0) + flags := unix.O_RDONLY | unix.O_CLOEXEC | unix.O_NOFOLLOW | unix.O_NONBLOCK + fd, err := unix.Open(path, flags|vacuumNoAtime, 0) + if err != nil && vacuumNoAtime != 0 { + // Native retries when preserving atime is unavailable to the caller. + fd, err = unix.Open(path, flags, 0) + } if err != nil { return nil, err } diff --git a/go/cmd/journalctl/vacuum_order.go b/go/cmd/journalctl/vacuum_order.go new file mode 100644 index 00000000..118b3b46 --- /dev/null +++ b/go/cmd/journalctl/vacuum_order.go @@ -0,0 +1,92 @@ +//go:build cgo + +package main + +import "sort" + +type vacuumRun struct{ next, end int } + +type vacuumOrder struct { + candidates []vacuumCandidate + heads []vacuumRun +} + +func newVacuumOrder(candidates []vacuumCandidate) *vacuumOrder { + // A pairwise sequence/time comparator can cycle after a clock rollback. + sort.Slice(candidates, func(i, j int) bool { + a, b := candidates[i], candidates[j] + if a.haveSeqno != b.haveSeqno { + return a.haveSeqno + } + if a.haveSeqno { + if a.seqnumID != b.seqnumID { + return a.seqnumID < b.seqnumID + } + if a.seqnum != b.seqnum { + return a.seqnum < b.seqnum + } + } + return a.name < b.name + }) + o := &vacuumOrder{candidates: candidates} + for start := 0; start < len(candidates); { + end := start + 1 + if candidates[start].haveSeqno { + for end < len(candidates) && candidates[end].haveSeqno && candidates[end].seqnumID == candidates[start].seqnumID { + end++ + } + } + o.heads = append(o.heads, vacuumRun{start, end}) + start = end + } + for i := len(o.heads)/2 - 1; i >= 0; i-- { + o.down(i) + } + return o +} + +func (o *vacuumOrder) less(i, j int) bool { + a, b := o.candidates[o.heads[i].next], o.candidates[o.heads[j].next] + if a.realtime != b.realtime { + return a.realtime < b.realtime + } + if a.haveSeqno != b.haveSeqno { + return a.haveSeqno + } + if a.haveSeqno && a.seqnumID != b.seqnumID { + return a.seqnumID < b.seqnumID + } + return a.name < b.name +} + +func (o *vacuumOrder) down(i int) { + for { + child := 2*i + 1 + if child >= len(o.heads) { + return + } + if child+1 < len(o.heads) && o.less(child+1, child) { + child++ + } + if !o.less(child, i) { + return + } + o.heads[i], o.heads[child] = o.heads[child], o.heads[i] + i = child + } +} + +func (o *vacuumOrder) next() *vacuumCandidate { + if len(o.heads) == 0 { + return nil + } + candidate := &o.candidates[o.heads[0].next] + o.heads[0].next++ + if o.heads[0].next == o.heads[0].end { + o.heads[0] = o.heads[len(o.heads)-1] + o.heads = o.heads[:len(o.heads)-1] + } + // The newly exposed timestamp may move backward; only the root changed. + o.down(0) + return candidate +} diff --git a/go/cmd/journalctl/vacuum_order_test.go b/go/cmd/journalctl/vacuum_order_test.go new file mode 100644 index 00000000..b1f58c48 --- /dev/null +++ b/go/cmd/journalctl/vacuum_order_test.go @@ -0,0 +1,43 @@ +//go:build cgo + +package main + +import ( + "reflect" + "testing" +) + +func TestVacuumHistoryOrder(t *testing.T) { + // A 0 { + next = nodes[i-1] + } + binary.LittleEndian.PutUint64(data[offset+32:], next) + } + } + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + r = mustOpenReaderFile(t, path) + defer r.Close() + _, err = r.QueryUnique("MESSAGE") + if cycle && !errors.Is(err, errCorruptObject) { + t.Fatalf("QueryUnique error=%v", err) + } + if !cycle && err != nil { + t.Fatal(err) + } + if err := r.QueryUniqueState("MESSAGE"); err != nil { + t.Fatal(err) + } + count := 0 + for count < 16 { + _, ok, nextErr := r.EnumerateUniquePayload() + err = nextErr + if err != nil || !ok { + break + } + count++ + } + if cycle && !errors.Is(err, errCorruptObject) { + t.Fatalf("stateful error=%v", err) + } + if !cycle && err != nil { + t.Fatal(err) + } + if !cycle && count != 3 { + t.Fatalf("acyclic values=%d", count) + } + if cycle { + if _, ok, err := r.EnumerateUniquePayload(); err != nil || ok { + t.Fatalf("failed iterator not exhausted: %v %v", ok, err) + } + } + }) + } +} diff --git a/go/journal/directory_reader.go b/go/journal/directory_reader.go index d1f1fa30..f3f22133 100644 --- a/go/journal/directory_reader.go +++ b/go/journal/directory_reader.go @@ -16,21 +16,25 @@ const directoryUniqueCacheCapacity = 8 // DirectoryReader reads multiple journal files in journal order. A // DirectoryReader is not safe for concurrent use by multiple goroutines. type DirectoryReader struct { - files []*Reader - index int - filter *filterBuilder - realtimeSeek *uint64 - realtimeSeekBound *directoryRealtimeSeekBound - candidates []*directoryCandidate - currentKey *directoryEntryKey - direction Direction - hasDirection bool - bootNewest map[UUID]directoryBootNewest - nonOverlapping bool - uniqueCache map[directoryUniqueCacheKey][][]byte - uniqueCacheOrder []directoryUniqueCacheKey - uniqueState *directoryUniqueState - uniqueCacheBuilds int + files []*Reader + index int + filter *filterBuilder + realtimeSeek *uint64 + realtimeSeekBound *directoryRealtimeSeekBound + candidates []*directoryCandidate + currentKey *directoryEntryKey + direction Direction + hasDirection bool + filteredCandidates bool + merging bool + retryCopies bool + bootNewest map[UUID]directoryBootNewest + nonOverlapping bool + orderDirty bool + uniqueCache map[directoryUniqueCacheKey][][]byte + uniqueCacheOrder []directoryUniqueCacheKey + uniqueState *directoryUniqueState + uniqueCacheBuilds int } func OpenDirectory(path string) (*DirectoryReader, error) { @@ -111,14 +115,18 @@ func newDirectoryReader(readers []*Reader, allowEmpty bool) (*DirectoryReader, e return readers[i].header.headEntrySeqnum < readers[j].header.headEntrySeqnum }) - return &DirectoryReader{ + dr := &DirectoryReader{ files: readers, index: -1, candidates: make([]*directoryCandidate, len(readers)), bootNewest: buildDirectoryBootNewest(readers), nonOverlapping: directoryFilesNonOverlapping(readers), uniqueCache: make(map[directoryUniqueCacheKey][][]byte), - }, nil + } + for _, reader := range readers { + reader.directoryOrderDirty = &dr.orderDirty + } + return dr, nil } func directoryFilesNonOverlapping(readers []*Reader) bool { @@ -265,16 +273,17 @@ func isASCIIHex(ch rune) bool { func buildDirectoryBootNewest(readers []*Reader) map[UUID]directoryBootNewest { boots := make(map[UUID]directoryBootNewest) for _, r := range readers { - bootID := r.header.tailEntryBootID + header := r.effectiveHeader() + bootID := header.tailEntryBootID if uuidIsZero(bootID) { continue } current, ok := boots[bootID] - if !ok || r.header.tailEntryMonotonic > current.monotonic { + if !ok || header.tailEntryMonotonic > current.monotonic { boots[bootID] = directoryBootNewest{ - machineID: r.header.machineID, - monotonic: r.header.tailEntryMonotonic, - realtime: r.header.tailEntryRealtime, + machineID: header.machineID, + monotonic: header.tailEntryMonotonic, + realtime: header.tailEntryRealtime, } } } @@ -406,35 +415,48 @@ func (dr *DirectoryReader) currentReader() (*Reader, error) { } func (dr *DirectoryReader) AddMatch(data []byte) { - if dr.filter == nil { - dr.filter = &filterBuilder{} + dr.invalidateFilterCandidates() + for _, reader := range dr.files { + reader.AddMatch(data) + dr.filter = reader.filter } - dr.filter.addMatch(data) - dr.resetMergeState() } -func (dr *DirectoryReader) AddDisjunction() { - if dr.filter == nil { - dr.filter = &filterBuilder{} +func (dr *DirectoryReader) AddDisjunction() error { + dr.invalidateFilterCandidates() + for _, reader := range dr.files { + if err := reader.AddDisjunction(); err != nil { + return err + } + dr.filter = reader.filter } - dr.filter.addDisjunction() - dr.resetMergeState() + return nil } -func (dr *DirectoryReader) AddConjunction() { - if dr.filter == nil { - dr.filter = &filterBuilder{} +func (dr *DirectoryReader) AddConjunction() error { + dr.invalidateFilterCandidates() + for _, reader := range dr.files { + if err := reader.AddConjunction(); err != nil { + return err + } + dr.filter = reader.filter } - dr.filter.addConjunction() - dr.resetMergeState() + return nil } func (dr *DirectoryReader) FlushMatches() { + for _, reader := range dr.files { + reader.FlushMatches() + } dr.filter = nil + dr.realtimeSeek = nil dr.resetMergeState() } func (dr *DirectoryReader) GetRealtimeUsec() (uint64, error) { + if dr.index < 0 || dr.index >= len(dr.files) { + return 0, errEndOfEntries + } if dr.currentKey != nil { return dr.currentKey.realtime, nil } @@ -446,6 +468,9 @@ func (dr *DirectoryReader) GetRealtimeUsec() (uint64, error) { } func (dr *DirectoryReader) GetCursor() (string, error) { + if dr.index < 0 || dr.index >= len(dr.files) { + return "", errEndOfEntries + } if dr.currentKey != nil { return formatCursorFromDirectoryKey(*dr.currentKey), nil } @@ -656,6 +681,9 @@ func (dr *DirectoryReader) uniqueCacheKey(fieldName string) directoryUniqueCache h.tailEntryMonotonic, h.tailEntryBootID.String(), ) + if r.recovery.Incomplete() { + fmt.Fprintf(&b, ":recovery:%d:%d:%d:%d", r.recovery.Reason, r.recovery.FirstLostOffset, r.recovery.RecoveredEntries, r.fileSize) + } } return directoryUniqueCacheKey(b.String()) } @@ -677,6 +705,7 @@ func (dr *DirectoryReader) EnumerateFields() (map[string]struct{}, error) { } func (dr *DirectoryReader) SeekHead() error { + dr.merging = false dr.index = -1 dr.realtimeSeek = nil dr.realtimeSeekBound = nil @@ -692,6 +721,7 @@ func (dr *DirectoryReader) SeekHead() error { } func (dr *DirectoryReader) SeekTail() error { + dr.merging = false dr.index = -1 dr.realtimeSeek = nil dr.realtimeSeekBound = nil @@ -707,6 +737,7 @@ func (dr *DirectoryReader) SeekTail() error { } func (dr *DirectoryReader) SeekRealtimeUsec(usec uint64) error { + dr.merging = false value := usec dr.realtimeSeek = &value dr.realtimeSeekBound = nil @@ -716,10 +747,80 @@ func (dr *DirectoryReader) SeekRealtimeUsec(usec uint64) error { return nil } +type directoryReaderPosition struct { + index int + realtime *uint64 + override uint64 +} + +func (r *Reader) directoryPosition() directoryReaderPosition { + position := directoryReaderPosition{index: r.entryIndex, realtime: r.realtimeSeek} + if r.filter != nil { + position.override = r.filter.overrideOffset + } + return position +} + +func (r *Reader) restoreDirectoryPosition(position directoryReaderPosition) { + r.clearCurrentEntryState() + r.entryIndex, r.realtimeSeek = position.index, position.realtime + if r.filter != nil { + r.filter.overrideOffset = position.override + r.filter.walkValid = false + r.filter.retry = true + invalidateFilterWalk(r.filter.expr) + } +} + func (dr *DirectoryReader) stepMerged(direction Direction, applyFilter bool) (bool, error) { + if dr.retryCopies { + // Getters may have cached damaged bytes again before the caller repaired them. + for _, reader := range dr.files { + reader.clearCurrentEntryState() + if reader.accessor != nil { + reader.accessor.invalidateCopiedWindows() + } + } + dr.retryCopies = false + } + index := dr.index + var selected *Reader + var before directoryReaderPosition + if dr.currentKey != nil && index >= 0 && index < len(dr.files) { + selected = dr.files[index] + before = selected.directoryPosition() + } else { + index = -1 + } + ok, err := dr.stepMergedInner(direction, applyFilter) + if err != nil { + dr.retryCopies = true + dr.index = index + if selected != nil { + selected.restoreDirectoryPosition(before) + // Recompute only the candidate whose child was restored. + dr.candidates[index] = nil + } else if !dr.merging { + dr.hasDirection = false + } + } + return ok, err +} + +func (dr *DirectoryReader) stepMergedInner(direction Direction, applyFilter bool) (bool, error) { + filtered := applyFilter && dr.filter != nil + if dr.filteredCandidates != filtered { + dr.filteredCandidates = filtered + if !dr.hasSequentialPosition() { + dr.hasDirection = false + } + dr.resetCandidates() + } if dr.canStepSequential(direction, applyFilter) { - return dr.stepSequential(direction) + return dr.stepSequential(direction, applyFilter) } + dr.merging = true +retry: if err := dr.prepareMergeDirection(direction); err != nil { return false, err } @@ -729,6 +830,10 @@ func (dr *DirectoryReader) stepMerged(direction Direction, applyFilter bool) (bo if err := dr.fillCandidate(i, direction, applyFilter); err != nil { return false, err } + if dr.orderDirty { + dr.restartRecoveryMerge() + goto retry + } candidate := dr.candidates[i] if candidate == nil { continue @@ -757,8 +862,21 @@ func (dr *DirectoryReader) stepMerged(direction Direction, applyFilter bool) (bo return true, nil } +func (dr *DirectoryReader) restartRecoveryMerge() { + dr.orderDirty = false + dr.bootNewest = buildDirectoryBootNewest(dr.files) + // A recovery transition invalidates an existing sequential traversal proof. + dr.nonOverlapping = false + if dr.currentKey == nil && dr.realtimeSeekBound != nil { + usec := dr.realtimeSeekBound.usec + dr.realtimeSeek = &usec + } + dr.hasDirection = false + dr.resetCandidates() +} + func (dr *DirectoryReader) canStepSequential(direction Direction, applyFilter bool) bool { - if !dr.nonOverlapping || dr.realtimeSeek != nil || applyFilter && dr.filter != nil { + if !dr.nonOverlapping || dr.merging || dr.realtimeSeek != nil || !dr.hasDirection && dr.currentKey != nil { return false } if dr.hasDirection && dr.direction != direction && dr.currentKey != nil { @@ -767,7 +885,7 @@ func (dr *DirectoryReader) canStepSequential(direction Direction, applyFilter bo return true } -func (dr *DirectoryReader) stepSequential(direction Direction) (bool, error) { +func (dr *DirectoryReader) stepSequential(direction Direction, applyFilter bool) (bool, error) { if len(dr.files) == 0 { dr.clearDirectoryPosition() return false, nil @@ -778,9 +896,9 @@ func (dr *DirectoryReader) stepSequential(direction Direction) (bool, error) { } } if direction == DirectionForward { - return dr.stepSequentialForward() + return dr.stepSequentialForward(applyFilter) } - return dr.stepSequentialBackward() + return dr.stepSequentialBackward(applyFilter) } func (dr *DirectoryReader) clearDirectoryPosition() { @@ -814,43 +932,57 @@ func seekReaderBoundary(r *Reader, direction Direction) error { return r.SeekTail() } -func (dr *DirectoryReader) stepSequentialForward() (bool, error) { +func (dr *DirectoryReader) stepSequentialForward(applyFilter bool) (bool, error) { if dr.index < 0 { dr.index = 0 } for dr.index < len(dr.files) { - ok, err := dr.stepSequentialReader(DirectionForward) + ok, err := dr.stepSequentialReader(DirectionForward, applyFilter) if err != nil || ok { return ok, err } dr.index++ } - dr.clearDirectoryPosition() + dr.finishSequentialEnd(applyFilter) return false, nil } -func (dr *DirectoryReader) stepSequentialBackward() (bool, error) { +func (dr *DirectoryReader) stepSequentialBackward(applyFilter bool) (bool, error) { if dr.index >= len(dr.files) { dr.index = len(dr.files) - 1 } for dr.index >= 0 { - ok, err := dr.stepSequentialReader(DirectionBackward) + ok, err := dr.stepSequentialReader(DirectionBackward, applyFilter) if err != nil || ok { return ok, err } dr.index-- } - dr.clearDirectoryPosition() + dr.finishSequentialEnd(applyFilter) return false, nil } -func (dr *DirectoryReader) stepSequentialReader(direction Direction) (bool, error) { - ok, err := stepReaderRaw(dr.files[dr.index], direction) +func (dr *DirectoryReader) finishSequentialEnd(applyFilter bool) { + dr.index = -1 + if !applyFilter || dr.filter == nil { + dr.currentKey = nil + } +} + +func (dr *DirectoryReader) stepSequentialReader(direction Direction, applyFilter bool) (bool, error) { + r := dr.files[dr.index] + before := r.directoryPosition() + ok, err := stepReader(r, direction, applyFilter) + if err == nil && dr.orderDirty { + dr.restartRecoveryMerge() + return dr.stepMergedInner(direction, applyFilter) + } if err != nil || !ok { return ok, err } - key, err := dr.files[dr.index].currentDirectoryEntryKey() + key, err := r.currentDirectoryEntryKey() if err != nil { + r.restoreDirectoryPosition(before) return false, err } dr.currentKey = &key @@ -918,24 +1050,21 @@ func (dr *DirectoryReader) fillCandidate(readerIndex int, direction Direction, a r := dr.files[readerIndex] for { - ok, err := stepReaderRaw(r, direction) + before := r.directoryPosition() + ok, err := stepReader(r, direction, applyFilter) if err != nil { return err } - if !ok { + if dr.orderDirty { return nil } - - matches, err := dr.readerEntryMatches(r, applyFilter) - if err != nil { - return err - } - if !matches { - continue + if !ok { + return nil } key, err := r.currentDirectoryEntryKey() if err != nil { + r.restoreDirectoryPosition(before) return err } if !dr.keyPassesRealtimeBound(key) { @@ -953,17 +1082,6 @@ func (dr *DirectoryReader) fillCandidate(readerIndex int, direction Direction, a } } -func (dr *DirectoryReader) readerEntryMatches(r *Reader, applyFilter bool) (bool, error) { - if !applyFilter || dr.filter == nil { - return true, nil - } - entry, err := r.GetEntry() - if err != nil { - return false, err - } - return dr.filter.matches(entry), nil -} - func (dr *DirectoryReader) keyPassesRealtimeBound(key directoryEntryKey) bool { if dr.realtimeSeekBound == nil { return true @@ -986,6 +1104,16 @@ func (dr *DirectoryReader) keyPassesCurrentPosition(key directoryEntryKey, direc return cmp < 0 } +func stepReader(r *Reader, direction Direction, applyFilter bool) (bool, error) { + if applyFilter { + if direction == DirectionForward { + return r.Step() + } + return r.StepBack() + } + return stepReaderRaw(r, direction) +} + func stepReaderRaw(r *Reader, direction Direction) (bool, error) { if direction == DirectionForward { err := r.Next() @@ -1081,7 +1209,29 @@ func cmpUint64(a, b uint64) int { return 0 } +func (dr *DirectoryReader) invalidateFilterCandidates() { + if dr.hasSequentialPosition() { + // Sequential children have no prefetched rows to reposition. + dr.resetCandidates() + return + } + if dr.currentKey == nil && dr.realtimeSeek == nil && dr.realtimeSeekBound != nil { + usec := dr.realtimeSeekBound.usec + dr.realtimeSeek = &usec + } + // Revalidate changed matches from the committed key, not prefetched rows. + dr.merging = dr.currentKey != nil + dr.hasDirection = false + dr.realtimeSeekBound = nil + dr.resetCandidates() +} + +func (dr *DirectoryReader) hasSequentialPosition() bool { + return dr.nonOverlapping && !dr.merging && dr.hasDirection && dr.currentKey != nil && dr.index >= 0 && dr.index < len(dr.files) +} + func (dr *DirectoryReader) resetMergeState() { + dr.merging = false dr.currentKey = nil dr.hasDirection = false dr.index = -1 @@ -1111,29 +1261,33 @@ func (dr *DirectoryReader) ListBoots() ([]BootInfo, error) { bootMap := make(map[string]*bootEntry) for _, r := range dr.files { - bootID := r.header.tailEntryBootID + header := r.effectiveHeader() + if r.recovery.Incomplete() && header.nEntries == 0 { + continue + } + bootID := header.tailEntryBootID key := bootID.String() if entry, ok := bootMap[key]; ok { - if r.header.headEntrySeqnum < entry.firstSeq { - entry.firstSeq = r.header.headEntrySeqnum + if header.headEntrySeqnum < entry.firstSeq { + entry.firstSeq = header.headEntrySeqnum } - if r.header.headEntryRealtime < entry.firstTime { - entry.firstTime = r.header.headEntryRealtime + if header.headEntryRealtime < entry.firstTime { + entry.firstTime = header.headEntryRealtime } - if r.header.tailEntrySeqnum > entry.lastSeq { - entry.lastSeq = r.header.tailEntrySeqnum + if header.tailEntrySeqnum > entry.lastSeq { + entry.lastSeq = header.tailEntrySeqnum } - if r.header.tailEntryRealtime > entry.lastTime { - entry.lastTime = r.header.tailEntryRealtime + if header.tailEntryRealtime > entry.lastTime { + entry.lastTime = header.tailEntryRealtime } } else { bootMap[key] = &bootEntry{ bootID: bootID, - firstSeq: r.header.headEntrySeqnum, - lastSeq: r.header.tailEntrySeqnum, - firstTime: r.header.headEntryRealtime, - lastTime: r.header.tailEntryRealtime, + firstSeq: header.headEntrySeqnum, + lastSeq: header.tailEntrySeqnum, + firstTime: header.headEntryRealtime, + lastTime: header.tailEntryRealtime, } } } diff --git a/go/journal/explorer.go b/go/journal/explorer.go index 6f3a7afa..3827cb06 100644 --- a/go/journal/explorer.go +++ b/go/journal/explorer.go @@ -1106,6 +1106,7 @@ func (r *Reader) findDataOffsetByPayload(payload []byte) (uint64, bool, error) { return 0, false, err } item := parseHashItem(itemBuf) + var chain chainGuard for offset := item.head; offset != 0; { header, err := r.readDataHeaderAt(offset) if err != nil { @@ -1120,6 +1121,9 @@ func (r *Reader) findDataOffsetByPayload(payload []byte) (uint64, bool, error) { return offset, true, nil } } + if !chain.advance(offset, header.nextHashOffset, false) { + return 0, false, errCorruptObject + } offset = header.nextHashOffset } return 0, false, nil @@ -1139,36 +1143,40 @@ func (r *Reader) readDataPayloadWithHeader(offset uint64, header dataHeader) ([] } func (r *Reader) firstDataEntryOffset(header dataHeader) (uint64, bool, error) { - if header.nEntries == 0 { - return 0, false, nil - } - if header.entryOffset != 0 { - return header.entryOffset, true, nil + if (header.entryOffset == 0) != (header.nEntries == 0) { + return 0, false, errCorruptObject } - if header.entryArrayOffset == 0 { - return 0, false, nil - } - _, offsets, _, err := r.readEntryArrayChunk(header.entryArrayOffset, 1) - if err != nil { - return 0, false, err + if r.recovery.Incomplete() { + var first uint64 + stop := errors.New("first recovered posting") + err := r.visitRecoveredDataEntryOffsets(header, func(offset uint64) error { first = offset; return stop }) + if errors.Is(err, stop) { + err = nil + } + return first, first != 0, err } - if len(offsets) == 0 { + if header.nEntries == 0 { return 0, false, nil } - return offsets[0], true, nil + return header.entryOffset, true, nil } func (r *Reader) visitDataEntryOffsets(header dataHeader, visit func(uint64) error) error { + if (header.entryOffset == 0) != (header.nEntries == 0) { + return errCorruptObject + } + if r.recovery.Incomplete() { + return r.visitRecoveredDataEntryOffsets(header, visit) + } if header.nEntries == 0 { return nil } - if header.entryOffset != 0 { - if err := visit(header.entryOffset); err != nil { - return err - } + if err := visit(header.entryOffset); err != nil { + return err } remaining := header.nEntries - 1 offset := header.entryArrayOffset + highest := offset for offset != 0 && remaining > 0 { arrayHeader, chunkOffsets, used, err := r.readEntryArrayChunk(offset, remaining) if err != nil { @@ -1180,8 +1188,20 @@ func (r *Reader) visitDataEntryOffsets(header dataHeader, visit func(uint64) err } } remaining -= used + if remaining > 0 && arrayHeader.nextArrayOffset != 0 { + if arrayHeader.nextArrayOffset <= highest { + if err := rejectArrayRevisit(r, header.entryArrayOffset, offset, arrayHeader.nextArrayOffset, errCorruptObject); err != nil { + return err + } + } else { + highest = arrayHeader.nextArrayOffset + } + } offset = arrayHeader.nextArrayOffset } + if remaining != 0 { + return errCorruptObject + } return nil } @@ -1190,11 +1210,15 @@ func (r *Reader) visitFieldDataObjects(field []byte, visit func(uint64, dataHead if err != nil || !ok { return err } + var chain chainGuard for offset != 0 { header, err := r.readDataHeaderAt(offset) if err != nil { return err } + if !chain.advance(offset, header.nextFieldOffset, true) { + return errCorruptObject + } payload, err := r.readDataPayloadWithHeader(offset, header) if err != nil { return err @@ -1209,40 +1233,68 @@ func (r *Reader) visitFieldDataObjects(field []byte, visit func(uint64, dataHead func (r *Reader) indexedCountFacetGroup(query ExplorerQuery, group facetPassGroup, candidates explorerCandidateSet, result *ExplorerResult) error { result.Stats.FacetRowsMatched += candidates.count + allCandidates := candidates.all + candidateOffsets := candidates.offsets + allPostings := allCandidates && !r.recovery.Incomplete() && uint64(len(r.entryOffsets)) == r.header.nEntries for _, facetIndex := range group.facetIndices { if facetIndex < 0 || facetIndex >= len(query.Facets) { continue } field := query.Facets[facetIndex] + fieldHasEquals := bytes.IndexByte(field, '=') >= 0 values := make(map[string]uint64) rowsWithField := make(map[uint64]struct{}) - err := r.visitFieldDataObjects(field, func(_ uint64, header dataHeader, payload []byte) error { - payloadField, value, ok := splitRawPayload(payload) - if !ok || !bytes.Equal(payloadField, field) { - return nil - } - var count uint64 - if err := r.visitDataEntryOffsets(header, func(entryOffset uint64) error { - result.Stats.DataRefsSeen++ - if candidates.contains(entryOffset) { - count++ - rowsWithField[entryOffset] = struct{}{} + var count, visited uint64 + visit := func(entryOffset uint64) error { + visited++ + if !allCandidates { + if _, ok := candidateOffsets[entryOffset]; !ok { + return nil } + } + count++ + rowsWithField[entryOffset] = struct{}{} + return nil + } + err := r.visitFieldDataObjects(field, func(offset uint64, header dataHeader, payload []byte) error { + if fieldHasEquals || len(payload) <= len(field) || payload[len(field)] != '=' || !bytes.Equal(payload[:len(field)], field) { return nil - }); err != nil { + } + value := payload[len(field)+1:] + var key string + if allPostings && header.nEntries != 0 { + key = string(value) + } + count = 0 + err := r.visitDataEntryOffsets(header, visit) + if err != nil { return err } + compressed := header.object.flag&(objectCompressedXZ|objectCompressedLZ4|objectCompressedZSTD) != 0 if count != 0 { - values[string(value)] += count + if !allPostings { + // Posting arrays can invalidate an uncompressed value. + if !compressed && header.nEntries > 1 && len(value) != 0 { + valueOffset := offset + (header.object.size - uint64(len(value))) + current, err := r.readSlice(valueOffset, uint64(len(value))) + if err != nil { + return err + } + value = current + } + key = string(value) + } + values[key] += count result.Stats.FacetUpdates += count } result.Stats.DataObjectsClassified++ result.Stats.DataPayloadsLoaded++ - if header.object.flag&(objectCompressedXZ|objectCompressedLZ4|objectCompressedZSTD) != 0 { + if compressed { result.Stats.PayloadsDecompressed++ } return nil }) + result.Stats.DataRefsSeen += visited if err != nil { return err } @@ -1260,39 +1312,113 @@ func (r *Reader) indexedCountHistogram(query ExplorerQuery, candidates explorerC return nil } field := result.Histogram.Field + fieldHasEquals := bytes.IndexByte(field, '=') >= 0 + allCandidates := candidates.all + candidateOffsets := candidates.offsets + ownInlineValue := false + if query.AfterRealtimeUsec == nil && query.BeforeRealtimeUsec == nil && len(result.Histogram.Buckets) != 0 { + first := result.Histogram.Buckets[0] + width := first.EndRealtimeUsec - first.StartRealtimeUsec + if width == 0 { + width = 1 + } + // Every possible timestamp must select a bucket before owning a singleton early. + ownInlineValue = (^uint64(0)-first.StartRealtimeUsec)/width <= uint64(^uint(0)>>1) + } rowsWithField := make(map[uint64]struct{}) - err := r.visitFieldDataObjects(field, func(_ uint64, header dataHeader, payload []byte) error { - payloadField, value, ok := splitRawPayload(payload) - if !ok || !bytes.Equal(payloadField, field) { + err := r.visitFieldDataObjects(field, func(offset uint64, header dataHeader, payload []byte) error { + if fieldHasEquals || len(payload) <= len(field) || payload[len(field)] != '=' || !bytes.Equal(payload[:len(field)], field) { return nil } - if err := r.visitDataEntryOffsets(header, func(entryOffset uint64) error { - result.Stats.DataRefsSeen++ - if !candidates.contains(entryOffset) { + value := payload[len(field)+1:] + compressed := header.object.flag&(objectCompressedXZ|objectCompressedLZ4|objectCompressedZSTD) != 0 + var err error + if header.nEntries == 1 && (compressed || ownInlineValue) { + // A singleton needs one key; raw bytes must be owned before ENTRY access. + err = r.visitDataEntryOffsets(header, func(entryOffset uint64) error { + result.Stats.DataRefsSeen++ + if !allCandidates { + if _, ok := candidateOffsets[entryOffset]; !ok { + return nil + } + } + var key string + if !compressed { + key = string(value) + } + entryHeader, err := r.readEntryHeaderAt(entryOffset) + if err != nil { + return err + } + rowsWithField[entryOffset] = struct{}{} + if !timestampInRange(query, entryHeader.realtime) { + return nil + } + bucket := histogramBucketIndex(result.Histogram, entryHeader.realtime) + if bucket >= 0 { + if compressed { + key = string(value) + } + result.Histogram.Buckets[bucket].Values[key]++ + result.Stats.HistogramUpdates++ + } return nil - } - entryHeader, err := r.readEntryHeaderAt(entryOffset) + }) if err != nil { return err } - rowsWithField[entryOffset] = struct{}{} - if !timestampInRange(query, entryHeader.realtime) { + if compressed { + result.Stats.PayloadsDecompressed++ + } + } else { + dataSize := header.object.size + var key string + keyReady := false + err = r.visitDataEntryOffsets(header, func(entryOffset uint64) error { + result.Stats.DataRefsSeen++ + if !allCandidates { + if _, ok := candidateOffsets[entryOffset]; !ok { + return nil + } + } + entryHeader, err := r.readEntryHeaderAt(entryOffset) + if err != nil { + return err + } + rowsWithField[entryOffset] = struct{}{} + if !timestampInRange(query, entryHeader.realtime) { + return nil + } + bucket := histogramBucketIndex(result.Histogram, entryHeader.realtime) + if bucket >= 0 { + if !keyReady { + current := value + // ENTRY reads may invalidate raw bytes, including the inline posting. + if !compressed && len(value) != 0 { + valueOffset := offset + (dataSize - uint64(len(value))) + var err error + current, err = r.readSlice(valueOffset, uint64(len(value))) + if err != nil { + return err + } + } + key = string(current) + keyReady = true + } + result.Histogram.Buckets[bucket].Values[key]++ + result.Stats.HistogramUpdates++ + } return nil + }) + if err != nil { + return err } - bucket := histogramBucketIndex(result.Histogram, entryHeader.realtime) - if bucket >= 0 { - result.Histogram.Buckets[bucket].Values[string(value)]++ - result.Stats.HistogramUpdates++ + if compressed { + result.Stats.PayloadsDecompressed++ } - return nil - }); err != nil { - return err } result.Stats.DataObjectsClassified++ result.Stats.DataPayloadsLoaded++ - if header.object.flag&(objectCompressedXZ|objectCompressedLZ4|objectCompressedZSTD) != 0 { - result.Stats.PayloadsDecompressed++ - } return nil }) if err != nil { @@ -1940,7 +2066,6 @@ func (r *Reader) setCurrentEntryOffset(offset uint64, direction Direction) bool } r.clearCurrentEntryState() r.entryIndex = index - r.direction = direction r.realtimeSeek = nil return true } @@ -2208,7 +2333,6 @@ func (r *Reader) seekExplorerBackwardRealtime(usec uint64) { } else { r.entryIndex = idx + 1 } - r.direction = DirectionBackward r.realtimeSeek = nil } diff --git a/go/journal/explorer_window_lifetime_test.go b/go/journal/explorer_window_lifetime_test.go new file mode 100644 index 00000000..ed7e1e7c --- /dev/null +++ b/go/journal/explorer_window_lifetime_test.go @@ -0,0 +1,623 @@ +package journal + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "reflect" + "strconv" + "strings" + "testing" + "time" +) + +type explorerWindowCase struct { + Path string + Kind string + Mode ReaderAccessMode + Compression int + TimeRange bool + ExpectError bool +} + +const explorerWindowMarker = "JOURNAL_SDK_EXPLORER_WINDOW_CASE" + +func runExplorerWindowChild(t *testing.T, c explorerWindowCase) { + t.Helper() + raw, err := json.Marshal(c) + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + // Isolate a possible mmap fault so every backend/layout still runs. + cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestExplorerIndexedWindowLifetime$", "-test.count=1") + cmd.Env = append(os.Environ(), explorerWindowMarker+"="+string(raw)) + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("child failed: %v\n%s", err, output) + } +} + +func explorerWindowFixture(t testing.TB, compact bool, compression, rows, cardinality int) string { + t.Helper() + path := filepath.Join(t.TempDir(), "explorer.journal") + opts := testOptions() + opts.Compact = compact + opts.Compression = compression + opts.CompressThresholdBytes = 32 + cadence := uint64(0) + opts.LivePublishEveryEntries = &cadence + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + for i := 0; i < rows; i++ { + selected := "skip" + if i%64 == 0 { + selected = "keep" + } + fields := []Field{StringField("SELECT", selected), StringField("MESSAGE", fmt.Sprintf("row-%04d", i))} + if i%5 != 0 { + fields = append(fields, StringField("VALUE", fmt.Sprintf("%04d-%s", i%cardinality, strings.Repeat("value-", 96)))) + } + if i%11 == 0 && i%5 != 0 { + fields = append(fields, StringField("VALUE", "extra-"+strings.Repeat("more-", 96))) + } + if err := w.Append(fields, EntryOptions{RealtimeUsec: 1000 + uint64(i), MonotonicUsec: 1 + uint64(i)}); err != nil { + _ = w.Close() + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + return path +} + +func explorerWindowQuery(kind string, sparse bool) ExplorerQuery { + q := DefaultExplorerQuery() + q.FieldMode = ExplorerFieldModeAllValues + q.Direction = DirectionForward + q.UseSourceRealtime = false + q.Limit = 3 + q.HistogramBuckets = 4 + if sparse { + q = q.WithFilter([]byte("SELECT"), []byte("keep")) + } + if kind == "facet" { + q = q.WithFacet([]byte("VALUE")) + } else { + q = q.WithHistogram([]byte("VALUE")) + } + return q +} + +func TestExplorerIndexedWindowLifetime(t *testing.T) { + if raw := os.Getenv(explorerWindowMarker); raw != "" { + var c explorerWindowCase + if err := json.Unmarshal([]byte(raw), &c); err != nil { + t.Fatal(err) + } + query := explorerWindowQuery(c.Kind, true) + if c.TimeRange { + after, before := uint64(1100), uint64(1800) + query.AfterRealtimeUsec, query.BeforeRealtimeUsec = &after, &before + } + if c.ExpectError { + query.Filters = nil + query.Limit = 0 + } + control, err := OpenFile(c.Path) + if err != nil { + t.Fatal(err) + } + want, err := control.ExploreWithStrategy(query, ExplorerStrategyTraversal) + _ = control.Close() + if err != nil { + t.Fatal(err) + } + control, err = OpenFile(c.Path) + if err != nil { + t.Fatal(err) + } + wantIndex, indexErr := control.ExploreWithStrategy(query, ExplorerStrategyIndex) + _ = control.Close() + expectedError := errInvalidJournal + if errors.Is(indexErr, errCorruptObject) { + expectedError = errCorruptObject + } + if c.ExpectError && !errors.Is(indexErr, expectedError) { + t.Fatalf("wide-window malformed index error = %v", indexErr) + } + if !c.ExpectError && indexErr != nil { + t.Fatal(indexErr) + } + window := uint64(4096) + if c.Mode == ReaderAccessReadAt { + window = 64 + } + r, err := OpenFileWithOptions(c.Path, DefaultReaderOptions().WithAccessMode(c.Mode).WithWindowSize(window).WithMaxWindows(1)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + before := r.AccessStats() + got, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if c.ExpectError { + if !errors.Is(err, expectedError) { + t.Fatalf("malformed index error = %v", err) + } + return + } + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(got.Facets, want.Facets) || !reflect.DeepEqual(got.Histogram, want.Histogram) || !reflect.DeepEqual(got.Rows, want.Rows) { + t.Fatalf("indexed/traversal mismatch: facet=%v histogram=%v rows=%v", reflect.DeepEqual(got.Facets, want.Facets), reflect.DeepEqual(got.Histogram, want.Histogram), reflect.DeepEqual(got.Rows, want.Rows)) + } + if got.Stats.FacetUpdates != want.Stats.FacetUpdates || got.Stats.HistogramUpdates != want.Stats.HistogramUpdates { + t.Fatalf("update counts differ: indexed=%+v traversal=%+v", got.Stats, want.Stats) + } + if got.Stats != wantIndex.Stats { + t.Fatalf("indexed statistics differ across access bounds: got=%+v want=%+v", got.Stats, wantIndex.Stats) + } + if c.Compression != CompressionNone && got.Stats.PayloadsDecompressed == 0 { + t.Fatal("fixture did not exercise compressed DATA") + } + after := r.AccessStats() + if after.ActiveWindows > 1 || after.RowArenaPeakBytes > after.RowArenaLimitBytes { + t.Fatalf("reader memory bound exceeded: %+v", after) + } + if after.SelectedAccessMode == ReaderAccessMmap && after.EvictionCount <= before.EvictionCount { + t.Fatalf("fixture did not force window eviction: %+v", after) + } + if c.Mode == ReaderAccessReadAt && after.TempCopyCount <= before.TempCopyCount { + t.Fatalf("fixture did not force temporary scratch reads: %+v", after) + } + return + } + for _, compact := range []bool{false, true} { + for _, compression := range []int{CompressionNone, CompressionZSTD, CompressionXZ, CompressionLZ4} { + path := explorerWindowFixture(t, compact, compression, 1024, 4) + for _, kind := range []string{"facet", "histogram"} { + for _, mode := range readerTestAccessModes() { + t.Run(fmt.Sprintf("compact=%v/compression=%d/%s/%s", compact, compression, kind, accessModeName(mode)), func(t *testing.T) { + runExplorerWindowChild(t, explorerWindowCase{Path: path, Kind: kind, Mode: mode, Compression: compression}) + }) + } + } + } + } +} + +func TestExplorerIndexedWindowValueShapes(t *testing.T) { + for _, compact := range []bool{false, true} { + entries := make([]explorerTestEntry, 1024) + for i := range entries { + value := "" + if i%3 != 0 { + value = "\x00\xff=" + strings.Repeat("binary-", 96) + } + if i < 100 || i > 800 { + value = "outside-" + strings.Repeat("range-", 96) + } + selected := "skip" + if i%64 == 0 { + selected = "keep" + } + entries[i] = explorerTestEntry{realtime: 1000 + uint64(i), payloads: [][]byte{[]byte("VALUE=" + value), []byte("SELECT=" + selected), []byte(fmt.Sprintf("MESSAGE=row-%04d", i))}} + } + path := createExplorerRawJournalWithOptions(t, entries, func(opts *Options) { opts.Compact = compact }) + for _, kind := range []string{"facet", "histogram"} { + for _, mode := range readerTestAccessModes() { + t.Run(fmt.Sprintf("compact=%v/%s/%s", compact, kind, accessModeName(mode)), func(t *testing.T) { + runExplorerWindowChild(t, explorerWindowCase{Path: path, Kind: kind, Mode: mode, TimeRange: true}) + }) + } + } + } +} + +func TestExplorerIndexedWindowSingletonValues(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, compression := range []int{CompressionNone, CompressionZSTD, CompressionXZ, CompressionLZ4} { + entries := make([]explorerTestEntry, 1024) + for i := range entries { + value := fmt.Sprintf("%04d\x00\xff=%s", i, strings.Repeat("singleton-", 64)) + if i == 64 { + value = "" + } + selected := "skip" + if i%64 == 0 { + selected = "keep" + } + entries[i] = explorerTestEntry{realtime: 1000 + uint64(i), payloads: [][]byte{[]byte("VALUE=" + value), []byte("SELECT=" + selected), []byte(fmt.Sprintf("MESSAGE=row-%04d", i))}} + } + path := createExplorerRawJournalWithOptions(t, entries, func(opts *Options) { + opts.Compact = compact + opts.Compression = compression + opts.CompressThresholdBytes = 32 + }) + for _, timeRange := range []bool{false, true} { + for _, mode := range readerTestAccessModes() { + t.Run(fmt.Sprintf("compact=%v/compression=%d/time=%v/%s", compact, compression, timeRange, accessModeName(mode)), func(t *testing.T) { + runExplorerWindowChild(t, explorerWindowCase{Path: path, Kind: "histogram", Mode: mode, Compression: compression, TimeRange: timeRange}) + }) + } + } + } + } +} + +func TestExplorerIndexedWindowSingletonBounds(t *testing.T) { + maxInt := uint64(^uint(0) >> 1) + upperStart := ^uint64(0) - maxInt + cases := []struct { + name string + realtime, start, end uint64 + buckets int + updates uint64 + }{ + {"maximum-int-index", maxInt, 0, 1, 1, 1}, + {"negative-int-index", maxInt + 1, 0, 1, 1, 0}, + {"zero-width", 1234, 0, 0, 1, 1}, + {"below-start", 1000, upperStart, upperStart + 1, 1, 1}, + {"maximum-quotient", ^uint64(0), upperStart, upperStart + 1, 1, 1}, + {"no-buckets", 1000, 0, 1, 0, 0}, + } + value := "\x00\xff=" + strings.Repeat("bounds-", 96) + for _, compact := range []bool{false, true} { + for _, c := range cases { + t.Run(fmt.Sprintf("compact=%v/%s", compact, c.name), func(t *testing.T) { + path := createExplorerRawJournalWithOptions(t, []explorerTestEntry{{realtime: c.realtime, payloads: [][]byte{[]byte("VALUE=" + value), []byte("PADDING=" + strings.Repeat("x", 8192))}}}, func(opts *Options) { opts.Compact = compact }) + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(ReaderAccessReadAt).WithWindowSize(64).WithMaxWindows(1)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + histogram := &ExplorerHistogram{Field: []byte("VALUE")} + for i := 0; i < c.buckets; i++ { + histogram.Buckets = append(histogram.Buckets, ExplorerHistogramBucket{StartRealtimeUsec: c.start, EndRealtimeUsec: c.end, Values: make(map[string]uint64)}) + } + query := explorerWindowQuery("histogram", false) + result := ExplorerResult{Histogram: histogram} + if err := r.indexedCountHistogram(query, explorerCandidateSet{all: true, count: 1}, &result); err != nil { + t.Fatal(err) + } + if result.Stats.DataRefsSeen != 1 || result.Stats.HistogramUpdates != c.updates { + t.Fatalf("unexpected statistics: %+v", result.Stats) + } + var total uint64 + for _, bucket := range histogram.Buckets { + for key, count := range bucket.Values { + if key != value { + t.Fatalf("singleton value changed: %q", key) + } + total += count + } + } + if total != c.updates { + t.Fatalf("bucket updates = %d, want %d", total, c.updates) + } + }) + } + } +} + +func TestExplorerIndexedWindowReadErrors(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, fault := range []string{"posting", "entry"} { + path := explorerWindowFixture(t, compact, CompressionNone, 1024, 4) + r := mustOpenReaderFile(t, path) + offset, found, err := r.findDataOffsetByPayload([]byte("VALUE=0001-" + strings.Repeat("value-", 96))) + _ = r.Close() + if err != nil || !found { + t.Fatalf("find fixture DATA: found=%v err=%v", found, err) + } + pointer := offset + 48 + if fault == "entry" { + pointer = offset + 40 + } + // Point at a valid DATA object where an ENTRY_ARRAY or ENTRY is required. + corruptHeaderUint64(t, path, int64(pointer), offset) + for _, kind := range []string{"facet", "histogram"} { + if fault == "entry" && kind != "histogram" { + continue + } + for _, mode := range readerTestAccessModes() { + t.Run(fmt.Sprintf("compact=%v/%s/%s/%s", compact, fault, kind, accessModeName(mode)), func(t *testing.T) { + runExplorerWindowChild(t, explorerWindowCase{Path: path, Kind: kind, Mode: mode, ExpectError: true}) + }) + } + } + } + } +} + +func TestExplorerIndexedWindowPartialFacetStats(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, prefix := range []bool{false, true} { + entries := []explorerTestEntry{ + {realtime: 1000, payloads: [][]byte{[]byte("VALUE=shared")}}, + {realtime: 2000, payloads: [][]byte{[]byte("VALUE=shared")}}, + } + if prefix { + // FIELD chains prepend new DATA, so append the successful control last. + entries = append(entries, explorerTestEntry{realtime: 3000, payloads: [][]byte{[]byte("VALUE=first")}}) + } + path := createExplorerRawJournalWithOptions(t, entries, func(opts *Options) { opts.Compact = compact }) + r := mustOpenReaderFile(t, path) + offset, found, err := r.findDataOffsetByPayload([]byte("VALUE=shared")) + if err == nil && found && prefix { + first, firstFound, firstErr := r.findDataOffsetByPayload([]byte("VALUE=first")) + head, headFound, headErr := r.findFieldHeadDataOffset([]byte("VALUE")) + if firstErr != nil || headErr != nil || !firstFound || !headFound || head != first { + _ = r.Close() + t.Fatalf("successful DATA must precede failed walk: first=%d head=%d errors=%v/%v", first, head, firstErr, headErr) + } + } + _ = r.Close() + if err != nil || !found { + t.Fatalf("find fixture DATA: found=%v err=%v", found, err) + } + corruptHeaderUint64(t, path, int64(offset+48), offset) + for _, all := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/prefix=%v/all=%v", compact, prefix, all), func(t *testing.T) { + r := mustOpenReaderFile(t, path) + defer r.Close() + selectedOffset := r.entryOffsets[0] + if prefix { + selectedOffset = r.entryOffsets[len(r.entryOffsets)-1] + } + candidates := explorerCandidateSet{all: all, count: 1, offsets: map[uint64]struct{}{selectedOffset: {}}} + if all { + candidates.count = uint64(len(r.entryOffsets)) + } + result := ExplorerResult{Facets: make(map[string]map[string]uint64)} + err := r.indexedCountFacetGroup(explorerWindowQuery("facet", false), facetPassGroup{facetIndices: []int{0}}, candidates, &result) + refs, updates, classified := uint64(1), uint64(0), uint64(0) + if prefix { + refs, updates, classified = 2, 1, 1 + } + if !errors.Is(err, errInvalidJournal) || result.Stats.DataRefsSeen != refs || result.Stats.FacetUpdates != updates || result.Stats.DataObjectsClassified != classified || result.Stats.DataPayloadsLoaded != classified { + t.Fatalf("partial posting error/counts: err=%v stats=%+v", err, result.Stats) + } + }) + } + } + } +} + +func TestExplorerIndexedWindowPartialHistogramStats(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, compression := range []int{CompressionNone, CompressionLZ4} { + payload := []byte("VALUE=" + strings.Repeat("shared-", 96)) + path := createExplorerRawJournalWithOptions(t, []explorerTestEntry{ + {realtime: 1000, payloads: [][]byte{payload}}, + }, func(opts *Options) { + opts.Compact = compact + opts.Compression = compression + opts.CompressThresholdBytes = 32 + }) + r := mustOpenReaderFile(t, path) + offset, found, err := r.findDataOffsetByPayload(payload) + if err != nil || !found { + _ = r.Close() + t.Fatalf("find fixture DATA: found=%v err=%v", found, err) + } + header, err := r.readDataHeaderAt(offset) + _ = r.Close() + if err != nil || header.nEntries != 1 || (compression == CompressionLZ4 && header.object.flag&objectCompressedLZ4 == 0) { + t.Fatalf("incorrect singleton fixture: header=%+v err=%v", header, err) + } + corruptHeaderUint64(t, path, int64(offset+40), offset) + for _, all := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/compression=%d/all=%v", compact, compression, all), func(t *testing.T) { + r := mustOpenReaderFile(t, path) + defer r.Close() + query := explorerWindowQuery("histogram", false) + histogram := newExplorerHistogram([]byte("VALUE"), query) + result := ExplorerResult{Histogram: &histogram} + candidates := explorerCandidateSet{all: all, count: 1, offsets: map[uint64]struct{}{offset: {}}} + err := r.indexedCountHistogram(query, candidates, &result) + if !errors.Is(err, errCorruptObject) || result.Stats.DataRefsSeen != 1 || result.Stats.HistogramUpdates != 0 || result.Stats.DataObjectsClassified != 0 || result.Stats.DataPayloadsLoaded != 0 || result.Stats.PayloadsDecompressed != 0 { + t.Fatalf("partial ENTRY error/counts: err=%v stats=%+v", err, result.Stats) + } + }) + } + } + } +} + +func TestExplorerIndexedWindowPayloadFieldMatch(t *testing.T) { + cases := []struct { + name, field, payload string + wrongObject bool + }{ + {name: "wrong-name", field: "VALUE", payload: "OTHER=value"}, + {name: "wrong-boundary", field: "VALUE", payload: "VALUE?value"}, + {name: "early-delimiter", field: "VALUE", payload: "VA=UE=value"}, + {name: "empty-parsed-name", field: "VALUE", payload: "=ALUE=value"}, + {name: "requested-delimiter", field: "VA=UE", payload: "VA=UE=value"}, + {name: "requested-delimiter-wrong-object", field: "VA=UE", payload: "VA=UE=value", wrongObject: true}, + } + for _, compact := range []bool{false, true} { + for _, c := range cases { + path := createExplorerRawJournalWithOptions(t, []explorerTestEntry{ + {realtime: 1000, payloads: [][]byte{[]byte("VALUE=value")}}, + }, func(opts *Options) { + opts.Compact = compact + opts.FieldHashTableBuckets = 1 + }) + r := mustOpenReaderFile(t, path) + dataOffset, found, err := r.findDataOffsetByPayload([]byte("VALUE=value")) + if err != nil || !found { + _ = r.Close() + t.Fatalf("find fixture DATA: found=%v err=%v", found, err) + } + payloadOffset := dataOffset + r.dataPayloadOffset() + fieldHash := r.hash([]byte(c.field)) + bucket, err := r.readSlice(r.header.fieldHashTableOffset, hashItemSize) + if err != nil { + _ = r.Close() + t.Fatal(err) + } + fieldOffset := parseHashItem(bucket).head + _ = r.Close() + if c.field != "VALUE" { + corruptHeaderUint64(t, path, int64(fieldOffset+16), fieldHash) + } + if c.wrongObject { + corruptHeaderUint64(t, path, int64(fieldOffset+32), fieldOffset) + } + file, err := os.OpenFile(path, os.O_RDWR, 0) + if err != nil { + t.Fatal(err) + } + for offset, data := range map[uint64][]byte{ + fieldOffset + fieldObjectHeaderSize: []byte(c.field), + payloadOffset: []byte(c.payload), + } { + if n, err := file.WriteAt(data, int64(offset)); err != nil || n != len(data) { + _ = file.Close() + t.Fatalf("patch fixture bytes: n=%d err=%v", n, err) + } + } + if err := file.Close(); err != nil { + t.Fatal(err) + } + for _, kind := range []string{"facet", "histogram"} { + t.Run(fmt.Sprintf("compact=%v/%s/%s", compact, c.name, kind), func(t *testing.T) { + r := mustOpenReaderFile(t, path) + defer r.Close() + field := []byte(c.field) + head, found, err := r.findFieldHeadDataOffset(field) + wantHead := dataOffset + if c.wrongObject { + wantHead = fieldOffset + } + if err != nil || !found || head != wantHead { + t.Fatalf("fixture FIELD was not found: head=%d found=%v err=%v", head, found, err) + } + query := explorerWindowQuery(kind, false) + query.Facets, query.Histogram = nil, nil + result := ExplorerResult{Facets: make(map[string]map[string]uint64)} + candidates := explorerCandidateSet{all: true, count: 1} + if kind == "facet" { + query = query.WithFacet(field) + err = r.indexedCountFacetGroup(query, facetPassGroup{facetIndices: []int{0}}, candidates, &result) + } else { + query = query.WithHistogram(field) + histogram := newExplorerHistogram(field, query) + result.Histogram = &histogram + err = r.indexedCountHistogram(query, candidates, &result) + } + if c.wrongObject { + if !errors.Is(err, errCorruptObject) || result.Stats.FacetUpdates+result.Stats.HistogramUpdates != 0 { + t.Fatalf("native FIELD chain error/counts: err=%v stats=%+v", err, result.Stats) + } + } else if err != nil || result.Stats.FacetUpdates+result.Stats.HistogramUpdates != 1 { + t.Fatalf("unset update: err=%v stats=%+v", err, result.Stats) + } + if result.Stats.DataRefsSeen != 0 || result.Stats.DataObjectsClassified != 0 || result.Stats.DataPayloadsLoaded != 0 || result.Stats.PayloadsDecompressed != 0 { + t.Fatalf("mismatched payload was aggregated: %+v", result.Stats) + } + }) + } + } + } +} + +func BenchmarkExplorerIndexedValueOwnership(b *testing.B) { + for _, cardinality := range []int{4, 1024} { + for _, kind := range []string{"facet", "histogram"} { + for _, sparse := range []bool{false, true} { + b.Run(fmt.Sprintf("cardinality=%d/%s/sparse=%s", cardinality, kind, strconv.FormatBool(sparse)), func(b *testing.B) { + path := explorerWindowFixture(b, false, CompressionNone, 2048, cardinality) + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + b.Fatal(err) + } + defer r.Close() + q := explorerWindowQuery(kind, sparse) + q.Limit = 0 + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + result, err := r.ExploreWithStrategy(q, ExplorerStrategyIndex) + if err != nil || result.Stats.FacetUpdates+result.Stats.HistogramUpdates == 0 { + b.Fatalf("incomplete query: %v %+v", err, result.Stats) + } + } + b.StopTimer() + }) + } + } + } +} + +func BenchmarkExplorerIndexedCompressedValueOwnership(b *testing.B) { + corpus := os.Getenv("JOURNAL_SDK_EXPLORER_COMPRESSED_CORPUS") + if corpus == "" { + b.Skip("set JOURNAL_SDK_EXPLORER_COMPRESSED_CORPUS to a prebuilt fixture directory") + } + for _, layout := range []string{"regular", "compact"} { + for _, codec := range []string{"zstd", "xz", "lz4"} { + for _, shape := range []string{"low", "high", "singleton"} { + for _, kind := range []string{"facet", "histogram"} { + b.Run(fmt.Sprintf("%s/%s/%s/%s", layout, codec, shape, kind), func(b *testing.B) { + path := filepath.Join(corpus, layout, codec, shape+".journal") + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + b.Fatal(err) + } + defer r.Close() + query := explorerWindowQuery(kind, shape != "low") + query.Limit = 0 + indexed, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if err != nil { + b.Fatal(err) + } + traversal, err := r.ExploreWithStrategy(query, ExplorerStrategyTraversal) + if err != nil { + b.Fatal(err) + } + if !reflect.DeepEqual(indexed.Facets, traversal.Facets) || !reflect.DeepEqual(indexed.Histogram, traversal.Histogram) || !reflect.DeepEqual(indexed.Rows, traversal.Rows) || !reflect.DeepEqual(indexed.ColumnFields, traversal.ColumnFields) { + b.Fatal("indexed/traversal results differ") + } + updates := uint64(2197) + if shape == "high" { + updates = 34 + } else if shape == "singleton" { + updates = 32 + } + if indexed.Stats.FacetUpdates+indexed.Stats.HistogramUpdates != updates || indexed.Stats.PayloadsDecompressed == 0 { + b.Fatalf("incorrect or uncompressed fixture: %+v", indexed.Stats) + } + if stats := r.AccessStats(); stats.EvictionCount != 0 || stats.TempCopyCount != 0 { + b.Fatalf("prebuilt performance control must fit its accessor window: %+v", stats) + } + encoded, err := json.Marshal(indexed.Stats) + if err != nil { + b.Fatal(err) + } + b.Logf("indexed-stats %s", encoded) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + result, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if err != nil || result.Stats.FacetUpdates+result.Stats.HistogramUpdates != updates { + b.Fatalf("incomplete query: %v %+v", err, result.Stats) + } + } + b.StopTimer() + }) + } + } + } + } +} diff --git a/go/journal/facade.go b/go/journal/facade.go index b7c0f049..79fb42b4 100644 --- a/go/journal/facade.go +++ b/go/journal/facade.go @@ -18,6 +18,7 @@ var ( ) type sdReader interface { + RecoveryStatuses() []FileRecoveryStatus io.Closer GetRealtimeUsec() (uint64, error) GetCursor() (string, error) @@ -30,8 +31,8 @@ type sdReader interface { EnumerateUniquePayload() ([]byte, bool, error) EnumerateFields() (map[string]struct{}, error) AddMatch(data []byte) - AddDisjunction() - AddConjunction() + AddDisjunction() error + AddConjunction() error FlushMatches() SeekHead() error SeekTail() error @@ -160,14 +161,12 @@ func SdJournalAddMatch(j *sdJournal, data []byte) error { func SdJournalAddDisjunction(j *sdJournal) error { j.resetIterators() - j.reader.AddDisjunction() - return nil + return j.reader.AddDisjunction() } func SdJournalAddConjunction(j *sdJournal) error { j.resetIterators() - j.reader.AddConjunction() - return nil + return j.reader.AddConjunction() } func SdJournalFlushMatches(j *sdJournal) error { @@ -286,11 +285,19 @@ func SdJournalGetMonotonicUsec(j *sdJournal) (uint64, UUID, error) { } func SdJournalGetCursor(j *sdJournal) (string, error) { - return j.reader.GetCursor() + cursor, err := j.reader.GetCursor() + if errors.Is(err, errEndOfEntries) { + return "", ErrNoEntry + } + return cursor, err } func SdJournalTestCursor(j *sdJournal, cursor string) (bool, error) { - return j.reader.TestCursor(cursor) + ok, err := j.reader.TestCursor(cursor) + if errors.Is(err, ErrEndOfEntries) { + return false, nil + } + return ok, err } func SdJournalSetOutputMode(j *sdJournal, mode string) { @@ -320,14 +327,14 @@ func (j *sdJournal) AddMatch(data []byte) { j.reader.AddMatch(data) } -func (j *sdJournal) AddDisjunction() { +func (j *sdJournal) AddDisjunction() error { j.resetIterators() - j.reader.AddDisjunction() + return j.reader.AddDisjunction() } -func (j *sdJournal) AddConjunction() { +func (j *sdJournal) AddConjunction() error { j.resetIterators() - j.reader.AddConjunction() + return j.reader.AddConjunction() } func (j *sdJournal) FlushMatches() { @@ -539,6 +546,9 @@ func SdJournalGetCursorWithRealtime(j *sdJournal, realtime uint64) (string, erro func SdJournalSeekCursor(j *sdJournal, cursor string) error { j.resetIterators() + if reader, ok := j.reader.(*Reader); ok { + return reader.SeekCursor(cursor) + } want, err := parseCursorLocation(cursor, true) if err != nil { return ErrInvalidCursor @@ -561,15 +571,19 @@ func SdJournalSeekCursor(j *sdJournal, cursor string) error { return err } if !ok { + // An unsuccessful seek has no committed row; keep forward exhaustion. + if directory, ok := j.reader.(*DirectoryReader); ok { + directory.clearDirectoryPosition() + } return nil } - entry, err := j.reader.GetEntry() + current, err := j.reader.GetCursor() if err != nil { return err } - got, err := parseCursorLocation(entry.Cursor, false) + got, err := parseCursorLocation(current, false) if err != nil { return err } diff --git a/go/journal/facade_test.go b/go/journal/facade_test.go index a2f5e5ab..aa465d06 100644 --- a/go/journal/facade_test.go +++ b/go/journal/facade_test.go @@ -3,6 +3,7 @@ package journal import ( "bytes" "errors" + "fmt" "path/filepath" "reflect" "testing" @@ -616,3 +617,133 @@ func containsPayload(payloads [][]byte, want []byte) bool { } return false } + +func TestSdJournalSeekCursorPastEnd(t *testing.T) { + for _, directory := range []bool{false, true} { + for _, filtered := range []bool{false, true} { + t.Run(fmt.Sprintf("directory=%v/filtered=%v", directory, filtered), func(t *testing.T) { + dir := t.TempDir() + first := filepath.Join(dir, "a.journal") + createMessageJournal(t, first, []messageRow{{"keep", 1000}, {"keep", 1001}}) + var j *sdJournal + var err error + if directory { + createMessageJournal(t, filepath.Join(dir, "b.journal"), []messageRow{{"keep", 2000}, {"keep", 2001}}) + j, err = SdJournalOpenDirectory(dir, 0) + } else { + j, err = SdJournalOpen(first, 0) + } + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { j.Close() }) + if filtered { + if err := SdJournalAddMatch(j, []byte("MESSAGE=keep")); err != nil { + t.Fatal(err) + } + } + if err := SdJournalSeekCursor(j, "t=ffffffffffffff"); err != nil { + t.Fatal(err) + } + for i := 0; i < 2; i++ { + if n, err := SdJournalNext(j); n != 0 || err != nil { + t.Fatalf("next=%d/%v", n, err) + } + if ok, err := j.reader.TestCursor("t=ffffffffffffff"); ok || !errors.Is(err, ErrEndOfEntries) { + t.Fatalf("direct EOF cursor=%v/%v", ok, err) + } + if cursor, err := SdJournalGetCursor(j); cursor != "" || !errors.Is(err, ErrNoEntry) { + t.Fatalf("facade EOF get cursor=%q/%v", cursor, err) + } + if ok, err := SdJournalTestCursor(j, "t=ffffffffffffff"); ok || err != nil { + t.Fatalf("facade EOF cursor=%v/%v", ok, err) + } + } + if n, err := SdJournalPrevious(j); n != 1 || err != nil { + t.Fatalf("previous=%d/%v", n, err) + } + want := uint64(1001) + if directory { + want = 2001 + } + if got, err := SdJournalGetRealtimeUsec(j); got != want || err != nil { + t.Fatalf("time=%d/%v want%d", got, err, want) + } + }) + } + } +} + +// Keep unrelated reader failures distinct from absence of a current position. +type cursorErrorReader struct { + sdReader + err error +} + +func (r cursorErrorReader) GetCursor() (string, error) { return "", r.err } + +func TestSdJournalGetCursorPreservesReadError(t *testing.T) { + want := errors.New("cursor read failed") + j := &sdJournal{reader: cursorErrorReader{err: want}} + if _, err := SdJournalGetCursor(j); !errors.Is(err, want) { + t.Fatalf("get cursor error=%v, want %v", err, want) + } +} + +func TestSdJournalFilteredSequenceCursorPastEnd(t *testing.T) { + for _, fileCount := range []int{0, 1, 2} { + directory := fileCount > 0 + t.Run(fmt.Sprintf("directory=%v/files=%d", directory, fileCount), func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "a.journal") + createMessageJournal(t, path, []messageRow{{"keep", 1000}, {"keep", 1001}}) + wantLast := uint64(1001) + if fileCount == 2 { + createMessageJournal(t, filepath.Join(dir, "b.journal"), []messageRow{{"keep", 900}, {"keep", 1100}}) + wantLast = 1100 + } + var j *sdJournal + var err error + if directory { + j, err = SdJournalOpenDirectory(dir, 0) + } else { + j, err = SdJournalOpenFiles([]string{path}, 0) + } + if err != nil { + t.Fatal(err) + } + defer j.Close() + if err := SdJournalAddMatch(j, []byte("MESSAGE=keep")); err != nil { + t.Fatal(err) + } + if n, err := SdJournalNext(j); n != 1 || err != nil { + t.Fatalf("first=%d/%v", n, err) + } + cursor, err := SdJournalGetCursor(j) + if err != nil { + t.Fatal(err) + } + location, err := parseCursorLocation(cursor, true) + if err != nil { + t.Fatal(err) + } + if err := SdJournalSeekCursor(j, fmt.Sprintf("s=%s;i=ffffffff", location.seqnumID)); err != nil { + t.Fatal(err) + } + if cursor, err := SdJournalGetCursor(j); cursor != "" || !errors.Is(err, ErrNoEntry) { + t.Fatalf("past-tail cursor=%q/%v", cursor, err) + } + for i := 0; i < 2; i++ { + if n, err := SdJournalNext(j); n != 0 || err != nil { + t.Fatalf("EOF=%d/%v", n, err) + } + } + if n, err := SdJournalPrevious(j); n != 1 || err != nil { + t.Fatalf("previous=%d/%v", n, err) + } + if time, err := SdJournalGetRealtimeUsec(j); time != wantLast || err != nil { + t.Fatalf("last=%d/%v", time, err) + } + }) + } +} diff --git a/go/journal/file_open_native_linux.go b/go/journal/file_open_native_linux.go new file mode 100644 index 00000000..8f08b7a0 --- /dev/null +++ b/go/journal/file_open_native_linux.go @@ -0,0 +1,28 @@ +//go:build linux + +package journal + +import ( + "os" + "syscall" +) + +func openNativeReaderFile(path string) (*os.File, error) { + for { + fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_CLOEXEC, 0) + if err == syscall.EINTR { + continue + } + if err != nil { + return nil, &os.PathError{Op: "open", Path: path, Err: err} + } + // Native journals use positioned access; a blocking descriptor avoids + // poller setup that cannot succeed for a regular file. + f := os.NewFile(uintptr(fd), path) + if f == nil { + _ = syscall.Close(fd) + return nil, &os.PathError{Op: "open", Path: path, Err: syscall.EINVAL} + } + return f, nil + } +} diff --git a/go/journal/file_open_native_linux_test.go b/go/journal/file_open_native_linux_test.go new file mode 100644 index 00000000..c47b9f0a --- /dev/null +++ b/go/journal/file_open_native_linux_test.go @@ -0,0 +1,149 @@ +//go:build linux + +package journal + +import ( + "bytes" + "errors" + "os" + "path/filepath" + "syscall" + "testing" +) + +func TestNativeReaderFileDescriptor(t *testing.T) { + path := filepath.Join(t.TempDir(), "native-descriptor") + want := []byte("positioned journal bytes") + if err := os.WriteFile(path, want, 0o600); err != nil { + t.Fatal(err) + } + f, err := openNativeReaderFile(path) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = f.Close() }) + flags, _, errno := syscall.Syscall(syscall.SYS_FCNTL, f.Fd(), syscall.F_GETFL, 0) + if errno != 0 { + t.Fatal(errno) + } + if flags&syscall.O_ACCMODE != syscall.O_RDONLY || flags&syscall.O_NONBLOCK != 0 { + t.Fatalf("descriptor flags = %#x; want read-only, blocking", flags) + } + flags, _, errno = syscall.Syscall(syscall.SYS_FCNTL, f.Fd(), syscall.F_GETFD, 0) + if errno != 0 || flags&syscall.FD_CLOEXEC == 0 { + t.Fatalf("descriptor inheritance flags = %#x, %v; want close-on-exec", flags, errno) + } + got := make([]byte, len(want)) + if n, err := f.ReadAt(got, 0); err != nil || n != len(got) || !bytes.Equal(got, want) { + t.Fatalf("ReadAt = %d, %v, %q; want %q", n, err, got, want) + } + if n, err := f.WriteAt([]byte("x"), 0); n != 0 || !errors.Is(err, syscall.EBADF) { + t.Fatalf("WriteAt = %d, %v; want read-only descriptor error", n, err) + } + if err := f.Close(); err != nil { + t.Fatal(err) + } + if _, err := f.ReadAt(got, 0); !errors.Is(err, os.ErrClosed) { + t.Fatalf("ReadAt after Close = %v; want closed file", err) + } +} + +func TestNativeReaderFilePaths(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "native-path") + if err := os.WriteFile(path, []byte("native bytes"), 0o600); err != nil { + t.Fatal(err) + } + link := filepath.Join(dir, "native-link") + if err := os.Symlink(path, link); err != nil { + t.Fatal(err) + } + bytePath := filepath.Join(dir, "native-\xff") + if err := os.WriteFile(bytePath, nil, 0o600); err != nil { + t.Fatal(err) + } + for name, input := range map[string]string{"regular": path, "symlink": link, "directory": dir, "byte-name": bytePath} { + t.Run(name, func(t *testing.T) { + f, err := openNativeReaderFile(input) + if err != nil { + t.Fatal(err) + } + defer func() { _ = f.Close() }() + got, err := f.Stat() + if err != nil { + t.Fatal(err) + } + want, err := os.Stat(input) + if err != nil { + t.Fatal(err) + } + if !os.SameFile(got, want) || got.Mode() != want.Mode() || f.Name() != input { + t.Fatalf("descriptor identity/name differs from requested path") + } + }) + } +} + +func TestNativeReaderFileErrors(t *testing.T) { + dir := t.TempDir() + file := filepath.Join(dir, "not-directory") + if err := os.WriteFile(file, nil, 0o600); err != nil { + t.Fatal(err) + } + broken := filepath.Join(dir, "broken-link") + if err := os.Symlink(filepath.Join(dir, "missing"), broken); err != nil { + t.Fatal(err) + } + for _, test := range []struct { + name, path string + err error + }{ + {"missing", filepath.Join(dir, "missing"), syscall.ENOENT}, + {"empty", "", syscall.ENOENT}, + {"not-directory", filepath.Join(file, "child"), syscall.ENOTDIR}, + {"nul", filepath.Join(dir, "invalid\x00name"), syscall.EINVAL}, + {"broken-link", broken, syscall.ENOENT}, + } { + t.Run(test.name, func(t *testing.T) { + f, err := openNativeReaderFile(test.path) + if f != nil { + _ = f.Close() + t.Fatal("failed open returned a file") + } + var pathErr *os.PathError + if !errors.As(err, &pathErr) || pathErr.Op != "open" || pathErr.Path != test.path || !errors.Is(err, test.err) { + t.Fatalf("open error = %#v; want typed path error matching %v", err, test.err) + } + }) + } +} + +func TestNativeReaderFileLargeOffset(t *testing.T) { + path := filepath.Join(t.TempDir(), "native-sparse") + const offset int64 = 1<<31 + 4096 + want := []byte("large-offset bytes") + writer, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + if _, err := writer.WriteAt(want, offset); err != nil { + _ = writer.Close() + t.Fatal(err) + } + if err := writer.Close(); err != nil { + t.Fatal(err) + } + f, err := openNativeReaderFile(path) + if err != nil { + t.Fatal(err) + } + defer func() { _ = f.Close() }() + st, err := f.Stat() + if err != nil || st.Size() != offset+int64(len(want)) { + t.Fatalf("large file Stat = %v, %v", st, err) + } + got := make([]byte, len(want)) + if n, err := f.ReadAt(got, offset); err != nil || n != len(got) || !bytes.Equal(got, want) { + t.Fatalf("large-offset ReadAt = %d, %v, %q", n, err, got) + } +} diff --git a/go/journal/file_open_native_other.go b/go/journal/file_open_native_other.go new file mode 100644 index 00000000..ec346549 --- /dev/null +++ b/go/journal/file_open_native_other.go @@ -0,0 +1,9 @@ +//go:build !linux + +package journal + +import "os" + +func openNativeReaderFile(path string) (*os.File, error) { + return openReaderFile(path) +} diff --git a/go/journal/indexed_snapshot.go b/go/journal/indexed_snapshot.go index 33cfe66c..74282d76 100644 --- a/go/journal/indexed_snapshot.go +++ b/go/journal/indexed_snapshot.go @@ -75,7 +75,7 @@ func OpenIndexedSnapshot(ctx context.Context, path string, opts IndexedSnapshotO if ro == (ReaderOptions{}) { ro = DefaultReaderOptions() } - r, err := openFileWithOptions(path, ro.WithSnapshot(true), false) + r, err := openFileWithPolicy(path, ro.WithSnapshot(true), false, false) if err != nil { return nil, err } diff --git a/go/journal/reader.go b/go/journal/reader.go index 23125db7..623c2d6b 100644 --- a/go/journal/reader.go +++ b/go/journal/reader.go @@ -165,21 +165,19 @@ func (o ReaderOptions) normalized() ReaderOptions { type Reader struct { file *os.File header journalHeader - path string - cleanupPath string options ReaderOptions accessor readerAccessor fileSize uint64 + arenaEnd uint64 + entryExtent uint64 + + entrySlotHeaderBuffer [objectHeaderSize]byte // Cached from immutable per-file layout flags; live refresh updates mutable // header counters/tails, but compact vs regular layout cannot change. - entryItemSizeBytes uint64 - offsetArrayItemSizeBytes uint64 - dataPayloadOffsetBytes uint64 - - cursor uint64 - position int - direction Direction + entryItemSizeBytes uint8 + offsetArrayItemSizeBytes uint8 + dataPayloadOffsetBytes uint8 entryOffsets []uint64 entryIndex int @@ -188,17 +186,23 @@ type Reader struct { currentHeader entryHeader currentHeaderOffset uint64 currentHeaderValid bool + entryDataActive bool + allowRecovery bool + staged bool filter *filterBuilder entryDataOffsets []uint64 entryDataOffsetsEntry uint64 entryDataIndex int - entryDataActive bool uniqueField string uniqueHeadOffset uint64 uniqueCurrentOffset uint64 + uniqueChain chainGuard + + recovery *readerRecoveryState + directoryOrderDirty *bool } type readerRefreshSnapshot struct { @@ -207,6 +211,7 @@ type readerRefreshSnapshot struct { index int fileSize uint64 visible readerAccessorVisibleSnapshot + recovery *readerRecoveryState } func OpenFile(path string) (*Reader, error) { @@ -214,12 +219,16 @@ func OpenFile(path string) (*Reader, error) { } func OpenFileWithOptions(path string, opts ReaderOptions) (*Reader, error) { - return openFileWithOptions(path, opts, true) + return openFileWithPolicy(path, opts, true, true) } // openFileWithOptions opens a journal file. loadEntries=false is only for // header and FIELD/DATA-index operations that never traverse ENTRY arrays. func openFileWithOptions(path string, opts ReaderOptions, loadEntries bool) (*Reader, error) { + return openFileWithPolicy(path, opts, loadEntries, true) +} + +func openFileWithPolicy(path string, opts ReaderOptions, loadEntries, allowRecovery bool) (*Reader, error) { opts = opts.normalized() f, cleanupPath, err := openJournalFile(path) if err != nil { @@ -246,14 +255,15 @@ func openFileWithOptions(path string, opts ReaderOptions, loadEntries bool) (*Re return nil, errUnsupportedJournal } + fileSize := accessor.size() r := &Reader{ - file: f, - header: header, - path: path, - cleanupPath: cleanupPath, - options: opts, - accessor: accessor, - fileSize: accessor.size(), + file: f, + header: header, + staged: cleanupPath != "", + options: opts, + accessor: accessor, + fileSize: fileSize, + allowRecovery: allowRecovery, } r.configureLayout() @@ -268,6 +278,11 @@ func openFileWithOptions(path string, opts ReaderOptions, loadEntries bool) (*Re } func (r *Reader) configureLayout() { + r.arenaEnd = r.header.headerSize + r.header.arenaSize + if r.arenaEnd < r.header.headerSize { + r.arenaEnd = 0 + } + r.entryExtent = minUint64(r.arenaEnd, r.fileSize) if r.header.isCompact() { r.entryItemSizeBytes = compactEntryItemSize r.offsetArrayItemSizeBytes = compactOffsetArrayItemSize @@ -285,13 +300,17 @@ func (r *Reader) Close() error { accessErr = r.accessor.close() r.accessor = nil } - closeErr := closeJournalFile(r.file, r.cleanupPath) + var cleanupPath string + if r.staged { + cleanupPath = r.file.Name() + } + closeErr := closeJournalFile(r.file, cleanupPath) return errors.Join(accessErr, closeErr) } func openJournalFile(path string) (*os.File, string, error) { if !strings.HasSuffix(path, ".zst") { - f, err := openReaderFile(path) + f, err := openNativeReaderFile(path) return f, "", err } @@ -518,10 +537,13 @@ func (r *Reader) clearCurrentEntryState() { } func (r *Reader) currentEntryOffset() (uint64, error) { - if r.entryIndex < 0 || r.entryIndex >= len(r.entryOffsets) { - return 0, errEndOfEntries + if uint(r.entryIndex) < uint(len(r.entryOffsets)) { + return r.entryOffsets[r.entryIndex], nil } - return r.entryOffsets[r.entryIndex], nil + if r.entryIndex == -2 && r.filter != nil && r.filter.overrideOffset != 0 { + return r.filter.overrideOffset, nil + } + return 0, errEndOfEntries } func (r *Reader) readCurrentHeader() (journalHeader, uint64, error) { @@ -533,11 +555,12 @@ func (r *Reader) readCurrentHeader() (journalHeader, uint64, error) { } func (r *Reader) Refresh() (bool, error) { + r.clearRecoveryEOF() return r.refreshEntryOffsets() } func (r *Reader) refreshEntryOffsets() (bool, error) { - if r.cleanupPath != "" || r.options.Bounds == ReaderBoundsSnapshot { + if r.staged || r.options.Bounds == ReaderBoundsSnapshot { return false, nil } snapshot := r.refreshSnapshot() @@ -556,15 +579,27 @@ func (r *Reader) refreshEntryOffsets() (bool, error) { r.applyRefreshState(header, size) if err := r.loadEntryArray(); err != nil { r.restoreRefreshSnapshot(snapshot) - return false, nil + return false, err + } + if r.directoryOrderDirty != nil && (snapshot.recovery.Incomplete() || r.recovery.Incomplete()) && + (!sameRecoveryState(snapshot.recovery, r.recovery)) { + *r.directoryOrderDirty = true } r.entryIndex = snapshot.index + if snapshot.recovery.Incomplete() && len(r.entryOffsets) > len(snapshot.offsets) && snapshot.index >= len(snapshot.offsets) { + r.entryIndex = len(snapshot.offsets) - 1 + } r.clampEntryIndex() - return true, nil + if r.filter != nil && + (!sameRecoveryState(snapshot.recovery, r.recovery)) { + r.filter.stale = true + } + return header != snapshot.header || size != snapshot.fileSize || !sameRecoveryState(r.recovery, snapshot.recovery) || len(r.entryOffsets) != len(snapshot.offsets), nil } func (r *Reader) sameEntryArrayState(header journalHeader, size uint64) bool { - return size == r.fileSize && + return !r.recovery.Incomplete() && size == r.fileSize && + (r.entryOffsets == nil || uint64(len(r.entryOffsets)) == header.nEntries) && header.nEntries == r.header.nEntries && header.tailEntryArrayOffset == r.header.tailEntryArrayOffset && header.tailEntryArrayNEntries == r.header.tailEntryArrayNEntries @@ -583,44 +618,78 @@ func (r *Reader) refreshSnapshot() readerRefreshSnapshot { index: r.entryIndex, fileSize: r.fileSize, visible: r.accessor.snapshotVisibleBounds(), + recovery: r.recovery, } } func (r *Reader) applyRefreshState(header journalHeader, size uint64) { r.header = header - r.configureLayout() r.fileSize = size + r.configureLayout() } func (r *Reader) restoreRefreshSnapshot(snapshot readerRefreshSnapshot) { r.header = snapshot.header - r.configureLayout() r.entryOffsets = snapshot.offsets r.entryIndex = snapshot.index + r.recovery = snapshot.recovery + if r.accessor != nil { + physical := r.accessor.snapshotVisibleBounds().fileSize + if physical < snapshot.fileSize { + r.currentHeaderValid = false + } + snapshot.fileSize = minUint64(snapshot.fileSize, physical) + snapshot.visible.fileSize = minUint64(snapshot.visible.fileSize, physical) + } r.fileSize = snapshot.fileSize + r.configureLayout() if r.accessor != nil { r.accessor.restoreVisibleBounds(snapshot.visible) } } func (r *Reader) loadEntryArray() error { + previousRecovery := r.recovery + r.recovery = nil if r.header.entryArrayOffset == 0 { - r.entryOffsets = nil + r.entryOffsets = []uint64{} + if r.header.nEntries != 0 { + if r.allowRecovery { + return r.retainRecoveredPrefix(nil, RecoveryMissingEntryArray, 0, previousRecovery) + } + return errCorruptObject + } return nil } - var offsets []uint64 + offsets := []uint64{} offset := r.header.entryArrayOffset + highest := offset remaining := r.header.nEntries for offset != 0 && remaining > 0 { header, chunkOffsets, toRead, err := r.readEntryArrayChunk(offset, remaining) if err != nil { - return err + return r.recoverEntryArrayFailure(offset, remaining, offsets, chunkOffsets, err, previousRecovery) } offsets = append(offsets, chunkOffsets...) remaining -= toRead + if remaining > 0 && header.nextArrayOffset != 0 { + if header.nextArrayOffset <= highest { + if err := rejectArrayRevisit(r, r.header.entryArrayOffset, offset, header.nextArrayOffset, errCorruptObject); err != nil { + return err + } + } else { + highest = header.nextArrayOffset + } + } offset = header.nextArrayOffset } + if remaining != 0 { + if r.allowRecovery { + return r.retainRecoveredPrefix(offsets, RecoveryMissingEntryArray, offset, previousRecovery) + } + return errCorruptObject + } r.entryOffsets = offsets r.entryIndex = -1 @@ -634,13 +703,23 @@ func (r *Reader) readEntryArrayChunk(offset uint64, remaining uint64) (offsetArr } toRead := minUint64(remaining, capacity) itemSize := r.offsetArrayItemSize() - buf := make([]byte, toRead*itemSize) - if err := r.readAt(buf, offset+offsetArrayObjectHeaderSize); err != nil { - return offsetArrayHeader{}, nil, 0, err + start := offset + offsetArrayObjectHeaderSize + bodySize := header.object.size - offsetArrayObjectHeaderSize + if start > r.fileSize || bodySize > r.fileSize-start { + return header, nil, 0, invalidEntryBodyBounds(start, bodySize, r.fileSize) + } + // Header capacity bounds this product by the validated object size. + byteCount := toRead * itemSize + if byteCount > uint64(int(^uint(0)>>1)) { + return header, nil, 0, errCorruptObject + } + buf := make([]byte, byteCount) + if err := r.readAt(buf, start); err != nil { + return header, nil, 0, err } offsets, err := r.validEntryOffsetsFromArray(buf, itemSize) if err != nil { - return offsetArrayHeader{}, nil, 0, err + return header, offsets, toRead, err } return header, offsets, toRead, nil } @@ -657,11 +736,11 @@ func (r *Reader) validEntryOffsetsFromArray(buf []byte, itemSize uint64) ([]uint for pos := 0; pos < len(buf); pos += int(itemSize) { off := entryOffsetArrayItem(buf[pos:], itemSize) if off == 0 { - continue + return offsets, readerTailError{0} } valid, err := r.validEntryObjectOffset(off) if err != nil { - return nil, err + return offsets, err } if valid { offsets = append(offsets, off) @@ -677,45 +756,80 @@ func entryOffsetArrayItem(src []byte, itemSize uint64) uint64 { return binary.LittleEndian.Uint64(src[:regularOffsetArrayItemSize]) } +func (r *Reader) readEntrySlotHeader(offset uint64) ([]byte, error) { + windowSize := r.options.WindowSize + if windowSize == 0 { + windowSize = defaultReaderWindowSize + } + if windowSize >= objectHeaderSize && windowSize&(windowSize-1) == 0 && + offset&(windowSize-1) <= windowSize-objectHeaderSize { + return r.readSlice(offset, objectHeaderSize) + } + // Crossing temp slices issue a file read even when both chunks are cached. + header := r.entrySlotHeaderBuffer[:] + if err := r.readAt(header, offset); err != nil { + return nil, err + } + return header, nil +} + func (r *Reader) validEntryObjectOffset(offset uint64) (bool, error) { - headerBuf := make([]byte, objectHeaderSize) - if err := r.readAt(headerBuf, offset); err != nil { - return false, err + if offset < r.header.headerSize || offset&7 != 0 || offset > r.arenaEnd || r.arenaEnd-offset < objectHeaderSize { + return false, errCorruptObject } - objHeader, err := parseObjectHeader(headerBuf) + headerBuf, err := r.readEntrySlotHeader(offset) if err != nil { return false, err } - if objHeader.typ == 0 && objHeader.size == 0 { - return false, nil + if len(headerBuf) < objectHeaderSize { + return false, errInvalidJournal } - if objHeader.typ != objectTypeEntry { + typ, size := headerBuf[0], binary.LittleEndian.Uint64(headerBuf[8:16]) + if typ != objectTypeEntry { + return false, invalidEntrySlotHeader(headerBuf, offset) + } + if size < entryObjectHeaderSize || size&(r.entryItemSize()-1) != 0 { return false, errCorruptObject } + if offset > r.entryExtent || size > r.entryExtent-offset { + return false, r.invalidEntryExtent(offset, size) + } return true, nil } func (r *Reader) readOffsetArrayHeader(offset uint64) (offsetArrayHeader, uint64, error) { - buf := make([]byte, offsetArrayObjectHeaderSize) - if err := r.readAt(buf, offset); err != nil { + if offset < r.header.headerSize || offset&7 != 0 || offset > r.arenaEnd || offsetArrayObjectHeaderSize > r.arenaEnd-offset { + return offsetArrayHeader{}, 0, errCorruptObject + } + buf, err := r.readArraySlotHeader(offset) + if err != nil { return offsetArrayHeader{}, 0, err } header, err := parseOffsetArrayHeader(buf) if err != nil { return offsetArrayHeader{}, 0, err } - if header.object.typ != objectTypeEntryArray || header.object.size < offsetArrayObjectHeaderSize { - return offsetArrayHeader{}, 0, errInvalidJournal + if header.object.typ != objectTypeEntryArray || header.object.size <= offsetArrayObjectHeaderSize { + return offsetArrayHeader{}, 0, invalidArraySlotHeader(buf, offset) } itemSize := r.offsetArrayItemSize() - if (header.object.size-offsetArrayObjectHeaderSize)%itemSize != 0 { + bodySize := header.object.size - offsetArrayObjectHeaderSize + if bodySize&(itemSize-1) != 0 { return offsetArrayHeader{}, 0, errInvalidJournal } - capacity := (header.object.size - offsetArrayObjectHeaderSize) / itemSize + if header.object.size > r.arenaEnd-offset { + return offsetArrayHeader{}, 0, errCorruptObject + } + capacity := bodySize >> ((itemSize >> 2) + 1) return header, capacity, nil } func (r *Reader) Next() error { + if r.entryIndex == -2 { + if err := r.normalizeFilteredPosition(DirectionForward); err != nil { + return err + } + } r.clearCurrentEntryState() if r.realtimeSeek != nil { return r.nextFromRealtimeSeek() @@ -724,11 +838,11 @@ func (r *Reader) Next() error { r.entryIndex = -1 } r.entryIndex++ - r.direction = DirectionForward if r.entryIndex >= len(r.entryOffsets) { return r.nextAfterTailRefresh() } + r.clearRecoveryEOF() return nil } @@ -739,7 +853,6 @@ func (r *Reader) nextFromRealtimeSeek() error { if err != nil { return err } - r.direction = DirectionForward idx, err = r.refreshRealtimeSeekIndex(usec, idx) if err != nil { return err @@ -764,24 +877,38 @@ func (r *Reader) refreshRealtimeSeekIndex(usec uint64, idx int) (int, error) { } func (r *Reader) nextAfterTailRefresh() error { + unchanged, err := r.unchangedRecoveryEOF(DirectionForward) + if err != nil { + return err + } + if unchanged { + r.entryIndex = len(r.entryOffsets) + return errEndOfEntries + } oldLen := len(r.entryOffsets) changed, err := r.refreshEntryOffsets() if err != nil { return err } if changed && oldLen < len(r.entryOffsets) { - if r.entryIndex > oldLen { - r.entryIndex = oldLen - } + r.entryIndex = oldLen if r.entryIndex < len(r.entryOffsets) { + r.clearRecoveryEOF() return nil } } r.entryIndex = len(r.entryOffsets) + r.rememberRecoveryEOF(DirectionForward) return errEndOfEntries } func (r *Reader) Previous() error { + r.clearRecoveryEOF() + if r.entryIndex == -2 { + if err := r.normalizeFilteredPosition(DirectionBackward); err != nil { + return err + } + } r.clearCurrentEntryState() if r.realtimeSeek != nil { idx, err := r.lastRealtimeIndexAtOrBefore(*r.realtimeSeek) @@ -789,7 +916,6 @@ func (r *Reader) Previous() error { if err != nil { return err } - r.direction = DirectionBackward if idx < 0 { r.entryIndex = -1 return errStartOfEntries @@ -800,7 +926,6 @@ func (r *Reader) Previous() error { if r.entryIndex > len(r.entryOffsets) { r.entryIndex = len(r.entryOffsets) } - r.direction = DirectionBackward r.entryIndex-- if r.entryIndex < 0 { @@ -811,23 +936,25 @@ func (r *Reader) Previous() error { } func (r *Reader) SeekHead() error { + r.clearRecoveryEOF() r.clearCurrentEntryState() r.entryIndex = -1 - r.direction = DirectionForward r.realtimeSeek = nil return nil } func (r *Reader) SeekTail() error { + r.clearRecoveryEOF() r.clearCurrentEntryState() r.entryIndex = len(r.entryOffsets) - r.direction = DirectionBackward r.realtimeSeek = nil return nil } func (r *Reader) SeekRealtimeUsec(usec uint64) error { + r.clearRecoveryEOF() r.clearCurrentEntryState() + r.entryIndex = -1 value := usec r.realtimeSeek = &value return nil @@ -914,10 +1041,10 @@ func (r *Reader) readEntryDataOffsetsAt(offset uint64, dst []uint64) (entryHeade } func (r *Reader) currentEntryDataOffsets() ([]uint64, error) { - if r.entryIndex < 0 || r.entryIndex >= len(r.entryOffsets) { - return nil, errEndOfEntries + offset, err := r.currentEntryOffset() + if err != nil { + return nil, err } - offset := r.entryOffsets[r.entryIndex] if r.entryDataOffsetsEntry == offset && r.entryDataOffsets != nil { return r.entryDataOffsets, nil } @@ -951,15 +1078,15 @@ func (r *Reader) hash(payload []byte) uint64 { } func (r *Reader) entryItemSize() uint64 { - return r.entryItemSizeBytes + return uint64(r.entryItemSizeBytes) } func (r *Reader) offsetArrayItemSize() uint64 { - return r.offsetArrayItemSizeBytes + return uint64(r.offsetArrayItemSizeBytes) } func (r *Reader) dataPayloadOffset() uint64 { - return r.dataPayloadOffsetBytes + return uint64(r.dataPayloadOffsetBytes) } func parseEntryHeader(src []byte) (entryHeader, error) { @@ -984,28 +1111,34 @@ func parseEntryHeader(src []byte) (entryHeader, error) { } func (r *Reader) AddMatch(data []byte) { + r.clearRecoveryEOF() if r.filter == nil { r.filter = &filterBuilder{} } r.filter.addMatch(data) } -func (r *Reader) AddDisjunction() { +func (r *Reader) AddDisjunction() error { + r.clearRecoveryEOF() if r.filter == nil { r.filter = &filterBuilder{} } - r.filter.addDisjunction() + r.clearCurrentEntryState() + return r.filter.addDisjunction(r) } -func (r *Reader) AddConjunction() { +func (r *Reader) AddConjunction() error { + r.clearRecoveryEOF() if r.filter == nil { r.filter = &filterBuilder{} } - r.filter.addConjunction() + r.clearCurrentEntryState() + return r.filter.addConjunction(r) } func (r *Reader) FlushMatches() { r.filter = nil + _ = r.SeekHead() } func (r *Reader) GetRealtimeUsec() (uint64, error) { @@ -1052,12 +1185,13 @@ func (r *Reader) SeekCursor(cursor string) error { } for { - if err := r.Next(); err != nil { - if errors.Is(err, errEndOfEntries) { - return nil - } + ok, err := r.Step() + if err != nil { return err } + if !ok { + return r.SeekTail() + } current, err := r.GetCursor() if err != nil { return err @@ -1092,49 +1226,95 @@ func (r *Reader) TestCursor(cursor string) (bool, error) { } func (r *Reader) Step() (bool, error) { - for { - if err := r.Next(); err != nil { - if errors.Is(err, errEndOfEntries) { - return false, nil - } - return false, err - } + if r.filter != nil { + return r.stepFiltered(DirectionForward) + } + err := r.Next() + if err == nil { + return true, nil + } + if errors.Is(err, errEndOfEntries) { + return false, nil + } + return err == nil, err +} - if r.filter == nil { - return true, nil - } +func (r *Reader) StepBack() (bool, error) { + if r.filter != nil { + return r.stepFiltered(DirectionBackward) + } + err := r.Previous() + if err == nil { + return true, nil + } + if errors.Is(err, errStartOfEntries) { + return false, nil + } + return err == nil, err +} - entry, err := r.GetEntry() - if err != nil { - return false, err +//go:noinline +func (r *Reader) recoverEntryArrayFailure(offset, remaining uint64, offsets, chunkOffsets []uint64, err error, previousRecovery *readerRecoveryState) error { + lost, ok := readerTailFailure(err) + if !ok || !r.allowRecovery { + return err + } + reason := RecoveryMissingEntry + if lost == offset || lost == offset+offsetArrayObjectHeaderSize { + reason = RecoveryMissingEntryArray + if lost != offset { + chunkOffsets, err = r.recoverPartialArray(offset, remaining) + if err != nil { + if _, ok := readerTailFailure(err); !ok { + return err + } + } } + } + return r.retainRecoveredPrefix(append(offsets, chunkOffsets...), reason, lost, previousRecovery) +} - if r.filter.matches(entry) { - return true, nil - } +//go:noinline +func (r *Reader) invalidEntryExtent(offset, size uint64) error { + if size > r.arenaEnd-offset { + return errCorruptObject } + return readerBoundsError{offset, size, r.fileSize} } -func (r *Reader) StepBack() (bool, error) { - for { - if err := r.Previous(); err != nil { - if errors.Is(err, errStartOfEntries) { - return false, nil - } - return false, err - } +//go:noinline +func invalidEntrySlotHeader(header []byte, offset uint64) error { + if binary.LittleEndian.Uint64(header[:8]) == 0 && binary.LittleEndian.Uint64(header[8:16]) == 0 { + return readerTailError{offset} + } + return errCorruptObject +} - if r.filter == nil { - return true, nil - } +//go:noinline +func invalidEntryBodyBounds(offset, size, fileSize uint64) error { + return readerBoundsError{offset, size, fileSize} +} - entry, err := r.GetEntry() - if err != nil { - return false, err - } +func (r *Reader) readArraySlotHeader(offset uint64) ([]byte, error) { + windowSize := r.options.WindowSize + if windowSize == 0 { + windowSize = defaultReaderWindowSize + } + if windowSize >= offsetArrayObjectHeaderSize && windowSize&(windowSize-1) == 0 && + offset&(windowSize-1) <= windowSize-offsetArrayObjectHeaderSize { + return r.readSlice(offset, offsetArrayObjectHeaderSize) + } + buf := make([]byte, offsetArrayObjectHeaderSize) + if err := r.readAt(buf, offset); err != nil { + return nil, err + } + return buf, nil +} - if r.filter.matches(entry) { - return true, nil - } +//go:noinline +func invalidArraySlotHeader(header []byte, offset uint64) error { + if binary.LittleEndian.Uint64(header[:8])|binary.LittleEndian.Uint64(header[8:16])|binary.LittleEndian.Uint64(header[16:24]) == 0 { + return readerTailError{offset} } + return errInvalidJournal } diff --git a/go/journal/reader_access.go b/go/journal/reader_access.go index 5995e811..83be218e 100644 --- a/go/journal/reader_access.go +++ b/go/journal/reader_access.go @@ -52,11 +52,13 @@ type readerAccessorVisibleSnapshot struct { type readerAccessor interface { selectedMode() ReaderAccessMode size() uint64 + initialSize() uint64 readAt(dst []byte, offset uint64) error tempSlice(offset, size uint64) ([]byte, error) rowSlice(offset, size uint64) ([]byte, error) rowCopy(src []byte) ([]byte, error) clearRow() error + invalidateCopiedWindows() snapshotVisibleBounds() readerAccessorVisibleSnapshot restoreVisibleBounds(readerAccessorVisibleSnapshot) refreshVisibleBounds() (journalHeader, bool, uint64, error) @@ -96,11 +98,13 @@ type rollingReaderAccessor struct { backend readerWindowBackend - requestedMode ReaderAccessMode - selectedModeID ReaderAccessMode - fallbackReason string + requestedMode uint8 + selectedModeID uint8 + powerOfTwoWindow bool + fallbackReason string - fileSize uint64 + fileSize uint64 + initialFileSize uint64 windowSize uint64 maxWindows int @@ -154,7 +158,7 @@ func newReaderAccessor(file *os.File, opts ReaderOptions) (readerAccessor, Reade if err != nil { return nil, ReaderAccessStats{}, err } - return accessor, accessor.stats(), nil + return accessor, accessor.statsValue, nil } mmapBackend, err := newMmapReaderBackend(file) @@ -162,7 +166,7 @@ func newReaderAccessor(file *os.File, opts ReaderOptions) (readerAccessor, Reade accessor, mmapErr := newRollingReaderAccessor(file, opts, mmapBackend, fileSize, "") if mmapErr == nil { if probeErr := accessor.probeInitialWindow(); probeErr == nil { - return accessor, accessor.stats(), nil + return accessor, accessor.statsValue, nil } else { _ = accessor.close() err = probeErr @@ -185,7 +189,7 @@ func newReaderAccessor(file *os.File, opts ReaderOptions) (readerAccessor, Reade if readAtErr != nil { return nil, ReaderAccessStats{}, readAtErr } - return accessor, accessor.stats(), nil + return accessor, accessor.statsValue, nil } func newRollingReaderAccessor(file *os.File, opts ReaderOptions, backend readerWindowBackend, fileSize uint64, fallbackReason string) (*rollingReaderAccessor, error) { @@ -201,10 +205,12 @@ func newRollingReaderAccessor(file *os.File, opts ReaderOptions, backend readerW a := &rollingReaderAccessor{ file: file, backend: backend, - requestedMode: opts.AccessMode, - selectedModeID: backend.mode(), + requestedMode: uint8(opts.AccessMode), + selectedModeID: uint8(backend.mode()), + powerOfTwoWindow: opts.WindowSize&(opts.WindowSize-1) == 0, fallbackReason: fallbackReason, fileSize: fileSize, + initialFileSize: fileSize, windowSize: opts.WindowSize, maxWindows: opts.MaxWindows, maxRowArenaBytes: opts.MaxRowArenaBytes, @@ -233,7 +239,7 @@ func (a *rollingReaderAccessor) probeInitialWindow() error { } func (a *rollingReaderAccessor) selectedMode() ReaderAccessMode { - return a.selectedModeID + return ReaderAccessMode(a.selectedModeID) } func (a *rollingReaderAccessor) size() uint64 { @@ -274,8 +280,8 @@ func (a *rollingReaderAccessor) tempSlice(offset, size uint64) ([]byte, error) { if size == 0 { return emptySlice(), nil } - if err := a.checkBounds(offset, size); err != nil { - return nil, err + if !a.containsRead(offset, size) { + return nil, a.invalidReadBounds(offset, size) } if a.rangeFitsWindow(offset, size) { window, err := a.getWindow(offset, size, false) @@ -294,8 +300,8 @@ func (a *rollingReaderAccessor) rowSlice(offset, size uint64) ([]byte, error) { if size == 0 { return emptySlice(), nil } - if err := a.checkBounds(offset, size); err != nil { - return nil, err + if !a.containsRead(offset, size) { + return nil, a.invalidReadBounds(offset, size) } if a.rangeFitsWindow(offset, size) { window, err := a.getWindow(offset, size, true) @@ -365,18 +371,25 @@ func (a *rollingReaderAccessor) refreshVisibleBounds() (journalHeader, bool, uin return journalHeader{}, false, oldSize, fmt.Errorf("%w: negative reader file size", errInvalidJournal) } newSize := uint64(info.Size()) + if newSize < oldSize { + a.fileSize = newSize + if a.selectedModeID == uint8(ReaderAccessReadAt) { + a.markWindowsStale() + } + a.refreshStats() + } header, err := readHeaderFromFileAt(a.file, newSize) if err != nil { - return journalHeader{}, false, oldSize, err + return journalHeader{}, false, newSize, err } a.fileSize = newSize if err := a.backend.refreshFileSize(a.file, newSize); err != nil { - a.fileSize = oldSize + a.fileSize = minUint64(oldSize, newSize) a.refreshStats() return journalHeader{}, false, oldSize, err } changed := newSize != oldSize - if a.selectedModeID == ReaderAccessReadAt { + if a.selectedModeID == uint8(ReaderAccessReadAt) { a.markWindowsStale() } a.refreshStats() @@ -422,6 +435,9 @@ func (a *rollingReaderAccessor) rangeFitsWindow(offset, size uint64) bool { } func (a *rollingReaderAccessor) windowBase(offset uint64) uint64 { + if a.powerOfTwoWindow { + return offset & -a.windowSize + } return (offset / a.windowSize) * a.windowSize } @@ -517,6 +533,13 @@ func (a *rollingReaderAccessor) markWindowsStale() { a.discardStaleUnpinnedWindows() } +func (a *rollingReaderAccessor) invalidateCopiedWindows() { + if a.selectedModeID == uint8(ReaderAccessReadAt) { + a.markWindowsStale() + a.refreshStats() + } +} + func (a *rollingReaderAccessor) discardStaleUnpinnedWindows() { for i := 0; i < len(a.windows); { window := a.windows[i] @@ -653,15 +676,16 @@ func (a *rollingReaderAccessor) updateRowArenaPeak() { } } -func (a *rollingReaderAccessor) checkBounds(offset, size uint64) error { - end, ok := checkedAdd(offset, size) - if !ok || end > a.fileSize { - return fmt.Errorf("%w: reader access out of bounds", errInvalidJournal) - } - if end > uint64(int(^uint(0)>>1)) && size > uint64(int(^uint(0)>>1)) { - return fmt.Errorf("%w: reader request too large", errInvalidJournal) +func (a *rollingReaderAccessor) containsRead(offset, size uint64) bool { + return offset <= a.fileSize && size <= a.fileSize-offset && size <= uint64(int(^uint(0)>>1)) +} + +//go:noinline +func (a *rollingReaderAccessor) invalidReadBounds(offset, size uint64) error { + if offset > a.fileSize || size > a.fileSize-offset { + return readerBoundsError{offset, size, a.fileSize} } - return nil + return fmt.Errorf("%w: reader request too large", errInvalidJournal) } func (a *rollingReaderAccessor) refreshStats() { @@ -675,8 +699,8 @@ func (a *rollingReaderAccessor) refreshStats() { pinned++ } } - a.statsValue.RequestedAccessMode = a.requestedMode - a.statsValue.SelectedAccessMode = a.selectedModeID + a.statsValue.RequestedAccessMode = ReaderAccessMode(a.requestedMode) + a.statsValue.SelectedAccessMode = ReaderAccessMode(a.selectedModeID) a.statsValue.FallbackReason = a.fallbackReason a.statsValue.FileSize = a.fileSize a.statsValue.WindowSize = a.windowSize @@ -690,7 +714,7 @@ func (a *rollingReaderAccessor) refreshStats() { } func (a *rollingReaderAccessor) recordWindowOpen() { - if a.selectedModeID == ReaderAccessReadAt { + if a.selectedModeID == uint8(ReaderAccessReadAt) { a.statsValue.ReadWindowCount++ return } @@ -698,7 +722,7 @@ func (a *rollingReaderAccessor) recordWindowOpen() { } func (a *rollingReaderAccessor) recordWindowClose() { - if a.selectedModeID == ReaderAccessMmap { + if a.selectedModeID == uint8(ReaderAccessMmap) { a.statsValue.UnmapCount++ } } @@ -785,3 +809,5 @@ func readFileAtFull(file *os.File, dst []byte, offset uint64) error { func emptySlice() []byte { return []byte{} } + +func (a *rollingReaderAccessor) initialSize() uint64 { return a.initialFileSize } diff --git a/go/journal/reader_accessor_stats_test.go b/go/journal/reader_accessor_stats_test.go new file mode 100644 index 00000000..ffba83b9 --- /dev/null +++ b/go/journal/reader_accessor_stats_test.go @@ -0,0 +1,87 @@ +package journal + +import ( + "fmt" + "os" + "path/filepath" + "runtime" + "testing" +) + +func TestReaderAccessorInitialStats(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessAuto, ReaderAccessMmap, ReaderAccessReadAt} { + for _, size := range []int{0, 256} { + t.Run(fmt.Sprintf("mode=%d/size=%d", mode, size), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "initial-stats") + if err := os.WriteFile(path, make([]byte, size), 0o600); err != nil { + t.Fatal(err) + } + file, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer func() { _ = file.Close() }() + opts := DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(64).WithMaxWindows(2).WithSnapshot(true) + accessor, got, err := newReaderAccessor(file, opts) + emptyWindowsMapping := runtime.GOOS == "windows" && size == 0 && mode != ReaderAccessReadAt + if emptyWindowsMapping && mode == ReaderAccessMmap { + if accessor != nil { + _ = accessor.close() + } + if err == nil || accessor != nil { + t.Fatal("empty Windows file must reject explicit mmap construction") + } + return + } + if err != nil { + t.Fatal(err) + } + defer func() { _ = accessor.close() }() + selected := ReaderAccessMmap + if mode == ReaderAccessReadAt || emptyWindowsMapping { + selected = ReaderAccessReadAt + } + if got != accessor.stats() || got.RequestedAccessMode != mode || got.FileSize != uint64(size) || + got.WindowSize != 64 || got.MaxWindows != 2 || got.RowPinnedWindows != 0 || + got.RowArenaBytes != 0 || got.RowArenaLimitBytes != opts.MaxRowArenaBytes || got.SelectedAccessMode != selected { + t.Fatalf("initial accessor statistics mismatch: %+v", got) + } + mapped := mode != ReaderAccessReadAt && size != 0 + if mapped { + if got.SelectedAccessMode != ReaderAccessMmap || got.ActiveWindows != 1 || got.MapCount != 1 || got.MappedBytes != 64 { + t.Fatalf("initial mapped statistics mismatch: %+v", got) + } + } else if got.ActiveWindows != 0 || got.MapCount != 0 || got.MappedBytes != 0 { + t.Fatalf("initial empty/lazy statistics mismatch: %+v", got) + } + if (got.FallbackReason != "") != emptyWindowsMapping || got.ReadWindowCount != 0 || got.UnmapCount != 0 || got.ReadBufferBytes != 0 { + t.Fatalf("unexpected initial accessor work: %+v", got) + } + }) + } + } +} + +func TestReaderAccessorInitialFallbackStats(t *testing.T) { + path := filepath.Join(t.TempDir(), "initial-fallback") + if err := os.WriteFile(path, make([]byte, 256), 0o600); err != nil { + t.Fatal(err) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer func() { _ = file.Close() }() + opts := DefaultReaderOptions().WithAccessMode(ReaderAccessAuto).WithWindowSize(64).WithMaxWindows(2).WithSnapshot(true) + accessor, got, err := newReaderAccessor(file, opts) + if err != nil { + t.Fatal(err) + } + defer func() { _ = accessor.close() }() + if got != accessor.stats() || got.RequestedAccessMode != ReaderAccessAuto || got.SelectedAccessMode != ReaderAccessReadAt || + got.FallbackReason == "" || got.FileSize != 256 || got.WindowSize != 64 || got.MaxWindows != 2 || + got.ActiveWindows != 0 || got.MapCount != 0 || got.UnmapCount != 0 || got.ReadWindowCount != 0 || + got.MappedBytes != 0 || got.ReadBufferBytes != 0 || got.RowArenaLimitBytes != opts.MaxRowArenaBytes { + t.Fatalf("initial fallback statistics mismatch: %+v", got) + } +} diff --git a/go/journal/reader_array_cycle_test.go b/go/journal/reader_array_cycle_test.go new file mode 100644 index 00000000..e3adba63 --- /dev/null +++ b/go/journal/reader_array_cycle_test.go @@ -0,0 +1,182 @@ +package journal + +import ( + "encoding/binary" + "errors" + "fmt" + "os" + "path/filepath" + "testing" +) + +func TestCountedArrayCyclesRejectRequiredRevisit(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + for _, postings := range []bool{false, true} { + for _, descending := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/postings=%v/descending=%v", compact, postings, descending), func(t *testing.T) { + data := append([]byte(nil), source...) + head, next := arrays[0].offset, arrays[1].offset + var postingHeader uint64 + count := uint64(len(arrays[0].entries) + len(arrays[1].entries) + 1) + if postings { + probe, _ := filterTestReader(t, source, ReaderAccessReadAt, false) + common := filterTestDataOffset(t, probe, "COMMON=all") + postingHeader = common + head = binary.LittleEndian.Uint64(data[common+48:]) + next = binary.LittleEndian.Uint64(data[head+16:]) + width := uint64(8) + if compact { + width = 4 + } + count = 2 + (binary.LittleEndian.Uint64(data[head+8:])-24)/width + (binary.LittleEndian.Uint64(data[next+8:])-24)/width + binary.LittleEndian.PutUint64(data[common+56:], count) + } else { + binary.LittleEndian.PutUint64(data[152:], count) + } + if descending { + head, next = next, head + if postings { + binary.LittleEndian.PutUint64(data[postingHeader+48:], head) + } else { + binary.LittleEndian.PutUint64(data[176:], head) + } + } + binary.LittleEndian.PutUint64(data[head+16:], next) + binary.LittleEndian.PutUint64(data[next+16:], head) + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + if errors.Is(err, errCorruptObject) { + return + } + t.Fatal(err) + } + defer r.Close() + if postings { + r.AddMatch([]byte("COMMON=all")) + } + for i := uint64(0); i <= count; i++ { + ok, err := r.Step() + if err != nil { + if !errors.Is(err, errCorruptObject) { + t.Fatal(err) + } + return + } + if !ok { + t.Fatal("required array revisit returned successful EOF") + } + } + t.Fatal("required array revisit returned duplicate rows") + }) + } + } + } +} + +func TestCountedArrayAcceptsAcyclicBackwardLink(t *testing.T) { + for _, compact := range []bool{false, true} { + data, arrays := recoveryTestFixture(t, compact) + a, b := arrays[0], arrays[1] + entries := append(append([]uint64(nil), a.entries...), b.entries...) + width := uint64(8) + if compact { + width = 4 + } + put := func(at, value uint64) { + if compact { + binary.LittleEndian.PutUint32(data[at:], uint32(value)) + } else { + binary.LittleEndian.PutUint64(data[at:], value) + } + } + for i, entry := range entries { + offset, slot := b.offset, i + if i >= len(b.entries) { + offset, slot = a.offset, i-len(b.entries) + } + put(offset+24+uint64(slot)*width, entry) + } + binary.LittleEndian.PutUint64(data[152:], uint64(len(entries))) + binary.LittleEndian.PutUint64(data[176:], b.offset) + binary.LittleEndian.PutUint64(data[b.offset+16:], a.offset) + binary.LittleEndian.PutUint64(data[a.offset+16:], 0) + r, _ := filterTestReader(t, data, ReaderAccessReadAt, false) + got := recoveryTestTimes(t, r, false) + if len(got) != len(entries) { + t.Fatalf("acyclic backward link: %v", got) + } + for i, time := range got { + if time != 1000+uint64(i) { + t.Fatalf("acyclic backward link row %d: %d", i, time) + } + } + } +} + +func TestCountedArrayIgnoresUnusedTerminalLink(t *testing.T) { + data, arrays := recoveryTestFixture(t, false) + count := len(arrays[0].entries) + len(arrays[1].entries) + binary.LittleEndian.PutUint64(data[152:], uint64(count)) + binary.LittleEndian.PutUint64(data[arrays[1].offset+16:], arrays[1].offset) + r, _ := filterTestReader(t, data, ReaderAccessReadAt, false) + if got := recoveryTestTimes(t, r, false); len(got) != count { + t.Fatalf("unused tail: %d rows, want %d", len(got), count) + } +} + +func TestWriterCountedArrayRejectsRevisitBeforeMutation(t *testing.T) { + for _, tailLookup := range []bool{false, true} { + for _, descending := range []bool{false, true} { + t.Run(fmt.Sprintf("tail=%v/descending=%v", tailLookup, descending), func(t *testing.T) { + w, err := Create(filepath.Join(t.TempDir(), "system.journal"), testOptions()) + if err != nil { + t.Fatal(err) + } + defer w.Close() + a, err := w.allocateOffsetArray(4) + if err != nil { + t.Fatal(err) + } + b, err := w.allocateOffsetArray(8) + if err != nil { + t.Fatal(err) + } + if err := w.writeUint64At(a+16, b); err != nil { + t.Fatal(err) + } + if err := w.writeUint64At(b+16, a); err != nil { + t.Fatal(err) + } + head := a + if descending { + head = b + } + before := make([]byte, 32) + if err := w.readAt(before, head+24); err != nil { + t.Fatal(err) + } + if tailLookup { + w.header.entryArrayOffset, w.header.nEntries, w.header.tailEntryArrayOffset = head, 13, 0 + _, err = w.entryArrayTailOffset() + } else { + _, _, err = w.appendToDataEntryArray(head, 13, 1024) + } + if !errors.Is(err, errInvalidJournal) { + t.Fatalf("required array revisit: %v", err) + } + after := make([]byte, len(before)) + if err := w.readAt(after, head+24); err != nil { + t.Fatal(err) + } + if string(after) != string(before) { + t.Fatal("rejected cyclic append changed the array") + } + }) + } + } +} diff --git a/go/journal/reader_cursor_direction.go b/go/journal/reader_cursor_direction.go new file mode 100644 index 00000000..4f277996 --- /dev/null +++ b/go/journal/reader_cursor_direction.go @@ -0,0 +1,489 @@ +package journal + +import "sort" + +func parseDirectionalCursor(cursor string) (parsedCursorLocation, error) { + want, err := parseCursorLocation(cursor, true) + if err != nil { + return want, err + } + for _, id := range []string{want.seqnumID, want.bootID} { + if id == "" { + continue + } + if len(id) != 32 { + return want, ErrInvalidCursor + } + for _, c := range id { + if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') { + return want, ErrInvalidCursor + } + } + } + want.seqnumSet = want.seqnumSet && want.seqnumID != "" + want.monotonicSet = want.monotonicSet && want.bootID != "" + return want, nil +} + +// SeekCursorDirection selects the first entry at or beyond cursor in direction. +// Existing SeekCursor retains its original forward-positioning behavior. +func (r *Reader) SeekCursorDirection(cursor string, direction Direction) error { + want, err := parseDirectionalCursor(cursor) + if err != nil { + return ErrInvalidCursor + } + if direction != DirectionForward && direction != DirectionBackward { + return ErrUnsupported + } + return r.seekCursorDirection(want, direction) +} + +func (r *Reader) seekCursorDirection(want parsedCursorLocation, direction Direction) error { + if r.accessor == nil { + return errInvalidJournal + } + r.clearRecoveryEOF() + r.clearCurrentEntryState() + if r.filter != nil && r.filter.retry { + r.accessor.invalidateCopiedWindows() + r.filter.retry = false + } + for { + before := r.RecoveryStatus() + var offset uint64 + var err error + if r.filter != nil { + if err = r.filter.build(r); err == nil { + offset, err = seekFilterCursor(r.filter.expr, r, want, direction) + } + } else { + offset, err = r.cursorInitialOffset(nil, want, direction) + } + var hdr entryHeader + if err == nil && offset != 0 { + hdr, err = r.readEntryHeaderAt(offset) + if err == nil { + err = r.validateFilteredEntryHeader(offset, hdr) + } + } + if err != nil { + if filterTailFailure(err) { + changed, recoveryErr := r.refreshFilterRecovery() + if recoveryErr != nil { + return recoveryErr + } + if changed { + continue + } + } + if r.filter != nil { + r.filter.walkValid, r.filter.retry = false, true + } + return err + } + if before != r.RecoveryStatus() { + continue + } + if offset == 0 { + if direction == DirectionForward { + return r.SeekTail() + } + return r.SeekHead() + } + index := sort.Search(len(r.entryOffsets), func(i int) bool { return r.entryOffsets[i] >= offset }) + if index >= len(r.entryOffsets) || r.entryOffsets[index] != offset { + if r.filter == nil { + return errCorruptObject + } + index = -2 + } + r.entryIndex, r.realtimeSeek = index, nil + r.currentHeader, r.currentHeaderOffset, r.currentHeaderValid = hdr, offset, true + if r.filter != nil { + f := r.filter + f.overrideOffset, f.walkOffset, f.walkValid = offset, offset, true + f.backward, f.retry = direction == DirectionBackward, false + } + return nil + } +} + +type cursorMetadataKey uint8 + +const ( + cursorSequence cursorMetadataKey = iota + cursorRealtime + cursorMonotonic + cursorOffset +) + +func (r *Reader) cursorKey(offset uint64, key cursorMetadataKey) (uint64, error) { + if key == cursorOffset { + return offset, nil + } + header, err := r.readEntryHeaderAt(offset) + if err != nil { + return 0, err + } + switch key { + case cursorSequence: + return header.seqnum, nil + case cursorMonotonic: + return header.monotonic, nil + default: + return header.realtime, nil + } +} + +func (r *Reader) cursorBound(posting *filterPostingCursor, key cursorMetadataKey, needle uint64, direction Direction) (uint64, error) { + if posting != nil { + return posting.seekCursorKey(r, key, needle, direction) + } + if uint64(len(r.entryOffsets)) != r.effectiveHeader().nEntries { + if err := r.refreshFilterOffsets(); err != nil { + return 0, err + } + } + left, right := 0, len(r.entryOffsets) + for left < right { + mid := left + (right-left)/2 + value, err := r.cursorKey(r.entryOffsets[mid], key) + if err != nil { + return 0, err + } + if value < needle || direction == DirectionBackward && value == needle { + left = mid + 1 + } else { + right = mid + } + } + if direction == DirectionBackward { + left-- + } + if left < 0 || left >= len(r.entryOffsets) { + return 0, nil + } + return r.entryOffsets[left], nil +} + +func (r *Reader) cursorBootPosting(boot string) (*filterPostingCursor, error) { + offset, found, err := r.findDataOffsetByPayload([]byte("_BOOT_ID=" + boot)) + if err != nil || !found { + return nil, err + } + posting, err := r.newFilterPostingCursor(offset) + return &posting, err +} + +func (r *Reader) cursorMonotonicOffset(posting *filterPostingCursor, boot string, needle uint64, direction Direction) (uint64, error) { + boots, err := r.cursorBootPosting(boot) + if err != nil || boots == nil { + return 0, err + } + offset, err := r.cursorBound(boots, cursorMonotonic, needle, direction) + if err != nil || offset == 0 || posting == nil { + return offset, err + } + for { + candidate, err := posting.seekRealtime(r, offset, direction) + if err != nil || candidate == 0 { + return 0, err + } + if candidate == offset { + return offset, nil + } + offset, err = boots.seekRealtime(r, candidate, direction) + if err != nil || offset == 0 { + return 0, err + } + if offset == candidate { + return offset, nil + } + } +} + +func (r *Reader) cursorAdjacentBoot(posting *filterPostingCursor, boot string, direction Direction) (uint64, error) { + var previous uint64 + for { + boots, err := r.cursorBootPosting(boot) + if err != nil || boots == nil { + return 0, err + } + boundary, needle, opposite := uint64(0), uint64(0), DirectionForward + if direction == DirectionForward { + needle, opposite = ^uint64(0), DirectionBackward + } + boundary, err = boots.seekRealtime(r, needle, opposite) + if err != nil || boundary == 0 { + return 0, err + } + if direction == DirectionForward { + if boundary == ^uint64(0) { + return 0, nil + } + boundary++ + } else { + boundary-- + } + offset, err := r.cursorBound(nil, cursorOffset, boundary, direction) + if err != nil || offset == 0 || posting == nil { + return offset, err + } + if previous != 0 && (direction == DirectionForward && offset <= previous || direction == DirectionBackward && offset >= previous) { + return 0, errCorruptObject + } + previous = offset + header, err := r.readEntryHeaderAt(offset) + if err != nil { + return 0, err + } + boot = header.bootID.String() + needle = 0 + if direction == DirectionBackward { + needle = ^uint64(0) + } + candidate, err := r.cursorMonotonicOffset(posting, boot, needle, direction) + if err != nil || candidate != 0 { + return candidate, err + } + } +} + +func (r *Reader) cursorInitialOffset(posting *filterPostingCursor, want parsedCursorLocation, direction Direction) (uint64, error) { + if want.seqnumSet && want.seqnumID == r.header.seqnumID.String() { + return r.cursorBound(posting, cursorSequence, want.seqnum, direction) + } + if want.monotonicSet { + offset, err := r.cursorMonotonicOffset(posting, want.bootID, want.monotonic, direction) + if err != nil || offset != 0 { + return offset, err + } + } + if want.realtimeSet { + return r.cursorBound(posting, cursorRealtime, want.realtime, direction) + } + if want.monotonicSet { + return r.cursorAdjacentBoot(posting, want.bootID, direction) + } + needle := uint64(0) + if direction == DirectionBackward { + needle = ^uint64(0) + } + return r.cursorBound(posting, cursorOffset, needle, direction) +} + +func seekFilterCursor(expr filterExpr, r *Reader, want parsedCursorLocation, direction Direction) (uint64, error) { + switch value := expr.(type) { + case *matchExpr: + return r.cursorInitialOffset(&value.posting, want, direction) + case falseExpr: + return 0, nil + case orExpr: + var best uint64 + for _, child := range value { + offset, err := seekFilterCursor(child, r, want, direction) + if err != nil { + return 0, err + } + if offset != 0 && (best == 0 || direction == DirectionForward && offset < best || direction == DirectionBackward && offset > best) { + best = offset + } + } + return best, nil + case andExpr: + var best uint64 + for _, child := range value { + offset, err := seekFilterCursor(child, r, want, direction) + if err != nil || offset == 0 { + return 0, err + } + if best == 0 || direction == DirectionForward && offset > best || direction == DirectionBackward && offset < best { + best = offset + } + } + if best == 0 { + return 0, nil + } + return value.seek(r, best, direction) + default: + return 0, ErrMalformedFilter + } +} + +func (p *filterPostingCursor) seekCursorKey(r *Reader, key cursorMetadataKey, needle uint64, direction Direction) (uint64, error) { + p.node, p.slot, p.value = -1, 0, p.inline + if p.count == 0 { + return 0, nil + } + inline, err := r.cursorKey(p.inline, key) + if err != nil { + return 0, err + } + if direction == DirectionForward && inline >= needle { + return p.inline, nil + } + if direction == DirectionBackward && inline > needle { + return 0, nil + } + if p.count == 1 { + if direction == DirectionBackward { + return p.inline, nil + } + return 0, nil + } + for index := 0; ; index++ { + if index == len(p.nodes) { + if err := p.appendNode(r); err != nil { + return 0, err + } + } + node := p.nodes[index] + left, right := uint64(0), node.length + for left < right { + mid := left + (right-left)/2 + value, err := p.readSlot(r, index, mid) + if err != nil { + return 0, err + } + bound, err := r.cursorKey(value, key) + if err != nil { + return 0, err + } + if bound < needle || direction == DirectionBackward && bound == needle { + left = mid + 1 + } else { + right = mid + } + } + if direction == DirectionBackward && left == 0 { + return p.value, nil + } + slot := left + if direction == DirectionBackward || left == node.length { + slot-- + } + value, err := p.readSlot(r, index, slot) + if err != nil { + return 0, err + } + // Retain the last visited posting even when a sibling wins an OR query. + p.node, p.slot, p.value = index, slot, value + if left < node.length { + return value, nil + } + if node.remaining == node.length { + if direction == DirectionBackward { + return value, nil + } + return 0, nil + } + } +} + +// SeekCursorDirection positions a merged reader using each file's native index. +func (dr *DirectoryReader) SeekCursorDirection(cursor string, direction Direction) error { + want, err := parseDirectionalCursor(cursor) + if err != nil { + return ErrInvalidCursor + } + if direction != DirectionForward && direction != DirectionBackward { + return ErrUnsupported + } + for { + dr.clearDirectoryPosition() + dr.realtimeSeek, dr.realtimeSeekBound = nil, nil + dr.resetCandidates() + dr.direction, dr.hasDirection, dr.merging = direction, true, true + dr.filteredCandidates = dr.filter != nil + dr.orderDirty = false + best, exact := -1, -1 + for i, reader := range dr.files { + if err := reader.seekCursorDirection(want, direction); err != nil { + return err + } + key, err := reader.currentDirectoryEntryKey() + if err == errEndOfEntries { + continue + } + if err != nil { + return err + } + dr.candidates[i] = &directoryCandidate{readerIndex: i, key: key} + if want.seqnumSet && key.seqnum == want.seqnum && key.seqnumID.String() == want.seqnumID { + exact = i + } + if best < 0 { + best = i + continue + } + cmp := dr.compareEntryKeys(key, dr.candidates[best].key) + if direction == DirectionForward && cmp < 0 || direction == DirectionBackward && cmp > 0 { + best = i + } + } + if dr.orderDirty { + dr.bootNewest = buildDirectoryBootNewest(dr.files) + continue + } + if exact >= 0 { + best = exact + } + if best < 0 { + return nil + } + key := dr.candidates[best].key + // Files without an initial candidate can become eligible at the selected row. + nextWant := parsedCursorLocation{seqnumID: key.seqnumID.String(), seqnum: key.seqnum, seqnumSet: true, + bootID: key.bootID.String(), monotonic: key.monotonic, monotonicSet: true, realtime: key.realtime, realtimeSet: true} + for i, reader := range dr.files { + if i == best { + continue + } + if candidate := dr.candidates[i]; candidate != nil { + cmp := dr.compareEntryKeys(candidate.key, key) + if direction == DirectionForward && cmp > 0 || direction == DirectionBackward && cmp < 0 { + continue + } + dr.candidates[i] = nil + } + if err := reader.seekCursorDirection(nextWant, direction); err != nil { + return err + } + candidateKey, err := reader.currentDirectoryEntryKey() + if err == errEndOfEntries { + continue + } + if err != nil { + return err + } + cmp := dr.compareEntryKeys(candidateKey, key) + if direction == DirectionForward && cmp > 0 || direction == DirectionBackward && cmp < 0 { + dr.candidates[i] = &directoryCandidate{readerIndex: i, key: candidateKey} + } + } + if dr.orderDirty { + dr.bootNewest = buildDirectoryBootNewest(dr.files) + continue + } + dr.index, dr.currentKey = best, &key + dr.candidates[best] = nil + return nil + } +} + +// SdJournalSeekCursorDirection is an eager directional extension to the facade. +func SdJournalSeekCursorDirection(j *sdJournal, cursor string, direction Direction) error { + j.resetIterators() + switch reader := j.reader.(type) { + case *Reader: + return reader.SeekCursorDirection(cursor, direction) + case *DirectoryReader: + return reader.SeekCursorDirection(cursor, direction) + default: + return ErrUnsupported + } +} + +func (j *sdJournal) SeekCursorDirection(cursor string, direction Direction) error { + return SdJournalSeekCursorDirection(j, cursor, direction) +} diff --git a/go/journal/reader_cursor_direction_test.go b/go/journal/reader_cursor_direction_test.go new file mode 100644 index 00000000..f100d38a --- /dev/null +++ b/go/journal/reader_cursor_direction_test.go @@ -0,0 +1,242 @@ +package journal + +import ( + "errors" + "fmt" + "path/filepath" + "syscall" + "testing" +) + +func TestDirectionalCursorNativeBoundaries(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "system.journal") + opts := testOptions() + writer, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + boots := []UUID{{1}, {2}, {3}} + for i := 0; i < 6; i++ { + timestamp := uint64(1000 + i) + if i == 3 { + timestamp = 1002 + } + message := "other" + if i%2 == 1 { + message = "keep" + } + boot := boots[i/2] + if err := writer.Append([]Field{StringField("ROW", fmt.Sprint(i)), StringField("MESSAGE", message), StringField("_BOOT_ID", boot.String())}, EntryOptions{RealtimeUsec: timestamp, MonotonicUsec: uint64(i%2 + 1), BootID: boot}); err != nil { + t.Fatal(err) + } + } + if err := writer.Close(); err != nil { + t.Fatal(err) + } + seq := opts.SeqnumID.String() + cases := []struct { + cursor string + direction Direction + raw, filtered string + }{ + {"t=3ea", DirectionForward, "2", "3"}, + {"t=3ea", DirectionBackward, "3", "3"}, + {"s=" + seq + ";i=1;t=3ed", DirectionBackward, "0", ""}, + {"s=" + seq + ";i=ffffffff", DirectionBackward, "5", "5"}, + {"s=" + seq + ";i=ffffffff", DirectionForward, "", ""}, + {"s=ffffffffffffffffffffffffffffffff;i=3", DirectionBackward, "5", "5"}, + {"s=ffffffffffffffffffffffffffffffff;i=3;m=0", DirectionBackward, "5", "5"}, + {"b=" + boots[1].String() + ";m=0", DirectionBackward, "1", "1"}, + {"b=" + boots[1].String() + ";m=0", DirectionForward, "2", "3"}, + {"b=" + boots[1].String() + ";m=ffffffff", DirectionForward, "4", "5"}, + {"b=ffffffffffffffffffffffffffffffff;m=0", DirectionBackward, "", ""}, + {"b=ffffffffffffffffffffffffffffffff;m=0;t=3ea", DirectionBackward, "3", "3"}, + } + for _, directory := range []bool{false, true} { + for _, filtered := range []bool{false, true} { + for _, tc := range cases { + t.Run(fmt.Sprintf("dir=%v/filter=%v/%s/%d", directory, filtered, tc.cursor, tc.direction), func(t *testing.T) { + var j *sdJournal + var err error + if directory { + j, err = SdJournalOpenDirectory(dir, 0) + } else { + j, err = SdJournalOpen(path, 0) + } + if err != nil { + t.Fatal(err) + } + defer j.Close() + if filtered { + if err := SdJournalAddMatch(j, []byte("MESSAGE=keep")); err != nil { + t.Fatal(err) + } + } + if err := SdJournalSeekCursorDirection(j, tc.cursor, tc.direction); err != nil { + t.Fatal(err) + } + want := tc.raw + if filtered { + want = tc.filtered + } + if want == "" { + if _, err := SdJournalGetCursor(j); !errors.Is(err, ErrNoEntry) { + t.Fatalf("expected absent cursor: %v", err) + } + return + } + entry, err := SdJournalGetEntry(j) + if err != nil { + t.Fatal(err) + } + if got := string(entry.Fields["ROW"]); got != want { + t.Fatalf("row=%s want%s", got, want) + } + }) + } + } + } +} + +func TestDirectionalCursorIndexedErrorsAndRecovery(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("compact=%v/access=%d", compact, mode), func(t *testing.T) { + r, _ := filterTestReader(t, source, mode, false) + unrelated := filterTestDataOffset(t, r, "MESSAGE=row-0001") + r.AddMatch([]byte("COMMON=all")) + trace := &filterAccessTrace{readerAccessor: r.accessor, forbidden: map[uint64]bool{unrelated: true}, faultAt: r.entryOffsets[500], fail: true} + r.accessor = trace + cursor := fmt.Sprintf("s=%s;i=%x", r.header.seqnumID.String(), 501) + if err := r.SeekCursorDirection(cursor, DirectionBackward); !errors.Is(err, syscall.EIO) { + t.Fatalf("I/O error=%v", err) + } + trace.fail = false + trace.calls = 0 + if err := r.SeekCursorDirection(cursor, DirectionBackward); err != nil { + t.Fatal(err) + } + if got, err := r.GetRealtimeUsec(); got != 1500 || err != nil { + t.Fatalf("time=%d/%v", got, err) + } + if trace.calls > 160 { + t.Fatalf("indexed seek performed %d accessor reads", trace.calls) + } + t.Logf("indexed metadata seek: %d accessor reads for1005 entries", trace.calls) + tail := arrays[len(arrays)-1].entries + recovered, _ := filterTestReader(t, source[:tail[len(tail)-1]], mode, false) + recovered.AddMatch([]byte("COMMON=all")) + if err := recovered.SeekCursorDirection("t=ffffffffffffffff", DirectionBackward); err != nil { + t.Fatal(err) + } + if got, err := recovered.GetRealtimeUsec(); got != 2003 || err != nil { + t.Fatalf("recovered last=%d/%v", got, err) + } + if !recovered.RecoveryStatus().Incomplete() { + t.Fatal("recovery status missing") + } + }) + } + } +} + +func TestDirectionalCursorExactSequenceOverridesBootOrder(t *testing.T) { + dir := t.TempDir() + for i, times := range [][]uint64{{1100, 1200}, {1150, 1180}} { + opts := testOptions() + opts.SeqnumID = UUID{byte(10 + i)} + opts.BootID = UUID{byte(20 + i)} + w, err := Create(filepath.Join(dir, fmt.Sprintf("%d.journal", i)), opts) + if err != nil { + t.Fatal(err) + } + for n, time := range times { + if err := w.Append([]Field{StringField("MESSAGE", fmt.Sprintf("%d-%d", i, n)), StringField("_BOOT_ID", opts.BootID.String())}, EntryOptions{RealtimeUsec: time, MonotonicUsec: uint64(n + 1)}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + single, err := OpenFile(filepath.Join(dir, "0.journal")) + if err != nil { + t.Fatal(err) + } + if ok, err := single.Step(); !ok || err != nil { + t.Fatalf("first=%v/%v", ok, err) + } + cursor, err := single.GetCursor() + if err != nil { + t.Fatal(err) + } + single.Close() + dr, err := OpenDirectory(dir) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if err := dr.SeekCursorDirection(cursor, DirectionForward); err != nil { + t.Fatal(err) + } + for _, want := range []string{"0-0", "0-1"} { + entry, err := dr.GetEntry() + if err != nil { + t.Fatal(err) + } + if got := string(entry.Fields["MESSAGE"]); got != want { + t.Fatalf("got%s want%s", got, want) + } + ok, err := dr.Step() + if err != nil { + t.Fatal(err) + } + if ok != (want == "0-0") { + t.Fatalf("step after%s=%v", want, ok) + } + } +} + +func TestDirectionalCursorClosedReader(t *testing.T) { + source, _ := recoveryTestFixture(t, false) + initial, path := filterTestReader(t, source, ReaderAccessMmap, false) + if err := initial.Close(); err != nil { + t.Fatal(err) + } + for _, directory := range []bool{false, true} { + for _, state := range []string{"raw", "pending", "compiled"} { + t.Run(fmt.Sprintf("directory=%v/%s", directory, state), func(t *testing.T) { + var j *sdJournal + var err error + if directory { + j, err = SdJournalOpenDirectory(filepath.Dir(path), 0) + } else { + j, err = SdJournalOpen(path, 0) + } + if err != nil { + t.Fatal(err) + } + if state != "raw" { + if err := SdJournalAddMatch(j, []byte("COMMON=all")); err != nil { + t.Fatal(err) + } + } + if state == "compiled" { + if err := j.SeekCursorDirection("t=1", DirectionForward); err != nil { + t.Fatal(err) + } + } + if err := j.Close(); err != nil { + t.Fatal(err) + } + for _, direction := range []Direction{DirectionForward, DirectionBackward, DirectionForward} { + if err := j.SeekCursorDirection("t=1", direction); !errors.Is(err, errInvalidJournal) { + t.Fatalf("closed seek: %v", err) + } + } + }) + } + } +} diff --git a/go/journal/reader_directory_transition_test.go b/go/journal/reader_directory_transition_test.go new file mode 100644 index 00000000..3229dc53 --- /dev/null +++ b/go/journal/reader_directory_transition_test.go @@ -0,0 +1,200 @@ +package journal + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestDirectoryFilterReplacementKeepsSequentialDiscovery(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, installed := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/installed=%v", compact, installed), func(t *testing.T) { + dir := t.TempDir() + for index := 0; index < 3; index++ { + opts := testOptions() + opts.Compact = compact + opts.FileID[0] += byte(index) + opts.HeadSeqnum = uint64(3*index + 1) + w, err := Create(filepath.Join(dir, fmt.Sprintf("%d.journal", index)), opts) + if err != nil { + t.Fatal(err) + } + for row := 1; row <= 3; row++ { + time := uint64(3*index+row) * 1000 + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: time, MonotonicUsec: time}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + dr, err := OpenDirectoryWithOptions(dir, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if !dr.nonOverlapping { + t.Fatal("fixture must select sequential traversal") + } + if installed { + dr.AddMatch([]byte("COMMON=all")) + } + for time := uint64(1000); time <= 4000; time += 1000 { + if ok, err := positionTestStep(dr, nil, false); err != nil || !ok { + t.Fatalf("initial step: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != time { + t.Fatalf("initial time: %d/%v, want %d", got, err, time) + } + } + file, err := os.OpenFile(filepath.Join(dir, "2.journal"), os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + offset := int64(dr.files[2].entryOffsets[0]) + if _, err := file.WriteAt([]byte{1}, offset); err != nil { + t.Fatal(err) + } + dr.AddMatch([]byte("COMMON=all")) + for _, time := range []uint64{5000, 6000} { + if ok, err := positionTestStep(dr, nil, false); err != nil || !ok { + t.Fatalf("nearer healthy step: %v/%v, want %d", ok, err, time) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != time { + t.Fatalf("nearer healthy time: %d/%v, want %d", got, err, time) + } + } + if ok, err := positionTestStep(dr, nil, false); err == nil || ok { + t.Fatalf("damaged required ENTRY must fail: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != 6000 { + t.Fatalf("error changed selected row: %d/%v", got, err) + } + if _, err := file.WriteAt([]byte{3}, offset); err != nil { + t.Fatal(err) + } + if ok, err := positionTestStep(dr, nil, false); err != nil || !ok { + t.Fatalf("repair retry: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != 7000 { + t.Fatalf("repair time: %d/%v", got, err) + } + }) + } + } +} + +func TestDirectoryRecoveryMergeErrorRetainsPrefetchedRow(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v", compact, mode, facadeAPI), func(t *testing.T) { + dir := t.TempDir() + for index := 0; index < 3; index++ { + opts := testOptions() + opts.Compact = compact + opts.FileID[0] += byte(index) + opts.HeadSeqnum = uint64(3*index + 1) + w, err := Create(filepath.Join(dir, fmt.Sprintf("%d.journal", index)), opts) + if err != nil { + t.Fatal(err) + } + for row := 1; row <= 3; row++ { + time := uint64(3*index+row) * 1000 + fields := []Field{StringField("MESSAGE", fmt.Sprintf("row-%d", time)), StringField("COMMON", "all"), StringField("PADDING", strings.Repeat("p", 2048)+fmt.Sprint(time))} + if err := w.Append(fields, EntryOptions{RealtimeUsec: time, MonotonicUsec: time}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + dr, err := OpenDirectoryWithOptions(dir, DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(1024).WithMaxWindows(1).WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if !dr.nonOverlapping { + t.Fatal("fixture must start with sequential traversal") + } + dr.AddMatch([]byte("COMMON=all")) + if err := dr.SeekTail(); err != nil { + t.Fatal(err) + } + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + } + for time := uint64(9000); time >= 4000; time -= 1000 { + if ok, err := positionTestStep(dr, facade, true); err != nil || !ok { + t.Fatalf("initial step: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != time { + t.Fatalf("initial time: %d/%v, want %d", got, err, time) + } + } + cursor, err := dr.GetCursor() + if err != nil { + t.Fatal(err) + } + for index, row := range map[int]int{0: 2, 2: 1} { + r := dr.files[index] + offset := r.entryOffsets[row] + if _, err := r.accessor.tempSlice(0, 1); err != nil { + t.Fatal(err) + } + file, err := os.OpenFile(filepath.Join(dir, fmt.Sprintf("%d.journal", index)), os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + damage := []byte{2} + if index == 0 { + damage = make([]byte, 16) + } + if _, err := file.WriteAt(damage, int64(offset)); err != nil { + t.Fatal(err) + } + } + if ok, err := positionTestStep(dr, facade, true); err == nil || ok { + t.Fatalf("merge probe must fail: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != 4000 { + t.Fatalf("selected time: %d/%v, want 4000", got, err) + } + if got, err := dr.GetCursor(); err != nil || got != cursor { + t.Fatalf("selected cursor changed: %q/%v", got, err) + } + if len(dr.RecoveryStatuses()) != 1 || dr.nonOverlapping { + t.Fatal("missing tail must switch the directory to merging") + } + file, err := os.OpenFile(filepath.Join(dir, "2.journal"), os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteAt([]byte{3}, int64(dr.files[2].entryOffsets[1])); err != nil { + t.Fatal(err) + } + for _, time := range []uint64{2000, 1000} { + if ok, err := positionTestStep(dr, facade, true); err != nil || !ok { + t.Fatalf("repaired retry: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != time { + t.Fatalf("retry time: %d/%v, want %d", got, err, time) + } + } + if ok, err := positionTestStep(dr, facade, true); err != nil || ok { + t.Fatalf("recovered EOF: %v/%v", ok, err) + } + }) + } + } + } +} diff --git a/go/journal/reader_entry.go b/go/journal/reader_entry.go index e96949e7..56de3425 100644 --- a/go/journal/reader_entry.go +++ b/go/journal/reader_entry.go @@ -7,12 +7,12 @@ import ( ) func (r *Reader) GetEntry() (*Entry, error) { - if r.entryIndex < 0 || r.entryIndex >= len(r.entryOffsets) { - return nil, errEndOfEntries + offset, err := r.currentEntryOffset() + if err != nil { + return nil, err } r.entryDataActive = false - offset := r.entryOffsets[r.entryIndex] return r.readEntryAt(offset) } @@ -342,6 +342,7 @@ func (r *Reader) findFieldHeadDataOffset(field []byte) (uint64, bool, error) { return 0, false, err } item := parseHashItem(itemBuf) + var chain chainGuard for offset := item.head; offset != 0; { header, payload, err := r.readFieldObjectAt(offset) if err != nil { @@ -350,6 +351,9 @@ func (r *Reader) findFieldHeadDataOffset(field []byte) (uint64, bool, error) { if header.hash == hash && bytes.Equal(payload, field) { return header.headDataOffset, true, nil } + if !chain.advance(offset, header.nextHashOffset, false) { + return 0, false, errCorruptObject + } offset = header.nextHashOffset } return 0, false, nil diff --git a/go/journal/reader_filter.go b/go/journal/reader_filter.go index 3c95aa77..a3266962 100644 --- a/go/journal/reader_filter.go +++ b/go/journal/reader_filter.go @@ -2,56 +2,107 @@ package journal import ( "bytes" + "errors" "sort" ) +var ErrMalformedFilter = errors.New("malformed journal filter") + type filterBuilder struct { - level0 []filterExpr - level1 []filterExpr - current [][]byte + level0 []filterExpr + level1 []filterExpr + current [][]byte + pending bool + expr filterExpr + stale bool + walkValid bool + backward bool + retry bool + walkOffset uint64 + overrideOffset uint64 } type filterExpr interface { - matches(*Entry) bool -} - -type matchExpr struct { - field string - value []byte + seek(*Reader, uint64, Direction) (uint64, error) + reset(*Reader, Direction) error + refresh(*Reader) (filterExpr, error) } +type matchExpr struct{ posting filterPostingCursor } type andExpr []filterExpr - type orExpr []filterExpr - type falseExpr struct{} func (f *filterBuilder) addMatch(data []byte) { - f.current = append(f.current, append([]byte(nil), data...)) + f.pending = true + if bytes.IndexByte(data, '=') >= 0 { + f.current = append(f.current, append([]byte(nil), data...)) + } } -func (f *filterBuilder) addDisjunction() { - f.commitCurrent() +func (f *filterBuilder) addDisjunction(r *Reader) error { + f.pending = true + return f.commitCurrent(r) } -func (f *filterBuilder) addConjunction() { - f.commitCurrent() +func (f *filterBuilder) addConjunction(r *Reader) error { + f.pending = true + if err := f.commitCurrent(r); err != nil { + return err + } f.commitLevel1() + return nil } -func (f *filterBuilder) matches(entry *Entry) bool { - expr := f.finalExpr() - if expr == nil { - return true +func (f *filterBuilder) build(r *Reader) error { + if f.pending { + if err := f.commitCurrent(r); err != nil { + return err + } + f.commitLevel1() + if len(f.level0) == 0 { + return ErrMalformedFilter + } + expr := collapseFilterAnd(f.level0) + if r.recovery.Incomplete() || f.stale { + var err error + expr, err = refreshStableFilter(r, expr) + if err != nil { + return err + } + } + f.expr = expr + f.level0 = nil + f.pending = false + f.stale = false + f.walkValid = false + } else if f.stale { + expr, err := refreshStableFilter(r, f.expr) + if err != nil { + return err + } + f.expr = expr + f.stale = false + f.walkValid = false } - return expr.matches(entry) + return nil } -func (f *filterBuilder) commitCurrent() { - if expr := buildCurrentFilterExpr(f.current); expr != nil { +func (f *filterBuilder) commitCurrent(r *Reader) error { + if len(f.current) != 0 { + if err := r.prepareFilterGroup(); err != nil { + return err + } + } + expr, err := buildCurrentFilterExpr(r, f.current) + if err != nil { + return err + } + if expr != nil { f.level1 = append(f.level1, expr) } f.current = nil + return nil } func (f *filterBuilder) commitLevel1() { @@ -61,102 +112,198 @@ func (f *filterBuilder) commitLevel1() { f.level1 = nil } -func (f *filterBuilder) finalExpr() filterExpr { - level0 := append([]filterExpr(nil), f.level0...) - level1 := append([]filterExpr(nil), f.level1...) - if expr := buildCurrentFilterExpr(f.current); expr != nil { - level1 = append(level1, expr) - } - if expr := buildLevel1FilterExpr(level1); expr != nil { - level0 = append(level0, expr) +func collapseFilterAnd(parts []filterExpr) filterExpr { + if len(parts) == 1 { + return parts[0] } - if len(level0) == 0 { + return andExpr(parts) +} + +func buildLevel1FilterExpr(parts []filterExpr) filterExpr { + if len(parts) == 0 { return nil } - if len(level0) == 1 { - return level0[0] + if len(parts) == 1 { + return parts[0] } - return andExpr(level0) + return orExpr(parts) } -func buildLevel1FilterExpr(level1 []filterExpr) filterExpr { - if len(level1) == 0 { - return nil +func buildCurrentFilterExpr(r *Reader, matches [][]byte) (filterExpr, error) { + if len(matches) == 0 { + return nil, nil } - if len(level1) == 1 { - return level1[0] + sort.SliceStable(matches, func(i, j int) bool { + return bytes.Compare(filterMatchKey(matches[i]), filterMatchKey(matches[j])) < 0 + }) + var parts []filterExpr + for start := 0; start < len(matches); { + end := start + 1 + for end < len(matches) && bytes.Equal(filterMatchKey(matches[start]), filterMatchKey(matches[end])) { + end++ + } + values := make([]filterExpr, 0, end-start) + for _, payload := range matches[start:end] { + if r.header.dataHashTableOffset == 0 || r.header.dataHashTableSize < hashItemSize { + return nil, errCorruptObject + } + offset, found, err := r.findDataOffsetByPayload(payload) + if err != nil { + return nil, err + } + var expr filterExpr = falseExpr{} + if found { + posting, err := r.newFilterPostingCursor(offset) + if err != nil { + return nil, err + } + if posting.declared != 0 { + expr = &matchExpr{posting: posting} + } + } + values = append(values, expr) + } + parts = append(parts, buildLevel1FilterExpr(values)) + start = end } - return orExpr(level1) + return collapseFilterAnd(parts), nil } -func buildCurrentFilterExpr(matches [][]byte) filterExpr { - if len(matches) == 0 { - return nil +func filterMatchKey(payload []byte) []byte { return payload[:bytes.IndexByte(payload, '=')] } + +func (m *matchExpr) seek(r *Reader, needle uint64, direction Direction) (uint64, error) { + return m.posting.seek(r, needle, direction) +} +func (m *matchExpr) reset(r *Reader, direction Direction) error { return m.posting.reset(r, direction) } +func (m *matchExpr) refresh(r *Reader) (filterExpr, error) { + posting, err := r.newFilterPostingCursor(m.posting.dataOffset) + if err != nil { + return nil, err } - byField := make(map[string][]filterExpr) - var fields []string - for _, item := range matches { - eq := bytes.IndexByte(item, '=') - if eq < 0 { - return falseExpr{} + return &matchExpr{posting: posting}, nil +} + +func refreshStableFilter(r *Reader, expr filterExpr) (filterExpr, error) { + for { + before := r.recovery + next, err := expr.refresh(r) + if err != nil { + return nil, err } - field := string(item[:eq]) - if _, ok := byField[field]; !ok { - fields = append(fields, field) + expr = next + if sameRecoveryState(before, r.recovery) { + return expr, nil } - byField[field] = append(byField[field], matchExpr{ - field: field, - value: append([]byte(nil), item[eq+1:]...), - }) } - sort.Strings(fields) +} - parts := make([]filterExpr, 0, len(fields)) - for _, field := range fields { - values := byField[field] - if len(values) == 1 { - parts = append(parts, values[0]) - } else { - parts = append(parts, orExpr(values)) +// Ordinary retries retain the installed DATA population while rereading nodes. +func invalidateFilterWalk(expr filterExpr) { + switch value := expr.(type) { + case *matchExpr: + p := &value.posting + p.nodes = p.nodes[:0] + p.highest = 0 + p.node, p.slot, p.value = -1, 0, p.inline + case andExpr: + for _, child := range value { + invalidateFilterWalk(child) + } + case orExpr: + for _, child := range value { + invalidateFilterWalk(child) } } - if len(parts) == 1 { - return parts[0] +} + +func prepareResolvedFilterWalk(expr filterExpr, direction Direction) { + switch value := expr.(type) { + case *matchExpr: + p := &value.posting + p.node, p.slot, p.value = -1, 0, p.inline + if direction == DirectionBackward { + // Resolved backward expressions prepare only the leaves they visit. + p.node, p.value = -2, 0 + } + case andExpr: + for _, child := range value { + prepareResolvedFilterWalk(child, direction) + } + case orExpr: + for _, child := range value { + prepareResolvedFilterWalk(child, direction) + } } - return andExpr(parts) } -func (m matchExpr) matches(entry *Entry) bool { - if entry.FieldValues != nil { - for _, value := range entry.FieldValues[m.field] { - if bytes.Equal(value, m.value) { - return true +func (a andExpr) seek(r *Reader, needle uint64, direction Direction) (uint64, error) { + for { + previous := needle + for i := range a { + index := i + if direction == DirectionBackward { + index = len(a) - 1 - i + } + value, err := a[index].seek(r, needle, direction) + if err != nil || value == 0 { + return 0, err } + needle = value + } + if needle == previous { + return needle, nil } - return false } - value, ok := entry.Fields[m.field] - return ok && bytes.Equal(value, m.value) } -func (a andExpr) matches(entry *Entry) bool { - for _, expr := range a { - if !expr.matches(entry) { - return false +func (o orExpr) seek(r *Reader, needle uint64, direction Direction) (uint64, error) { + var best uint64 + for _, expr := range o { + value, err := expr.seek(r, needle, direction) + if err != nil { + return 0, err + } + if value != 0 && (best == 0 || direction == DirectionForward && value < best || direction == DirectionBackward && value > best) { + best = value } } - return true + return best, nil } -func (o orExpr) matches(entry *Entry) bool { - for _, expr := range o { - if expr.matches(entry) { - return true +func resetFilterChildren(r *Reader, children []filterExpr, direction Direction) error { + for _, child := range children { + if err := child.reset(r, direction); err != nil { + return err } } - return false + return nil +} +func (a andExpr) reset(r *Reader, direction Direction) error { + return resetFilterChildren(r, a, direction) +} +func (o orExpr) reset(r *Reader, direction Direction) error { + return resetFilterChildren(r, o, direction) } -func (falseExpr) matches(*Entry) bool { - return false +func refreshFilterChildren(r *Reader, children []filterExpr) ([]filterExpr, error) { + next := make([]filterExpr, len(children)) + for i, child := range children { + var err error + next[i], err = child.refresh(r) + if err != nil { + return nil, err + } + } + return next, nil +} +func (a andExpr) refresh(r *Reader) (filterExpr, error) { + children, err := refreshFilterChildren(r, a) + return andExpr(children), err +} +func (o orExpr) refresh(r *Reader) (filterExpr, error) { + children, err := refreshFilterChildren(r, o) + return orExpr(children), err } +func (falseExpr) seek(*Reader, uint64, Direction) (uint64, error) { return 0, nil } +func (falseExpr) reset(*Reader, Direction) error { return nil } +func (f falseExpr) refresh(*Reader) (filterExpr, error) { return f, nil } diff --git a/go/journal/reader_filter_boundary_test.go b/go/journal/reader_filter_boundary_test.go new file mode 100644 index 00000000..13ea7843 --- /dev/null +++ b/go/journal/reader_filter_boundary_test.go @@ -0,0 +1,384 @@ +package journal + +import ( + "fmt" + "os" + "path/filepath" + "testing" +) + +func directoryFilterBoundaryFixture(t *testing.T, compact, merged bool) string { + t.Helper() + dir := t.TempDir() + times := [2][3]uint64{{1000, 2000, 3000}, {4000, 5000, 6000}} + if merged { + times = [2][3]uint64{{1000, 3000, 5000}, {2000, 4000, 6000}} + } + for i, rows := range times { + opts := testOptions() + opts.Compact = compact + opts.FileID[0] += byte(i) + if merged { + opts.SeqnumID[0] += byte(i) + } else { + opts.HeadSeqnum = uint64(3*i + 1) + } + w, err := Create(filepath.Join(dir, fmt.Sprintf("%d.journal", i)), opts) + if err != nil { + t.Fatal(err) + } + for _, time := range rows { + if err := w.Append([]Field{StringField("COMMON", "all"), StringField("MESSAGE", fmt.Sprintf("row-%d", time))}, EntryOptions{RealtimeUsec: time, MonotonicUsec: time}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + return dir +} + +func TestDirectoryFilterMutationKeepsEmittedAnchor(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, merged := range []bool{false, true} { + dir := directoryFilterBoundaryFixture(t, compact, merged) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, backward := range []bool{false, true} { + for _, mutation := range []string{"install", "replace", "and", "or"} { + t.Run(fmt.Sprintf("compact=%v/merge=%v/mode=%d/facade=%v/back=%v/%s", compact, merged, mode, facadeAPI, backward, mutation), func(t *testing.T) { + dr, err := OpenDirectoryWithOptions(dir, DefaultReaderOptions().WithAccessMode(mode).WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if dr.nonOverlapping == merged { + t.Fatal("fixture selects wrong directory path") + } + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + } + addValue := func(value string) { + if facade != nil { + if err := SdJournalAddMatch(facade, []byte(value)); err != nil { + t.Fatal(err) + } + } else { + dr.AddMatch([]byte(value)) + } + } + if mutation == "replace" { + addValue("COMMON=all") + } + for want := uint64(1000); want <= 3000; want += 1000 { + if ok, err := positionTestStep(dr, facade, false); err != nil || !ok { + t.Fatalf("initial step=%v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("initial time=%d/%v, want %d", got, err, want) + } + } + if mutation == "replace" { + addValue("MESSAGE=row-1000") + addValue("MESSAGE=row-5000") + } else if mutation == "or" { + addValue("MESSAGE=row-1000") + } else { + addValue("COMMON=all") + } + if mutation == "and" || mutation == "or" { + if facade != nil { + if mutation == "and" { + err = SdJournalAddConjunction(facade) + } else { + err = SdJournalAddDisjunction(facade) + } + } else if mutation == "and" { + err = dr.AddConjunction() + } else { + err = dr.AddDisjunction() + } + if err != nil { + t.Fatal(err) + } + if mutation == "and" { + addValue("MESSAGE=row-1000") + } + addValue("MESSAGE=row-5000") + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != 3000 { + t.Fatalf("mutation changed selected row: %d/%v", got, err) + } + wants := []uint64{4000, 5000, 6000} + if backward { + wants = []uint64{2000, 1000} + } + if mutation != "install" { + wants = []uint64{5000} + if backward { + wants = []uint64{1000} + } + } + for _, want := range wants { + if ok, err := positionTestStep(dr, facade, backward); err != nil || !ok { + t.Fatalf("mutated step=%v/%v, want %d", ok, err, want) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("mutated time=%d/%v, want %d", got, err, want) + } + } + if ok, err := positionTestStep(dr, facade, backward); err != nil || ok { + t.Fatalf("mutated EOF=%v/%v", ok, err) + } + }) + } + } + } + } + } + } +} + +func TestDirectoryFilterMutationKeepsFailedRealtimeSeek(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, merged := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/merge=%v/mode=%d/facade=%v/back=%v", compact, merged, mode, facadeAPI, backward), func(t *testing.T) { + dir := directoryFilterBoundaryFixture(t, compact, merged) + dr, err := OpenDirectoryWithOptions(dir, DefaultReaderOptions().WithAccessMode(mode).WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + } + addMatch := func() { + if facade != nil { + if err := SdJournalAddMatch(facade, []byte("COMMON=all")); err != nil { + t.Fatal(err) + } + } else { + dr.AddMatch([]byte("COMMON=all")) + } + } + addMatch() + if err := dr.SeekRealtimeUsec(3500); err != nil { + t.Fatal(err) + } + fileIndex, rowIndex, want := 1, 0, uint64(4000) + if backward { + fileIndex, rowIndex, want = 0, 2, 3000 + } + if merged { + rowIndex = 1 + } + path := filepath.Join(dir, fmt.Sprintf("%d.journal", fileIndex)) + var offset uint64 + for _, reader := range dr.files { + if reader.file.Name() == path { + offset = reader.entryOffsets[rowIndex] + } + } + if offset == 0 { + t.Fatal("missing target ENTRY") + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteAt([]byte{objectTypeData}, int64(offset)); err != nil { + t.Fatal(err) + } + if ok, err := positionTestStep(dr, facade, backward); err == nil || ok { + t.Fatalf("damaged seek=%v/%v", ok, err) + } + addMatch() + if _, err := file.WriteAt([]byte{objectTypeEntry}, int64(offset)); err != nil { + t.Fatal(err) + } + if ok, err := positionTestStep(dr, facade, backward); err != nil || !ok { + t.Fatalf("repaired seek=%v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("repaired time=%d/%v, want %d", got, err, want) + } + }) + } + } + } + } + } +} + +func TestDirectoryFlushMatchesDiscardsPendingRealtimeSeek(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, merged := range []bool{false, true} { + dir := directoryFilterBoundaryFixture(t, compact, merged) + for _, facadeAPI := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/merge=%v/facade=%v", compact, merged, facadeAPI), func(t *testing.T) { + dr, err := OpenDirectory(dir) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if dr.nonOverlapping == merged { + t.Fatal("fixture selects wrong directory path") + } + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + if err := SdJournalAddMatch(facade, []byte("COMMON=all")); err != nil { + t.Fatal(err) + } + if err := SdJournalSeekRealtimeUsec(facade, 5500); err != nil { + t.Fatal(err) + } + if err := SdJournalFlushMatches(facade); err != nil { + t.Fatal(err) + } + } else { + dr.AddMatch([]byte("COMMON=all")) + if err := dr.SeekRealtimeUsec(5500); err != nil { + t.Fatal(err) + } + dr.FlushMatches() + } + if ok, err := positionTestStep(dr, facade, false); err != nil || !ok { + t.Fatalf("flushed step: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("flushed time=%d/%v, want 1000", got, err) + } + }) + } + } + } +} + +func TestDirectoryFilteredEOFReversalKeepsNearestMatch(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, merged := range []bool{false, true} { + dir := directoryFilterBoundaryFixture(t, compact, merged) + for _, facadeAPI := range []bool{false, true} { + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/merge=%v/facade=%v/back=%v", compact, merged, facadeAPI, backward), func(t *testing.T) { + dr, err := OpenDirectory(dir) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if dr.nonOverlapping == merged { + t.Fatal("fixture selects wrong directory path") + } + dr.AddMatch([]byte("COMMON=all")) + if backward { + if err := dr.SeekTail(); err != nil { + t.Fatal(err) + } + } + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + } + for i := uint64(1); i <= 6; i++ { + want := 1000 * i + if backward { + want = 7000 - want + } + if ok, err := positionTestStep(dr, facade, backward); err != nil || !ok { + t.Fatalf("step=%v/%v, want %d", ok, err, want) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("time=%d/%v, want %d", got, err, want) + } + } + for i := 0; i < 2; i++ { + if ok, err := positionTestStep(dr, facade, backward); err != nil || ok { + t.Fatalf("EOF %d=%v/%v", i, ok, err) + } + } + want := uint64(5000) + if backward { + want = 2000 + } + if ok, err := positionTestStep(dr, facade, !backward); err != nil || !ok { + t.Fatalf("reverse=%v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("reverse time=%d/%v, want %d", got, err, want) + } + }) + } + } + } + } +} + +func TestFlushMatchesClearsSelectedRow(t *testing.T) { + dir := directoryFilterBoundaryFixture(t, false, false) + for _, directory := range []bool{false, true} { + for _, facadeAPI := range []bool{false, true} { + t.Run(fmt.Sprintf("directory=%v/facade=%v", directory, facadeAPI), func(t *testing.T) { + var reader sdReader + var step interface { + Step() (bool, error) + StepBack() (bool, error) + } + if directory { + dr, err := OpenDirectory(dir) + if err != nil { + t.Fatal(err) + } + reader, step = dr, dr + } else { + r, err := OpenFile(filepath.Join(dir, "0.journal")) + if err != nil { + t.Fatal(err) + } + reader, step = r, r + } + defer reader.Close() + reader.AddMatch([]byte("COMMON=all")) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(reader) + } + if ok, err := positionTestStep(step, facade, false); err != nil || !ok { + t.Fatalf("first: %v/%v", ok, err) + } + if _, err := reader.GetEntry(); err != nil { + t.Fatal(err) + } + if facade != nil { + if err := SdJournalFlushMatches(facade); err != nil { + t.Fatal(err) + } + } else { + reader.FlushMatches() + } + if _, err := reader.GetRealtimeUsec(); err == nil { + t.Fatal("flushed reader retains timestamp") + } + if _, err := reader.GetCursor(); err == nil { + t.Fatal("flushed reader retains cursor") + } + if _, err := reader.GetEntry(); err == nil { + t.Fatal("flushed reader retains entry") + } + if ok, err := positionTestStep(step, facade, false); err != nil || !ok { + t.Fatalf("rewound: %v/%v", ok, err) + } + if got, err := reader.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("rewound time=%d/%v", got, err) + } + }) + } + } +} diff --git a/go/journal/reader_filter_native_test.go b/go/journal/reader_filter_native_test.go new file mode 100644 index 00000000..75aec90f --- /dev/null +++ b/go/journal/reader_filter_native_test.go @@ -0,0 +1,828 @@ +package journal + +import ( + "bytes" + "encoding/binary" + "errors" + "fmt" + "os" + "path/filepath" + "slices" + "syscall" + "testing" +) + +type filterAccessTrace struct { + readerAccessor + forbidden map[uint64]bool + faultAt uint64 + fail bool + calls int +} + +func (a *filterAccessTrace) check(offset, size uint64) error { + a.calls++ + if a.fail && offset <= a.faultAt && a.faultAt-offset < size { + return syscall.EIO + } + for at := range a.forbidden { + if offset <= at && at-offset < size { + return fmt.Errorf("unexpected unrelated DATA read at %d", at) + } + } + return nil +} +func (a *filterAccessTrace) readAt(dst []byte, offset uint64) error { + if err := a.check(offset, uint64(len(dst))); err != nil { + return err + } + return a.readerAccessor.readAt(dst, offset) +} +func (a *filterAccessTrace) tempSlice(offset, size uint64) ([]byte, error) { + if err := a.check(offset, size); err != nil { + return nil, err + } + return a.readerAccessor.tempSlice(offset, size) +} + +func filterTestReader(t *testing.T, data []byte, mode ReaderAccessMode, live bool) (*Reader, string) { + t.Helper() + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(1024).WithMaxWindows(1).WithSnapshot(!live)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + if r.accessor != nil { + if err := r.Close(); err != nil { + t.Error(err) + } + } + }) + return r, path +} + +func filterTestDataOffset(t *testing.T, r *Reader, payload string) uint64 { + t.Helper() + offset, found, err := r.findDataOffsetByPayload([]byte(payload)) + if err != nil || !found { + t.Fatalf("lookup %s: %v/%v", payload, found, err) + } + return offset +} + +func TestNativeFilterSelectiveAccessAndRestart(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("%v/%d", compact, mode), func(t *testing.T) { + r, _ := filterTestReader(t, source, mode, false) + unrelated := filterTestDataOffset(t, r, "MESSAGE=row-0001") + common := filterTestDataOffset(t, r, "COMMON=all") + array := binary.LittleEndian.Uint64(source[common+48:]) + trace := &filterAccessTrace{readerAccessor: r.accessor, forbidden: map[uint64]bool{unrelated: true}} + r.accessor = trace + r.AddMatch([]byte("SPARSE=some")) + if got := recoveryTestTimes(t, r, false); !slices.Equal(got, []uint64{1000, 1500, 2004}) { + t.Fatalf("sparse=%v", got) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 2004 { + t.Fatalf("EOF position=%d/%v", got, err) + } + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + if got := recoveryTestTimes(t, r, true); !slices.Equal(got, []uint64{2004, 1500, 1000}) { + t.Fatalf("reverse=%v", got) + } + for _, tc := range []struct { + time uint64 + back bool + want uint64 + }{{1250, false, 1500}, {1750, true, 1500}, {0, false, 1000}, {9999, true, 2004}} { + if err := r.SeekRealtimeUsec(tc.time); err != nil { + t.Fatal(err) + } + var ok bool + var err error + if tc.back { + ok, err = r.StepBack() + } else { + ok, err = r.Step() + } + if err != nil || !ok { + t.Fatalf("seek=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != tc.want { + t.Fatalf("time=%d/%v want%d", got, err, tc.want) + } + } + r.FlushMatches() + r.AddMatch([]byte("COMMON=all")) + trace.faultAt, trace.fail = array, true + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("first dense step eagerly read array: %v/%v", ok, err) + } + if ok, err := r.Step(); ok || !errors.Is(err, syscall.EIO) { + t.Fatalf("posting error=%v/%v", ok, err) + } + trace.fail = false + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("posting retry=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1001 { + t.Fatalf("retry time=%d/%v", got, err) + } + r.FlushMatches() + r.AddMatch([]byte("MISSING=value")) + for _, offset := range r.entryOffsets { + trace.forbidden[offset] = true + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("absent=%v/%v", ok, err) + } + }) + } + } +} + +func TestNativeFilterBuildRetryAndMalformed(t *testing.T) { + source, _ := recoveryTestFixture(t, false) + r, path := filterTestReader(t, source, ReaderAccessReadAt, false) + common := filterTestDataOffset(t, r, "COMMON=all") + trace := &filterAccessTrace{readerAccessor: r.accessor, faultAt: common, fail: true} + r.accessor = trace + r.AddMatch([]byte("COMMON=all")) + if err := r.AddConjunction(); !errors.Is(err, syscall.EIO) { + t.Fatalf("operator error=%v", err) + } + trace.fail = false + if err := r.AddConjunction(); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("build retry=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("first=%d/%v", got, err) + } + r.AddMatch([]byte("MESSAGE=row-0500")) + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("replacement=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1500 { + t.Fatalf("replacement time=%d/%v", got, err) + } + for _, op := range []string{"invalid", "and", "or"} { + r.FlushMatches() + switch op { + case "invalid": + r.AddMatch([]byte("invalid")) + case "and": + if err := r.AddConjunction(); err != nil { + t.Fatal(err) + } + case "or": + if err := r.AddDisjunction(); err != nil { + t.Fatal(err) + } + } + if ok, err := r.Step(); ok || !errors.Is(err, ErrMalformedFilter) { + t.Fatalf("%s=%v/%v", op, ok, err) + } + } + r.FlushMatches() + r.AddMatch([]byte("COMMON=all")) + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(make([]byte, 16), 104) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("remove hash table=%v/%v", err, closeErr) + } + if ok, err := r.Step(); ok || err == nil { + t.Fatalf("missing hash table=%v/%v", ok, err) + } +} + +func TestNativeFilterRecoveredSparsePositionAndRepair(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + last := arrays[len(arrays)-1].entries[len(arrays[len(arrays)-1].entries)-1] + damaged := bytes.Clone(source) + clear(damaged[last : last+16]) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("%v/%d", compact, mode), func(t *testing.T) { + r, path := filterTestReader(t, damaged, mode, true) + r.AddMatch([]byte("SPARSE=some")) + for _, want := range []uint64{1000, 1500} { + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("step=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("time=%d/%v", got, err) + } + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("EOF=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1500 { + t.Fatalf("EOF position=%d/%v", got, err) + } + if ok, err := r.StepBack(); err != nil || !ok { + t.Fatalf("reverse=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("reverse position=%d/%v", got, err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("forward=%v/%v", ok, err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("second EOF=%v/%v", ok, err) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(source[last:last+16], int64(last)) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("repair=%v/%v", err, closeErr) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("unchanged EOF=%v/%v", ok, err) + } + if err := r.SeekRealtimeUsec(1501); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("repair step=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 2004 { + t.Fatalf("repair position=%d/%v", got, err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("repair repeated=%v/%v", ok, err) + } + if r.RecoveryStatus().Incomplete() { + t.Fatal("repair remains degraded") + } + }) + } + } +} + +func TestNativeFilterDirectoryCandidateModesAndMutation(t *testing.T) { + dir := t.TempDir() + a := filepath.Join(dir, "a.journal") + b := filepath.Join(dir, "b.journal") + createMessageJournal(t, a, []messageRow{{"keep", 1000}, {"other", 1002}, {"other", 1004}}) + createMessageJournal(t, b, []messageRow{{"other", 1001}, {"keep", 1003}, {"other", 1005}}) + dr, err := OpenFiles([]string{a, b}) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + dr.AddMatch([]byte("MESSAGE=keep")) + for _, tc := range []struct { + filtered bool + want uint64 + }{{true, 1000}, {false, 1001}, {true, 1003}, {false, 1004}} { + var ok bool + var err error + if tc.filtered { + ok, err = dr.Step() + } else { + err = dr.Next() + ok = err == nil + } + if err != nil || !ok { + t.Fatalf("step=%v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != tc.want { + t.Fatalf("time=%d/%v want%d", got, err, tc.want) + } + } + dr.FlushMatches() + if ok, err := dr.Step(); err != nil || !ok { + t.Fatalf("flush=%v/%v", ok, err) + } + dr.AddMatch([]byte("MESSAGE=keep")) + if ok, err := dr.Step(); err != nil || !ok { + t.Fatalf("mutation=%v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != 1003 { + t.Fatalf("mutation time=%d/%v", got, err) + } + cursor, err := dr.GetCursor() + if err != nil { + t.Fatal(err) + } + if ok, err := dr.Step(); err != nil || ok { + t.Fatalf("end=%v/%v", ok, err) + } + if _, err := dr.GetRealtimeUsec(); err == nil { + t.Fatal("directory EOF retained a current row") + } + if ok, err := dr.TestCursor(cursor); !errors.Is(err, ErrEndOfEntries) || ok { + t.Fatalf("EOF cursor=%v/%v", ok, err) + } +} + +func TestNativeFilterSnapshotRepairAndAfterOpenLoss(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + last := arrays[len(arrays)-1].entries[len(arrays[len(arrays)-1].entries)-1] + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("snapshot/%v/%d", compact, mode), func(t *testing.T) { + damaged := bytes.Clone(source) + clear(damaged[last : last+16]) + r, path := filterTestReader(t, damaged, mode, false) + r.AddMatch([]byte("MESSAGE=row-1004")) + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("missing row=%v/%v", ok, err) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(source[last:last+16], int64(last)) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("repair=%v/%v", err, closeErr) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("unchanged EOF=%v/%v", ok, err) + } + if err := r.SeekRealtimeUsec(2004); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("snapshot repaired row=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 2004 { + t.Fatalf("snapshot time=%d/%v", got, err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("snapshot repeated=%v/%v", ok, err) + } + }) + } + t.Run(fmt.Sprintf("after-open/%v", compact), func(t *testing.T) { + r, path := filterTestReader(t, source, ReaderAccessMmap, true) + pair := filterTestDataOffset(t, r, "TAIL_PAIR=some") + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + if _, err := file.WriteAt(make([]byte, 16), int64(last)); err != nil { + t.Fatal(err) + } + if _, err := file.WriteAt(make([]byte, 8), int64(pair+48)); err != nil { + t.Fatal(err) + } + if err := file.Close(); err != nil { + t.Fatal(err) + } + r.AddMatch([]byte("TAIL_PAIR=some")) + r.AddMatch([]byte("COMMON=all")) + if got := recoveryTestTimes(t, r, false); !slices.Equal(got, []uint64{2003}) { + t.Fatalf("surviving pair=%v", got) + } + if r.RecoveryStatus().RecoveredEntries != 1004 { + t.Fatalf("status=%+v", r.RecoveryStatus()) + } + }) + } +} + +func TestNativeFilterOrphanAndDuplicatePostingParity(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("orphan/%v/%d", compact, mode), func(t *testing.T) { + damaged := bytes.Clone(source) + binary.LittleEndian.PutUint64(damaged[152:], 1) + r, _ := filterTestReader(t, damaged, mode, false) + r.AddMatch([]byte("MESSAGE=row-0500")) + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("orphan=%v/%v", ok, err) + } + entry, err := r.GetEntry() + if err != nil || entry.Realtime != 1500 || string(entry.Fields["MESSAGE"]) != "row-0500" { + t.Fatalf("orphan entry=%v/%v", entry, err) + } + cursor, err := r.GetCursor() + if err != nil { + t.Fatal(err) + } + if err := r.SeekCursor(cursor); err != nil { + t.Fatal(err) + } + // Realtime seeking uses the global array, so the valid orphan is + // reachable by head/tail exact traversal, not by this time bound. + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + if ok, err := r.StepBack(); err != nil || !ok { + t.Fatalf("reverse orphan=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1500 { + t.Fatalf("reverse orphan time=%d/%v", got, err) + } + if err := r.Previous(); err != nil { + t.Fatal(err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("raw before orphan=%d/%v", got, err) + } + }) + } + t.Run(fmt.Sprintf("duplicate/%v", compact), func(t *testing.T) { + r, _ := filterTestReader(t, source, ReaderAccessMmap, false) + common := filterTestDataOffset(t, r, "COMMON=all") + array := binary.LittleEndian.Uint64(source[common+48:]) + damaged := bytes.Clone(source) + slot := damaged[array+24:] + if compact { + binary.LittleEndian.PutUint32(slot, uint32(arrays[0].entries[0])) + } else { + binary.LittleEndian.PutUint64(slot, arrays[0].entries[0]) + } + candidate, _ := filterTestReader(t, damaged, ReaderAccessMmap, false) + candidate.AddMatch([]byte("COMMON=all")) + got := recoveryTestTimes(t, candidate, false) + if len(got) != 1004 || got[0] != 1000 || got[1] != 1002 || got[len(got)-1] != 2004 { + t.Fatalf("duplicate traversal count=%d", len(got)) + } + }) + } +} + +func TestNativeFilterDirectoryModeSwitchPreservesNonoverlapPosition(t *testing.T) { + dir := t.TempDir() + a := filepath.Join(dir, "a.journal") + b := filepath.Join(dir, "b.journal") + for _, tc := range []struct { + path string + head uint64 + rows []messageRow + }{ + {a, 1, []messageRow{{"other", 1000}, {"keep", 1001}, {"other", 1002}}}, + {b, 4, []messageRow{{"other", 2000}, {"keep", 2001}, {"other", 2002}}}, + } { + opts := testOptions() + opts.HeadSeqnum = tc.head + w, err := Create(tc.path, opts) + if err != nil { + t.Fatal(err) + } + for i, row := range tc.rows { + if err := w.Append([]Field{StringField("MESSAGE", row.message)}, EntryOptions{RealtimeUsec: row.realtime, MonotonicUsec: tc.head + uint64(i)}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + dr, err := OpenFiles([]string{a, b}) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if !dr.nonOverlapping { + t.Fatal("fixture did not exercise sequential proof") + } + dr.AddMatch([]byte("MESSAGE=keep")) + if ok, err := dr.Step(); err != nil || !ok { + t.Fatalf("filtered=%v/%v", ok, err) + } + for _, want := range []uint64{1002, 2000, 2001, 2002} { + if err := dr.Next(); err != nil { + t.Fatal(err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("raw=%d/%v want%d", got, err, want) + } + } +} + +func TestNativeFilterDoesNotDecodeUnrelatedCompressedData(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + r, _ := filterTestReader(t, source, ReaderAccessMmap, false) + message := filterTestDataOffset(t, r, "MESSAGE=row-0000") + damaged := bytes.Clone(source) + damaged[message+1] = objectCompressedZSTD + candidate, _ := filterTestReader(t, damaged, ReaderAccessReadAt, false) + candidate.AddMatch([]byte("SPARSE=some")) + if ok, err := candidate.Step(); err != nil || !ok { + t.Fatalf("metadata movement=%v/%v", ok, err) + } + cursor, err := candidate.GetCursor() + if err != nil { + t.Fatal(err) + } + if err := candidate.SeekCursor(cursor); err != nil { + t.Fatal(err) + } + if got, err := candidate.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("cursor time=%d/%v", got, err) + } + if _, err := candidate.GetEntry(); err == nil { + t.Fatal("payload control did not detect malformed compression") + } + } +} + +func TestNativeFilterRecoveredTailAnchorSurvivesRepair(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("%d/%v", mode, backward), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "source.journal") + createMessageJournal(t, path, []messageRow{{"only", 1000}}) + source, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + array := binary.LittleEndian.Uint64(source[176:]) + entry := binary.LittleEndian.Uint64(source[array+24:]) + damaged := bytes.Clone(source) + clear(damaged[entry : entry+16]) + r, target := filterTestReader(t, damaged, mode, false) + r.AddMatch([]byte("MESSAGE=only")) + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + file, err := os.OpenFile(target, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(source[entry:entry+16], int64(entry)) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("repair=%v/%v", err, closeErr) + } + var ok bool + if backward { + ok, err = r.StepBack() + } else { + ok, err = r.Step() + } + if err != nil || ok != backward { + t.Fatalf("tail step=%v/%v", ok, err) + } + time, timeErr := r.GetRealtimeUsec() + if backward && (timeErr != nil || time != 1000) { + t.Fatalf("tail time=%d/%v", time, timeErr) + } + if !backward && timeErr == nil { + t.Fatal("tail became a resolved row after repair") + } + }) + } + } +} + +func TestNativeFilterNewGroupsObserveHealthyAppend(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, grow := range []bool{false, true} { + t.Run(fmt.Sprintf("%v/%d/grow=%v", compact, mode, grow), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + opts := testOptions() + opts.Compact = compact + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + defer w.Close() + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: 1000, MonotonicUsec: 1}); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(1024).WithMaxWindows(1)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + oldSize := r.fileSize + r.AddMatch([]byte("TARGET=new")) + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("absent=%v/%v", ok, err) + } + fields := []Field{StringField("COMMON", "all"), StringField("TARGET", "new")} + if grow { + fields = append([]Field{StringField("LARGE", string(bytes.Repeat([]byte{'x'}, int(oldSize)+4096)))}, fields...) + } + if err := w.Append(fields, EntryOptions{RealtimeUsec: 1001, MonotonicUsec: 2}); err != nil { + t.Fatal(err) + } + if grow { + info, err := os.Stat(path) + if err != nil || uint64(info.Size()) <= oldSize { + t.Fatalf("no physical growth: %v", err) + } + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("resolved absent leaf changed=%v/%v", ok, err) + } + trace := &filterAccessTrace{readerAccessor: r.accessor, forbidden: map[uint64]bool{r.entryOffsets[0]: true}} + r.accessor = trace + r.AddMatch([]byte("TARGET=new")) + if err := r.AddConjunction(); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("new group=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1001 { + t.Fatalf("new group time=%d/%v", got, err) + } + trace.forbidden = nil + if err := r.Previous(); err != nil { + t.Fatal(err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("raw before new selected row=%d/%v", got, err) + } + r.FlushMatches() + r.AddMatch([]byte("COMMON=all")) + if got := recoveryTestTimes(t, r, false); !slices.Equal(got, []uint64{1000, 1001}) { + t.Fatalf("existing DATA=%v", got) + } + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: 1002, MonotonicUsec: 3}); err != nil { + t.Fatal(err) + } + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + if got := recoveryTestTimes(t, r, false); !slices.Equal(got, []uint64{1000, 1001}) { + t.Fatalf("captured population changed=%v", got) + } + r.AddMatch([]byte("COMMON=all")) + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("replacement existing DATA=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1002 { + t.Fatalf("replacement time=%d/%v", got, err) + } + }) + } + } + } +} + +func TestNativeFilterGroupVisibilityKeepsRawGrowth(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, initial := range []int{0, 1} { + for _, failBuild := range []bool{false, true} { + for _, clearFilter := range []bool{false, true} { + for _, refresh := range []bool{false, true} { + t.Run(fmt.Sprintf("%v/%d/initial=%d/failure=%v/clear=%v/refresh=%v", compact, mode, initial, failBuild, clearFilter, refresh), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + opts := testOptions() + opts.Compact = compact + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + defer w.Close() + appendRow := func(stamp uint64) { + t.Helper() + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: stamp, MonotonicUsec: stamp}); err != nil { + t.Fatal(err) + } + } + if initial != 0 { + appendRow(1000) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + if initial != 0 { + if err := r.Next(); err != nil { + t.Fatal(err) + } + } + appendRow(1001) + if failBuild { + probe, err := OpenFile(path) + if err != nil { + t.Fatal(err) + } + data := filterTestDataOffset(t, probe, "COMMON=all") + if err := probe.Close(); err != nil { + t.Fatal(err) + } + trace := &filterAccessTrace{readerAccessor: r.accessor, faultAt: data, fail: true} + r.accessor = trace + r.AddMatch([]byte("COMMON=all")) + if err := r.AddDisjunction(); !errors.Is(err, syscall.EIO) { + t.Fatalf("group failure=%v", err) + } + trace.fail = false + } else { + r.AddMatch([]byte("MISSING=value")) + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("absent=%v/%v", ok, err) + } + } + if clearFilter { + r.FlushMatches() + } + if refresh { + if changed, err := r.Refresh(); err != nil || !changed { + t.Fatalf("population refresh=%v/%v", changed, err) + } + } + want := []uint64{1001} + if clearFilter && initial != 0 { + want = []uint64{1000, 1001} + } + for _, stamp := range want { + if err := r.Next(); err != nil { + t.Fatal(err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != stamp { + t.Fatalf("raw row=%d/%v want%d", got, err, stamp) + } + } + if err := r.Next(); !errors.Is(err, errEndOfEntries) { + t.Fatalf("raw EOF=%v", err) + } + }) + } + } + } + } + } + } +} + +func TestNativeFilterValidatesSelectedEntryBodyAfterOpen(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, damage := range []string{"partial-body", "unaligned-body", "outside-arena"} { + t.Run(fmt.Sprintf("%v/%d/%s", compact, mode, damage), func(t *testing.T) { + r, path := filterTestReader(t, source, mode, true) + last := r.entryOffsets[len(r.entryOffsets)-1] + if damage == "partial-body" { + if err := os.Truncate(path, int64(last+entryObjectHeaderSize)); err != nil { + t.Fatal(err) + } + } else { + size := binary.LittleEndian.Uint64(source[last+8:]) + 1 + if damage == "outside-arena" { + size = r.header.headerSize + r.header.arenaSize + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + var buf [8]byte + binary.LittleEndian.PutUint64(buf[:], size) + _, err = file.WriteAt(buf[:], int64(last+8)) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("modify=%v/%v", err, closeErr) + } + } + r.AddMatch([]byte("SPARSE=some")) + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + ok, err := r.StepBack() + if damage != "partial-body" { + if ok || !errors.Is(err, errCorruptObject) || r.RecoveryStatus().Incomplete() { + t.Fatalf("malformed selected row=%v/%v/%+v", ok, err, r.RecoveryStatus()) + } + return + } + if err != nil || !ok { + t.Fatalf("recovered selection=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1500 { + t.Fatalf("surviving sparse row=%d/%v", got, err) + } + if status := r.RecoveryStatus(); !status.Incomplete() || status.RecoveredEntries != 1004 { + t.Fatalf("recovery=%+v", status) + } + }) + } + } + } +} diff --git a/go/journal/reader_filter_position_test.go b/go/journal/reader_filter_position_test.go new file mode 100644 index 00000000..0046e18b --- /dev/null +++ b/go/journal/reader_filter_position_test.go @@ -0,0 +1,764 @@ +package journal + +import ( + "encoding/binary" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestDirectoryFilteredMovementAfterClose(t *testing.T) { + for _, facadeAPI := range []bool{false, true} { + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("facade=%v/back=%v", facadeAPI, backward), func(t *testing.T) { + dr, err := OpenDirectory(createDirectoryReaderJournals(t, 1, 1)) + if err != nil { + t.Fatal(err) + } + dr.AddMatch([]byte("PRIORITY=6")) + if backward { + if err := dr.SeekTail(); err != nil { + t.Fatal(err) + } + } + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + err = SdJournalClose(facade) + } else { + err = dr.Close() + } + if err != nil { + t.Fatal(err) + } + for attempt := 0; attempt < 2; attempt++ { + var ok bool + if facade != nil { + var n int + if backward { + n, err = SdJournalPrevious(facade) + } else { + n, err = SdJournalNext(facade) + } + ok = n != 0 + } else if backward { + ok, err = dr.StepBack() + } else { + ok, err = dr.Step() + } + if ok || !errors.Is(err, errInvalidJournal) { + t.Fatalf("closed movement %d: %v/%v, want invalid journal", attempt, ok, err) + } + } + }) + } + } +} + +func TestDirectoryPositionAfterEntryLoadError(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, filtered := range []bool{false, true} { + for _, backward := range []bool{false, true} { + for _, merged := range []bool{false, true} { + // Unfiltered backward time-search error propagation is tracked in SOW-0045. + if merged && backward && !filtered { + continue + } + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v/filter=%v/back=%v/merge=%v", compact, mode, facadeAPI, filtered, backward, merged), func(t *testing.T) { + dir := t.TempDir() + times := [2][3]uint64{{1000, 2000, 3000}, {4000, 5000, 6000}} + if merged { + times = [2][3]uint64{{1000, 3000, 5000}, {2000, 4000, 6000}} + } + for i, rows := range times { + opts := testOptions() + opts.Compact = compact + if merged { + opts.SeqnumID[0] += byte(i) + } else { + opts.HeadSeqnum = uint64(3*i + 1) + } + opts.FileID[0] += byte(i) + w, err := Create(filepath.Join(dir, fmt.Sprintf("%d.journal", i)), opts) + if err != nil { + t.Fatal(err) + } + for _, time := range rows { + fields := []Field{StringField("MESSAGE", fmt.Sprintf("row-%d", time)), StringField("COMMON", "all"), StringField("PADDING", strings.Repeat("p", 2048)+fmt.Sprint(time))} + if err := w.Append(fields, EntryOptions{RealtimeUsec: time, MonotonicUsec: time}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + dr, err := OpenDirectoryWithOptions(dir, DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(1024).WithMaxWindows(1).WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if dr.nonOverlapping == merged { + t.Fatal("fixture selects wrong directory path") + } + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + } + if filtered { + dr.AddMatch([]byte("COMMON=all")) + } + step := func(back bool) (bool, error) { + if facade != nil { + if back { + n, err := SdJournalPrevious(facade) + return n != 0, err + } + n, err := SdJournalNext(facade) + return n != 0, err + } + if back { + return dr.StepBack() + } + return dr.Step() + } + before, targetFile, targetRow := uint64(3000), 1, 0 + initialBack := backward + if merged { + initialBack = !backward + if !backward { + targetRow = 1 + } + } else if backward { + before, targetFile, targetRow = 4000, 0, 2 + } + if initialBack { + if err := dr.SeekTail(); err != nil { + t.Fatal(err) + } + } + for { + if ok, err := step(initialBack); err != nil || !ok { + t.Fatalf("initial movement: %v/%v", ok, err) + } + if time, err := dr.GetRealtimeUsec(); err != nil { + t.Fatal(err) + } else if time == before { + break + } + } + cursor, err := dr.GetCursor() + if err != nil { + t.Fatal(err) + } + wantPayload := fmt.Sprintf("MESSAGE=row-%d", before) + assertPosition := func() { + if time, err := dr.GetRealtimeUsec(); err != nil || time != before { + t.Fatalf("position: %d/%v, want %d", time, err, before) + } + if got, err := dr.GetCursor(); err != nil || got != cursor { + t.Fatalf("cursor changed: %q/%v", got, err) + } + payload, ok, err := dr.GetEntryPayload([]byte("MESSAGE")) + if err != nil || !ok || string(payload) != wantPayload { + t.Fatalf("payload: %q/%v/%v, want %q", payload, ok, err, wantPayload) + } + } + assertPosition() + r := dr.files[targetFile] + offset := r.entryOffsets[targetRow] + // Evict a prefetched copy so this call observes the controlled damage. + if _, err := r.accessor.tempSlice(0, 1); err != nil { + t.Fatal(err) + } + file, err := os.OpenFile(filepath.Join(dir, fmt.Sprintf("%d.journal", targetFile)), os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteAt([]byte{2}, int64(offset)); err != nil { + t.Fatal(err) + } + if ok, err := step(backward); err == nil || ok { + t.Fatalf("damaged entry: %v/%v", ok, err) + } + assertPosition() + if _, err := file.WriteAt([]byte{3}, int64(offset)); err != nil { + t.Fatal(err) + } + assertPosition() + for _, want := range []uint64{before + 1000, before + 2000} { + if backward { + want = 2*before - want + } + if ok, err := step(backward); err != nil || !ok { + t.Fatalf("retry: %v/%v", ok, err) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("retry time: %d/%v, want %d", got, err, want) + } + } + }) + } + } + } + } + } + } +} + +func positionTestStep(r interface { + Step() (bool, error) + StepBack() (bool, error) +}, facade *sdJournal, backward bool) (bool, error) { + if facade != nil { + var n int + var err error + if backward { + n, err = SdJournalPrevious(facade) + } else { + n, err = SdJournalNext(facade) + } + return n != 0, err + } + if backward { + return r.StepBack() + } + return r.Step() +} + +func TestDirectoryInitialEntryErrorRetry(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, filtered := range []bool{false, true} { + for _, scenario := range []string{"head", "tail", "time-forward", "time-backward"} { + if !filtered && strings.HasPrefix(scenario, "time-") { + continue + } + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v/filter=%v/%s", compact, mode, facadeAPI, filtered, scenario), func(t *testing.T) { + dir := t.TempDir() + for i := 0; i < 2; i++ { + opts := testOptions() + opts.Compact = compact + opts.HeadSeqnum = uint64(3*i + 1) + opts.FileID[0] += byte(i) + w, err := Create(filepath.Join(dir, fmt.Sprintf("%d.journal", i)), opts) + if err != nil { + t.Fatal(err) + } + for j := 0; j < 3; j++ { + time := uint64(1000 * (3*i + j + 1)) + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: time, MonotonicUsec: time}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + dr, err := OpenDirectoryWithOptions(dir, DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(1024).WithMaxWindows(1).WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(dr) + } + if filtered { + dr.AddMatch([]byte("COMMON=all")) + } + backward := scenario == "tail" || scenario == "time-backward" + fileIndex, rowIndex := 0, 0 + want := []uint64{1000, 2000, 3000, 4000, 5000, 6000} + switch scenario { + case "tail": + fileIndex, rowIndex = 1, 2 + if err := dr.SeekTail(); err != nil { + t.Fatal(err) + } + case "time-forward": + fileIndex, rowIndex = 1, 0 + want = want[1:] + if err := dr.SeekRealtimeUsec(1500); err != nil { + t.Fatal(err) + } + case "time-backward": + fileIndex, rowIndex = 1, 1 + want = want[:5] + if err := dr.SeekRealtimeUsec(5500); err != nil { + t.Fatal(err) + } + } + step := func() (bool, error) { + if facade != nil { + if backward { + n, err := SdJournalPrevious(facade) + return n != 0, err + } + n, err := SdJournalNext(facade) + return n != 0, err + } + if backward { + return dr.StepBack() + } + return dr.Step() + } + offset := dr.files[fileIndex].entryOffsets[rowIndex] + file, err := os.OpenFile(filepath.Join(dir, fmt.Sprintf("%d.journal", fileIndex)), os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := dr.files[fileIndex].accessor.tempSlice(0, 1); err != nil { + t.Fatal(err) + } + if _, err := file.WriteAt([]byte{2}, int64(offset)); err != nil { + t.Fatal(err) + } + if ok, err := step(); err == nil || ok { + t.Fatalf("initial damage: %v/%v", ok, err) + } + if _, err := dr.GetCursor(); err == nil { + t.Fatal("failed first movement selected a row") + } + if _, err := file.WriteAt([]byte{3}, int64(offset)); err != nil { + t.Fatal(err) + } + for i := range want { + expected := want[i] + if backward { + expected = want[len(want)-1-i] + } + if ok, err := step(); err != nil || !ok { + t.Fatalf("retry: %v/%v, want %d", ok, err, expected) + } + if got, err := dr.GetRealtimeUsec(); err != nil || got != expected { + t.Fatalf("retry time: %d/%v, want %d", got, err, expected) + } + } + if ok, err := step(); err != nil || ok { + t.Fatalf("EOF: %v/%v", ok, err) + } + }) + } + } + } + } + } +} + +func positionTestWant(t *testing.T, r *Reader, facade *sdJournal, backward bool, want uint64) { + t.Helper() + if ok, err := positionTestStep(r, facade, backward); err != nil || !ok { + t.Fatalf("step backward=%v: %v/%v", backward, ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != want { + t.Fatalf("backward=%v: time=%d/%v, want %d", backward, got, err, want) + } +} + +func TestNativeFilterReverseAnchor(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, conjunction := range []bool{false, true} { + for _, scenario := range []string{"install", "install-eof", "filtered-eof"} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v/and=%v/%s", compact, mode, facadeAPI, conjunction, scenario), func(t *testing.T) { + r, _ := filterTestReader(t, source, mode, false) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + if scenario != "filtered-eof" { + for i := 0; i < 1005; i++ { + positionTestWant(t, r, facade, false, 1000+uint64(i)) + } + } + if conjunction { + r.AddMatch([]byte("COMMON=all")) + } + r.AddMatch([]byte("SPARSE=some")) + if scenario == "filtered-eof" { + for _, want := range []uint64{1000, 1500, 2004} { + positionTestWant(t, r, facade, false, want) + } + } + if scenario != "install" { + if ok, err := positionTestStep(r, facade, false); err != nil || ok { + t.Fatalf("EOF=%v/%v", ok, err) + } + } + positionTestWant(t, r, facade, true, 1500) + positionTestWant(t, r, facade, false, 2004) + positionTestWant(t, r, facade, true, 1500) + positionTestWant(t, r, facade, true, 1000) + if ok, err := positionTestStep(r, facade, true); err != nil || ok { + t.Fatalf("BOF=%v/%v", ok, err) + } + positionTestWant(t, r, facade, false, 1500) + }) + } + } + } + } + } +} + +func TestReaderPositionAfterEntryLoadError(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + var entries []uint64 + for _, array := range arrays { + entries = append(entries, array.entries...) + } + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, sparse := range []bool{false, true} { + for _, backward := range []bool{false, true} { + for _, damage := range []string{"type", "size"} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v/sparse=%v/back=%v/%s", compact, mode, facadeAPI, sparse, backward, damage), func(t *testing.T) { + r, path := filterTestReader(t, source, mode, false) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + before := uint64(1499) + if backward { + before = 1501 + } + if sparse { + r.AddMatch([]byte("SPARSE=some")) + before = 1000 + if backward { + before = 2004 + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + } + } else { + r.AddMatch([]byte("COMMON=all")) + if err := r.SeekRealtimeUsec(before); err != nil { + t.Fatal(err) + } + } + positionTestWant(t, r, facade, backward, before) + offset, damaged := entries[500], []byte{2} + if damage == "size" { + offset += 8 + damaged = make([]byte, 8) + binary.LittleEndian.PutUint64(damaged, 8) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteAt(damaged, int64(offset)); err != nil { + t.Fatal(err) + } + if ok, err := positionTestStep(r, facade, backward); err == nil || ok { + t.Fatalf("damaged entry=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != before { + t.Fatalf("position while damaged=%d/%v, want %d", got, err, before) + } + if _, err := file.WriteAt(source[offset:offset+uint64(len(damaged))], int64(offset)); err != nil { + t.Fatal(err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != before { + t.Fatalf("position after error=%d/%v, want %d", got, err, before) + } + positionTestWant(t, r, facade, backward, 1500) + }) + } + } + } + } + } + } +} + +func TestNativeFilterNestedDirectionChange(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v", compact, mode, facadeAPI), func(t *testing.T) { + r, _ := filterTestReader(t, source, mode, false) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + if err := r.SeekRealtimeUsec(1500); err != nil { + t.Fatal(err) + } + positionTestWant(t, r, facade, false, 1500) + r.AddMatch([]byte("MESSAGE=row-0500")) + r.AddMatch([]byte("SPARSE=some")) + if err := r.AddDisjunction(); err != nil { + t.Fatal(err) + } + r.AddMatch([]byte("TAIL_PAIR=some")) + positionTestWant(t, r, facade, false, 2003) + positionTestWant(t, r, facade, true, 1500) + positionTestWant(t, r, facade, false, 2003) + positionTestWant(t, r, facade, false, 2004) + positionTestWant(t, r, facade, true, 2003) + positionTestWant(t, r, facade, true, 1500) + }) + } + } + } +} + +func TestNativeFilterResolvedBackwardSkipsUnvisitedLeaf(t *testing.T) { + source, _ := recoveryTestFixture(t, false) + probe, _ := filterTestReader(t, source, ReaderAccessReadAt, false) + common := filterTestDataOffset(t, probe, "COMMON=all") + array := binary.LittleEndian.Uint64(source[common+48:]) + source[array] = objectTypeData + for _, facadeAPI := range []bool{false, true} { + t.Run(fmt.Sprintf("facade=%v", facadeAPI), func(t *testing.T) { + r, _ := filterTestReader(t, source, ReaderAccessReadAt, false) + if err := r.SeekRealtimeUsec(1500); err != nil { + t.Fatal(err) + } + positionTestWant(t, r, nil, false, 1500) + r.AddMatch([]byte("COMMON=all")) + r.AddMatch([]byte("ZZ_ABSENT=x")) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + if ok, err := positionTestStep(r, facade, true); err != nil || ok { + t.Fatalf("resolved absent AND: %v/%v", ok, err) + } + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + if ok, err := positionTestStep(r, facade, true); err == nil || ok { + t.Fatalf("tail must prepare every leaf: %v/%v", ok, err) + } + }) + } +} + +func TestNativeFilterResolvedEOFDoesNotRescan(t *testing.T) { + source, _ := recoveryTestFixture(t, false) + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("back=%v", backward), func(t *testing.T) { + r, _ := filterTestReader(t, source, ReaderAccessReadAt, false) + anchor := uint64(2004) + if backward { + anchor = 1000 + } + if err := r.SeekRealtimeUsec(anchor); err != nil { + t.Fatal(err) + } + positionTestWant(t, r, nil, false, anchor) + r.AddMatch([]byte("COMMON=all")) + trace := &filterAccessTrace{readerAccessor: r.accessor} + r.accessor = trace + if ok, err := positionTestStep(r, nil, backward); err != nil || ok { + t.Fatalf("first boundary: %v/%v", ok, err) + } + calls := trace.calls + for i := 0; i < 2; i++ { + if ok, err := positionTestStep(r, nil, backward); err != nil || ok { + t.Fatalf("repeated boundary: %v/%v", ok, err) + } + } + if trace.calls != calls { + t.Fatalf("boundary reread postings: %d -> %d", calls, trace.calls) + } + if err := r.SeekRealtimeUsec(1500); err != nil { + t.Fatal(err) + } + if err := r.Next(); err != nil { + t.Fatal(err) + } + want := uint64(1501) + if backward { + want = 1499 + } + positionTestWant(t, r, nil, backward, want) + }) + } +} + +func TestNativeFilterRealtimeProbeBound(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v/back=%v", compact, mode, facadeAPI, backward), func(t *testing.T) { + r, _ := filterTestReader(t, source, mode, false) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + r.AddMatch([]byte("COMMON=all")) + if err := r.filter.build(r); err != nil { + t.Fatal(err) + } + trace := &filterAccessTrace{readerAccessor: r.accessor} + r.accessor = trace + if err := r.SeekRealtimeUsec(1500); err != nil { + t.Fatal(err) + } + positionTestWant(t, r, facade, backward, 1500) + t.Logf("time-seek reads=%d", trace.calls) + if trace.calls > 120 { + t.Fatalf("time seek made %d reads; want at most 120 for 1005 postings", trace.calls) + } + want := uint64(1501) + if backward { + want = 1499 + } + positionTestWant(t, r, facade, backward, want) + }) + } + } + } + } +} + +func TestNativeFilterRealtimeDuplicateBounds(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + index := uint64(0) + for _, array := range arrays { + for _, entry := range array.entries { + binary.LittleEndian.PutUint64(source[entry+24:], 1000+2*(index/7)) + index++ + } + } + binary.LittleEndian.PutUint64(source[192:], 1000+2*((index-1)/7)) + for _, live := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/live=%v/mode=%d/facade=%v", compact, live, mode, facadeAPI), func(t *testing.T) { + r, _ := filterTestReader(t, source, mode, live) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + r.AddMatch([]byte("COMMON=all")) + for _, tc := range []struct { + time uint64 + back bool + seq uint64 + }{{0, false, 1}, {0, true, 0}, {999, true, 0}, {1000, false, 1}, {1000, true, 7}, {1001, false, 8}, {1001, true, 7}, {1142, false, 498}, {1142, true, 504}, {1143, false, 505}, {1143, true, 504}, {1286, false, 1002}, {1286, true, 1005}, {1287, false, 0}, {^uint64(0), false, 0}, {^uint64(0), true, 1005}} { + if err := r.SeekRealtimeUsec(tc.time); err != nil { + t.Fatal(err) + } + ok, err := positionTestStep(r, facade, tc.back) + if err != nil || ok != (tc.seq != 0) { + t.Fatalf("seek %d/back=%v: %v/%v, want seq%d", tc.time, tc.back, ok, err, tc.seq) + } + if ok { + entry, err := r.GetEntry() + if err != nil || entry.Seqnum != tc.seq { + t.Fatalf("seek %d/back=%v: entry=%v/%v, want seq%d", tc.time, tc.back, entry, err, tc.seq) + } + } + } + }) + } + } + } + } +} + +func TestNativeFilterRealtimeSlotDiscoveryAndRetry(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + probe, _ := filterTestReader(t, source, ReaderAccessReadAt, false) + data := filterTestDataOffset(t, probe, "COMMON=all") + if err := probe.Close(); err != nil { + t.Fatal(err) + } + width := uint64(8) + if compact { + width = 4 + } + array := binary.LittleEndian.Uint64(source[data+48:]) + first := array + 24 + remaining := binary.LittleEndian.Uint64(source[data+56:]) - 1 + middle := first + minUint64(remaining, (binary.LittleEndian.Uint64(source[array+8:])-24)/width)/2*width + var last uint64 + for remaining > 0 { + count := minUint64(remaining, (binary.LittleEndian.Uint64(source[array+8:])-24)/width) + last = array + 24 + (count-1)*width + remaining -= count + array = binary.LittleEndian.Uint64(source[array+16:]) + } + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, facadeAPI := range []bool{false, true} { + for _, backward := range []bool{false, true} { + for _, probed := range []bool{false, true} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/facade=%v/back=%v/probed=%v", compact, mode, facadeAPI, backward, probed), func(t *testing.T) { + at := first + if backward { + at = last + } + if probed { + at = middle + } + damaged := append([]byte(nil), source...) + clear(damaged[at : at+width]) + r, path := filterTestReader(t, damaged, mode, false) + var facade *sdJournal + if facadeAPI { + facade = newSdJournal(r) + } + r.AddMatch([]byte("COMMON=all")) + if err := r.SeekRealtimeUsec(1500); err != nil { + t.Fatal(err) + } + if probed { + if ok, err := positionTestStep(r, facade, backward); err == nil || ok { + t.Fatalf("visited zero slot=%v/%v", ok, err) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(source[at:at+width], int64(at)) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("repair=%v/%v", err, closeErr) + } + } + positionTestWant(t, r, facade, backward, 1500) + if !probed { + if backward { + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + } else { + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + positionTestWant(t, r, facade, false, 1000) + } + if ok, err := positionTestStep(r, facade, backward); err == nil || ok { + t.Fatalf("ordinary traversal skipped zero slot=%v/%v", ok, err) + } + } + }) + } + } + } + } + } +} diff --git a/go/journal/reader_filter_posting.go b/go/journal/reader_filter_posting.go new file mode 100644 index 00000000..5fe3a311 --- /dev/null +++ b/go/journal/reader_filter_posting.go @@ -0,0 +1,262 @@ +package journal + +// Posting cursors retain scalar node metadata, never mapped slices or a copy of +// every matching offset. The inline posting makes the first healthy step lazy. +type filterPostingCursor struct { + dataOffset uint64 + declared uint64 + count uint64 + inline uint64 + head uint64 + nodes []filterPostingNode + highest uint64 + node int + slot uint64 + value uint64 +} + +type filterPostingNode struct { + offset uint64 + next uint64 + remaining uint64 + length uint64 +} + +func (r *Reader) newFilterPostingCursor(offset uint64) (filterPostingCursor, error) { + posting, err := r.buildFilterPostingCursor(offset) + if err != nil && filterTailFailure(err) { + changed, recoveryErr := r.refreshFilterRecovery() + if recoveryErr != nil { + return filterPostingCursor{}, recoveryErr + } + if changed { + return r.buildFilterPostingCursor(offset) + } + } + return posting, err +} + +func (r *Reader) buildFilterPostingCursor(offset uint64) (filterPostingCursor, error) { + header, err := r.readDataHeaderAt(offset) + if err != nil { + return filterPostingCursor{}, err + } + p := filterPostingCursor{dataOffset: offset, declared: header.nEntries, count: header.nEntries, inline: header.entryOffset, head: header.entryArrayOffset, node: -1, value: header.entryOffset} + if (p.declared == 0) != (p.inline == 0) { + return filterPostingCursor{}, readerTailError{p.inline} + } + if p.declared == 0 { + return p, nil + } + if r.recovery.Incomplete() { + p.count, err = r.recoveredFilterPostingCount(header) + if err != nil { + return filterPostingCursor{}, err + } + } + if p.count > 1 && p.head == 0 { + return filterPostingCursor{}, readerTailError{0} + } + return p, nil +} + +func (r *Reader) recoveredFilterPostingCount(header dataHeader) (uint64, error) { + if len(r.entryOffsets) == 0 || header.nEntries == 0 { + return 0, nil + } + last := r.entryOffsets[len(r.entryOffsets)-1] + if header.entryOffset > last { + return 0, nil + } + previous, count := header.entryOffset, uint64(1) + remaining := header.nEntries - 1 + highest := header.entryArrayOffset + for offset := header.entryArrayOffset; remaining > 0; { + if offset == 0 { + if previous == last { + return count, nil + } + return 0, errCorruptObject + } + array, capacity, err := r.readOffsetArrayHeader(offset) + if err != nil { + if _, tail := readerTailFailure(err); tail && previous == last { + return count, nil + } + return 0, err + } + wanted := minUint64(remaining, capacity) + start := offset + offsetArrayObjectHeaderSize + available := uint64(0) + if start <= r.fileSize { + available = minUint64(wanted, (r.fileSize-start)/r.offsetArrayItemSize()) + } + for i := uint64(0); i < available; i++ { + entry, err := r.filterPostingSlot(offset, i) + if err != nil { + return 0, err + } + if entry == 0 { + if previous == last { + return count, nil + } + return 0, errCorruptObject + } + if entry%8 != 0 || entry <= previous { + return 0, errCorruptObject + } + if entry > last { + return count, nil + } + previous = entry + count++ + } + if available < wanted { + if previous == last { + return count, nil + } + return 0, readerBoundsError{start, wanted * r.offsetArrayItemSize(), r.fileSize} + } + remaining -= wanted + if remaining > 0 && array.nextArrayOffset != 0 { + if array.nextArrayOffset <= highest { + if err := rejectArrayRevisit(r, header.entryArrayOffset, offset, array.nextArrayOffset, errCorruptObject); err != nil { + return 0, err + } + } else { + highest = array.nextArrayOffset + } + } + offset = array.nextArrayOffset + } + return count, nil +} + +func (r *Reader) filterPostingSlot(offset, index uint64) (uint64, error) { + size := r.offsetArrayItemSize() + position, ok := checkedAdd(offset, offsetArrayObjectHeaderSize) + if !ok || index > (^uint64(0)-position)/size { + return 0, errCorruptObject + } + buf, err := r.readSlice(position+index*size, size) + if err != nil { + return 0, err + } + return entryOffsetArrayItem(buf, size), nil +} + +func (p *filterPostingCursor) appendNode(r *Reader) error { + remaining, offset := p.count-1, p.head + if len(p.nodes) != 0 { + last := p.nodes[len(p.nodes)-1] + remaining = last.remaining - last.length + offset = last.next + if remaining > 0 && offset <= p.highest { + for _, node := range p.nodes { + if node.offset == offset { + return errCorruptObject + } + } + } + } + if remaining == 0 || offset == 0 { + return readerTailError{offset} + } + array, capacity, err := r.readOffsetArrayHeader(offset) + if err != nil { + return err + } + if !r.recovery.Incomplete() && (offset > r.fileSize || array.object.size > r.fileSize-offset) { + return readerBoundsError{offset, array.object.size, r.fileSize} + } + p.nodes = append(p.nodes, filterPostingNode{offset: offset, next: array.nextArrayOffset, remaining: remaining, length: minUint64(remaining, capacity)}) + p.highest = max(p.highest, offset) + return nil +} + +func (p *filterPostingCursor) reset(r *Reader, direction Direction) error { + p.node, p.slot, p.value = -1, 0, p.inline + if p.count < 2 || direction == DirectionForward { + return nil + } + for len(p.nodes) == 0 || p.nodes[len(p.nodes)-1].remaining > p.nodes[len(p.nodes)-1].length { + if err := p.appendNode(r); err != nil { + return err + } + } + index := len(p.nodes) - 1 + slot := p.nodes[index].length - 1 + p.node, p.slot, p.value = index, slot, 0 + return nil +} + +func (p *filterPostingCursor) readSlot(r *Reader, node int, slot uint64) (uint64, error) { + value, err := r.filterPostingSlot(p.nodes[node].offset, slot) + if err != nil { + return 0, err + } + if value == 0 { + return 0, readerTailError{0} + } + return value, nil +} + +func (p *filterPostingCursor) seek(r *Reader, needle uint64, direction Direction) (uint64, error) { + if p.count == 0 { + return 0, nil + } + if p.value == 0 { + if p.node == -2 { + if err := p.reset(r, DirectionBackward); err != nil { + return 0, err + } + } + if p.value == 0 { + value, err := p.readSlot(r, p.node, p.slot) + if err != nil { + return 0, err + } + p.value = value + } + } + for { + if direction == DirectionForward && p.value >= needle || direction == DirectionBackward && p.value <= needle { + return p.value, nil + } + node, slot := p.node, p.slot + if direction == DirectionForward { + if node < 0 { + if p.count == 1 { + return 0, nil + } + node, slot = 0, 0 + } else if slot+1 < p.nodes[node].length { + slot++ + } else if p.nodes[node].remaining > p.nodes[node].length { + node, slot = node+1, 0 + } else { + return 0, nil + } + if node == len(p.nodes) { + if err := p.appendNode(r); err != nil { + return 0, err + } + } + } else if node < 0 { + return 0, nil + } else if slot > 0 { + slot-- + } else if node > 0 { + node-- + slot = p.nodes[node].length - 1 + } else { + p.node, p.slot, p.value = -1, 0, p.inline + continue + } + value, err := p.readSlot(r, node, slot) + if err != nil { + return 0, err + } + p.node, p.slot, p.value = node, slot, value + } +} diff --git a/go/journal/reader_filter_realtime.go b/go/journal/reader_filter_realtime.go new file mode 100644 index 00000000..7c49ab18 --- /dev/null +++ b/go/journal/reader_filter_realtime.go @@ -0,0 +1,103 @@ +package journal + +func seekFilterRealtime(expr filterExpr, r *Reader, needle uint64, direction Direction) (uint64, error) { + switch value := expr.(type) { + case *matchExpr: + return value.posting.seekRealtime(r, needle, direction) + case falseExpr: + return 0, nil + case andExpr: + initial := needle + for i := range value { + index := i + if direction == DirectionBackward { + index = len(value) - 1 - i + } + candidate, err := seekFilterRealtime(value[index], r, needle, direction) + if err != nil || candidate == 0 { + return 0, err + } + needle = candidate + } + if needle == initial { + return needle, nil + } + return value.seek(r, needle, direction) + case orExpr: + var best uint64 + for _, child := range value { + candidate, err := seekFilterRealtime(child, r, needle, direction) + if err != nil { + return 0, err + } + if candidate != 0 && (best == 0 || direction == DirectionForward && candidate < best || direction == DirectionBackward && candidate > best) { + best = candidate + } + } + return best, nil + default: + return 0, ErrMalformedFilter + } +} + +func (p *filterPostingCursor) seekRealtime(r *Reader, needle uint64, direction Direction) (uint64, error) { + p.node, p.slot, p.value = -1, 0, p.inline + if p.count == 0 { + return 0, nil + } + if direction == DirectionForward && p.inline >= needle { + return p.inline, nil + } + if direction == DirectionBackward && p.inline > needle { + return 0, nil + } + if p.count == 1 { + if direction == DirectionBackward { + return p.inline, nil + } + return 0, nil + } + for index := 0; ; index++ { + if index == len(p.nodes) { + if err := p.appendNode(r); err != nil { + return 0, err + } + } + node := p.nodes[index] + left, right := uint64(0), node.length + for left < right { + mid := left + (right-left)/2 + value, err := p.readSlot(r, index, mid) + if err != nil { + return 0, err + } + if value < needle || direction == DirectionBackward && value == needle { + left = mid + 1 + } else { + right = mid + } + } + if direction == DirectionBackward && left == 0 { + return p.value, nil + } + slot := left + if direction == DirectionBackward || left == node.length { + slot-- + } + value, err := p.readSlot(r, index, slot) + if err != nil { + return 0, err + } + // Retain the last visited posting even when a sibling wins an OR query. + p.node, p.slot, p.value = index, slot, value + if left < node.length { + return value, nil + } + if node.remaining == node.length { + if direction == DirectionBackward { + return value, nil + } + return 0, nil + } + } +} diff --git a/go/journal/reader_filter_recovery.go b/go/journal/reader_filter_recovery.go new file mode 100644 index 00000000..7d3c7856 --- /dev/null +++ b/go/journal/reader_filter_recovery.go @@ -0,0 +1,79 @@ +package journal + +import ( + "errors" + "io" +) + +func filterTailFailure(err error) bool { + _, eligible := readerTailFailure(err) + return eligible || errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) +} + +// A new group sees current native indexes. Existing compiled leaves keep their +// captured population; refreshing visibility must not walk every ENTRY. +func (r *Reader) prepareFilterGroup() error { + snapshot := r.refreshSnapshot() + header, size, err := r.readCurrentHeader() + if err != nil { + r.restoreRefreshSnapshot(snapshot) + return err + } + if r.options.Bounds == ReaderBoundsSnapshot || r.staged { + size = minUint64(size, r.accessor.initialSize()) + r.accessor.restoreVisibleBounds(readerAccessorVisibleSnapshot{fileSize: size}) + } + r.applyRefreshState(header, size) + return nil +} + +// Recovery must reread published bytes even when the file size and header +// counters are unchanged. Snapshot readers retain their original byte extent. +func (r *Reader) refreshFilterRecovery() (bool, error) { + if !r.allowRecovery || r.accessor == nil { + return false, nil + } + snapshot := r.refreshSnapshot() + header, size, err := r.readCurrentHeader() + if err != nil { + r.restoreRefreshSnapshot(snapshot) + return false, err + } + if r.options.Bounds == ReaderBoundsSnapshot || r.staged { + size = minUint64(size, r.accessor.initialSize()) + r.accessor.restoreVisibleBounds(readerAccessorVisibleSnapshot{fileSize: size}) + } + r.applyRefreshState(header, size) + if err := r.loadEntryArray(); err != nil { + r.restoreRefreshSnapshot(snapshot) + return false, err + } + r.entryIndex = snapshot.index + if snapshot.index >= 0 && snapshot.index >= len(snapshot.offsets) { + r.entryIndex = len(r.entryOffsets) + } else if snapshot.index >= 0 && snapshot.index >= len(r.entryOffsets) && r.filter != nil { + r.filter.overrideOffset = snapshot.offsets[snapshot.index] + r.entryIndex = -2 + } + r.clampEntryIndex() + changed := !sameRecoveryState(r.recovery, snapshot.recovery) + if changed { + if r.filter != nil { + r.filter.stale = true + r.filter.walkValid = false + } + if r.directoryOrderDirty != nil { + *r.directoryOrderDirty = true + } + } + return changed, nil +} + +func (r *Reader) refreshFilterOffsets() error { + if r.options.Bounds == ReaderBoundsSnapshot || r.staged { + _, err := r.refreshFilterRecovery() + return err + } + _, err := r.refreshEntryOffsets() + return err +} diff --git a/go/journal/reader_filter_refresh_test.go b/go/journal/reader_filter_refresh_test.go new file mode 100644 index 00000000..393d838d --- /dev/null +++ b/go/journal/reader_filter_refresh_test.go @@ -0,0 +1,373 @@ +package journal + +// Tests that rewrite damaged source bytes exercise synthetic fault recovery, +// not a supported in-place journal repair lifecycle. Actual publication tests +// use Writer.Append; accessor I/O injection separately checks retry behavior. + +import ( + "bytes" + "encoding/binary" + "errors" + "fmt" + "os" + "path/filepath" + "syscall" + "testing" +) + +func TestNativeFilterPendingRecoveryBuildKeepsAllGroups(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("%v/%d", compact, mode), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + opts := testOptions() + opts.Compact = compact + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + for i, fields := range [][]Field{{StringField("COMMON", "all")}, {StringField("COMMON", "all"), StringField("TARGET", "only")}} { + if err := w.Append(fields, EntryOptions{RealtimeUsec: 1000 + uint64(i), MonotonicUsec: 100 + uint64(i)}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + source, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + target := filterTestDataOffset(t, r, "TARGET=only") + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if err := file.Truncate(int64(target + 8)); err != nil { + t.Fatal(err) + } + if _, err := r.Refresh(); err != nil { + t.Fatal(err) + } + if !r.RecoveryStatus().Incomplete() { + t.Fatal("expected one-row prefix") + } + r.AddMatch([]byte("COMMON=all")) + if err := r.AddConjunction(); err != nil { + t.Fatal(err) + } + partial := bytes.Clone(source) + binary.LittleEndian.PutUint64(partial[target+40:], 0) + if _, err := file.WriteAt(partial, 0); err != nil { + t.Fatal(err) + } + r.AddMatch([]byte("TARGET=only")) + if ok, err := r.Step(); ok || err == nil { + t.Fatalf("broken DATA=%v/%v", ok, err) + } + if r.RecoveryStatus().Incomplete() { + t.Fatal("ENTRY repair was not observed") + } + if _, err := file.WriteAt(source[target+40:target+48], int64(target+40)); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("repaired pending AND=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1001 { + t.Fatalf("row=%d/%v", got, err) + } + }) + } + } +} + +func TestNativeFilterKnownEmptyDataKeepsRecoveryIdentity(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("%v/%d", compact, mode), func(t *testing.T) { + r, path := filterTestReader(t, source, mode, true) + common := filterTestDataOffset(t, r, "COMMON=all") + last := r.entryOffsets[len(r.entryOffsets)-1] + r.AddMatch([]byte("COMMON=all")) + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("initial=%v/%v", ok, err) + } + partial := bytes.Clone(source) + clear(partial[common+40 : common+64]) + clear(partial[last : last+16]) + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteAt(partial, 0); err != nil { + t.Fatal(err) + } + if err := file.Truncate(int64(last + 8)); err != nil { + t.Fatal(err) + } + if _, err := r.Refresh(); err != nil { + t.Fatal(err) + } + if !r.RecoveryStatus().Incomplete() { + t.Fatal("expected missing tail") + } + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("empty retained DATA=%v/%v", ok, err) + } + if _, err := file.WriteAt(source, 0); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("unchanged EOF=%v/%v", ok, err) + } + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("repaired retained DATA=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1000 { + t.Fatalf("row=%d/%v", got, err) + } + }) + } + } +} + +func nativeFilterLiveReader(t *testing.T, mode ReaderAccessMode, initial int) (*Writer, *Reader, string) { + t.Helper() + path := filepath.Join(t.TempDir(), "system.journal") + w, err := Create(path, testOptions()) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = w.Close() }) + for i := 0; i < initial; i++ { + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: 1000 + uint64(i), MonotonicUsec: 100 + uint64(i)}); err != nil { + t.Fatal(err) + } + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = r.Close() }) + return w, r, path +} + +func TestNativeFilterIORetryPreservesCapturedPopulation(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprint(mode), func(t *testing.T) { + w, r, _ := nativeFilterLiveReader(t, mode, 2) + r.AddMatch([]byte("COMMON=all")) + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("first=%v/%v", ok, err) + } + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: 1002, MonotonicUsec: 102}); err != nil { + t.Fatal(err) + } + trace := &filterAccessTrace{readerAccessor: r.accessor, faultAt: r.entryOffsets[1], fail: true} + r.accessor = trace + if ok, err := r.Step(); ok || !errors.Is(err, syscall.EIO) { + t.Fatalf("injected failure=%v/%v", ok, err) + } + trace.fail = false + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("retry=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1001 { + t.Fatalf("retry row=%d/%v", got, err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("installed population widened=%v/%v", ok, err) + } + }) + } +} + +func TestNativeFilterRealtimeRecoveryRebuildsBeforeFirstResult(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, backward := range []bool{false, true} { + t.Run(fmt.Sprintf("%d/backward=%v", mode, backward), func(t *testing.T) { + w, r, path := nativeFilterLiveReader(t, mode, 1) + if err := w.Append([]Field{StringField("TARGET", "new")}, EntryOptions{RealtimeUsec: 1001, MonotonicUsec: 101}); err != nil { + t.Fatal(err) + } + probe, err := OpenFile(path) + if err != nil { + t.Fatal(err) + } + last := probe.entryOffsets[1] + if err := probe.Close(); err != nil { + t.Fatal(err) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(make([]byte, 16), int64(last)) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("remove tail=%v/%v", err, closeErr) + } + r.AddMatch([]byte("TARGET=new")) + if err := r.SeekRealtimeUsec(1000); err != nil { + t.Fatal(err) + } + var ok bool + if backward { + ok, err = r.StepBack() + } else { + ok, err = r.Step() + } + if err != nil || ok { + t.Fatalf("first recovered result=%v/%v", ok, err) + } + if status := r.RecoveryStatus(); !status.Incomplete() || status.RecoveredEntries != 1 { + t.Fatalf("recovery=%+v", status) + } + }) + } + } +} + +func TestNativeFilterRecoveryViewStaysConsistentAcrossLeaves(t *testing.T) { + w, r, path := nativeFilterLiveReader(t, ReaderAccessReadAt, 1) + if err := w.Append([]Field{StringField("COMMON", "all"), StringField("TARGET", "only")}, EntryOptions{RealtimeUsec: 1001, MonotonicUsec: 101}); err != nil { + t.Fatal(err) + } + r.AddMatch([]byte("COMMON=all")) + r.AddMatch([]byte("TARGET=only")) + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("build at tail=%v/%v", ok, err) + } + source, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + target := filterTestDataOffset(t, r, "TARGET=only") + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + if err := os.Truncate(path, int64(target)); err != nil { + t.Fatal(err) + } + if _, err := r.Refresh(); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err == nil || ok { + t.Fatalf("missing required DATA=%v/%v", ok, err) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(source, 0) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("repair=%v/%v", err, closeErr) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("consistent repair=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1001 { + t.Fatalf("repaired row=%d/%v", got, err) + } + if r.RecoveryStatus().Incomplete() { + t.Fatal("complete repair remained degraded") + } +} + +func TestNativeFilterSnapshotRestoresOriginalExtent(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("compact=%v/%d", compact, mode), func(t *testing.T) { + r, path := filterTestReader(t, source, mode, false) + last := r.entryOffsets[len(r.entryOffsets)-1] + if err := os.Truncate(path, int64(last+8)); err != nil { + t.Fatal(err) + } + r.AddMatch([]byte("SPARSE=some")) + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + if ok, err := r.StepBack(); err != nil || !ok || !r.RecoveryStatus().Incomplete() { + t.Fatalf("observed shrink=%v/%v/%+v", ok, err, r.RecoveryStatus()) + } + file, err := os.OpenFile(path, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + _, err = file.WriteAt(append(bytes.Clone(source), make([]byte, 4096)...), 0) + closeErr := file.Close() + if err != nil || closeErr != nil { + t.Fatalf("restore and grow=%v/%v", err, closeErr) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("restored snapshot=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 2004 { + t.Fatalf("restored row=%d/%v", got, err) + } + if status := r.RecoveryStatus(); status.Incomplete() { + t.Fatalf("restored status=%+v", status) + } + if got := r.accessor.size(); got != uint64(len(source)) { + t.Fatalf("snapshot extent=%d want%d", got, len(source)) + } + }) + } + } +} + +func TestNativeFilterSnapshotNewBuilderSeesPreallocatedPublication(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + path := filepath.Join(t.TempDir(), "system.journal") + w, err := Create(path, testOptions()) + if err != nil { + t.Fatal(err) + } + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: 1000, MonotonicUsec: 100}); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode).WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + if err := w.Append([]Field{StringField("TARGET", "new")}, EntryOptions{RealtimeUsec: 1001, MonotonicUsec: 101}); err != nil { + t.Fatal(err) + } + r.AddMatch([]byte("TARGET=new")) + if err := r.SeekRealtimeUsec(1000); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("new snapshot filter=%v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1001 { + t.Fatalf("new snapshot row=%d/%v", got, err) + } + if err := r.Close(); err != nil { + t.Fatal(err) + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } +} diff --git a/go/journal/reader_filter_step.go b/go/journal/reader_filter_step.go new file mode 100644 index 00000000..2f862996 --- /dev/null +++ b/go/journal/reader_filter_step.go @@ -0,0 +1,220 @@ +package journal + +import ( + "errors" + "sort" +) + +func (r *Reader) normalizeFilteredPosition(direction Direction) error { + if r.entryIndex != -2 || r.filter == nil { + return nil + } + offset := r.filter.overrideOffset + if uint64(len(r.entryOffsets)) != r.effectiveHeader().nEntries { + if err := r.refreshFilterOffsets(); err != nil { + return err + } + } + r.entryIndex = sort.Search(len(r.entryOffsets), func(i int) bool { return r.entryOffsets[i] >= offset }) + if direction == DirectionForward && (r.entryIndex >= len(r.entryOffsets) || r.entryOffsets[r.entryIndex] != offset) { + r.entryIndex-- + } + return nil +} + +func (r *Reader) stepFiltered(direction Direction) (bool, error) { + r.clearCurrentEntryState() + if r.accessor == nil { + return false, errInvalidJournal + } + if r.recovery != nil && r.recovery.eofValid { + if unchanged, err := r.unchangedRecoveryEOF(direction); err != nil || unchanged { + return false, err + } + } + f := r.filter + if f.retry { + // Retry with fresh bytes, including any cached by getters after failure. + r.accessor.invalidateCopiedWindows() + f.retry = false + } + for attempt := 0; ; attempt++ { + if err := f.build(r); err != nil { + f.retry = true + return false, err + } + value, err := r.filteredOffset(direction) + if err == nil && value != 0 { + index := r.entryIndex + if direction == DirectionForward { + index++ + } else { + index-- + } + if index < 0 || index >= len(r.entryOffsets) || r.entryOffsets[index] != value { + index = sort.Search(len(r.entryOffsets), func(i int) bool { return r.entryOffsets[i] >= value }) + } + if index >= len(r.entryOffsets) || r.entryOffsets[index] != value { + index = -2 + } + var header entryHeader + if err == nil { + header, err = r.readEntryHeaderAt(value) + if errors.Is(err, errCorruptObject) { + _, classified := r.validEntryObjectOffset(value) + if classified != nil { + err = classified + } + } + } + if err == nil { + err = r.validateFilteredEntryHeader(value, header) + } + if err == nil { + r.clearRecoveryEOF() + r.entryIndex = index + f.overrideOffset = value + r.currentHeader, r.currentHeaderOffset, r.currentHeaderValid = header, value, true + r.realtimeSeek = nil + f.walkOffset, f.walkValid = value, true + f.backward = direction == DirectionBackward + f.retry = false + return true, nil + } + } + if err != nil { + f.retry = true + f.walkValid = false + invalidateFilterWalk(f.expr) + if !filterTailFailure(err) || !r.allowRecovery { + return false, err + } + } + if attempt != 0 || err == nil && !r.recovery.Incomplete() { + if err == nil { + r.rememberRecoveryEOF(direction) + } + return false, err + } + changed, refreshErr := r.refreshFilterRecovery() + if refreshErr != nil { + return false, refreshErr + } + if !changed { + if err == nil { + r.rememberRecoveryEOF(direction) + } + return false, err + } + } +} + +func (r *Reader) validateFilteredEntryHeader(offset uint64, header entryHeader) error { + size := header.object.size + if offset < r.header.headerSize || offset&7 != 0 || size < entryObjectHeaderSize || + (size-entryObjectHeaderSize)&(r.entryItemSize()-1) != 0 { + return errCorruptObject + } + arenaEnd, ok := checkedAdd(r.header.headerSize, r.header.arenaSize) + if !ok || offset > arenaEnd || size > arenaEnd-offset { + return errCorruptObject + } + if offset > r.fileSize || size > r.fileSize-offset { + return readerBoundsError{offset, size, r.fileSize} + } + return nil +} + +func (r *Reader) filteredOffset(direction Direction) (uint64, error) { + f := r.filter + if r.effectiveHeader().nEntries == 0 && r.realtimeSeek == nil { + return 0, nil + } + needle := uint64(1) + reset := !f.walkValid || f.backward != (direction == DirectionBackward) + if r.realtimeSeek != nil { + var index int + var err error + if direction == DirectionForward { + index, err = r.filterRealtimeIndex(*r.realtimeSeek, false) + } else { + index, err = r.filterRealtimeIndex(*r.realtimeSeek, true) + index-- + } + if err != nil { + return 0, err + } + if f.stale { + if err := f.build(r); err != nil { + return 0, err + } + } + if index < 0 || index >= len(r.entryOffsets) { + return 0, nil + } + f.walkValid = false + return seekFilterRealtime(f.expr, r, r.entryOffsets[index], direction) + } else if r.entryIndex < 0 && r.entryIndex != -2 { + if direction == DirectionBackward { + return 0, nil + } + reset = true + } else if r.entryIndex >= len(r.entryOffsets) { + if direction == DirectionForward { + return 0, nil + } + needle = ^uint64(0) + reset = true + } else { + current, err := r.currentEntryOffset() + if err != nil { + return 0, err + } + reset = reset || f.walkOffset != current + if direction == DirectionForward { + if current == ^uint64(0) { + return 0, nil + } + needle = current + 1 + } else { + needle = current - 1 + } + if reset { + prepareResolvedFilterWalk(f.expr, direction) + f.walkOffset, f.walkValid = current, true + f.backward = direction == DirectionBackward + } + return f.expr.seek(r, needle, direction) + } + if reset { + f.walkValid = false + if err := f.expr.reset(r, direction); err != nil { + return 0, err + } + } + return f.expr.seek(r, needle, direction) +} + +// The filtered seek path must propagate metadata read failures instead of +// converting a failed binary-search comparison into an apparent result. +func (r *Reader) filterRealtimeIndex(usec uint64, after bool) (int, error) { + if uint64(len(r.entryOffsets)) != r.effectiveHeader().nEntries { + if err := r.refreshFilterOffsets(); err != nil { + return 0, err + } + } + left, right := 0, len(r.entryOffsets) + for left < right { + mid := left + (right-left)/2 + realtime, err := r.entryRealtimeAtIndex(mid) + if err != nil { + return 0, err + } + if realtime < usec || after && realtime == usec { + left = mid + 1 + } else { + right = mid + } + } + return left, nil +} diff --git a/go/journal/reader_header_access_test.go b/go/journal/reader_header_access_test.go new file mode 100644 index 00000000..609b4c1f --- /dev/null +++ b/go/journal/reader_header_access_test.go @@ -0,0 +1,316 @@ +package journal + +import ( + "bytes" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "testing" +) + +type entrySlotHeaderAccessTrace struct { + readerAccessor + readCalls, sliceCalls int + fail bool +} + +func (a *entrySlotHeaderAccessTrace) readAt(dst []byte, offset uint64) error { + a.readCalls++ + if a.fail { + copy(dst, bytes.Repeat([]byte{0xee}, 8)) + return io.ErrUnexpectedEOF + } + return a.readerAccessor.readAt(dst, offset) +} + +func (a *entrySlotHeaderAccessTrace) tempSlice(offset, size uint64) ([]byte, error) { + a.sliceCalls++ + if a.fail { + return nil, io.ErrUnexpectedEOF + } + return a.readerAccessor.tempSlice(offset, size) +} + +func entrySlotHeaderTestReader(t testing.TB, mode ReaderAccessMode, windowSize uint64, windows int, prepare func([]byte)) (*Reader, []byte) { + t.Helper() + payload := make([]byte, 256) + for i := range payload { + payload[i] = byte(13*i + 7) + } + if prepare != nil { + prepare(payload) + } + path := filepath.Join(t.TempDir(), "header-bytes") + if err := os.WriteFile(path, payload, 0o600); err != nil { + t.Fatal(err) + } + file, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + opts := DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(windowSize).WithMaxWindows(windows).WithSnapshot(true) + accessor, _, err := newReaderAccessor(file, opts) + if err != nil { + _ = file.Close() + t.Fatal(err) + } + r := &Reader{file: file, accessor: accessor, options: opts, fileSize: accessor.size()} + t.Cleanup(func() { + if r.accessor != nil { + if err := r.Close(); err != nil { + t.Error(err) + } + } + }) + return r, payload +} + +func TestEntrySlotHeaderAccess(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, windowSize := range []uint64{64, 63, 8, 0} { + for _, test := range []struct { + name string + offset uint64 + closed bool + bad bool + }{ + {name: "within", offset: 16}, + {name: "last-fitting", offset: 48}, + {name: "crossing", offset: 56}, + {name: "short", offset: 248, bad: true}, + {name: "overflow", offset: ^uint64(0) - 7, bad: true}, + {name: "closed-within", offset: 16, closed: true, bad: true}, + {name: "closed-crossing", offset: 56, closed: true, bad: true}, + } { + t.Run(fmt.Sprintf("mode=%d/window=%d/%s", mode, windowSize, test.name), func(t *testing.T) { + r, payload := entrySlotHeaderTestReader(t, mode, windowSize, 2, nil) + if test.closed { + if err := r.Close(); err != nil { + t.Fatal(err) + } + } + header, err := r.readEntrySlotHeader(test.offset) + if test.bad { + if header != nil || !errors.Is(err, errInvalidJournal) { + t.Fatalf("read = %x, %v; want nil, invalid journal", header, err) + } + return + } + if err != nil || !bytes.Equal(header, payload[test.offset:test.offset+objectHeaderSize]) { + t.Fatalf("header mismatch at %d: %x, %v", test.offset, header, err) + } + }) + } + } + } +} + +func TestEntrySlotHeaderCachedRouting(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, test := range []struct { + window, offset uint64 + slice bool + }{ + {64, 16, true}, {64, 48, true}, {64, 56, false}, + {63, 16, false}, {63, 56, false}, {8, 16, false}, + {0, 16, true}, {0, 56, true}, + } { + t.Run(fmt.Sprintf("mode=%d/window=%d/offset=%d", mode, test.window, test.offset), func(t *testing.T) { + r, payload := entrySlotHeaderTestReader(t, mode, test.window, 2, nil) + var warm [objectHeaderSize]byte + if err := r.readAt(warm[:], test.offset); err != nil { + t.Fatal(err) + } + trace := &entrySlotHeaderAccessTrace{readerAccessor: r.accessor} + r.accessor = trace + before := r.AccessStats() + for i := 0; i < 4; i++ { + header, err := r.readEntrySlotHeader(test.offset) + if err != nil || !bytes.Equal(header, payload[test.offset:test.offset+objectHeaderSize]) { + t.Fatalf("cached header mismatch: %x, %v", header, err) + } + } + after := r.AccessStats() + if trace.sliceCalls != map[bool]int{true: 4, false: 0}[test.slice] || + trace.readCalls != map[bool]int{true: 0, false: 4}[test.slice] { + t.Fatalf("route: slices=%d readAt=%d; native slice=%v", trace.sliceCalls, trace.readCalls, test.slice) + } + if after.TempCopyCount != before.TempCopyCount || after.MapCount != before.MapCount || + after.ReadWindowCount != before.ReadWindowCount || after.EvictionCount != before.EvictionCount { + t.Fatalf("cached header introduced I/O or eviction: before=%+v after=%+v", before, after) + } + }) + } + } +} + +func TestEntrySlotHeaderPartialFailureRetry(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, windowSize := range []uint64{64, 63} { + for _, offset := range []uint64{16, 56} { + t.Run(fmt.Sprintf("mode=%d/window=%d/offset=%d", mode, windowSize, offset), func(t *testing.T) { + r, payload := entrySlotHeaderTestReader(t, mode, windowSize, 2, nil) + if _, err := r.readEntrySlotHeader(offset); err != nil { + t.Fatal(err) + } + trace := &entrySlotHeaderAccessTrace{readerAccessor: r.accessor, fail: true} + r.accessor = trace + if header, err := r.readEntrySlotHeader(offset); header != nil || !errors.Is(err, io.ErrUnexpectedEOF) { + t.Fatalf("failed read = %x, %v; want nil, unexpected EOF", header, err) + } + trace.fail = false + for _, retry := range []uint64{120, 16} { + header, err := r.readEntrySlotHeader(retry) + if err != nil || !bytes.Equal(header, payload[retry:retry+objectHeaderSize]) { + t.Fatalf("retry header mismatch at %d: %x, %v", retry, header, err) + } + } + }) + } + } + } +} + +func TestEntrySlotHeaderRetainedRowsAndEviction(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, pinned := range []bool{false, true} { + t.Run(fmt.Sprintf("mode=%d/pinned=%v", mode, pinned), func(t *testing.T) { + r, payload := entrySlotHeaderTestReader(t, mode, 64, 1, nil) + var row []byte + if pinned { + var err error + row, err = r.readRowSlice(0, 8) + if err != nil { + t.Fatal(err) + } + } + before := r.AccessStats() + for _, offset := range []uint64{80, 144, 56, 120, 184, 16} { + header, err := r.readEntrySlotHeader(offset) + if err != nil || !bytes.Equal(header, payload[offset:offset+objectHeaderSize]) { + t.Fatalf("header mismatch at %d: %x, %v", offset, header, err) + } + if pinned && !bytes.Equal(row, payload[:8]) { + t.Fatal("header access changed retained row bytes") + } + } + after := r.AccessStats() + if after.ActiveWindows > 1 || after.MappedBytes > 4096 || after.ReadBufferBytes > 64 { + t.Fatalf("header reads exceeded cache bounds: %+v", after) + } + if pinned && after.RowPinnedWindows != 1 { + t.Fatalf("header reads changed row pins: %+v", after) + } + if !pinned && after.EvictionCount <= before.EvictionCount { + t.Fatal("header sequence did not exercise eviction") + } + }) + } + } +} + +func TestEntrySlotHeaderRawBytes(t *testing.T) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, window := range []uint64{64, 63} { + for _, offset := range []uint64{16, 56} { + for _, test := range []struct { + name string + typ, flags byte + }{ + {"zero", 0, 0}, {"flagged-zero", 0, 0x80}, {"data", objectTypeData, 0}, + } { + t.Run(fmt.Sprintf("mode=%d/window=%d/offset=%d/%s", mode, window, offset, test.name), func(t *testing.T) { + r, payload := entrySlotHeaderTestReader(t, mode, window, 2, func(payload []byte) { + clear(payload[offset : offset+objectHeaderSize]) + payload[offset], payload[offset+1] = test.typ, test.flags + }) + header, err := r.readEntrySlotHeader(offset) + if err != nil || !bytes.Equal(header, payload[offset:offset+objectHeaderSize]) { + t.Fatalf("raw header mismatch: %x, %v", header, err) + } + }) + } + } + } + } +} + +func TestEntrySlotHeaderLiveRefresh(t *testing.T) { + for _, compact := range []bool{false, true} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, window := range []uint64{0, 8, 63, 64} { + t.Run(fmt.Sprintf("compact=%v/mode=%d/window=%d", compact, mode, window), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "live.journal") + opts := testOptions() + opts.Compact = compact + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + defer w.Close() + readerOpts := DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(window).WithMaxWindows(1) + r, err := OpenFileWithOptions(path, readerOpts) + if err != nil { + t.Fatal(err) + } + defer r.Close() + if len(r.entryOffsets) != 0 || r.options.WindowSize == 0 { + t.Fatal("empty open did not normalize options or loaded entries") + } + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + appendMessage(t, w, "row-1", 1001, 1) + if changed, err := r.Refresh(); err != nil || !changed || len(r.entryOffsets) != 1 { + t.Fatalf("first refresh = %v, %v; entries=%d", changed, err, len(r.entryOffsets)) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("first Step = %v, %v", ok, err) + } + held, found, err := r.GetRaw([]byte("MESSAGE")) + if err != nil || !found || string(held) != "row-1" { + t.Fatalf("first MESSAGE = %q, %v, %v", held, found, err) + } + snapshot, err := OpenFileWithOptions(path, readerOpts.WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer snapshot.Close() + initialArrays := w.header.nEntryArrays + for i := 2; i <= 33; i++ { + appendMessage(t, w, fmt.Sprintf("row-%d", i), 1000+uint64(i), uint64(i)) + } + if w.header.nEntryArrays <= initialArrays { + t.Fatal("appends did not exercise ENTRY-array growth") + } + if changed, err := r.Refresh(); err != nil || !changed || len(r.entryOffsets) != 33 { + t.Fatalf("growth refresh = %v, %v; entries=%d", changed, err, len(r.entryOffsets)) + } + if string(held) != "row-1" { + t.Fatalf("refresh changed retained MESSAGE: %q", held) + } + for i := 2; i <= 33; i++ { + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("Step %d = %v, %v", i, ok, err) + } + value, found, err := r.GetRaw([]byte("MESSAGE")) + if err != nil || !found || string(value) != fmt.Sprintf("row-%d", i) { + t.Fatalf("MESSAGE %d = %q, %v, %v", i, value, found, err) + } + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("live EOF = %v, %v", ok, err) + } + if ok, err := snapshot.Step(); err != nil || !ok { + t.Fatalf("snapshot first = %v, %v", ok, err) + } + if ok, err := snapshot.Step(); err != nil || ok { + t.Fatalf("snapshot EOF = %v, %v", ok, err) + } + }) + } + } + } +} diff --git a/go/journal/reader_header_bench_test.go b/go/journal/reader_header_bench_test.go new file mode 100644 index 00000000..d27a25f1 --- /dev/null +++ b/go/journal/reader_header_bench_test.go @@ -0,0 +1,192 @@ +package journal + +import ( + "encoding/binary" + "fmt" + "os" + "path/filepath" + "runtime" + "testing" + "unsafe" +) + +var readerHeaderAllocationSink *Reader + +func BenchmarkReaderAllocation(b *testing.B) { + b.ReportAllocs() + for i := 0; i < b.N; i++ { + readerHeaderAllocationSink = new(Reader) + } +} + +func TestReaderAllocationShape(t *testing.T) { + const count = 4096 + readers := make([]*Reader, count) + var before, after runtime.MemStats + runtime.GC() + runtime.ReadMemStats(&before) + for i := range readers { + readers[i] = new(Reader) + } + runtime.ReadMemStats(&after) + runtime.KeepAlive(readers) + found := false + for i, class := range after.BySize { + if delta := class.Mallocs - before.BySize[i].Mallocs; delta >= count { + t.Logf("reader-size=%d heap-class=%d allocations=%d", unsafe.Sizeof(Reader{}), class.Size, delta) + found = true + } + } + if !found { + t.Fatal("retained Reader allocations did not identify an allocation class") + } +} + +func readerOpenBenchmarkFixture(b *testing.B, compact bool, size string) (string, int) { + b.Helper() + if size == "small" { + if root := os.Getenv("JOURNAL_SDK_READER_OPEN_SMALL_CORPUS"); root != "" { + layout := "regular" + if compact { + layout = "compact" + } + return filepath.Join(root, layout, "system.journal"), 5 + } + } + if size == "large" { + root := os.Getenv("JOURNAL_SDK_READER_OPEN_CORPUS") + if root == "" { + b.Skip("set JOURNAL_SDK_READER_OPEN_CORPUS to the immutable 100k-row corpus") + } + layout := "regular" + if compact { + layout = "compact" + } + return filepath.Join(root, layout, "system.journal"), 100000 + } + rows := 0 + if size == "small" { + rows = 5 + } + opts := testOptions() + opts.Compact = compact + cadence := uint64(0) + opts.LivePublishEveryEntries = &cadence + path := filepath.Join(b.TempDir(), "open.journal") + w, err := Create(path, opts) + if err != nil { + b.Fatal(err) + } + for i := 0; i < rows; i++ { + if err := w.Append([]Field{StringField("MESSAGE", fmt.Sprintf("row-%d", i))}, + EntryOptions{RealtimeUsec: 1000 + uint64(i), MonotonicUsec: uint64(i + 1)}); err != nil { + _ = w.Close() + b.Fatal(err) + } + } + if err := w.Close(); err != nil { + b.Fatal(err) + } + return path, rows +} + +func BenchmarkReaderOpen(b *testing.B) { + for _, compact := range []bool{false, true} { + for _, size := range []string{"empty", "small", "large"} { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + b.Run(fmt.Sprintf("compact=%v/size=%s/mode=%d", compact, size, mode), func(b *testing.B) { + b.StopTimer() + path, rows := readerOpenBenchmarkFixture(b, compact, size) + opts := DefaultReaderOptions().WithAccessMode(mode).WithSnapshot(true) + check := func() { + r, err := OpenFileWithOptions(path, opts) + if err != nil { + b.Fatal(err) + } + if r.header.nEntries != uint64(rows) || len(r.entryOffsets) != rows { + _ = r.Close() + b.Fatalf("open counts = %d/%d, want %d", r.header.nEntries, len(r.entryOffsets), rows) + } + if err := r.Close(); err != nil { + b.Fatal(err) + } + } + check() + b.ReportAllocs() + b.ResetTimer() + b.StartTimer() + for i := 0; i < b.N; i++ { + check() + } + b.StopTimer() + }) + } + } + } +} + +func BenchmarkReaderEntryHeader(b *testing.B) { + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + for _, test := range []struct { + name string + window uint64 + windows int + offsets []uint64 + pinned bool + }{ + {"within", 64, 2, []uint64{16}, false}, + {"crossing", 64, 2, []uint64{56}, false}, + {"non-power", 63, 2, []uint64{16}, false}, + {"tiny", 8, 2, []uint64{16}, false}, + {"default", 0, 2, []uint64{16}, false}, + {"eviction", 64, 1, []uint64{16, 80, 144, 56, 120, 184}, false}, + {"pinned", 64, 1, []uint64{56}, true}, + } { + b.Run(fmt.Sprintf("mode=%d/%s", mode, test.name), func(b *testing.B) { + b.StopTimer() + payload := make([]byte, 512) + for _, offset := range test.offsets { + payload[offset] = objectTypeEntry + binary.LittleEndian.PutUint64(payload[offset+8:offset+16], entryObjectHeaderSize) + } + path := filepath.Join(b.TempDir(), "entry-headers") + if err := os.WriteFile(path, payload, 0o600); err != nil { + b.Fatal(err) + } + file, err := os.Open(path) + if err != nil { + b.Fatal(err) + } + opts := DefaultReaderOptions().WithAccessMode(mode).WithWindowSize(test.window).WithMaxWindows(test.windows).WithSnapshot(true).normalized() + accessor, _, err := newReaderAccessor(file, opts) + if err != nil { + _ = file.Close() + b.Fatal(err) + } + r := &Reader{file: file, accessor: accessor, options: opts, fileSize: accessor.size(), + header: journalHeader{arenaSize: uint64(len(payload))}} + r.configureLayout() + b.Cleanup(func() { _ = r.Close() }) + if test.pinned { + if _, err := r.readRowSlice(0, 8); err != nil { + b.Fatal(err) + } + } + for _, offset := range test.offsets { + if valid, err := r.validEntryObjectOffset(offset); err != nil || !valid { + b.Fatalf("entry preflight at %d = %v, %v", offset, valid, err) + } + } + b.ReportAllocs() + b.ResetTimer() + b.StartTimer() + for i := 0; i < b.N; i++ { + if valid, err := r.validEntryObjectOffset(test.offsets[i%len(test.offsets)]); err != nil || !valid { + b.Fatalf("entry check = %v, %v", valid, err) + } + } + b.StopTimer() + }) + } + } +} diff --git a/go/journal/reader_recovery.go b/go/journal/reader_recovery.go new file mode 100644 index 00000000..b40d7c91 --- /dev/null +++ b/go/journal/reader_recovery.go @@ -0,0 +1,344 @@ +package journal + +import ( + "encoding/binary" + "errors" + "fmt" + "sort" +) + +// RecoveryReason identifies an observed unavailable journal tail. +type RecoveryReason uint8 + +const ( + RecoveryNone RecoveryReason = iota + RecoveryMissingEntry + RecoveryMissingEntryArray +) + +// RecoveryStatus describes an observed intact prefix. A zero value reports no +// observed loss; it does not certify journal integrity. +type RecoveryStatus struct { + Reason RecoveryReason + FirstLostOffset uint64 + DeclaredEntries uint64 + RecoveredEntries uint64 +} + +func (s RecoveryStatus) Incomplete() bool { return s.Reason != RecoveryNone } + +// FileRecoveryStatus associates observed loss with a journal's file ID. +type FileRecoveryStatus struct { + FileID UUID + Status RecoveryStatus +} + +type readerRecoveryState struct { + RecoveryStatus + tail entryHeader + eofCount uint64 + eofDirection Direction + eofValid bool +} + +func (s *readerRecoveryState) Incomplete() bool { return s != nil } +func (s *readerRecoveryState) status() RecoveryStatus { + if s == nil { + return RecoveryStatus{} + } + return s.RecoveryStatus +} +func sameRecoveryState(a, b *readerRecoveryState) bool { + return a == b || (a != nil && b != nil && a.RecoveryStatus == b.RecoveryStatus && a.tail == b.tail) +} + +func (r *Reader) clearRecoveryEOF() { + if r.recovery != nil { + r.recovery.eofValid = false + } +} + +func (r *Reader) rememberRecoveryEOF(direction Direction) { + if r.recovery != nil { + r.recovery.eofCount = r.header.nEntries + r.recovery.eofDirection = direction + r.recovery.eofValid = true + } +} + +func (r *Reader) unchangedRecoveryEOF(direction Direction) (bool, error) { + state := r.recovery + if state == nil || !state.eofValid || state.eofDirection != direction { + return false, nil + } + // Read the live counter without reloading copied index windows or scanning entries. + var count [8]byte + if _, err := r.file.ReadAt(count[:], 152); err != nil { + return false, err + } + return binary.LittleEndian.Uint64(count[:]) == state.eofCount, nil +} + +func (r *Reader) RecoveryStatus() RecoveryStatus { return r.recovery.status() } + +func (r *Reader) RecoveryStatuses() []FileRecoveryStatus { + if !r.recovery.Incomplete() { + return nil + } + return []FileRecoveryStatus{{FileID: r.header.fileID, Status: r.recovery.status()}} +} + +func (dr *DirectoryReader) RecoveryStatuses() []FileRecoveryStatus { + var out []FileRecoveryStatus + for _, reader := range dr.files { + if reader.recovery.Incomplete() { + out = append(out, FileRecoveryStatus{FileID: reader.header.fileID, Status: reader.recovery.status()}) + } + } + return out +} + +// SdJournalRecoveryStatuses returns diagnostics without changing iteration. +func SdJournalRecoveryStatuses(j *sdJournal) []FileRecoveryStatus { + return j.reader.RecoveryStatuses() +} + +type readerBoundsError struct{ offset, size, fileSize uint64 } + +type readerTailError struct{ offset uint64 } + +func (e readerTailError) Error() string { + return fmt.Sprintf("%v: unpublished reader tail", errCorruptObject) +} +func (e readerTailError) Unwrap() error { return errCorruptObject } + +func readerTailFailure(err error) (uint64, bool) { + if bound, ok := tailBoundsFailure(err); ok { + return bound.offset, true + } + var missing readerTailError + if errors.As(err, &missing) { + return missing.offset, true + } + return 0, false +} + +func (e readerBoundsError) Error() string { + return fmt.Sprintf("%v: reader access out of bounds", errInvalidJournal) +} + +func (e readerBoundsError) Unwrap() error { return errInvalidJournal } + +func tailBoundsFailure(err error) (readerBoundsError, bool) { + var bound readerBoundsError + if !errors.As(err, &bound) { + return bound, false + } + _, valid := checkedAdd(bound.offset, bound.size) + return bound, valid && (bound.offset > bound.fileSize || bound.size > bound.fileSize-bound.offset) +} + +func (r *Reader) recoverPartialArray(offset, remaining uint64) ([]uint64, error) { + header, capacity, err := r.readOffsetArrayHeader(offset) + if err != nil { + return nil, err + } + start, ok := checkedAdd(offset, offsetArrayObjectHeaderSize) + if !ok || start > r.fileSize { + return nil, readerBoundsError{offset, header.object.size, r.fileSize} + } + available := minUint64(minUint64(remaining, capacity), (r.fileSize-start)/r.offsetArrayItemSize()) + byteCount := available * r.offsetArrayItemSize() + if byteCount > uint64(int(^uint(0)>>1)) { + return nil, errCorruptObject + } + buf := make([]byte, byteCount) + if err := r.readAt(buf, start); err != nil { + return nil, err + } + return r.validEntryOffsetsFromArray(buf, r.offsetArrayItemSize()) +} + +func (r *Reader) retainRecoveredPrefix(offsets []uint64, reason RecoveryReason, lost uint64, previous *readerRecoveryState) error { + if err := r.validateRecoveredSuffix(); err != nil { + return err + } + status := RecoveryStatus{reason, lost, r.header.nEntries, uint64(len(offsets))} + r.entryOffsets = offsets + r.entryIndex = -1 + var tail entryHeader + if len(offsets) != 0 { + header, err := r.readEntryHeaderAt(offsets[len(offsets)-1]) + if err != nil { + return err + } + tail = header + } + if previous != nil && previous.RecoveryStatus == status && previous.tail == tail { + r.recovery = previous + } else { + r.recovery = &readerRecoveryState{RecoveryStatus: status, tail: tail} + } + return nil +} + +// A missing tail may end a prefix, but must not hide a readable later entry. +// This additional native-index pass runs only after a failed tail access. +func (r *Reader) validateRecoveredSuffix() error { + remaining := r.header.nEntries + missing := false + previous := uint64(0) + highest := r.header.entryArrayOffset + for offset := r.header.entryArrayOffset; remaining > 0; { + if offset == 0 { + return nil + } + header, capacity, err := r.readOffsetArrayHeader(offset) + if err != nil { + if _, ok := readerTailFailure(err); ok { + return nil + } + return err + } + wanted := minUint64(remaining, capacity) + start := offset + offsetArrayObjectHeaderSize + available := uint64(0) + if start <= r.fileSize { + available = minUint64(wanted, (r.fileSize-start)/r.offsetArrayItemSize()) + } + for i := uint64(0); i < available; i++ { + buf, err := r.readSlice(start+i*r.offsetArrayItemSize(), r.offsetArrayItemSize()) + if err != nil { + return err + } + entry := entryOffsetArrayItem(buf, r.offsetArrayItemSize()) + if entry == 0 { + missing = true + continue + } + valid, err := r.validEntryObjectOffset(entry) + if err != nil { + if _, ok := readerTailFailure(err); !ok { + return err + } + missing = true + continue + } + if valid && missing { + return errCorruptObject + } + if valid && entry <= previous { + return errCorruptObject + } + previous = entry + } + missing = missing || available < wanted + remaining -= wanted + if remaining > 0 && header.nextArrayOffset != 0 { + if header.nextArrayOffset <= highest { + if err := rejectArrayRevisit(r, r.header.entryArrayOffset, offset, header.nextArrayOffset, errCorruptObject); err != nil { + return err + } + } else { + highest = header.nextArrayOffset + } + } + offset = header.nextArrayOffset + } + return nil +} + +func (r *Reader) effectiveHeader() journalHeader { + header := r.header + if r.recovery.Incomplete() { + header.nEntries = r.recovery.RecoveredEntries + header.tailEntrySeqnum = r.recovery.tail.seqnum + header.tailEntryRealtime = r.recovery.tail.realtime + header.tailEntryMonotonic = r.recovery.tail.monotonic + header.tailEntryBootID = r.recovery.tail.bootID + } + return header +} + +func (r *Reader) recoveredEntryPresent(offset uint64) bool { + index := sort.Search(len(r.entryOffsets), func(i int) bool { return r.entryOffsets[i] >= offset }) + return index < len(r.entryOffsets) && r.entryOffsets[index] == offset +} + +func (r *Reader) visitRecoveredDataEntryOffsets(header dataHeader, visit func(uint64) error) error { + if header.nEntries == 0 || len(r.entryOffsets) == 0 { + return nil + } + last := r.entryOffsets[len(r.entryOffsets)-1] + if header.entryOffset == 0 { + return errCorruptObject + } + if header.entryOffset > last { + return nil + } + previous := header.entryOffset + if header.entryOffset != 0 && r.recoveredEntryPresent(header.entryOffset) { + if err := visit(header.entryOffset); err != nil { + return err + } + } + remaining := header.nEntries - 1 + highest := header.entryArrayOffset + for offset := header.entryArrayOffset; offset != 0 && remaining > 0; { + array, offsets, used, err := r.readEntryArrayChunk(offset, remaining) + lost := err != nil + knownBeyondTail := false + if err != nil { + lostOffset, recoverable := readerTailFailure(err) + if !recoverable { + return err + } + knownBeyondTail = lostOffset > last && lostOffset != offset && lostOffset != offset+offsetArrayObjectHeaderSize + if lostOffset == offset+offsetArrayObjectHeaderSize { + offsets, err = r.recoverPartialArray(offset, remaining) + if err != nil { + if entryLost, recoverable := readerTailFailure(err); !recoverable { + return err + } else { + knownBeyondTail = entryLost > last && entryLost != offset && entryLost != offset+offsetArrayObjectHeaderSize + } + } + } + } + for _, entry := range offsets { + if entry > last { + return nil + } + if entry <= previous { + return errCorruptObject + } + previous = entry + if r.recoveredEntryPresent(entry) { + if visitErr := visit(entry); visitErr != nil { + return visitErr + } + } + } + if lost { + if !knownBeyondTail && previous < last { + return errCorruptObject + } + return nil + } + remaining -= used + if remaining > 0 && array.nextArrayOffset != 0 { + if array.nextArrayOffset <= highest { + if err := rejectArrayRevisit(r, header.entryArrayOffset, offset, array.nextArrayOffset, errCorruptObject); err != nil { + return err + } + } else { + highest = array.nextArrayOffset + } + } + offset = array.nextArrayOffset + } + if remaining > 0 && previous < last { + return errCorruptObject + } + return nil +} diff --git a/go/journal/reader_recovery_eof_test.go b/go/journal/reader_recovery_eof_test.go new file mode 100644 index 00000000..b3fe4432 --- /dev/null +++ b/go/journal/reader_recovery_eof_test.go @@ -0,0 +1,96 @@ +package journal + +import ( + "bytes" + "fmt" + "os" + "path/filepath" + "testing" + "unsafe" +) + +type recoveryEOFCountingAccessor struct { + readerAccessor + reads int +} + +func (a *recoveryEOFCountingAccessor) readAt(dst []byte, offset uint64) error { + a.reads++ + return a.readerAccessor.readAt(dst, offset) +} + +func TestRecoveryEOFSkipsUnchangedPrefix(t *testing.T) { + source, arrays := recoveryTestFixture(t, false) + last := arrays[len(arrays)-1].entries[len(arrays[len(arrays)-1].entries)-1] + for _, mode := range readerTestAccessModes() { + for _, filtered := range []bool{false, true} { + t.Run(fmt.Sprintf("%d/%v", mode, filtered), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + damaged := bytes.Clone(source) + clear(damaged[last : last+16]) + if err := os.WriteFile(path, damaged, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { r.Close() }) + if filtered { + r.AddMatch([]byte("COMMON=all")) + } + if got := len(recoveryTestTimes(t, r, false)); got != 1004 { + t.Fatalf("prefix=%d", got) + } + probe := &recoveryEOFCountingAccessor{readerAccessor: r.accessor} + r.accessor = probe + for i := 0; i < 10; i++ { + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("EOF=%v/%v", ok, err) + } + } + if probe.reads > 10 { + t.Fatalf("unchanged EOF reread prefix: %d reads", probe.reads) + } + allocs := testing.AllocsPerRun(10, func() { + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("EOF=%v/%v", ok, err) + } + }) + t.Logf("EOF allocations/poll=%g; Reader=%d bytes; recovery state=%d bytes", allocs, unsafe.Sizeof(Reader{}), unsafe.Sizeof(readerRecoveryState{})) + after, err := os.ReadFile(path) + if err != nil || !bytes.Equal(after, damaged) { + t.Fatalf("reader changed source: %v", err) + } + if ok, err := r.StepBack(); err != nil || !ok { + t.Fatalf("reverse=%v/%v", ok, err) + } + // Restoration is synthetic; the following real append changes the + // declared population, which must invalidate the terminal memo. + if got := recoveryTestTimes(t, r, false); filtered && len(got) != 1 { + t.Fatalf("forward after reverse=%v", got) + } + if err := os.WriteFile(path, source, 0o600); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("same-count restoration=%v/%v", ok, err) + } + w, err := OpenWithOptions(path, testOptions()) + if err != nil { + t.Fatal(err) + } + if err := w.Append([]Field{StringField("COMMON", "all")}, EntryOptions{RealtimeUsec: 3000, MonotonicUsec: 3000}); err != nil { + w.Close() + t.Fatal(err) + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + if got := recoveryTestTimes(t, r, false); len(got) != 2 || got[0] != 2004 || got[1] != 3000 { + t.Fatalf("changed-count rows=%v", got) + } + }) + } + } +} diff --git a/go/journal/reader_recovery_facet_test.go b/go/journal/reader_recovery_facet_test.go new file mode 100644 index 00000000..63911ddc --- /dev/null +++ b/go/journal/reader_recovery_facet_test.go @@ -0,0 +1,143 @@ +package journal + +import ( + "bytes" + "encoding/binary" + "fmt" + "os" + "path/filepath" + "reflect" + "testing" +) + +func recoveryUnusedArrayReader(tb testing.TB, compact bool) *Reader { + tb.Helper() + path := filepath.Join(tb.TempDir(), "system.journal") + opts := testOptions() + opts.Compact = compact + w, err := Create(path, opts) + if err != nil { + tb.Fatal(err) + } + tb.Cleanup(func() { _ = w.Close() }) + for i, fields := range [][]Field{ + {StringField("BASE", "one")}, + {{Name: "ORPHAN", Value: bytes.Repeat([]byte("x"), 512<<10)}}, + } { + if err := w.Append(fields, testEntryOptions(uint64(i+1))); err != nil { + tb.Fatal(err) + } + } + if err := w.Close(); err != nil { + tb.Fatal(err) + } + r, err := OpenFile(path) + if err != nil { + tb.Fatal(err) + } + header := r.header + first := r.entryOffsets[0] + entry, err := r.readEntryHeaderAt(first) + if err != nil { + _ = r.Close() + tb.Fatal(err) + } + if err := r.Close(); err != nil { + tb.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + tb.Fatal(err) + } + itemSize := uint64(8) + if compact { + itemSize = 4 + } + array := (uint64(len(data)) + 7) &^ 7 + if array > uint64(^uint32(0)) { + tb.Fatal("fixture exceeds cached-array offset width") + } + // Publish one entry, retaining prepared DATA beyond that population. + header.nEntries = 1 + header.entryArrayOffset = array + header.tailObjectOffset = array + header.nObjects++ + header.nEntryArrays++ + header.tailEntryArrayOffset = uint32(array) + header.tailEntryArrayNEntries = 1 + header.tailEntryOffset = first + header.tailEntrySeqnum = entry.seqnum + header.tailEntryRealtime = entry.realtime + header.tailEntryMonotonic = entry.monotonic + header.tailEntryBootID = entry.bootID + header.arenaSize = array + 24 + 2*itemSize - header.headerSize + // The committed slot survives; only the unused second slot is unavailable. + data = append(data, make([]byte, int(array+24+itemSize)-len(data))...) + putHeader(data, header) + data[array] = objectTypeEntryArray + binary.LittleEndian.PutUint64(data[array+8:], 24+2*itemSize) + if compact { + binary.LittleEndian.PutUint32(data[array+24:], uint32(first)) + } else { + binary.LittleEndian.PutUint64(data[array+24:], first) + } + if err := os.WriteFile(path, data, 0o600); err != nil { + tb.Fatal(err) + } + r, err = OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + tb.Fatal(err) + } + tb.Cleanup(func() { _ = r.Close() }) + status := r.RecoveryStatus() + if status.Reason != RecoveryMissingEntryArray || status.DeclaredEntries != 1 || status.RecoveredEntries != 1 || len(r.entryOffsets) != 1 { + tb.Fatalf("unexpected recovered population: %+v", status) + } + return r +} + +func recoveryUnusedArrayQuery() ExplorerQuery { + query := DefaultExplorerQuery().WithFacet([]byte("ORPHAN")) + query.Limit = 0 + query.FieldMode = ExplorerFieldModeAllValues + query.UseSourceRealtime = false + return query +} + +func TestRecoveryEqualCountFacetWithoutPostings(t *testing.T) { + for _, compact := range []bool{false, true} { + t.Run(fmt.Sprint(compact), func(t *testing.T) { + r := recoveryUnusedArrayReader(t, compact) + result, err := r.ExploreWithStrategy(recoveryUnusedArrayQuery(), ExplorerStrategyIndex) + if err != nil { + t.Fatal(err) + } + want := map[string]map[string]uint64{"ORPHAN": {"-": 1}} + if !reflect.DeepEqual(result.Facets, want) || len(result.Rows) != 0 || result.Stats.DataRefsSeen != 0 || result.Stats.FacetUpdates != 1 { + t.Fatalf("unexpected recovered facet: %+v", result) + } + }) + } +} + +func BenchmarkRecoveryUnmatchedFacet(b *testing.B) { + for _, compact := range []bool{false, true} { + b.Run(fmt.Sprint(compact), func(b *testing.B) { + r := recoveryUnusedArrayReader(b, compact) + query := recoveryUnusedArrayQuery() + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + result, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if err != nil { + b.Fatal(err) + } + values := result.Facets["ORPHAN"] + if len(values) != 1 || values["-"] != 1 || result.Stats.DataRefsSeen != 0 || result.Stats.FacetUpdates != 1 { + b.Fatal("incorrect recovered facet") + } + } + b.StopTimer() + }) + } +} diff --git a/go/journal/reader_recovery_test.go b/go/journal/reader_recovery_test.go new file mode 100644 index 00000000..76ef41d7 --- /dev/null +++ b/go/journal/reader_recovery_test.go @@ -0,0 +1,729 @@ +package journal + +import ( + "bytes" + "context" + "encoding/binary" + "errors" + "fmt" + "os" + "path/filepath" + "slices" + "testing" +) + +type recoveryTestArray struct { + offset uint64 + entries []uint64 +} + +func recoveryTestFixture(t *testing.T, compact bool) ([]byte, []recoveryTestArray) { + t.Helper() + path := filepath.Join(t.TempDir(), "system.journal") + opts := testOptions() + opts.Compact = compact + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + for i := 0; i < 1005; i++ { + fields := []Field{StringField("MESSAGE", fmt.Sprintf("row-%04d", i)), StringField("COMMON", "all")} + if i == 0 || i == 500 || i == 1004 { + fields = append(fields, StringField("SPARSE", "some")) + } + if i >= 1003 { + fields = append(fields, StringField("TAIL_PAIR", "some")) + } + if err := w.Append(fields, EntryOptions{RealtimeUsec: 1000 + uint64(i), MonotonicUsec: 100 + uint64(i)}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + itemSize := uint64(8) + if compact { + itemSize = 4 + } + var arrays []recoveryTestArray + remaining := binary.LittleEndian.Uint64(data[152:]) + for offset := binary.LittleEndian.Uint64(data[176:]); remaining > 0; { + capacity := (binary.LittleEndian.Uint64(data[offset+8:]) - 24) / itemSize + count := minUint64(capacity, remaining) + a := recoveryTestArray{offset: offset} + for i := uint64(0); i < count; i++ { + a.entries = append(a.entries, entryOffsetArrayItem(data[offset+24+i*itemSize:], itemSize)) + } + arrays = append(arrays, a) + remaining -= count + offset = binary.LittleEndian.Uint64(data[offset+16:]) + } + return data, arrays +} + +func recoveryTestPrefix(data []byte, arrays []recoveryTestArray, cut uint64, compact bool) int { + itemSize := uint64(8) + if compact { + itemSize = 4 + } + count := 0 + for _, a := range arrays { + for i, offset := range a.entries { + if a.offset+24+uint64(i+1)*itemSize > cut || offset+binary.LittleEndian.Uint64(data[offset+8:]) > cut { + return count + } + count++ + } + } + return count +} + +func recoveryTestTimes(t *testing.T, reader *Reader, backward bool) []uint64 { + t.Helper() + var times []uint64 + for { + var ok bool + var err error + if backward { + ok, err = reader.StepBack() + } else { + ok, err = reader.Step() + } + if err != nil { + t.Fatal(err) + } + if !ok { + return times + } + value, err := reader.GetRealtimeUsec() + if err != nil { + t.Fatal(err) + } + times = append(times, value) + } +} + +func TestRecoverySnapshotSecondaryPostingShrink(t *testing.T) { + for _, compact := range []bool{false, true} { + source, arrays := recoveryTestFixture(t, compact) + for _, mode := range []ReaderAccessMode{ReaderAccessMmap, ReaderAccessReadAt} { + t.Run(fmt.Sprintf("compact=%v/mode=%d", compact, mode), func(t *testing.T) { + r, path := filterTestReader(t, source, mode, false) + data := filterTestDataOffset(t, r, "COMMON=all") + array := binary.LittleEndian.Uint64(source[data+48:]) + cut := array + 24 + r.offsetArrayItemSize() + wantCount := recoveryTestPrefix(source, arrays, cut, compact) + if wantCount != 2 { + t.Fatalf("fixture retains %d rows, want 2", wantCount) + } + r.AddMatch([]byte("COMMON=all")) + if err := os.Truncate(path, int64(cut)); err != nil { + t.Fatal(err) + } + if got := recoveryTestTimes(t, r, false); !slices.Equal(got, []uint64{1000, 1001}) { + t.Fatalf("prefix=%v", got) + } + if status := r.RecoveryStatus(); !status.Incomplete() || status.RecoveredEntries != 2 { + t.Fatalf("status=%+v", status) + } + if err := os.WriteFile(path, source, 0o600); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("unchanged EOF: %v/%v", ok, err) + } + if err := r.SeekRealtimeUsec(1002); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("reposition: %v/%v", ok, err) + } + if got, err := r.GetRealtimeUsec(); err != nil || got != 1002 { + t.Fatalf("repair time=%d/%v", got, err) + } + }) + } + } +} + +type partialArrayExtentAccessor struct { + readerAccessor + header []byte +} + +func (a *partialArrayExtentAccessor) tempSlice(offset, size uint64) ([]byte, error) { + if offset != 4096 || size != offsetArrayObjectHeaderSize { + return nil, errCorruptObject + } + return a.header, nil +} + +func TestRecoveryPartialArrayRejectsUnrepresentableAllocation(t *testing.T) { + maxInt := uint64(int(^uint(0) >> 1)) + const offset = uint64(4096) + bodySize := maxInt + 1 + regularOffsetArrayItemSize + header := make([]byte, offsetArrayObjectHeaderSize) + header[0] = objectTypeEntryArray + binary.LittleEndian.PutUint64(header[8:], offsetArrayObjectHeaderSize+bodySize) + r := Reader{ + header: journalHeader{headerSize: headerSize, arenaSize: offset + offsetArrayObjectHeaderSize + bodySize - headerSize}, + fileSize: offset + offsetArrayObjectHeaderSize + maxInt + 1, + accessor: &partialArrayExtentAccessor{header: header}, + } + r.configureLayout() + defer func() { + if failure := recover(); failure != nil { + t.Errorf("partial-array allocation panicked: %v", failure) + } + }() + if _, err := r.recoverPartialArray(offset, ^uint64(0)); !errors.Is(err, errCorruptObject) { + t.Fatalf("oversized recovery allocation: %v", err) + } +} + +func TestRecoveryPrefixReverseTimeAndExactFilter(t *testing.T) { + for _, compact := range []bool{false, true} { + data, arrays := recoveryTestFixture(t, compact) + last := arrays[len(arrays)-1] + lastEntry := last.entries[len(last.entries)-1] + itemSize := uint64(8) + if compact { + itemSize = 4 + } + for name, cut := range map[string]uint64{"entry-header": lastEntry, "entry-body": lastEntry + 32, "array-header": last.offset, "array-slots": last.offset + 24 + 3*itemSize} { + t.Run(fmt.Sprintf("%v/%s", compact, name), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, data[:cut], 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + forward := recoveryTestTimes(t, r, false) + want := recoveryTestPrefix(data, arrays, cut, compact) + if len(forward) != want { + t.Fatalf("forward=%d want=%d", len(forward), want) + } + status := r.RecoveryStatus() + if !status.Incomplete() || status.RecoveredEntries != uint64(want) || r.Header().nEntries != 1005 { + t.Fatalf("status=%+v", status) + } + if err := r.SeekTail(); err != nil { + t.Fatal(err) + } + reverse := recoveryTestTimes(t, r, true) + slices.Reverse(reverse) + if !slices.Equal(reverse, forward) { + t.Fatal("reverse differs") + } + if err := r.SeekRealtimeUsec(1000 + uint64(want/2)); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("time step: %v %v", ok, err) + } + value, err := r.GetRealtimeUsec() + if err != nil || value != 1000+uint64(want/2) { + t.Fatalf("time=%d %v", value, err) + } + r.AddMatch([]byte("COMMON=all")) + if err := r.SeekHead(); err != nil { + t.Fatal(err) + } + filtered := recoveryTestTimes(t, r, false) + if !slices.Equal(filtered, forward) { + t.Fatalf("filtered=%d forward=%d", len(filtered), len(forward)) + } + if err := VerifyFile(path); err == nil { + t.Fatal("VerifyFile accepted truncated journal") + } + if err := VerifyIndex(context.Background(), path); err == nil { + t.Fatal("VerifyIndex accepted truncated journal") + } + if snapshot, err := OpenIndexedSnapshot(context.Background(), path, IndexedSnapshotOptions{}); err == nil { + snapshot.Close() + t.Fatal("IndexedSnapshot accepted truncated journal") + } + before, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if w, err := Open(path); err == nil { + w.Close() + t.Fatal("append-open accepted truncated journal") + } + after, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(before, after) { + t.Fatal("rejected append changed bytes") + } + }) + } + } +} + +func TestRecoverySparsePostingBoundary(t *testing.T) { + for _, compact := range []bool{false, true} { + data, arrays := recoveryTestFixture(t, compact) + cut := arrays[len(arrays)-1].entries[len(arrays[len(arrays)-1].entries)-1] + payloadStart := uint64(64) + if compact { + payloadStart += 8 + } + var sparse uint64 + for at := binary.LittleEndian.Uint64(data[88:]); at+16 <= uint64(len(data)); { + size := binary.LittleEndian.Uint64(data[at+8:]) + if size < 16 || size > uint64(len(data))-at { + break + } + if data[at] == 1 && size >= payloadStart && bytes.Equal(data[at+payloadStart:at+size], []byte("SPARSE=some")) { + sparse = at + break + } + at += (size + 7) &^ 7 + } + if sparse == 0 { + t.Fatal("missing sparse DATA object") + } + for _, missingArray := range []bool{false, true} { + t.Run(fmt.Sprintf("%v/missing-array=%v", compact, missingArray), func(t *testing.T) { + damaged := bytes.Clone(data[:cut]) + if missingArray { + clear(damaged[sparse+48 : sparse+56]) + } + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, damaged, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + query := DefaultExplorerQuery().WithFilter([]byte("SPARSE"), []byte("some")).WithFacet([]byte("SPARSE")) + query.FieldMode = ExplorerFieldModeAllValues + query.UseSourceRealtime = false + result, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if missingArray { + if err == nil { + t.Fatal("unknown posting targets were silently omitted") + } + return + } + if err != nil { + t.Fatal(err) + } + assertExplorerFacetCount(t, result, "SPARSE", "some", 2) + }) + } + } +} + +func TestIndexedPostingsRejectMissingPointersWithoutPriorTraversal(t *testing.T) { + for _, compact := range []bool{false, true} { + source, _ := recoveryTestFixture(t, compact) + payloadStart := uint64(64) + if compact { + payloadStart = 72 + } + var common uint64 + for at := binary.LittleEndian.Uint64(source[88:]); at+16 <= uint64(len(source)); { + size := binary.LittleEndian.Uint64(source[at+8:]) + if size < 16 || size > uint64(len(source))-at { + break + } + if source[at] == 1 && size >= payloadStart && bytes.Equal(source[at+payloadStart:at+size], []byte("COMMON=all")) { + common = at + break + } + at += (size + 7) &^ 7 + } + if common == 0 { + t.Fatal("missing common DATA object") + } + array := binary.LittleEndian.Uint64(source[common+48:]) + for name, pointer := range map[string]uint64{"inline": common + 40, "head": common + 48, "next": array + 16} { + t.Run(fmt.Sprintf("%v/%s", compact, name), func(t *testing.T) { + damaged := bytes.Clone(source) + clear(damaged[pointer : pointer+8]) + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, damaged, 0o600); err != nil { + t.Fatal(err) + } + for _, mode := range []string{"filter", "facet", "histogram"} { + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + query := DefaultExplorerQuery() + query.Limit = 0 + query.FieldMode = ExplorerFieldModeAllValues + query.UseSourceRealtime = false + switch mode { + case "filter": + query.Limit = 1 + query = query.WithFilter([]byte("COMMON"), []byte("all")) + case "facet": + query = query.WithFacet([]byte("COMMON")) + case "histogram": + query = query.WithHistogram([]byte("COMMON")) + } + _, err = r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if err == nil { + t.Errorf("%s silently omitted declared postings", mode) + } + if r.RecoveryStatus().Incomplete() { + t.Error("a broken secondary index changed global recovery status") + } + r.Close() + } + }) + } + } +} + +func TestRecoveryMissingArrayAfterLastSurvivingPosting(t *testing.T) { + for _, compact := range []bool{false, true} { + data, arrays := recoveryTestFixture(t, compact) + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + pristine, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + pair, found, err := pristine.findDataOffsetByPayload([]byte("TAIL_PAIR=some")) + pristine.Close() + if err != nil || !found { + t.Fatalf("tail pair lookup: %v/%v", found, err) + } + lastArray := arrays[len(arrays)-1] + last := lastArray.entries[len(lastArray.entries)-1] + clear(data[last : last+16]) + clear(data[pair+48 : pair+56]) + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)) + if err != nil { + t.Fatal(err) + } + query := DefaultExplorerQuery().WithFilter([]byte("TAIL_PAIR"), []byte("some")).WithFacet([]byte("TAIL_PAIR")) + query.FieldMode = ExplorerFieldModeAllValues + query.UseSourceRealtime = false + result, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if err != nil { + t.Fatal(err) + } + if len(result.Rows) != 1 || result.Rows[0].RealtimeUsec != 2003 { + t.Fatalf("unexpected surviving rows: %v", result.Rows) + } + assertExplorerFacetCount(t, result, "TAIL_PAIR", "some", 1) + if r.RecoveryStatus().RecoveredEntries != 1004 { + t.Fatal("wrong recovered population") + } + r.Close() + } +} + +func TestRecoveryRefreshReturnsRepairedTailOnce(t *testing.T) { + data, arrays := recoveryTestFixture(t, false) + last := arrays[len(arrays)-1] + cut := last.entries[len(last.entries)-1] + for _, explicit := range []bool{false, true} { + t.Run(fmt.Sprint(explicit), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, data[:cut], 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFile(path) + if err != nil { + t.Fatal(err) + } + defer r.Close() + if got := len(recoveryTestTimes(t, r, false)); got != 1004 { + t.Fatalf("prefix=%d", got) + } + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + if explicit { + if changed, err := r.Refresh(); err != nil || !changed { + t.Fatalf("refresh=%v %v", changed, err) + } + } + tail := recoveryTestTimes(t, r, false) + if !explicit { + if len(tail) != 0 || !r.RecoveryStatus().Incomplete() { + t.Fatalf("unchanged EOF=%v status=%+v", tail, r.RecoveryStatus()) + } + return + } + if !slices.Equal(tail, []uint64{2004}) { + t.Fatalf("repaired tail=%v", tail) + } + if r.RecoveryStatus().Incomplete() { + t.Fatal("complete repair still degraded") + } + }) + } +} + +func TestRecoveryRejectsInteriorHolesAndArenaPointers(t *testing.T) { + for _, compact := range []bool{false, true} { + data, arrays := recoveryTestFixture(t, compact) + last := arrays[len(arrays)-1] + cut := last.entries[len(last.entries)-1] + item := 8 + if compact { + item = 4 + } + for _, corruption := range []string{"zero-entry", "zero-slot", "arena-pointer", "nonzero-header"} { + t.Run(fmt.Sprintf("%v/%s", compact, corruption), func(t *testing.T) { + damaged := bytes.Clone(data[:cut]) + first := arrays[0].entries[0] + switch corruption { + case "zero-entry", "nonzero-header": + clear(damaged[first : first+16]) + if corruption == "nonzero-header" { + damaged[first+1] = 1 + } + case "zero-slot": + clear(damaged[arrays[0].offset+24 : arrays[0].offset+24+uint64(item)]) + default: + bad := binary.LittleEndian.Uint64(data[88:]) + binary.LittleEndian.Uint64(data[96:]) + 8 + slot := arrays[0].offset + 24 + if compact { + binary.LittleEndian.PutUint32(damaged[slot:], uint32(bad)) + } else { + binary.LittleEndian.PutUint64(damaged[slot:], bad) + } + } + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, damaged, 0o600); err != nil { + t.Fatal(err) + } + if reader, err := OpenFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true)); err == nil { + reader.Close() + t.Fatal("interior corruption recovered") + } + }) + } + } +} + +// External byte restoration is fault injection, not journald publication or a +// supported repair lifecycle. Initial damaged-prefix assertions remain required. +func TestRecoverySyntheticRestorationKeepsIndexedQueriesAndRefresh(t *testing.T) { + for _, compact := range []bool{false, true} { + data, arrays := recoveryTestFixture(t, compact) + last := arrays[len(arrays)-1] + item := uint64(8) + if compact { + item = 4 + } + for _, slot := range []bool{false, true} { + t.Run(fmt.Sprintf("%v/%v", compact, slot), func(t *testing.T) { + damaged := bytes.Clone(data) + if slot { + at := last.offset + 24 + uint64(len(last.entries)-1)*item + clear(damaged[at : at+item]) + } else { + at := last.entries[len(last.entries)-1] + clear(damaged[at : at+16]) + } + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, damaged, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFile(path) + if err != nil { + t.Fatal(err) + } + defer r.Close() + query := DefaultExplorerQuery().WithFilter([]byte("COMMON"), []byte("all")).WithFacet([]byte("COMMON")) + query.FieldMode = ExplorerFieldModeAllValues + query.UseSourceRealtime = false + result, err := r.ExploreWithStrategy(query, ExplorerStrategyIndex) + if err != nil { + t.Fatal(err) + } + assertExplorerFacetCount(t, result, "COMMON", "all", 1004) + r.AddMatch([]byte("MESSAGE=row-1004")) + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("missing exact row=%v %v", ok, err) + } + if changed, err := r.Refresh(); err != nil || changed { + t.Fatalf("unchanged degraded refresh=%v %v", changed, err) + } + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + if ok, err := r.Step(); err != nil || !ok { + t.Fatalf("repaired exact row=%v %v", ok, err) + } + if value, err := r.GetRealtimeUsec(); err != nil || value != 2004 { + t.Fatalf("repaired time=%d %v", value, err) + } + if ok, err := r.Step(); err != nil || ok { + t.Fatalf("duplicate repaired row=%v %v", ok, err) + } + if r.RecoveryStatus().Incomplete() { + t.Fatal("repair still degraded") + } + }) + } + } +} + +func TestRecoveryRefreshShrinkKeepsPhysicalBounds(t *testing.T) { + data, _ := recoveryTestFixture(t, false) + for _, mode := range readerTestAccessModes() { + t.Run(accessModeName(mode), func(t *testing.T) { + path := filepath.Join(t.TempDir(), "system.journal") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatal(err) + } + r, err := OpenFileWithOptions(path, DefaultReaderOptions().WithAccessMode(mode)) + if err != nil { + t.Fatal(err) + } + defer r.Close() + if ok, err := r.Step(); err != nil || !ok { + t.Fatal(ok, err) + } + if _, err := r.GetRealtimeUsec(); err != nil { + t.Fatal(err) + } + if err := os.Truncate(path, 64); err != nil { + t.Fatal(err) + } + if _, err := r.Refresh(); err == nil { + t.Fatal("header loss accepted") + } + if r.AccessStats().FileSize != 64 { + t.Fatal("stale physical extent") + } + if _, err := r.GetRealtimeUsec(); err == nil { + t.Fatal("stale ENTRY header") + } + }) + } +} + +func TestRecoveryDirectoryRepairPreservesPendingMergeRows(t *testing.T) { + for _, compact := range []bool{false, true} { + dir := t.TempDir() + aPath, bPath := filepath.Join(dir, "a.journal"), filepath.Join(dir, "b.journal") + for path, seqs := range map[string][]uint64{aPath: {1, 3}, bPath: {2}} { + opts := testOptions() + opts.Compact = compact + opts.HeadSeqnum = seqs[0] + w, err := Create(path, opts) + if err != nil { + t.Fatal(err) + } + for _, seq := range seqs { + if err := w.Append([]Field{StringField("MESSAGE", fmt.Sprint(seq))}, EntryOptions{Seqnum: seq, RealtimeUsec: seq * 1000, MonotonicUsec: seq}); err != nil { + t.Fatal(err) + } + } + if err := w.Close(); err != nil { + t.Fatal(err) + } + } + a, err := OpenFile(aPath) + if err != nil { + t.Fatal(err) + } + cut := a.entryOffsets[1] + a.Close() + bytes, err := os.ReadFile(aPath) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(aPath, bytes[:cut], 0o600); err != nil { + t.Fatal(err) + } + a, err = OpenFile(aPath) + if err != nil { + t.Fatal(err) + } + b, err := OpenFile(bPath) + if err != nil { + t.Fatal(err) + } + dr, err := newDirectoryReader([]*Reader{b, a}, false) + if err != nil { + t.Fatal(err) + } + defer dr.Close() + if dr.nonOverlapping { + t.Fatal("recovered prefix proved false disjointness") + } + if ok, err := dr.Step(); err != nil || !ok { + t.Fatal(ok, err) + } + if time, err := dr.GetRealtimeUsec(); err != nil || time != 1000 { + t.Fatal(time, err) + } + if err := os.WriteFile(aPath, bytes, 0o600); err != nil { + t.Fatal(err) + } + var times []uint64 + for { + ok, err := dr.Step() + if err != nil { + t.Fatal(err) + } + if !ok { + break + } + value, err := dr.GetRealtimeUsec() + if err != nil { + t.Fatal(err) + } + times = append(times, value) + } + if !slices.Equal(times, []uint64{2000, 3000}) { + t.Fatalf("repair order=%v", times) + } + if len(dr.RecoveryStatuses()) != 0 { + t.Fatal("directory status stale") + } + boots, err := dr.ListBoots() + if err != nil || len(boots) != 1 || boots[0].LastEntry != 3000 { + t.Fatalf("boots=%v %v", boots, err) + } + if err := dr.SeekTail(); err != nil { + t.Fatal(err) + } + times = nil + for { + ok, err := dr.StepBack() + if err != nil { + t.Fatal(err) + } + if !ok { + break + } + value, err := dr.GetRealtimeUsec() + if err != nil { + t.Fatal(err) + } + times = append(times, value) + } + if !slices.Equal(times, []uint64{3000, 2000, 1000}) { + t.Fatalf("reverse repair order=%v", times) + } + } +} diff --git a/go/journal/reader_test.go b/go/journal/reader_test.go index f3671a5e..c8822b48 100644 --- a/go/journal/reader_test.go +++ b/go/journal/reader_test.go @@ -548,8 +548,8 @@ func TestReaderRefreshFailureRestoresStateAndKeepsRowPayload(t *testing.T) { corruptHeaderUint64(t, path, 176, wantSize+1) changed, err := r.Refresh() - if err != nil || changed { - t.Fatalf("Refresh() = %v, %v; want false, nil rollback", changed, err) + if err == nil || changed { + t.Fatalf("Refresh() = %v, %v; want false, corruption error with rollback", changed, err) } if got := r.AccessStats().FileSize; got != wantSize { t.Fatalf("restored accessor file size = %d, want %d", got, wantSize) @@ -1123,7 +1123,9 @@ func TestReaderMatchDisjunction(t *testing.T) { defer r.Close() r.AddMatch([]byte("L3=ok")) - r.AddDisjunction() + if err := r.AddDisjunction(); err != nil { + t.Fatal(err) + } r.AddMatch([]byte("L4=yes")) count := 0 @@ -1190,7 +1192,9 @@ func TestReaderSystemdComplexMatchExpression(t *testing.T) { } defer r.Close() - addSystemdComplexMatchExpression(r) + if err := addSystemdComplexMatchExpression(r); err != nil { + t.Fatal(err) + } count := 0 for { @@ -1210,9 +1214,9 @@ func TestReaderSystemdComplexMatchExpression(t *testing.T) { func addSystemdComplexMatchExpression(r interface { AddMatch([]byte) - AddDisjunction() - AddConjunction() -}) { + AddDisjunction() error + AddConjunction() error +}) error { r.AddMatch([]byte{'A', '=', 1, 2}) r.AddMatch([]byte{'B', '=', 'C', 0, 'D'}) r.AddMatch([]byte("HALLO=WALDO")) @@ -1222,18 +1226,25 @@ func addSystemdComplexMatchExpression(r interface { r.AddMatch([]byte("QUUX=xxxxx")) r.AddMatch([]byte("QUUX=yyyyy")) r.AddMatch([]byte("PIFF=paff")) - r.AddDisjunction() + if err := r.AddDisjunction(); err != nil { + return err + } r.AddMatch([]byte("ONE=one")) r.AddMatch([]byte("ONE=two")) r.AddMatch([]byte("TWO=two")) - r.AddConjunction() + if err := r.AddConjunction(); err != nil { + return err + } r.AddMatch([]byte("L4_1=yes")) r.AddMatch([]byte("L4_1=ok")) r.AddMatch([]byte("L4_2=yes")) r.AddMatch([]byte("L4_2=ok")) - r.AddDisjunction() + if err := r.AddDisjunction(); err != nil { + return err + } r.AddMatch([]byte("L3=yes")) r.AddMatch([]byte("L3=ok")) + return nil } func TestReaderBinaryFields(t *testing.T) { diff --git a/go/journal/reader_unique.go b/go/journal/reader_unique.go index be7d6e4f..7e3e2f3d 100644 --- a/go/journal/reader_unique.go +++ b/go/journal/reader_unique.go @@ -2,6 +2,7 @@ package journal import ( "bytes" + "encoding/binary" "fmt" "unicode/utf8" ) @@ -26,11 +27,15 @@ func (r *Reader) VisitUnique(fieldName string, visit func([]byte) error) error { return err } + var chain chainGuard for offset != 0 { header, err := r.readDataHeaderAt(offset) if err != nil { return err } + if !chain.advance(offset, header.nextFieldOffset, true) { + return fmt.Errorf("%w: cyclic FIELD DATA chain", errCorruptObject) + } err = r.visitDataPayloadWithHeader(offset, header.object, func(payload []byte) error { if len(payload) <= len(field) || !bytes.Equal(payload[:len(field)], field) || payload[len(field)] != '=' { return fmt.Errorf("%w: field data object at offset %d does not match %q", errCorruptObject, offset, fieldName) @@ -67,6 +72,7 @@ func (r *Reader) QueryUniqueState(fieldName string) error { func (r *Reader) RestartUniqueState() error { r.uniqueCurrentOffset = r.uniqueHeadOffset + r.uniqueChain = chainGuard{} return nil } @@ -74,6 +80,7 @@ func (r *Reader) ClearUniqueState() { r.uniqueField = "" r.uniqueHeadOffset = 0 r.uniqueCurrentOffset = 0 + r.uniqueChain = chainGuard{} } func (r *Reader) EnumerateUniquePayload() ([]byte, bool, error) { @@ -82,11 +89,14 @@ func (r *Reader) EnumerateUniquePayload() ([]byte, bool, error) { } field := []byte(r.uniqueField) offset := r.uniqueCurrentOffset - header, err := r.readDataHeaderAt(offset) + header, err := r.readUniqueDataHeaderAt(offset) if err != nil { r.ClearUniqueState() return nil, false, err } + if !r.uniqueChain.advance(offset, header.nextFieldOffset, true) { + return r.uniqueCycleError() + } r.uniqueCurrentOffset = header.nextFieldOffset var out []byte @@ -129,11 +139,15 @@ func (r *Reader) EnumerateFields() (map[string]struct{}, error) { return r.enumerateFieldsByEntryScan() } item := parseHashItem(itemBuf) + var chain chainGuard for offset := item.head; offset != 0; { header, payload, err := r.readFieldObjectAt(offset) if err != nil { return r.enumerateFieldsByEntryScan() } + if !chain.advance(offset, header.nextHashOffset, false) { + return nil, fmt.Errorf("%w: cyclic FIELD hash chain", errCorruptObject) + } if utf8.Valid(payload) { fields[string(payload)] = struct{}{} } @@ -157,3 +171,33 @@ func (r *Reader) enumerateFieldsByEntryScan() (map[string]struct{}, error) { } return fields, nil } + +//go:noinline +func (r *Reader) uniqueCycleError() ([]byte, bool, error) { + r.ClearUniqueState() + return nil, false, fmt.Errorf("%w: cyclic FIELD DATA chain", errCorruptObject) +} + +// Unique enumeration needs no posting or hash metadata. +type uniqueDataHeader struct { + object objectHeader + nextFieldOffset uint64 +} + +func (r *Reader) readUniqueDataHeaderAt(offset uint64) (uniqueDataHeader, error) { + buf, err := r.readSlice(offset, dataObjectHeaderSize) + if err != nil { + return uniqueDataHeader{}, err + } + if len(buf) < dataObjectHeaderSize { + return uniqueDataHeader{}, errInvalidJournal + } + header := uniqueDataHeader{ + object: objectHeader{typ: buf[0], flag: buf[1], size: binary.LittleEndian.Uint64(buf[8:16])}, + nextFieldOffset: binary.LittleEndian.Uint64(buf[32:40]), + } + if header.object.typ != objectTypeData || header.object.size < r.dataPayloadOffset() { + return uniqueDataHeader{}, errCorruptObject + } + return header, nil +} diff --git a/go/journal/reader_unique_header_test.go b/go/journal/reader_unique_header_test.go new file mode 100644 index 00000000..05c35c50 --- /dev/null +++ b/go/journal/reader_unique_header_test.go @@ -0,0 +1,71 @@ +package journal + +import ( + "encoding/binary" + "errors" + "syscall" + "testing" +) + +type uniqueHeaderControlAccess struct { + readerAccessor + data []byte + err error + offset, extent uint64 + calls int +} + +func (a *uniqueHeaderControlAccess) tempSlice(offset, extent uint64) ([]byte, error) { + a.offset, a.extent = offset, extent + a.calls++ + return a.data, a.err +} + +func TestUniqueHeaderAccessorPreservesGenericContract(t *testing.T) { + for _, compact := range []bool{false, true} { + for length := 0; length <= dataObjectHeaderSize; length++ { + for _, typ := range []byte{0, objectTypeData, objectTypeEntry} { + for _, size := range []uint64{0, 63, 64, 71, 72, ^uint64(0)} { + for _, accessErr := range []error{nil, syscall.EIO} { + data := make([]byte, dataObjectHeaderSize) + data[0], data[1] = typ, 0xff + binary.LittleEndian.PutUint64(data[8:16], size) + binary.LittleEndian.PutUint64(data[32:40], 0xfedcba9876543210) + oldAccess := &uniqueHeaderControlAccess{data: data[:length], err: accessErr} + newAccess := &uniqueHeaderControlAccess{data: data[:length], err: accessErr} + oldReader, newReader := Reader{accessor: oldAccess}, Reader{accessor: newAccess} + if compact { + oldReader.header.incompatibleFlags = incompatibleCompact + newReader.header.incompatibleFlags = incompatibleCompact + } + oldReader.configureLayout() + newReader.configureLayout() + old, oldErr := oldReader.readDataHeaderAt(1) + got, gotErr := newReader.readUniqueDataHeaderAt(1) + if !errors.Is(gotErr, oldErr) || !errors.Is(oldErr, gotErr) { + t.Fatalf("compact=%v length=%d type=%d size=%d access=%v: errors %v / %v", compact, length, typ, size, accessErr, oldErr, gotErr) + } + if gotErr == nil && (got.object != old.object || got.nextFieldOffset != old.nextFieldOffset) { + t.Fatalf("different decoded unique header: %+v / %+v", old, got) + } + if oldAccess.calls != 1 || newAccess.calls != 1 || oldAccess.offset != 1 || newAccess.offset != 1 || oldAccess.extent != dataObjectHeaderSize || newAccess.extent != dataObjectHeaderSize { + t.Fatalf("different access request: %+v / %+v", oldAccess, newAccess) + } + for i := range data { + data[i] = 0 + } + if gotErr == nil && (got.object != old.object || got.nextFieldOffset != old.nextFieldOffset) { + t.Fatal("header retained borrowed storage") + } + } + } + } + } + } + var reader Reader + _, oldErr := reader.readDataHeaderAt(1) + _, gotErr := reader.readUniqueDataHeaderAt(1) + if !errors.Is(oldErr, errInvalidJournal) || !errors.Is(gotErr, oldErr) { + t.Fatalf("nil accessor: %v / %v", oldErr, gotErr) + } +} diff --git a/go/journal/reader_zstd_test.go b/go/journal/reader_zstd_test.go index 213fc00b..9312acd7 100644 --- a/go/journal/reader_zstd_test.go +++ b/go/journal/reader_zstd_test.go @@ -35,7 +35,10 @@ func TestReaderJournalZstdUsesTempAccessorAndCleansUp(t *testing.T) { if err != nil { t.Fatalf("OpenFileWithOptions(%s) error = %v", zstPath, err) } - cleanupPath := r.cleanupPath + cleanupPath := r.file.Name() + if !r.staged { + t.Fatal("expected a staged decompressed file") + } if cleanupPath == "" { t.Fatal("cleanupPath is empty; want streamed .journal.zst temp file") } diff --git a/go/journal/verify.go b/go/journal/verify.go index d7a78ca9..9019b8e3 100644 --- a/go/journal/verify.go +++ b/go/journal/verify.go @@ -27,6 +27,8 @@ func (e *VerificationError) Error() string { // This compatibility check tolerates some incomplete final-entry links and // can revisit DATA posting lists for repeated values. Use VerifyIndex for // complete native-index consistency with work proportional to the graph. +// Recovery is disabled: missing required entries are rejected rather than +// accepted as a recovered prefix. // For sealed journals, this validates structure only; use VerifyFileWithKey // when TAG/HMAC verification is required. func VerifyFile(path string) error { @@ -48,7 +50,7 @@ func verifyFileWithOptions(path string, opts ReaderOptions) error { } func openVerifyReader(path string, opts ReaderOptions) (*Reader, error) { - return OpenFileWithOptions(path, opts.WithSnapshot(true)) + return openFileWithPolicy(path, opts.WithSnapshot(true), true, false) } func verifyReaderStrict(r *Reader) error { diff --git a/go/journal/verify_index.go b/go/journal/verify_index.go index a3fa94ab..bfdb46ba 100644 --- a/go/journal/verify_index.go +++ b/go/journal/verify_index.go @@ -23,7 +23,7 @@ func VerifyIndex(ctx context.Context, path string) error { if err := ctx.Err(); err != nil { return err } - r, err := openFileWithOptions(path, DefaultReaderOptions().WithSnapshot(true), false) + r, err := openFileWithPolicy(path, DefaultReaderOptions().WithSnapshot(true), false, false) if err != nil { return &VerificationError{Reason: fmt.Sprintf("journal index verification failed: %v", err)} } diff --git a/go/journal/writer_arrays.go b/go/journal/writer_arrays.go index 6803fc89..aefc7b0d 100644 --- a/go/journal/writer_arrays.go +++ b/go/journal/writer_arrays.go @@ -72,6 +72,7 @@ func (w *Writer) entryArrayTailOffset() (uint64, error) { return tailOffset, nil } tailOffset = w.header.entryArrayOffset + highest := tailOffset for remaining := w.header.nEntries; ; { header, cap, err := w.readOffsetArrayHeader(tailOffset) if err != nil { @@ -81,6 +82,13 @@ func (w *Writer) entryArrayTailOffset() (uint64, error) { return tailOffset, nil } remaining -= cap + if header.nextArrayOffset <= highest { + if err := rejectArrayRevisit(w, w.header.entryArrayOffset, tailOffset, header.nextArrayOffset, errInvalidJournal); err != nil { + return 0, err + } + } else { + highest = header.nextArrayOffset + } tailOffset = header.nextArrayOffset } } @@ -91,11 +99,22 @@ func (w *Writer) entryArrayTailEntries(tailOffset uint64) (uint64, error) { return tailEntries, nil } tailEntries = w.header.nEntries + highest := w.header.entryArrayOffset for offset := w.header.entryArrayOffset; offset != 0 && offset != tailOffset; { h, c, err := w.readOffsetArrayHeader(offset) if err != nil { return 0, err } + if c > tailEntries { + return 0, errInvalidJournal + } + if h.nextArrayOffset != 0 && h.nextArrayOffset <= highest { + if err := rejectArrayRevisit(w, w.header.entryArrayOffset, offset, h.nextArrayOffset, errInvalidJournal); err != nil { + return 0, err + } + } else { + highest = max(highest, h.nextArrayOffset) + } tailEntries -= c offset = h.nextArrayOffset } @@ -169,7 +188,7 @@ func (w *Writer) readOffsetArrayHeader(offset uint64) (offsetArrayHeader, uint64 } func (w *Writer) offsetArrayCapacity(header offsetArrayHeader) (uint64, error) { - if header.object.typ != objectTypeEntryArray || header.object.size < offsetArrayObjectHeaderSize { + if header.object.typ != objectTypeEntryArray || header.object.size <= offsetArrayObjectHeaderSize { return 0, errInvalidJournal } itemSize := w.offsetArrayItemSize() @@ -387,6 +406,7 @@ func (w *Writer) writeCompactDataTail(dataOffset, tailOffset, tailEntries uint64 func (w *Writer) appendToDataEntryArray(arrayOffset, currentCount, entryOffset uint64) (uint64, uint64, error) { remaining := currentCount offset := arrayOffset + highest := offset for { header, cap, err := w.readOffsetArrayHeader(offset) if err != nil { @@ -412,6 +432,13 @@ func (w *Writer) appendToDataEntryArray(arrayOffset, currentCount, entryOffset u } return newOffset, 1, nil } + if header.nextArrayOffset <= highest { + if err := rejectArrayRevisit(w, arrayOffset, offset, header.nextArrayOffset, errInvalidJournal); err != nil { + return 0, 0, err + } + } else { + highest = header.nextArrayOffset + } offset = header.nextArrayOffset } } diff --git a/go/journal/writer_objects.go b/go/journal/writer_objects.go index e67cef94..7e9af573 100644 --- a/go/journal/writer_objects.go +++ b/go/journal/writer_objects.go @@ -166,6 +166,7 @@ func (w *Writer) appendHashItem(tableOffset, tableSize uint64, typ uint8, hash, func (w *Writer) updateHashChainDepth(typ uint8, head uint64) error { var depth uint64 + var chain chainGuard for offset := head; offset != 0; { var next uint64 switch typ { @@ -184,6 +185,9 @@ func (w *Writer) updateHashChainDepth(typ uint8, head uint64) error { default: return errInvalidJournal } + if !chain.advance(offset, next, false) { + return errInvalidJournal + } if next != 0 { depth++ } @@ -212,6 +216,7 @@ func (w *Writer) findData(hash uint64, payload []byte) (uint64, resolvedDataLink } depth := uint64(0) + var chain chainGuard for offset := item.head; offset != 0; { header, err := w.readDataHeader(offset) if err != nil { @@ -237,6 +242,9 @@ func (w *Writer) findData(hash uint64, payload []byte) (uint64, resolvedDataLink w.header.dataHashChainDepth = depth } } + if !chain.advance(offset, header.nextHashOffset, false) { + return 0, resolvedDataLinkState{}, false, errInvalidJournal + } offset = header.nextHashOffset } return 0, resolvedDataLinkState{}, false, nil @@ -269,6 +277,7 @@ func (w *Writer) findField(hash uint64, payload []byte) (uint64, bool, error) { } depth := uint64(0) + var chain chainGuard for offset := item.head; offset != 0; { header, stored, err := w.readFieldObject(offset) if err != nil { @@ -284,6 +293,9 @@ func (w *Writer) findField(hash uint64, payload []byte) (uint64, bool, error) { w.header.fieldHashChainDepth = depth } } + if !chain.advance(offset, header.nextHashOffset, false) { + return 0, false, errInvalidJournal + } offset = header.nextHashOffset } return 0, false, nil diff --git a/rust/Cargo.lock b/rust/Cargo.lock index 19ada219..9f011993 100644 --- a/rust/Cargo.lock +++ b/rust/Cargo.lock @@ -143,6 +143,12 @@ dependencies = [ "serde", ] +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + [[package]] name = "bitflags" version = "2.13.1" @@ -442,6 +448,37 @@ dependencies = [ "uuid", ] +[[package]] +name = "defmt" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e4a0795ac740cb54d7b3775c9c17345f6aaecf884c01733b39d6017738e0e3e1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "840bb056bd99c495c9188198a258ab787c79f306d864db3e6e8451442d6d8366" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "defmt-parser" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71958f69fb2de856c5dd51ac9377e986ca1aa5919f931bf5395a807277f889a0" +dependencies = [ + "thiserror", +] + [[package]] name = "digest" version = "0.11.3" @@ -567,7 +604,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db907f40a527ca2aa2f40a5f68b32ea58aa70f050cd233518e9ffd402cfba6ce" dependencies = [ "anyhow", - "bitflags", + "bitflags 2.13.1", "cmsketch", "equivalent", "foyer-common", @@ -880,7 +917,7 @@ version = "0.11.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "153be1941a183ec9ccd095ddbe17a8b8d435ef6c76e9e02451b933c3999af2c8" dependencies = [ - "bitflags", + "bitflags 2.13.1", "inotify-sys", "libc", ] @@ -900,7 +937,7 @@ version = "0.7.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9080b15e63775b9a2ac7dca720f7050a8b955e092ea0f6020a4a80f69998cdc0" dependencies = [ - "bitflags", + "bitflags 2.13.1", "cfg-if", "libc", ] @@ -935,6 +972,61 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jiff" +version = "0.2.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2b005715dcbeb0089a3c0dab99f2ff1cc3b2525323552703d648585d342a383" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb", + "jiff-tzdb-platform", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e52fe76043ccecc9005d2305ebaadf7d7fc0cc89ca6baa10a94d6bc68c7128c" +dependencies = [ + "defmt", + "log", +] + +[[package]] +name = "jiff-static" +version = "0.2.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cc9817253cf7c7ee4684451bd327e88d6f3658014e54a29198625590650695c" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa8377070c6bae868759445e5a77f66d84f0b72f3a054bfb00e6d038b8282da7" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + [[package]] name = "jobserver" version = "0.1.35" @@ -978,10 +1070,13 @@ name = "journalctl" version = "0.10.0" dependencies = [ "anyhow", + "cc", "chrono", "clap", "hex", + "jiff", "libc", + "pkg-config", "regex", "serde_json", "systemd-journal-sdk", @@ -1017,7 +1112,7 @@ version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "07293a4e297ac234359b510362495713f75ea345d5307140414f20c69ffeb087" dependencies = [ - "bitflags", + "bitflags 2.13.1", "libc", ] @@ -1191,7 +1286,7 @@ version = "8.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d3d07927151ff8575b7087f245456e549fea62edf0ec4e565a5ee50c8402bc3" dependencies = [ - "bitflags", + "bitflags 2.13.1", "fsevent-sys", "inotify", "kqueue", @@ -1209,7 +1304,7 @@ version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42b8cfee0e339a0337359f3c88165702ac6e600dc01c0cc9579a92d62b08477a" dependencies = [ - "bitflags", + "bitflags 2.13.1", ] [[package]] @@ -1356,6 +1451,21 @@ version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" +[[package]] +name = "portable-atomic" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" + +[[package]] +name = "portable-atomic-util" +version = "0.2.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10ab3eb7f3becc3a1cbc4f2c6f20267996cfc1a6467a873763411b136a122715" +dependencies = [ + "portable-atomic", +] + [[package]] name = "ppv-lite86" version = "0.2.21" @@ -1491,7 +1601,7 @@ version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ - "bitflags", + "bitflags 2.13.1", ] [[package]] @@ -1546,7 +1656,7 @@ version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" dependencies = [ - "bitflags", + "bitflags 2.13.1", "errno", "libc", "linux-raw-sys", diff --git a/rust/README.md b/rust/README.md index 92daa76f..2dd72c68 100644 --- a/rust/README.md +++ b/rust/README.md @@ -207,7 +207,7 @@ Current reader scope: - default reader options use live/windowed mmap with a 32 MiB window. Smaller windows are available for constrained environments, but high-cardinality indexed queries can become remap-bound with very small windows; -- file-backed `journalctl` output covers the stock v260.1 short family, +- file-backed `journalctl` output covers the stock v262 short family, `verbose`, `with-unit`, `cat`, `export`, `json`, `json-pretty`, `json-sse`, and `json-seq` modes; `--output-fields` projects requested fields while JSON/export retain stock metadata fields; @@ -509,7 +509,19 @@ inputs. `--vacuum-size`, `--vacuum-files`, and `--vacuum-time` operate on explicit `--directory` inputs, deleting only stock-recognized archived `.journal` and `.journal~` files while protecting active/current and unknown files. -The CLI implements the stock v260.1 output-mode family, `--output-fields` +The CLI targets systemd v262. Building it requires PCRE2-8 development files +(version 10.42 or later), `pkg-config`, and a native C toolchain. PCRE2 and the CLI timezone provider are not dependencies of SDK consumers. +The command links PCRE2 only; it never links system journal libraries. Native +library deployment must accompany dynamically linked command binaries. + +`-F`/`-N` with time, entry, cursor, grep, or explicit boot/invocation restrictions +now fail under v262 (bare `all` remains unscoped); +remove the restriction for whole-file enumeration, or read filtered rows with +the SDK. Grep supports PCRE2 byte syntax and returns exit 1 without a diagnostic +when no entry matches. See [Journalctl CLI](../docs/Journalctl-CLI.md) for migration +and platform prerequisites. + +The CLI implements the stock v262 output-mode family, `--output-fields` projection for `verbose`, `export`, JSON modes, and `cat`, and stock output controls such as `--all`, `--full`, `--no-full`, empty-result messages, and `--pager-end` implicit 1000-line tail behavior. `--output=help` prints the diff --git a/rust/src/adapter/main.rs b/rust/src/adapter/main.rs index 38c2a909..76adb16f 100644 --- a/rust/src/adapter/main.rs +++ b/rust/src/adapter/main.rs @@ -20,6 +20,7 @@ use std::process::Command; use std::time::Instant; mod helpers; +mod reader_probe; use helpers::*; const ADAPTER_VERSION: &str = "0.1.0"; @@ -164,9 +165,10 @@ fn run() -> AnyResult<()> { Some("run") => cmd_run(), Some("list") => cmd_list(), Some("probe") => cmd_probe(), + Some("inspect-reader") => reader_probe::run(), Some("__corrupt_probe") => cmd_corrupt_probe(), Some(other) => Err(anyhow!("unknown subcommand {other}")), - None => Err(anyhow!("usage: adapter [run|list|probe]")), + None => Err(anyhow!("usage: adapter [run|list|probe|inspect-reader]")), } } diff --git a/rust/src/adapter/reader_probe.rs b/rust/src/adapter/reader_probe.rs new file mode 100644 index 00000000..576bcbcd --- /dev/null +++ b/rust/src/adapter/reader_probe.rs @@ -0,0 +1,294 @@ +use anyhow::{Result, anyhow}; +use journal::{DirectoryReader, FileReader, ReaderOptions}; +use serde::Deserialize; +use serde_json::{Value, json}; + +#[derive(Deserialize)] +struct Request { + path: String, + operation: String, + #[serde(default)] + field: String, + #[serde(default)] + match_hex: String, + #[serde(default)] + realtime: u64, + #[serde(default)] + snapshot: bool, + #[serde(default)] + directory: bool, +} + +enum Reader { + File(FileReader), + Directory(DirectoryReader), +} + +macro_rules! call { + ($reader:expr, $method:ident $(, $argument:expr)*) => { + match $reader { + Reader::File(reader) => reader.$method($($argument),*), + Reader::Directory(reader) => reader.$method($($argument),*), + } + }; +} + +fn hex(bytes: &[u8]) -> String { + const DIGITS: &[u8] = b"0123456789abcdef"; + let mut out = String::with_capacity(bytes.len() * 2); + for &byte in bytes { + out.push(DIGITS[(byte >> 4) as usize] as char); + out.push(DIGITS[(byte & 15) as usize] as char); + } + out +} + +fn decode_hex(text: &str) -> Result> { + if !text.len().is_multiple_of(2) || !text.is_ascii() { + return Err(anyhow!("invalid hex input")); + } + text.as_bytes() + .chunks_exact(2) + .map(|pair| { + let high = (pair[0] as char) + .to_digit(16) + .ok_or_else(|| anyhow!("invalid hex input"))?; + let low = (pair[1] as char) + .to_digit(16) + .ok_or_else(|| anyhow!("invalid hex input"))?; + Ok((high * 16 + low) as u8) + }) + .collect() +} + +// Public SDK probe used by shared damaged-file subprocess tests. +pub(super) fn run() -> Result<()> { + let request: Request = serde_json::from_reader(std::io::stdin())?; + let mut result = + json!({"rows":[],"fields":[],"unique":[],"recovery_statuses":[],"error":false}); + if let Err(error) = inspect(&request, &mut result) { + result["error"] = Value::Bool(true); + result["error_message"] = Value::String(error.to_string()); + } + serde_json::to_writer(std::io::stdout(), &result)?; + println!(); + Ok(()) +} + +fn inspect(request: &Request, result: &mut Value) -> Result<()> { + if request.operation == "verify" { + return journal::verify_file(&request.path).map_err(Into::into); + } + if request.operation.starts_with("facade-") { + return inspect_facade(request, result); + } + let options = if request.snapshot { + ReaderOptions::snapshot() + } else { + ReaderOptions::default() + }; + let mut reader = if request.directory { + Reader::Directory(DirectoryReader::open_with_options(&request.path, options)?) + } else { + Reader::File(FileReader::open_with_options(&request.path, options)?) + }; + let inspected = inspect_open(&mut reader, request, result); + result["recovery_statuses"] = Value::Array( + call!(&reader, recovery_statuses) + .iter() + .map(|file| { + let s = file.status; + json!({"file_id":hex(&file.file_id),"status":{ + "reason":s.reason as u8,"first_lost_offset":s.first_lost_offset, + "declared_entries":s.declared_entries,"recovered_entries":s.recovered_entries}}) + }) + .collect(), + ); + inspected +} + +fn inspect_facade(request: &Request, result: &mut Value) -> Result<()> { + let options = if request.snapshot { + ReaderOptions::snapshot() + } else { + ReaderOptions::default() + }; + let mut reader = if request.directory { + journal::SdJournalOpenDirectoryWithOptions(&request.path, 0, options)? + } else { + journal::SdJournalOpenFileWithOptions(&request.path, 0, options)? + }; + let inspected = (|| -> Result<()> { + if !request.match_hex.is_empty() { + journal::SdJournalAddMatch(&mut reader, &decode_hex(&request.match_hex)?)?; + } + let operation = request.operation.strip_prefix("facade-").unwrap(); + match operation { + "fields" => { + result["fields"] = json!(journal::SdJournalEnumerateFields(&mut reader)?); + return Ok(()); + } + "unique" => { + for (_, value) in journal::SdJournalQueryUnique(&mut reader, &request.field)? { + result["unique"] + .as_array_mut() + .unwrap() + .push(Value::String(hex(&value))); + } + return Ok(()); + } + "unique-state" => { + journal::SdJournalQueryUniqueState(&mut reader, &request.field)?; + while let Some(value) = journal::SdJournalEnumerateAvailableUnique(&mut reader)? { + result["unique"] + .as_array_mut() + .unwrap() + .push(Value::String(hex(&value))); + } + return Ok(()); + } + "flush" => { + for _ in 0..2 { + if journal::SdJournalNext(&mut reader)? != 1 { + return Err(anyhow!("missing pre-flush match")); + } + } + journal::SdJournalFlushMatches(&mut reader)?; + } + "forward" => (), + "reverse" => journal::SdJournalSeekTail(&mut reader)?, + _ => return Err(anyhow!("unknown facade probe operation")), + } + loop { + let count = if operation == "reverse" { + journal::SdJournalPrevious(&mut reader)? + } else { + journal::SdJournalNext(&mut reader)? + }; + if count == 0 { + return Ok(()); + } + let entry = journal::SdJournalGetEntry(&mut reader)?; + let mut payloads = entry + .payloads + .iter() + .map(|value| hex(value)) + .collect::>(); + payloads.sort(); + result["rows"] + .as_array_mut() + .unwrap() + .push(json!({"realtime":entry.realtime,"payloads":payloads})); + } + })(); + result["recovery_statuses"] = Value::Array(journal::SdJournalRecoveryStatuses(&reader).iter().map(|file| { + let s = file.status; + json!({"file_id":hex(&file.file_id),"status":{"reason":s.reason as u8,"first_lost_offset":s.first_lost_offset,"declared_entries":s.declared_entries,"recovered_entries":s.recovered_entries}}) + }).collect()); + inspected +} + +fn inspect_open(reader: &mut Reader, request: &Request, result: &mut Value) -> Result<()> { + if !request.match_hex.is_empty() { + call!(&mut *reader, add_match, &decode_hex(&request.match_hex)?); + } + match request.operation.as_str() { + "explorer" | "explorer-filter" | "explorer-facet" | "explorer-histogram" => { + let Reader::File(file) = reader else { + return Err(anyhow!("explorer probe needs one file")); + }; + let mut query = journal::ExplorerQuery { + filters: vec![journal::ExplorerFilter::new( + request.field.as_bytes(), + [b"absent".as_slice()], + )], + field_mode: journal::ExplorerFieldMode::AllValues, + use_source_realtime: false, + ..Default::default() + }; + if request.operation != "explorer" { + query.filters.clear(); + query.limit = 0; + match request.operation.as_str() { + "explorer-filter" => { + query.limit = 1; + query.filters.push(journal::ExplorerFilter::new( + request.field.as_bytes(), + [b"all".as_slice()], + )); + } + "explorer-facet" => query.facets.push(request.field.as_bytes().to_vec()), + _ => query.histogram = Some(request.field.as_bytes().to_vec()), + } + } + file.explore_with_strategy(&query, journal::ExplorerStrategy::Index)?; + return Ok(()); + } + "fields" => { + result["fields"] = json!(call!(reader, enumerate_fields)?); + return Ok(()); + } + "unique" => { + call!( + reader, + visit_unique_values, + &request.field, + |value: &[u8]| { + result["unique"] + .as_array_mut() + .unwrap() + .push(Value::String(hex(value))); + Ok(()) + } + )?; + return Ok(()); + } + "unique-state" => { + call!(&mut *reader, query_unique_state, &request.field)?; + loop { + let value = match &mut *reader { + Reader::File(file) => file.enumerate_unique_payload(&request.field)?, + Reader::Directory(directory) => directory.enumerate_unique_payload()?, + }; + let Some(value) = value else { + return Ok(()); + }; + result["unique"] + .as_array_mut() + .unwrap() + .push(Value::String(hex(&value))); + } + } + "flush" => { + for _ in 0..2 { + if !call!(&mut *reader, next)? { + return Err(anyhow!("missing pre-flush match")); + } + } + call!(&mut *reader, flush_matches); + } + "reverse" => call!(&mut *reader, seek_tail), + "seek-forward" | "seek-reverse" => call!(&mut *reader, seek_realtime, request.realtime), + "forward" => (), + _ => return Err(anyhow!("unsupported probe operation {}", request.operation)), + } + loop { + let found = if request.operation == "reverse" || request.operation == "seek-reverse" { + call!(&mut *reader, previous)? + } else { + call!(&mut *reader, next)? + }; + if !found { + return Ok(()); + } + let realtime = call!(&mut *reader, get_realtime_usec)?; + let mut payloads = Vec::new(); + call!(&mut *reader, collect_entry_payloads, &mut payloads)?; + let mut payloads: Vec<_> = payloads.iter().map(|payload| hex(payload)).collect(); + payloads.sort(); + result["rows"] + .as_array_mut() + .unwrap() + .push(json!({"realtime":realtime,"payloads":payloads})); + } +} diff --git a/rust/src/cmd/journalctl/Cargo.toml b/rust/src/cmd/journalctl/Cargo.toml index 4ecefc2f..cd28828e 100644 --- a/rust/src/cmd/journalctl/Cargo.toml +++ b/rust/src/cmd/journalctl/Cargo.toml @@ -4,7 +4,7 @@ publish = false version.workspace = true edition.workspace = true rust-version.workspace = true -description = "Pure-Rust systemd journalctl rewrite" +description = "File-backed journalctl rewrite with native PCRE2" license.workspace = true [[bin]] @@ -13,6 +13,7 @@ path = "main.rs" [dependencies] journal = { package = "systemd-journal-sdk", path = "../../journal" } +journal-core = { workspace = true } clap = { workspace = true } anyhow = { workspace = true } chrono = { workspace = true } @@ -21,7 +22,11 @@ regex = { workspace = true } libc = { workspace = true } uuid = { workspace = true } serde_json = { workspace = true } +jiff = { version = "=0.2.38", features = ["tzdb-bundle-always"] } [dev-dependencies] -journal-core = { workspace = true } tempfile = { workspace = true } + +[build-dependencies] +cc = "1" +pkg-config = "0.3" diff --git a/rust/src/cmd/journalctl/build.rs b/rust/src/cmd/journalctl/build.rs new file mode 100644 index 00000000..f810e6ef --- /dev/null +++ b/rust/src/cmd/journalctl/build.rs @@ -0,0 +1,14 @@ +fn main() { + let target = std::env::var("CARGO_CFG_TARGET_OS").unwrap(); + if matches!(target.as_str(), "linux" | "macos" | "freebsd") { + println!("cargo:rerun-if-changed=native_unit_pattern.c"); + println!("cargo:rerun-if-changed=../../../../go/cmd/journalctl/unit_pattern_compat.h"); + cc::Build::new() + .file("native_unit_pattern.c") + .compile("journal_unit_pattern"); + } + pkg_config::Config::new() + .atleast_version("10.42") + .probe("libpcre2-8") + .expect("journalctl requires PCRE2-8 development files; the SDK has no PCRE2 dependency"); +} diff --git a/rust/src/cmd/journalctl/colors.rs b/rust/src/cmd/journalctl/colors.rs new file mode 100644 index 00000000..5000e672 --- /dev/null +++ b/rust/src/cmd/journalctl/colors.rs @@ -0,0 +1,171 @@ +use std::io::IsTerminal; + +#[derive(Clone, Copy, Debug, Default)] +pub(super) struct MatchRange { + pub start: usize, + pub end: usize, +} + +#[derive(Clone, Copy, Debug, Default)] +pub(super) struct Colors { + level: u16, + pub underline: bool, +} + +pub(super) const NORMAL: &str = "\x1b[0m"; +pub(super) const HIGHLIGHT: &str = "\x1b[0;1;39m"; +pub(super) const RED: &str = "\x1b[0;1;31m"; +pub(super) const BLUE: &str = "\x1b[0;34m"; +pub(super) const GREEN: &str = "\x1b[0;32m"; + +impl Colors { + pub fn detect() -> Self { + let mode = std::env::var("SYSTEMD_COLORS").unwrap_or_default(); + let level = match mode.as_str() { + "off" => 0, + "16" => 16, + "256" => 256, + "24bit" => 24, + _ => { + let force = super::parse_bool_option("SYSTEMD_COLORS", &mode).ok(); + if force == Some(false) { + return Self::default(); + } + if force != Some(true) { + let term = std::env::var_os("TERM"); + if std::env::var_os("NO_COLOR").is_some() + || term.as_deref().is_none_or(|v| v == "dumb") + || (std::process::id() != 1 + && !(std::io::stdout().is_terminal() + && std::io::stderr().is_terminal())) + { + return Self::default(); + } + } + match mode.as_str() { + "auto-16" => 16, + "auto-256" => 256, + "auto-24bit" => 24, + _ if std::env::var("COLORTERM") + .is_ok_and(|v| matches!(v.as_str(), "truecolor" | "24bit")) => + { + 24 + } + _ => 256, + } + } + }; + Self { + level, + underline: level != 0 + && std::env::var_os("TERM") + .as_deref() + .is_none_or(|v| v != "linux"), + } + } + + pub fn enabled(self) -> bool { + self.level != 0 + } + + pub fn grey(self) -> &'static str { + match self.level { + 16 => "\x1b[0;90m", + 24 => "\x1b[0;38:5:245m", + _ => "\x1b[0;90m\x1b[0;38:5:245m", + } + } + + pub fn log(self, priority: u8, audit: bool) -> (&'static str, &'static str, &'static str) { + if !self.enabled() { + return ("", "", ""); + } + let (mut on, highlight) = match priority { + 0..=3 => (RED, HIGHLIGHT), + 4 => ( + match self.level { + 16 => "\x1b[0;1;33m", + 24 => "\x1b[0;1;38:5:185m", + _ => "\x1b[0;1;33m\x1b[0;1;38:5:185m", + }, + HIGHLIGHT, + ), + 5 => (HIGHLIGHT, RED), + 7.. => (self.grey(), RED), + _ => ("", ""), + }; + if audit && on.is_empty() { + on = BLUE; + } + (on, if on.is_empty() { "" } else { NORMAL }, highlight) + } + + pub fn boot(self, id: &str) -> String { + if self.enabled() { + format!("{HIGHLIGHT}-- Boot {id} --{NORMAL}\n") + } else { + format!("-- Boot {id} --\n") + } + } + + pub fn table_header(self, id: &str) -> String { + if !self.enabled() || !self.underline { + return format!("IDX {id:<32} FIRST ENTRY LAST ENTRY\n"); + } + let on = if self.underline { "\x1b[4m" } else { "" }; + let fields = [ + "IDX".to_owned(), + format!("{id:<32}"), + format!("{:<27}", "FIRST ENTRY"), + format!("{:<27}", "LAST ENTRY"), + ]; + format!( + "{on}{}{NORMAL}\n", + fields.join(&format!("{NORMAL}{on} {NORMAL}{on}")) + ) + } +} + +// This scans an already valid JSON body; callers retain their existing framing. +pub(super) fn json(data: &[u8]) -> Vec { + let mut out = Vec::new(); + let mut i = 0; + while i < data.len() { + let start = i; + match data[i] { + b'"' => { + i += 1; + while i < data.len() && data[i] != b'"' { + if data[i] == b'\\' { + i += 1; + } + i += 1; + } + out.push(b'"'); + out.extend_from_slice(GREEN.as_bytes()); + out.extend_from_slice(&data[start + 1..i]); + out.extend_from_slice(NORMAL.as_bytes()); + out.push(b'"'); + i += 1; + } + b'-' | b'0'..=b'9' | b'n' | b't' | b'f' => { + let on = if matches!(data[i], b'n' | b't' | b'f') { + HIGHLIGHT + } else { + "\x1b[0;1;34m" + }; + while i < data.len() && !b",]} \t\r\n".contains(&data[i]) { + i += 1; + } + out.extend_from_slice(on.as_bytes()); + out.extend_from_slice(&data[start..i]); + out.extend_from_slice(NORMAL.as_bytes()); + } + byte => { + out.push(byte); + i += 1; + } + } + } + out +} diff --git a/rust/src/cmd/journalctl/file_glob.rs b/rust/src/cmd/journalctl/file_glob.rs new file mode 100644 index 00000000..2d1fd6d1 --- /dev/null +++ b/rust/src/cmd/journalctl/file_glob.rs @@ -0,0 +1,285 @@ +use anyhow::{Result, anyhow}; +use std::path::{Path, PathBuf}; + +#[cfg(any(target_os = "linux", target_os = "macos", target_os = "freebsd"))] +pub(super) struct ThreadLocale { + selected: libc::locale_t, + previous: libc::locale_t, +} + +#[cfg(any(target_os = "linux", target_os = "macos", target_os = "freebsd"))] +impl ThreadLocale { + pub(super) fn new() -> Result { + let mut selected = + unsafe { libc::newlocale(libc::LC_ALL_MASK, c"".as_ptr(), std::ptr::null_mut()) }; + if selected.is_null() { + selected = + unsafe { libc::newlocale(libc::LC_ALL_MASK, c"C".as_ptr(), std::ptr::null_mut()) }; + } + if selected.is_null() { + return Err(anyhow!("failed to initialize pattern locale")); + } + let previous = unsafe { libc::uselocale(selected) }; + if previous.is_null() { + unsafe { libc::freelocale(selected) }; + return Err(anyhow!("failed to select pattern locale")); + } + Ok(Self { selected, previous }) + } +} + +#[cfg(any(target_os = "linux", target_os = "macos", target_os = "freebsd"))] +impl Drop for ThreadLocale { + fn drop(&mut self) { + // Restore before releasing the locale still selected by this thread. + unsafe { + libc::uselocale(self.previous); + libc::freelocale(self.selected); + } + } +} + +pub(super) fn expand(pattern: &Path) -> Result> { + if !pattern + .as_os_str() + .as_encoded_bytes() + .iter() + .any(|b| b"*?[]{}".contains(b) || cfg!(unix) && *b == b'\\') + { + return Ok(Vec::new()); + } + expand_platform(pattern) +} + +#[cfg(any(target_os = "linux", target_os = "macos", target_os = "freebsd"))] +fn expand_platform(pattern: &Path) -> Result> { + use std::ffi::{CStr, CString, OsString}; + use std::os::unix::ffi::{OsStrExt, OsStringExt}; + + struct Matches(libc::glob_t); + impl Drop for Matches { + fn drop(&mut self) { + // glob owns all result strings, including partial error results. + unsafe { libc::globfree(&mut self.0) }; + } + } + + let pattern = CString::new(pattern.as_os_str().as_bytes())?; + let _locale = ThreadLocale::new()?; + // The zeroed glob result is valid for glob and globfree on failure. + let mut matches = Matches(unsafe { std::mem::zeroed() }); + #[cfg(target_os = "linux")] + let brace = libc::GLOB_BRACE; + // libc omits this flag from its Apple and FreeBSD bindings. + #[cfg(any(target_os = "macos", target_os = "freebsd"))] + let brace = 0x0080; + let status = unsafe { + libc::glob( + pattern.as_ptr(), + brace | libc::GLOB_NOCHECK | libc::GLOB_NOSORT, + None, + &mut matches.0, + ) + }; + if status != 0 { + return Err(anyhow!("file pattern expansion failed ({status})")); + } + if matches.0.gl_pathc == 0 { + return Ok(Vec::new()); + } + let count = usize::try_from(matches.0.gl_pathc)?; + let paths = unsafe { std::slice::from_raw_parts(matches.0.gl_pathv, count) }; + Ok(paths + .iter() + .map(|path| { + let bytes = unsafe { CStr::from_ptr(*path) }.to_bytes(); + PathBuf::from(OsString::from_vec(bytes.to_vec())) + }) + .collect()) +} + +#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")))] +fn expand_platform(pattern: &Path) -> Result> { + let text = pattern + .to_str() + .ok_or_else(|| anyhow!("file patterns require Unicode filenames on this platform"))?; + let mut paths = Vec::new(); + for alternative in brace_alternatives(text) { + paths.extend( + expand_unicode(Path::new(&alternative))? + .into_iter() + .filter(|p| p.exists()), + ); + } + Ok(paths) +} + +#[cfg(any( + test, + not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")) +))] +fn brace_alternatives(pattern: &str) -> Vec { + for (start, byte) in pattern.bytes().enumerate() { + if byte != b'{' { + continue; + } + let (mut depth, mut part) = (1, start + 1); + let mut alternatives = Vec::new(); + for (end, byte) in pattern.bytes().enumerate().skip(start + 1) { + match byte { + b'{' => depth += 1, + b',' if depth == 1 => { + alternatives.push(&pattern[part..end]); + part = end + 1; + } + b'}' => { + depth -= 1; + if depth != 0 { + continue; + } + if !alternatives.is_empty() { + alternatives.push(&pattern[part..end]); + return alternatives + .into_iter() + .flat_map(|value| { + brace_alternatives(&format!( + "{}{value}{}", + &pattern[..start], + &pattern[end + 1..] + )) + }) + .collect(); + } + break; + } + _ => {} + } + } + } + vec![pattern.to_owned()] +} + +#[cfg(any( + test, + not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")) +))] +fn expand_unicode(pattern: &Path) -> Result> { + let mut prefixes = Vec::::new(); + for component in pattern.components() { + match component { + std::path::Component::Prefix(prefix) => { + prefixes = vec![PathBuf::from(prefix.as_os_str())] + } + std::path::Component::RootDir => { + if prefixes.is_empty() { + prefixes.push(PathBuf::from(std::path::MAIN_SEPARATOR.to_string())); + } else { + for prefix in &mut prefixes { + prefix.push(std::path::MAIN_SEPARATOR.to_string()); + } + } + } + std::path::Component::CurDir | std::path::Component::ParentDir => { + if prefixes.is_empty() { + prefixes.push(PathBuf::new()); + } + for prefix in &mut prefixes { + prefix.push(component.as_os_str()); + } + } + std::path::Component::Normal(component) => { + if prefixes.is_empty() { + prefixes.push(PathBuf::new()); + } + let text = component + .to_str() + .ok_or_else(|| anyhow!("Unicode file pattern required"))?; + if super::is_glob_pattern(text) { + let matcher = regex::Regex::new(&super::glob_pattern_to_regex(text))?; + let mut next = Vec::new(); + for prefix in &prefixes { + let dir = if prefix.as_os_str().is_empty() { + Path::new(".") + } else { + prefix + }; + let Ok(entries) = std::fs::read_dir(dir) else { + continue; + }; + for entry in entries.flatten() { + let name = entry.file_name(); + let Some(name_text) = name.to_str() else { + continue; + }; + if name_text.starts_with('.') && !text.starts_with('.') { + continue; + } + if matcher.is_match(name_text) { + next.push(prefix.join(name)); + } + } + } + if next.is_empty() { + return Ok(Vec::new()); + } + prefixes = next; + } else { + for prefix in &mut prefixes { + prefix.push(component); + } + } + } + } + } + prefixes.sort(); + Ok(prefixes) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn unicode_brace_alternatives() { + assert_eq!( + brace_alternatives("{a,{b,c}}-é.journal"), + ["a-é.journal", "b-é.journal", "c-é.journal"] + ); + assert_eq!(brace_alternatives("{plain}.journal"), ["{plain}.journal"]); + assert_eq!( + brace_alternatives("{{a,b}}.journal"), + ["{a}.journal", "{b}.journal"] + ); + assert_eq!( + brace_alternatives("{a,}?.journal"), + ["a?.journal", "?.journal"] + ); + } + + #[test] + fn exhausted_unicode_wildcard_stays_empty() { + let directory = tempfile::tempdir().unwrap(); + for suffix in ["missing*/system.journal", "missing*/../system.journal"] { + assert!( + expand_unicode(&directory.path().join(suffix)) + .unwrap() + .is_empty() + ); + } + } + + #[test] + fn unicode_wildcards_require_explicit_dot() { + let directory = tempfile::tempdir().unwrap(); + std::fs::write(directory.path().join("é.journal"), b"").unwrap(); + std::fs::write(directory.path().join(".hidden.journal"), b"").unwrap(); + assert_eq!( + expand_unicode(&directory.path().join("?.journal")).unwrap(), + [directory.path().join("é.journal")] + ); + assert_eq!( + expand_unicode(&directory.path().join(".*.journal")).unwrap(), + [directory.path().join(".hidden.journal")] + ); + } +} diff --git a/rust/src/cmd/journalctl/invocation.rs b/rust/src/cmd/journalctl/invocation.rs new file mode 100644 index 00000000..e85c164b --- /dev/null +++ b/rust/src/cmd/journalctl/invocation.rs @@ -0,0 +1,409 @@ +use super::*; +use journal::{SdJournalGetData, SdJournalGetRealtimeUsec}; + +fn single_unit(journal: &mut SdJournal, args: &Args, option: &str) -> Result { + let (system, user) = raw_unit_specs(args); + match system.len() + user.len() { + 0 => { + return Err(anyhow!( + "Using {option} requires a unit. Please specify a unit name with -u/--unit=/--user-unit=." + )); + } + 1 => {} + _ => { + return Err(anyhow!( + "Using {option} with multiple units is not supported." + )); + } + } + let (system, user) = effective_unit_specs(args)?; + let (specs, fields) = if !user.is_empty() { + (user, &USER_UNIT_FIELDS_FULL[..3]) + } else { + (system, &SYSTEM_UNIT_FIELDS_FULL[..3]) + }; + let units = expand_unit_specs(journal, specs, fields)?; + if units.is_empty() { + return Err(anyhow!( + "No matching unit found for '{}' in journal.", + String::from_utf8_lossy(&specs[0]) + )); + } + if units.len() > 1 { + return Err(anyhow!( + "Multiple matching units found for '{}' in journal.", + String::from_utf8_lossy(&specs[0]) + )); + } + Ok(if !user.is_empty() { + (Vec::new(), units) + } else { + (units, Vec::new()) + }) +} + +fn boot_id(journal: &mut SdJournal, args: &Args) -> Result { + match effective_boot_descriptor(args) { + Some(boot) if boot.trim() != "all" => resolve_boot_id(journal, boot.trim()), + _ => Ok(String::new()), + } +} + +fn add_boot(journal: &mut SdJournal, boot: &str) -> Result<()> { + if !boot.is_empty() { + add_field_matches(journal, "_BOOT_ID", [boot.to_string()])?; + } + Ok(()) +} + +fn parse_id(value: &[u8]) -> Option<[u8; 16]> { + if value.len() > 37 { + return None; + } + let value = value.split(|c| *c == 0).next()?; + let mut clean = [0; 32]; + let value = match value.len() { + 32 => value, + 36 => { + if [8, 13, 18, 23].iter().any(|&i| value[i] != b'-') { + return None; + } + let mut i = 0; + for (pos, &c) in value.iter().enumerate() { + if [8, 13, 18, 23].contains(&pos) { + continue; + } + clean[i] = c; + i += 1; + } + &clean + } + _ => return None, + }; + let mut id = [0; 16]; + hex::decode_to_slice(value, &mut id).ok()?; + (id != [0; 16]).then_some(id) +} + +fn eligibility(journal: &mut SdJournal) -> Result>> { + let mut matches = Vec::new(); + for field in INVOCATION_ID_FIELDS { + SdJournalVisitUniqueValues(journal, field, |value| { + if parse_id(value).is_some() { + let mut payload = format!("{field}=").into_bytes(); + payload.extend_from_slice(value); + matches.push(payload); + } + Ok(()) + })?; + } + Ok(matches) +} + +fn primary_id(journal: &mut SdJournal) -> Result> { + for field in INVOCATION_ID_FIELDS { + let payload = match SdJournalGetData(journal, field) { + Ok(payload) => payload, + Err(FacadeError::NoEntry) => continue, + Err(err) => return Err(err.into()), + }; + if let Some(id) = payload.get(field.len() + 1..).and_then(parse_id) { + return Ok(Some(hex::encode(id))); + } + } + Ok(None) +} + +// Keep the discovery predicate fixed while a second indexed reader supplies +// global bounds. Metadata getters avoid expanding unrelated DATA payloads. +struct Walker { + scan: SdJournal, + bounds: Option, + boot: String, + older: bool, + done: bool, + anchor: Option, + previous: Option, + units: UnitSpecs, + eligibility: Vec>, + unrestricted: bool, +} + +impl Walker { + fn new( + input: &CliInput, + args: &Args, + option: Option<&str>, + older: bool, + show: bool, + ) -> Result { + let mut scan = input.open_journal()?; + // Show validates boot before unit; invocation tables acquire unit first. + let mut boot = if show { + boot_id(&mut scan, args)? + } else { + String::new() + }; + let (system, user) = match option { + Some(option) => single_unit(&mut scan, args, option)?, + None => (Vec::new(), Vec::new()), + }; + if !show { + boot = boot_id(&mut scan, args)?; + } + let eligibility = if option.is_some() { + eligibility(&mut scan)? + } else { + Vec::new() + }; + let done = option.is_some() && eligibility.is_empty(); + let mut walker = Self { + scan, + bounds: if done { + None + } else { + Some(input.open_journal()?) + }, + boot, + older, + done, + anchor: None, + previous: None, + units: (system, user), + eligibility, + unrestricted: false, + }; + if done { + return Ok(walker); + } + walker.reset_scan(false)?; + if older { + SdJournalSeekTail(&mut walker.scan)?; + } else { + SdJournalSeekHead(&mut walker.scan)?; + } + Ok(walker) + } + + fn add_unit_scope(journal: &mut SdJournal, units: &UnitSpecs) -> Result<()> { + for unit in &units.0 { + add_system_unit_match_groups(journal, unit, false)?; + } + let uid = current_uid_string(); + for unit in &units.1 { + add_user_unit_match_groups(journal, unit, uid.as_deref(), false)?; + } + if !units.0.is_empty() || !units.1.is_empty() { + SdJournalAddConjunction(journal)?; + } + Ok(()) + } + + fn reset_scan(&mut self, unrestricted: bool) -> Result<()> { + SdJournalFlushMatches(&mut self.scan)?; + if !unrestricted { + add_boot(&mut self.scan, &self.boot)?; + Self::add_unit_scope(&mut self.scan, &self.units)?; + } + for (i, payload) in self.eligibility.iter().enumerate() { + if i > 0 { + SdJournalAddDisjunction(&mut self.scan)?; + } + SdJournalAddMatch(&mut self.scan, payload)?; + } + if !self.eligibility.is_empty() { + SdJournalAddConjunction(&mut self.scan)?; + } + self.unrestricted = unrestricted; + Ok(()) + } + + fn bound(&mut self, id: &str) -> Result> { + let Some(j) = self.bounds.as_mut() else { + return Ok(None); + }; + SdJournalFlushMatches(j)?; + add_boot(j, &self.boot)?; + if id.is_empty() { + Self::add_unit_scope(j, &self.units)?; + } else { + add_invocation_matches(j, id)?; + } + SdJournalSeekHead(j)?; + if SdJournalNext(j)? == 0 { + return Ok(None); + } + let first_usec = SdJournalGetRealtimeUsec(j)?; + let mut anchor = if self.older { + SdJournalGetCursor(j)? + } else { + String::new() + }; + SdJournalSeekTail(j)?; + if SdJournalPrevious(j)? == 0 { + return Ok(None); + } + let last_usec = SdJournalGetRealtimeUsec(j)?; + if !self.older { + anchor = SdJournalGetCursor(j)?; + } + Ok(Some(( + InvocationEntry { + id: id.to_string(), + first_usec, + last_usec, + }, + anchor, + ))) + } + + fn step(&mut self) -> Result { + if let Some(anchor) = self.anchor.take() { + SdJournalSeekCursorDirection( + &mut self.scan, + &anchor, + if self.older { + journal::Direction::Backward + } else { + journal::Direction::Forward + }, + )?; + match SdJournalTestCursor(&self.scan, &anchor) { + Ok(false) => return Ok(true), + Ok(true) => {} + Err(FacadeError::NoEntry | FacadeError::EndOfEntries) => return Ok(false), + Err(err) => return Err(err.into()), + } + } + Ok(if self.older { + SdJournalPrevious(&mut self.scan)? + } else { + SdJournalNext(&mut self.scan)? + } != 0) + } + + fn next(&mut self) -> Result> { + if self.done { + return Ok(None); + } + if self.unrestricted { + self.reset_scan(false)?; + } + let mut broken = HashSet::new(); + loop { + if !self.step()? { + self.done = true; + return Ok(None); + } + let Some(id) = primary_id(&mut self.scan)? else { + continue; + }; + if self.previous.as_deref() == Some(&id) || broken.contains(&id) { + continue; + } + let Some((info, anchor)) = self.bound(&id)? else { + broken.insert(id); + let previous = self.previous.clone().unwrap_or_default(); + let Some((_, anchor)) = self.bound(&previous)? else { + return Err(anyhow!( + "Cannot seek to the previous invocation: No data available" + )); + }; + self.anchor = Some(anchor); + // Native discovery resumes without unit/boot matches after a broken ID. + self.reset_scan(true)?; + continue; + }; + self.anchor = Some(anchor); + self.previous = Some(id); + return Ok(Some(info)); + } + } +} + +pub(super) fn collect_invocations_from_input( + input: &CliInput, + args: &Args, + unit_option: Option<&str>, +) -> Result> { + let (older, max) = match parse_lines_limit(args.lines.as_deref())? { + Some(LinesLimit::Head(n)) => (false, n), + Some(LinesLimit::Tail(n)) => (true, n), + _ => (false, usize::MAX), + }; + if max == 0 { + let mut journal = input.open_journal()?; + if let Some(option) = unit_option { + single_unit(&mut journal, args, option)?; + } + boot_id(&mut journal, args)?; + return Ok(Vec::new()); + } + let mut walker = Walker::new(input, args, unit_option, older, false)?; + let mut out = Vec::new(); + let mut seen = HashSet::new(); + while out.len() < max { + let Some(info) = walker.next()? else { + break; + }; + if !seen.insert(info.id.clone()) { + break; + } + out.push(info); + } + if older { + out.reverse(); + } + Ok(out) +} + +pub(super) fn resolve_invocation_filter(input: &CliInput, args: &Args) -> Result> { + if args.invocation.is_none() && !args.invocation_latest { + return Ok(None); + } + args.resolved_invocation + .get_or_init(|| find_filter(input, args).map_err(|e| e.to_string())) + .as_ref() + .cloned() + .map_err(|e| anyhow!("{e}")) +} + +fn find_filter(input: &CliInput, args: &Args) -> Result> { + let Some(descriptor) = effective_invocation_descriptor(args) else { + return Ok(None); + }; + let Some((id, offset)) = parse_invocation_descriptor(descriptor)? else { + return Ok(None); + }; + let option = (id.is_empty() || offset != 0).then_some("-I/--invocation= with an offset"); + let mut walker = Walker::new(input, args, option, offset <= 0, true)?; + let not_found = || { + anyhow!( + "No journal entry found for the invocation ({}{offset:+}).", + id + ) + }; + let mut off = if offset <= 0 { 0 } else { 1 }; + if !id.is_empty() { + let Some((_, anchor)) = walker.bound(&id)? else { + return Err(not_found()); + }; + if offset == 0 { + return Ok(Some(id)); + } + walker.anchor = Some(anchor); + walker.previous = Some(id.clone()); + if offset < 0 { + off = -1; + } + } + loop { + let Some(info) = walker.next()? else { + return Err(not_found()); + }; + if off == offset { + return Ok(Some(info.id)); + } + off += if walker.older { -1 } else { 1 }; + } +} diff --git a/rust/src/cmd/journalctl/main.rs b/rust/src/cmd/journalctl/main.rs index ed13feb0..fefed544 100644 --- a/rust/src/cmd/journalctl/main.rs +++ b/rust/src/cmd/journalctl/main.rs @@ -1,19 +1,36 @@ +mod colors; +mod file_glob; +mod invocation; +use invocation::{collect_invocations_from_input, resolve_invocation_filter}; mod output; +mod output_links; +mod parser; +mod pcre; +mod terminal; +mod timestamp; +mod timestamp_zone; +mod unit_name; +mod unit_pattern; +mod vacuum_metadata; +mod vacuum_order; use anyhow::{Result, anyhow}; -use chrono::{Local, NaiveDate, NaiveDateTime, NaiveTime, TimeZone}; +use clap::builder::TypedValueParser; use clap::{Parser, ValueEnum}; use journal::{ Entry, FacadeError, FileHeader, FileReader, OutputMode, SdJournal, SdJournalAddConjunction, SdJournalAddDisjunction, SdJournalAddMatch, SdJournalEnumerateFields, SdJournalFlushMatches, - SdJournalGetEntry, SdJournalNext, SdJournalOpenDirectory, SdJournalOpenFiles, - SdJournalPrevious, SdJournalSeekCursor, SdJournalSeekHead, SdJournalSeekRealtimeUsec, - SdJournalSeekTail, SdJournalSetOutputMode, SdJournalTestCursor, SdJournalVisitUniqueValues, - parse_match_string, verify_file, verify_file_with_key, + SdJournalGetCursor, SdJournalGetEntry, SdJournalNext, SdJournalOpenDirectoryPath, + SdJournalOpenFilePaths, SdJournalPrevious, SdJournalSeekCursorDirection, SdJournalSeekHead, + SdJournalSeekRealtimeUsec, SdJournalSeekTail, SdJournalSetOutputMode, SdJournalTestCursor, + SdJournalVisitUniqueValues, parse_match_bytes, parse_match_string, verify_file, + verify_file_with_key, }; use output::{OutputOptions, OutputRenderer}; -use regex::{Regex, RegexBuilder}; +use regex::Regex; +use std::cell::{Cell, OnceCell}; use std::collections::{HashSet, VecDeque}; +use std::ffi::{OsStr, OsString}; use std::fs; use std::io::{ErrorKind, Read, Seek, SeekFrom, Write}; use std::path::{Path, PathBuf}; @@ -87,7 +104,7 @@ const OUTPUT_MODE_HELP_LIST: &[&str] = &[ "with-unit", ]; -const OFFICIAL_OPTION_SURFACE_HELP: &str = r#"Official systemd v260.1 option reference: +const OFFICIAL_OPTION_SURFACE_HELP: &str = r#"Official systemd v262 option reference: --system Show system journals when host-backed; no-op for explicit files. --user Show user journals when host-backed; no-op for explicit files. -M, --machine=CONTAINER Unsupported: requires host/container journal access. @@ -201,11 +218,26 @@ fn unsupported_reason(name: &str) -> &'static str { } } +type UnitSpecs = (Vec>, Vec>); + #[derive(Parser, Debug)] -#[command(name = "journalctl")] -#[command(about = "Pure-Rust file-backed journalctl subset")] +#[command(name = "journalctl", args_override_self = true)] +#[command(about = "Portable file-backed journalctl")] #[command(after_help = OFFICIAL_OPTION_SURFACE_HELP)] struct Args { + #[arg(skip)] + action: Option<&'static str>, + #[arg(skip)] + unit_specs: OnceCell>, + #[arg(skip)] + unit_matches: OnceCell>, + #[arg(skip)] + resolved_invocation: OnceCell, String>>, + #[arg(skip)] + since_usec: Option, + #[arg(skip)] + until_usec: Option, + #[arg(short = 'i', long = "file")] file: Vec, #[arg(short = 'D', long = "directory")] @@ -246,14 +278,14 @@ struct Args { verify_only: bool, #[arg(long = "verify-key")] verify_key: Option, - matches: Vec, + matches: Vec, - // Parser-recognized v260.1 options that the Rust CLI does not yet + // Parser-recognized v262 options that the Rust CLI does not yet // dispatch behavior for. They exist so the parser accepts every // official long option; downstream validation rejects the ones that // are intentionally unsupported in portable mode with a portable // unsupported message. This keeps the parser in lock-step with the - // shared v260.1 manifest under tests/parser-parity/. + // shared v262 manifest under tests/parser-parity/. #[arg(long = "system")] system: bool, #[arg(long = "user")] @@ -269,34 +301,34 @@ struct Args { #[arg(long = "image-policy")] image_policy: Option, #[arg(long = "namespace")] - namespace: Option, + namespace: Option, #[arg(short = 'c', long = "cursor")] cursor: Option, #[arg(long = "after-cursor")] after_cursor: Option, - #[arg(long = "cursor-file")] + #[arg(long = "cursor-file", value_parser = clap::builder::OsStringValueParser::new().map(PathBuf::from))] cursor_file: Option, #[arg(long = "this-boot")] this_boot: bool, #[arg(short = 'u', long = "unit")] - unit: Vec, + unit: Vec, #[arg(long = "user-unit")] - user_unit: Vec, + user_unit: Vec, #[arg(long = "invocation")] invocation: Option, #[arg(short = 'I')] invocation_latest: bool, #[arg(short = 't', long = "identifier")] - identifier: Vec, + identifier: Vec, #[arg(short = 'T', long = "exclude-identifier")] - exclude_identifier: Vec, + exclude_identifier: Vec, #[arg(short = 'p', long = "priority")] priority: Vec, #[arg(long = "facility")] facility: Vec, #[arg(short = 'g', long = "grep")] - grep: Option, + grep: Option, #[arg(long = "case-sensitive", num_args = 0..=1, default_missing_value = "true")] case_sensitive: Option, #[arg(short = 'k', long = "dmesg")] @@ -331,7 +363,7 @@ struct Args { #[arg(short = 'e', long = "pager-end")] pager_end: bool, #[arg(long = "output-fields")] - output_fields: Option, + output_fields: Vec, #[arg(long = "interval")] interval: Option, @@ -433,7 +465,9 @@ impl From for OutputMode { fn main() { if let Err(err) = run() { - eprintln!("Error: {err}"); + if !err.is::() { + eprintln!("Error: {err}"); + } exit(1); } } @@ -442,54 +476,29 @@ fn portable_unsupported(feature: &str, reason: &str) -> anyhow::Error { anyhow!("journalctl portable mode does not support {feature}: {reason}") } -fn enforce_stock_invalid_short_equals(raw_args: &[String]) -> Result<()> { - for arg in raw_args.iter().skip(1) { - if arg == "--" { - break; - } - if arg.starts_with("-n=") { - let value = &arg[2..]; - return Err(anyhow!("Failed to parse --lines='{value}'.")); - } - } - Ok(()) -} - fn run() -> Result<()> { - // nosemgrep: rust.lang.security.args.args -- CLI entry point parses argv; not an authorization boundary. - let raw_args: Vec = std::env::args().collect(); - enforce_stock_invalid_short_equals(&raw_args)?; - let full_width = resolve_full_width(&raw_args); - let args = Args::parse_from(preprocess_optional_boot_args(raw_args)); - if matches!(args.output, OutputModeArg::Help) { - print_output_mode_help(); + let Some(args) = parser::parse_cli(std::env::args_os().collect())? else { return Ok(()); - } - - // Enforce the v260.1 parser-required interaction rules. These run + }; + let full_width = args.full; + // Enforce the v262 parser-required interaction rules. These run // before any dispatch so the user sees an explicit conflict error. enforce_source_exclusivity(&args)?; - enforce_since_until_order(&args.since, &args.until)?; + if let (Some(s), Some(u)) = (args.since_usec, args.until_usec) { + if s > u { + return Err(anyhow!("--since= must be before --until=.")); + } + } enforce_cursor_source_exclusivity(&args)?; enforce_follow_reverse_conflict(&args)?; enforce_oldest_lines_conflict(&args)?; enforce_case_sensitive_value(&args)?; enforce_boot_descriptor_value(&args)?; - if args.version { - println!("journalctl (systemd-journal-sdk Rust rewrite)"); - println!("baseline: systemd v260.1 (c0a5a2516d28)"); - println!("portable file-backed mode"); - return Ok(()); - } - - if facility_help_requested(&args.facility) { - print_facility_help(args.quiet); - return Ok(()); - } - enforce_action_argument_restriction(&args)?; + enforce_field_enumeration_scope(&args)?; enforce_boot_merge_conflict(&args)?; + let post_filters = CliPostFilters::from_args(&args)?; enforce_portable_unsupported(&args)?; // Portable utility actions that do not require a journal file: @@ -532,122 +541,188 @@ fn run() -> Result<()> { return run_list_invocations(&input, &args); } - if args.verify || args.verify_only || args.verify_key.is_some() { - return run_verify_input(&input, args.verify_key.as_deref()); + if args.verify || args.verify_only { + return run_verify_input(&input, args.verify_key.as_deref(), effective_quiet(&args)); } if args.list_boots { return run_list_boots(&input, &args); } - let since_usec = parse_optional_timestamp(args.since.as_deref())?; - let until_usec = parse_optional_timestamp(args.until.as_deref())?; - let post_filters = CliPostFilters::from_args(&args)?; + let since_usec = args.since_usec; + let until_usec = args.until_usec; let cursor_control = CursorControl::from_args(&args)?; let output_options = OutputOptions::from_args(&args, full_width); + post_filters.capture_highlight.set( + (output_options.colors().enabled() || output_options.urlify()) + && (matches!(args.output, OutputModeArg::Cat) + || output_uses_exclude_identifier(args.output)), + ); - if args.follow { - let tail = parse_tail_count(args.tail.as_ref(), args.lines.as_deref()).unwrap_or(10); + if args.follow && is_show_action(&args) { + let (tail, all) = match parse_lines_limit(args.lines.as_deref())? { + Some(LinesLimit::All) => (0, true), + Some(LinesLimit::Tail(n) | LinesLimit::Head(n)) => (n, false), + None => match args.tail { + Some(n) => (n, false), + None if args.since.is_none() => (10, false), + None if args.pager_end => (1000, false), + None => (0, true), + }, + }; return run_follow( &input, &args, since_usec, until_usec, tail, + all || args.no_tail, &post_filters, &cursor_control, &output_options, ); } - let mut journal = open_filtered_journal(&input, &args, None)?; + let result = (|| -> Result<()> { + let mut journal = open_filtered_journal(&input, &args, None)?; - if args.fields { - let mut fields = - SdJournalEnumerateFields(&mut journal).map_err(|err| anyhow!("fields: {err}"))?; - fields.sort(); - for field in fields { - println!("{field}"); + if args.fields { + let mut fields = + SdJournalEnumerateFields(&mut journal).map_err(|err| anyhow!("fields: {err}"))?; + fields.sort(); + for field in fields { + println!("{field}"); + } + return Ok(()); } - return Ok(()); - } - if let Some(field) = args.field.as_deref() { - let mut stdout = std::io::stdout().lock(); - SdJournalVisitUniqueValues(&mut journal, field, |value| { - stdout - .write_all(value) - .map_err(|err| FacadeError::Other(err.to_string()))?; - stdout - .write_all(b"\n") - .map_err(|err| FacadeError::Other(err.to_string()))?; - Ok(()) - }) - .map_err(|err| anyhow!("field: {err}"))?; - return Ok(()); - } + if let Some(field) = args.field.as_deref() { + if !output::valid_field_name(field) { + return Err(anyhow!("Invalid field name: {field}")); + } + let limit = match parse_lines_limit(args.lines.as_deref())? { + Some(LinesLimit::Head(n) | LinesLimit::Tail(n)) => Some(n), + _ => None, + }; + if limit == Some(0) { + return Ok(()); + } + let mut count = 0usize; + let mut stopped_at_limit = false; + let mut stdout = std::io::stdout().lock(); + let result = SdJournalVisitUniqueValues(&mut journal, field, |value| { + if limit == Some(count) { + stopped_at_limit = true; + return Err(FacadeError::Other("unique-limit".into())); + } + let value = value.split(|&b| b == 0).next().unwrap_or_default(); + stdout + .write_all(value) + .map_err(|err| FacadeError::Other(err.to_string()))?; + stdout + .write_all(b"\n") + .map_err(|err| FacadeError::Other(err.to_string()))?; + count += 1; + Ok(()) + }); + if let Err(err) = result { + if !stopped_at_limit { + return Err(anyhow!("field: {err}")); + } + } + return Ok(()); + } - // If --lines is set, it acts as an alternative --head / --tail. - let grep_tail_reverse = grep_tail_implies_reverse(&args)?; - if let Some(limit) = parse_lines_limit(args.lines.as_deref())? { - return match limit { - LinesLimit::All => show_head_or_all_with_reverse( - &mut journal, - None, - since_usec, - until_usec, - args.reverse, - args.show_cursor, - effective_quiet(&args), - &post_filters, - &cursor_control, - &output_options, - ), - LinesLimit::Head(n) => show_head_or_all_with_reverse( - &mut journal, - Some(n), - since_usec, - until_usec, - false, - args.show_cursor, - effective_quiet(&args), - &post_filters, - &cursor_control, - &output_options, - ), - LinesLimit::Tail(n) => show_tail_with_reverse( + // If --lines is set, it acts as an alternative --head / --tail. + let grep_tail_reverse = grep_tail_implies_reverse(&args)?; + if let Some(limit) = parse_lines_limit(args.lines.as_deref())? { + return match limit { + LinesLimit::All => show_head_or_all_with_reverse( + &mut journal, + None, + since_usec, + until_usec, + args.reverse, + args.show_cursor, + effective_quiet(&args), + &post_filters, + &cursor_control, + &output_options, + ), + LinesLimit::Head(n) => show_head_or_all_with_reverse( + &mut journal, + Some(n), + since_usec, + until_usec, + false, + args.show_cursor, + effective_quiet(&args), + &post_filters, + &cursor_control, + &output_options, + ), + LinesLimit::Tail(n) + if cursor_control.seek.is_some() && !args.reverse && !grep_tail_reverse => + { + show_head_or_all_with_reverse( + &mut journal, + Some(n), + since_usec, + until_usec, + false, + args.show_cursor, + effective_quiet(&args), + &post_filters, + &cursor_control, + &output_options, + ) + } + LinesLimit::Tail(n) => show_tail_with_reverse( + &mut journal, + n, + since_usec, + until_usec, + args.reverse || grep_tail_reverse, + args.show_cursor, + effective_quiet(&args), + &post_filters, + &cursor_control, + &output_options, + ), + }; + } + + let reverse = args.reverse; + if let Some(n) = args.tail { + show_tail_with_reverse( &mut journal, n, since_usec, until_usec, - args.reverse || grep_tail_reverse, + reverse, args.show_cursor, effective_quiet(&args), &post_filters, &cursor_control, &output_options, - ), - }; - } - - let reverse = args.reverse; - if let Some(n) = args.tail { - show_tail_with_reverse( - &mut journal, - n, - since_usec, - until_usec, - reverse, - args.show_cursor, - effective_quiet(&args), - &post_filters, - &cursor_control, - &output_options, - ) - } else { - if args.pager_end && args.head.is_none() { - return show_tail_with_reverse( + ) + } else { + if args.pager_end && args.head.is_none() { + return show_tail_with_reverse( + &mut journal, + 1000, + since_usec, + until_usec, + reverse || grep_tail_reverse, + args.show_cursor, + effective_quiet(&args), + &post_filters, + &cursor_control, + &output_options, + ); + } + show_head_or_all_with_reverse( &mut journal, - 1000, + args.head, since_usec, until_usec, reverse, @@ -656,33 +731,13 @@ fn run() -> Result<()> { &post_filters, &cursor_control, &output_options, - ); - } - show_head_or_all_with_reverse( - &mut journal, - args.head, - since_usec, - until_usec, - reverse, - args.show_cursor, - effective_quiet(&args), - &post_filters, - &cursor_control, - &output_options, - ) - } -} - -fn resolve_full_width(args: &[String]) -> bool { - let mut full_width = true; - for arg in args { - match arg.as_str() { - "--no-full" => full_width = false, - "--full" | "-l" => full_width = true, - _ => {} + ) } + })(); + if result.is_ok() && is_show_action(&args) && args.grep.is_some() && !post_filters.shown.get() { + return Err(SilentNoData.into()); } - full_width + result } fn print_output_mode_help() { @@ -714,15 +769,11 @@ impl CliInput { fn open_journal(&self) -> Result { match self { - Self::Directory(path) => SdJournalOpenDirectory(&path.to_string_lossy(), 0) - .map_err(|err| anyhow!("open: {err}")), + Self::Directory(path) => { + SdJournalOpenDirectoryPath(path, 0).map_err(|err| anyhow!("open: {err}")) + } Self::Files(paths) => { - let strings: Vec = paths - .iter() - .map(|path| path.to_string_lossy().into_owned()) - .collect(); - let refs: Vec<&str> = strings.iter().map(String::as_str).collect(); - SdJournalOpenFiles(&refs, 0).map_err(|err| anyhow!("open: {err}")) + SdJournalOpenFilePaths(paths, 0).map_err(|err| anyhow!("open: {err}")) } } } @@ -747,7 +798,7 @@ fn resolve_file_inputs(values: &[PathBuf]) -> Result> { "stdin-backed journals require seekable mmap-capable file descriptors and are not supported in portable mode", )); } - let matches = expand_glob_path(value); + let matches = file_glob::expand(value)?; if matches.is_empty() { push_unique_file(&mut files, &mut seen, value.clone()); } else { @@ -766,87 +817,6 @@ fn push_unique_file(files: &mut Vec, seen: &mut HashSet, path: } } -fn expand_glob_path(pattern: &Path) -> Vec { - let pattern_text = pattern.to_string_lossy(); - if !is_glob_pattern(&pattern_text) { - return Vec::new(); - } - - let mut prefixes = Vec::::new(); - for component in pattern.components() { - match component { - std::path::Component::Prefix(prefix) => { - prefixes = vec![PathBuf::from(prefix.as_os_str())]; - } - std::path::Component::RootDir => { - if prefixes.is_empty() { - prefixes.push(PathBuf::from(std::path::MAIN_SEPARATOR.to_string())); - } else { - for prefix in &mut prefixes { - prefix.push(std::path::MAIN_SEPARATOR.to_string()); - } - } - } - std::path::Component::CurDir => { - if prefixes.is_empty() { - prefixes.push(PathBuf::from(".")); - } else { - for prefix in &mut prefixes { - prefix.push("."); - } - } - } - std::path::Component::ParentDir => { - if prefixes.is_empty() { - prefixes.push(PathBuf::from("..")); - } else { - for prefix in &mut prefixes { - prefix.push(".."); - } - } - } - std::path::Component::Normal(component) => { - if prefixes.is_empty() { - prefixes.push(PathBuf::new()); - } - let component_pattern = component.to_string_lossy(); - if is_glob_pattern(&component_pattern) { - let mut next = Vec::new(); - for prefix in &prefixes { - let dir = if prefix.as_os_str().is_empty() { - Path::new(".") - } else { - prefix.as_path() - }; - let Ok(entries) = fs::read_dir(dir) else { - continue; - }; - for entry in entries.flatten() { - let name = entry.file_name(); - let name_text = name.to_string_lossy(); - if name_text.starts_with('.') && !component_pattern.starts_with('.') { - continue; - } - if glob_pattern_matches(&component_pattern, &name_text) { - let mut path = prefix.clone(); - path.push(name); - next.push(path); - } - } - } - prefixes = next; - } else { - for prefix in &mut prefixes { - prefix.push(component); - } - } - } - } - } - prefixes.sort(); - prefixes -} - fn effective_quiet(args: &Args) -> bool { args.quiet || matches!( @@ -867,9 +837,9 @@ enum LinesLimit { Tail(usize), } -/// Parse `--lines=[+]N` per the v260.1 grammar. +/// Parse `--lines=[+]N` per the v262 grammar. /// Bare `--lines` is normalized to the official default tail count, 10. -/// Explicit empty `--lines=` is invalid, matching systemd v260.1. +/// Explicit empty `--lines=` is invalid, matching systemd v262. fn parse_lines_limit(value: Option<&str>) -> Result> { let Some(value) = value else { return Ok(None); @@ -885,9 +855,12 @@ fn parse_lines_limit(value: Option<&str>) -> Result> { } else { (false, value) }; - let n: usize = stripped - .parse() - .map_err(|_| anyhow!("Failed to parse --lines='{value}'."))?; + let n = + parse_native_i32(stripped).ok_or_else(|| anyhow!("Failed to parse --lines='{value}'."))?; + if n < 0 { + return Err(anyhow!("Failed to parse --lines='{value}'.")); + } + let n = n as usize; if oldest { Ok(Some(LinesLimit::Head(n))) } else { @@ -895,28 +868,58 @@ fn parse_lines_limit(value: Option<&str>) -> Result> { } } -/// Resolve the effective tail count for follow mode. Honors `--tail` first, -/// then `--lines`. Returns `None` if neither was supplied. -fn parse_tail_count(tail: Option<&usize>, lines: Option<&str>) -> Option { - if let Some(n) = tail { - return Some(*n); - } - match parse_lines_limit(lines).ok().flatten() { - Some(LinesLimit::Tail(n) | LinesLimit::Head(n)) => Some(n), - Some(LinesLimit::All) | None => None, - } -} - fn grep_tail_implies_reverse(args: &Args) -> Result { if args.grep.is_none() || args.follow { return Ok(false); } let Some(limit) = parse_lines_limit(args.lines.as_deref())? else { - return Ok(false); + return Ok(args.pager_end); }; Ok(matches!(limit, LinesLimit::Tail(_))) } +fn parse_native_number(value: &str) -> Option<(u64, bool)> { + let mut value = value.trim_start_matches(|c: char| c.is_ascii_whitespace()); + let mut base = 0; + if value.starts_with("0b") || value.starts_with("0B") { + base = 2; + } else if value.starts_with("0o") || value.starts_with("0O") { + base = 8; + } + if base != 0 { + value = value[2..].trim_start_matches(|c: char| c.is_ascii_whitespace()); + } + let negative = value.starts_with('-'); + value = value + .strip_prefix('+') + .or_else(|| value.strip_prefix('-')) + .unwrap_or(value); + if base == 0 { + base = 10; + if value.starts_with("0x") || value.starts_with("0X") { + base = 16; + value = &value[2..]; + } else if value.starts_with('0') { + base = 8; + } + } + if value.starts_with('+') || value.starts_with('-') { + return None; + } + Some((u64::from_str_radix(value, base).ok()?, negative)) +} + +fn parse_native_i32(value: &str) -> Option { + let (number, negative) = parse_native_number(value)?; + let number = i64::try_from(number).ok()?; + i32::try_from(if negative { -number } else { number }).ok() +} + +fn parse_native_u64(value: &str) -> Option { + let (number, negative) = parse_native_number(value)?; + (!negative || number == 0).then_some(number) +} + #[derive(Debug, Clone)] struct CursorSeek { cursor: String, @@ -1005,17 +1008,6 @@ fn enforce_source_exclusivity(args: &Args) -> Result<()> { Ok(()) } -fn enforce_since_until_order(since: &Option, until: &Option) -> Result<()> { - let since_usec = parse_optional_timestamp(since.as_deref())?; - let until_usec = parse_optional_timestamp(until.as_deref())?; - if let (Some(s), Some(u)) = (since_usec, until_usec) { - if s > u { - return Err(anyhow!("--since= must be before --until=.")); - } - } - Ok(()) -} - fn enforce_cursor_source_exclusivity(args: &Args) -> Result<()> { let mut count = 0; if args.cursor.is_some() { @@ -1063,6 +1055,9 @@ fn enforce_oldest_lines_conflict(args: &Args) -> Result<()> { } fn is_show_action(args: &Args) -> bool { + if let Some(action) = args.action { + return action == "show"; + } !args.version && !args.new_id128 && !args.fields @@ -1122,11 +1117,14 @@ fn enforce_action_argument_restriction(args: &Args) -> Result<()> { } Err(anyhow!( "Extraneous arguments starting with '{}'", - args.matches[0] + args.matches[0].to_string_lossy() )) } fn action_rejects_arguments(args: &Args) -> bool { + if let Some(action) = args.action { + return !matches!(action, "show" | "list-catalog" | "dump-catalog"); + } args.new_id128 || args.setup_keys || args.update_catalog @@ -1188,7 +1186,7 @@ fn enforce_portable_unsupported(args: &Args) -> Result<()> { if args.flush { return Err(portable_unsupported("--flush", unsupported_reason("flush"))); } - if args.rotate && has_vacuum_flags(args) { + if args.action == Some("rotate-and-vacuum") || args.rotate && has_vacuum_flags(args) { return Err(portable_unsupported( "--rotate with --vacuum-*", "official rotate-and-vacuum action requires journald rotation; portable mode can only vacuum explicit directories without rotation", @@ -1251,64 +1249,6 @@ fn enforce_portable_unsupported(args: &Args) -> Result<()> { Ok(()) } -fn preprocess_optional_boot_args(args: I) -> Vec -where - I: IntoIterator, -{ - let mut input = args.into_iter().peekable(); - let mut out = Vec::new(); - while let Some(arg) = input.next() { - if arg == "--boot" || arg == "-b" { - if let Some(next) = input.peek() { - if looks_like_boot_descriptor(next) { - let next = input.next().unwrap(); - out.push(format!("{arg}={next}")); - continue; - } - } - out.push(format!("{arg}=0")); - continue; - } - if arg == "--lines" || arg == "-n" { - if let Some(next) = input.peek() { - if looks_like_lines_descriptor(next) { - let next = input.next().unwrap(); - out.push(format!("{arg}={next}")); - continue; - } - } - out.push(format!("{arg}=10")); - continue; - } - out.push(arg); - } - out -} - -fn looks_like_boot_descriptor(value: &str) -> bool { - value == "all" - || value.parse::().is_ok() - || parse_boot_id_prefix(value) - .map(|id| { - let consumed = if value.as_bytes().get(8) == Some(&b'-') { - 36 - } else { - 32 - }; - let rest = &value[consumed..]; - !id.is_empty() && (rest.is_empty() || rest.parse::().is_ok()) - }) - .unwrap_or(false) -} - -fn looks_like_lines_descriptor(value: &str) -> bool { - if value == "all" { - return true; - } - let value = value.strip_prefix('+').unwrap_or(value); - !value.is_empty() && value.parse::().is_ok() -} - fn open_filtered_journal( input: &CliInput, args: &Args, @@ -1317,6 +1257,7 @@ fn open_filtered_journal( let mut journal = input.open_journal()?; if let Some(invocation_id) = resolve_invocation_filter(input, args)? { add_invocation_matches(&mut journal, &invocation_id)?; + apply_non_unit_matches(&mut journal, args)?; } else { apply_boot_match(&mut journal, args, boot_descriptor_override)?; apply_cli_matches(&mut journal, args)?; @@ -1333,10 +1274,10 @@ fn run_verify(path: &Path, verify_key: Option<&str>) -> Result<()> { } else { CliInput::Files(vec![path.to_path_buf()]) }; - run_verify_input(&input, verify_key) + run_verify_input(&input, verify_key, false) } -fn run_verify_input(input: &CliInput, verify_key: Option<&str>) -> Result<()> { +fn run_verify_input(input: &CliInput, verify_key: Option<&str>, quiet: bool) -> Result<()> { if verify_key.is_some_and(|key| !valid_verification_key(key)) { eprintln!("Failed to parse seed."); return Err(anyhow!("failed to parse seed")); @@ -1369,7 +1310,11 @@ fn run_verify_input(input: &CliInput, verify_key: Option<&str>) -> Result<()> { if sealed && verify_key.is_some() { match verify_file_with_key(file, verify_key.unwrap()) { - Ok(()) => eprintln!("PASS: {}", file.display()), + Ok(()) => { + if !quiet { + eprintln!("PASS: {}", file.display()); + } + } Err(err) => { eprintln!("FAIL: {} ({err})", file.display()); if first_err.is_none() { @@ -1402,7 +1347,9 @@ fn run_verify_input(input: &CliInput, verify_key: Option<&str>) -> Result<()> { } continue; } - eprintln!("PASS: {}", file.display()); + if !quiet { + eprintln!("PASS: {}", file.display()); + } } match first_err { @@ -1418,12 +1365,12 @@ fn is_file_sealed(path: &Path) -> Result { fn is_journal_file_name(path: &Path) -> bool { path.file_name() - .and_then(|name| name.to_str()) + .map(|name| name.as_encoded_bytes()) .is_some_and(|name| { - name.ends_with(".journal") - || name.ends_with(".journal~") - || name.ends_with(".journal.zst") - || name.ends_with(".journal~.zst") + name.ends_with(b".journal") + || name.ends_with(b".journal~") + || name.ends_with(b".journal.zst") + || name.ends_with(b".journal~.zst") }) } @@ -1518,7 +1465,7 @@ struct VacuumOptions { #[derive(Debug, Clone)] struct VacuumCandidate { path: PathBuf, - name: String, + name: OsString, usage: u64, seqnum_id: [u8; 16], seqnum: u64, @@ -1527,6 +1474,9 @@ struct VacuumCandidate { } fn has_vacuum_flags(args: &Args) -> bool { + if let Some(action) = args.action { + return matches!(action, "vacuum" | "rotate-and-vacuum"); + } args.vacuum_size.is_some() || args.vacuum_files.is_some() || args.vacuum_time.is_some() } @@ -1549,10 +1499,8 @@ fn parse_vacuum_options(args: &Args) -> Result { None => 0, }; let max_files = match args.vacuum_files.as_deref() { - Some(value) => value - .trim() - .parse::() - .map_err(|_| anyhow!("failed to parse --vacuum-files value: {value}"))?, + Some(value) => parse_native_u64(value) + .ok_or_else(|| anyhow!("failed to parse --vacuum-files value: {value}"))?, None => 0, }; let max_retention_usec = match args.vacuum_time.as_deref() { @@ -1568,29 +1516,53 @@ fn parse_vacuum_options(args: &Args) -> Result { } fn parse_vacuum_size(value: &str) -> Result { - let re = Regex::new(r"^\s*(\d+(?:\.\d+)?)\s*([A-Za-z]*)\s*$")?; - let captures = re - .captures(value) - .ok_or_else(|| anyhow!("failed to parse --vacuum-size value: {value}"))?; - let number = captures[1].parse::()?; - let unit = captures - .get(2) - .map(|m| m.as_str().to_ascii_lowercase()) - .unwrap_or_default(); - let multiplier = match unit.as_str() { - "" | "b" | "byte" | "bytes" => 1_u64, - "k" | "kb" | "kib" => 1024, - "m" | "mb" | "mib" => 1024_u64.pow(2), - "g" | "gb" | "gib" => 1024_u64.pow(3), - "t" | "tb" | "tib" => 1024_u64.pow(4), - "p" | "pb" | "pib" => 1024_u64.pow(5), - "e" | "eb" | "eib" => 1024_u64.pow(6), - _ => return Err(anyhow!("failed to parse --vacuum-size value: {value}")), - }; - if number < 0_f64 || number > u64::MAX as f64 / multiplier as f64 { - return Err(anyhow!("failed to parse --vacuum-size value: {value}")); + fn parse(value: &str) -> Option { + const UNITS: [&str; 8] = ["E", "P", "T", "G", "M", "K", "B", ""]; + const FACTORS: [u64; 8] = [1 << 60, 1 << 50, 1 << 40, 1 << 30, 1 << 20, 1 << 10, 1, 1]; + let mut remaining = value; + let mut first_unit = 0; + let mut total = 0_u64; + loop { + remaining = remaining.trim_start_matches([' ', '\t', '\n', '\r']); + if remaining.starts_with('-') { + return None; + } + let number = remaining.trim_start_matches([' ', '\t', '\n', '\r', '\x0b', '\x0c']); + let negative = number.starts_with('-'); + let number = number.strip_prefix(['+', '-']).unwrap_or(number); + let end = number.bytes().take_while(u8::is_ascii_digit).count(); + let mut whole = number[..end].parse::().ok()?; + if negative { + whole = whole.wrapping_neg(); + } + remaining = &number[end..]; + let mut fraction = 0.0; + if let Some(rest) = remaining.strip_prefix('.') { + let end = rest.bytes().take_while(u8::is_ascii_digit).count(); + if end > 0 { + fraction = rest[..end].parse::().ok()? as f64; + for _ in 0..end { + fraction /= 10.0; + } + } + remaining = &rest[end..]; + } + remaining = remaining.trim_start_matches([' ', '\t', '\n', '\r']); + let unit = (first_unit..UNITS.len()).find(|&i| remaining.starts_with(UNITS[i]))?; + let factor = FACTORS[unit]; + let ceiling = whole.checked_add(u64::from(fraction > 0.0))?; + if ceiling > u64::MAX / factor { + return None; + } + total = total.checked_add(whole * factor + (fraction * factor as f64) as u64)?; + remaining = &remaining[UNITS[unit].len()..]; + if remaining.is_empty() { + return Some(total); + } + first_unit = unit + 1; + } } - Ok((number * multiplier as f64) as u64) + parse(value).ok_or_else(|| anyhow!("failed to parse --vacuum-size value: {value}")) } fn vacuum_directory(dir: &Path, opts: VacuumOptions, quiet: bool) -> Result<()> { @@ -1610,7 +1582,8 @@ fn vacuum_directory(dir: &Path, opts: VacuumOptions, quiet: bool) -> Result<()> Ok(entry) => entry, Err(_) => continue, }; - let name = entry.file_name().to_string_lossy().into_owned(); + let native_name = entry.file_name(); + let name = native_name.to_string_lossy(); let path = entry.path(); let metadata = match fs::symlink_metadata(&path) { Ok(metadata) if metadata.is_file() => metadata, @@ -1663,15 +1636,17 @@ fn vacuum_directory(dir: &Path, opts: VacuumOptions, quiet: bool) -> Result<()> sum = sum.saturating_add(usage); } - candidates.sort_by(vacuum_candidate_cmp); let retention_limit = if opts.max_retention_usec > 0 { current_realtime_usec().saturating_sub(opts.max_retention_usec) } else { 0 }; + patch_vacuum_creation_times(&mut candidates, &opts, retention_limit, sum, active_files); + let candidate_count = candidates.len(); + let order = vacuum_order::VacuumOrder::new(&mut candidates); - for (idx, candidate) in candidates.iter().enumerate() { - let left = active_files.saturating_add((candidates.len() - idx) as u64); + for (idx, candidate) in order.enumerate() { + let left = active_files.saturating_add((candidate_count - idx) as u64); if (opts.max_retention_usec == 0 || candidate.realtime >= retention_limit) && (opts.max_use == 0 || sum <= opts.max_use) && (opts.max_files == 0 || left <= opts.max_files) @@ -1687,7 +1662,7 @@ fn vacuum_directory(dir: &Path, opts: VacuumOptions, quiet: bool) -> Result<()> eprintln!( "Deleted archived journal {}/{} ({}).", dir.display(), - candidate.name, + candidate.name.to_string_lossy(), format_journal_bytes(candidate.usage) ); } @@ -1697,7 +1672,7 @@ fn vacuum_directory(dir: &Path, opts: VacuumOptions, quiet: bool) -> Result<()> eprintln!( "Failed to delete archived journal {}/{}: {err}", dir.display(), - candidate.name + candidate.name.to_string_lossy() ); } Err(_) => {} @@ -1736,8 +1711,8 @@ fn parse_vacuum_candidate( }; return ( Some(VacuumCandidate { + name: path.file_name().unwrap_or_default().to_os_string(), path, - name: name.to_owned(), usage, seqnum_id, seqnum, @@ -1756,8 +1731,8 @@ fn parse_vacuum_candidate( }; return ( Some(VacuumCandidate { + name: path.file_name().unwrap_or_default().to_os_string(), path, - name: name.to_owned(), usage, seqnum_id: [0u8; 16], seqnum: 0, @@ -1777,23 +1752,6 @@ fn parse_id128_hex(value: &str) -> Result<[u8; 16]> { .map_err(|_| anyhow!("failed to parse id128")) } -fn vacuum_candidate_cmp(a: &VacuumCandidate, b: &VacuumCandidate) -> std::cmp::Ordering { - if a.have_seqno && b.have_seqno && a.seqnum_id == b.seqnum_id { - return a.seqnum.cmp(&b.seqnum); - } - match a.realtime.cmp(&b.realtime) { - std::cmp::Ordering::Equal => {} - order => return order, - } - if a.have_seqno && b.have_seqno { - match a.seqnum_id.cmp(&b.seqnum_id) { - std::cmp::Ordering::Equal => {} - order => return order, - } - } - a.name.cmp(&b.name) -} - fn vacuum_journal_file_empty(path: &Path, metadata: &fs::Metadata) -> Result { if metadata.len() < JOURNAL_HEADER_SIZE { return Ok(true); @@ -1811,7 +1769,18 @@ fn open_vacuum_candidate(path: &Path) -> Result { #[cfg(unix)] { use std::os::unix::fs::OpenOptionsExt; - opts.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW); + let flags = libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK; + #[cfg(target_os = "linux")] + let flags = flags | libc::O_NOATIME; + opts.custom_flags(flags); + #[cfg(target_os = "linux")] + match opts.open(path) { + Ok(file) => return Ok(file), + Err(_) => { + // Native retries when preserving atime is unavailable to the caller. + opts.custom_flags(flags & !libc::O_NOATIME); + } + } } opts.open(path) .map_err(|err| anyhow!("vacuum: open {}: {err}", path.display())) @@ -1823,6 +1792,38 @@ fn patch_vacuum_realtime(candidate: &mut VacuumCandidate, metadata: &fs::Metadat candidate.realtime = usec; } } + candidate.realtime = vacuum_metadata::oldest_realtime(metadata, candidate.realtime); +} + +fn patch_vacuum_creation_times( + candidates: &mut [VacuumCandidate], + opts: &VacuumOptions, + retention_limit: u64, + usage: u64, + active_files: u64, +) { + let Some(first) = candidates.first() else { + return; + }; + if retention_limit == 0 + && (opts.max_use == 0 || usage <= opts.max_use) + && (opts.max_files == 0 + || active_files.saturating_add(candidates.len() as u64) <= opts.max_files) + { + return; + } + let sequence_only = first.have_seqno + && candidates + .iter() + .all(|candidate| candidate.have_seqno && candidate.seqnum_id == first.seqnum_id); + for candidate in candidates { + // A shared sequence ID fixes order; only unresolved age can need I/O. + if sequence_only && (retention_limit == 0 || candidate.realtime < retention_limit) { + continue; + } + candidate.realtime = + vacuum_metadata::creation_realtime(&candidate.path, candidate.realtime); + } } fn current_realtime_usec() -> u64 { @@ -1856,7 +1857,12 @@ fn run_header_input(input: &CliInput) -> Result<()> { fn print_header(path: &Path, header: &FileHeader, disk_usage: u64) -> Result<()> { let data_buckets = header.data_hash_table_size / HASH_ITEM_SIZE; let field_buckets = header.field_hash_table_size / HASH_ITEM_SIZE; - println!("File path: {}", path.display()); + { + let mut stdout = std::io::stdout().lock(); + stdout.write_all(b"File path: ")?; + stdout.write_all(path.as_os_str().as_encoded_bytes())?; + stdout.write_all(b"\n")?; + } println!("File ID: {}", hex::encode(header.file_id)); println!("Machine ID: {}", hex::encode(header.machine_id)); println!("Boot ID: {}", hex::encode(header.tail_entry_boot_id)); @@ -1939,12 +1945,19 @@ fn print_header(path: &Path, header: &FileHeader, disk_usage: u64) -> Result<()> fn run_list_boots(input: &CliInput, args: &Args) -> Result<()> { let mut journal = input.open_journal()?; let boots = collect_boots(&mut journal)?; - if boots.is_empty() { - return Err(anyhow!("No boot found.")); - } let rows = select_boot_rows(&boots, args)?; + if rows.is_empty() { + return Err(no_data_error(effective_quiet(args), "No boot found.")); + } + if is_json_output(args.output) { + let rows: Vec<_> = rows.iter().map(|row| serde_json::json!({ + "index": row.index, "boot_id": row.boot_id, + "first_entry": table_timestamp(row.first_entry), "last_entry": table_timestamp(row.last_entry), + })).collect(); + return print_json_table(args.output, &rows); + } if !args.quiet { - println!("IDX BOOT ID FIRST ENTRY LAST ENTRY"); + print!("{}", colors::Colors::detect().table_header("BOOT ID")); } let index_width = if args.quiet { rows.iter() @@ -1969,12 +1982,22 @@ fn run_list_boots(input: &CliInput, args: &Args) -> Result<()> { fn run_list_invocations(input: &CliInput, args: &Args) -> Result<()> { let invocations = collect_invocations_from_input(input, args, Some("--list-invocations"))?; - if invocations.is_empty() { - return Err(anyhow!("No invocation ID found.")); - } let rows = select_invocation_rows(&invocations, args)?; + if rows.is_empty() { + return Err(no_data_error( + effective_quiet(args), + "No invocation ID found.", + )); + } + if is_json_output(args.output) { + let rows: Vec<_> = rows.iter().map(|(index, row)| serde_json::json!({ + "index": index, "invocation_id": row.id, + "first_entry": table_timestamp(row.first_usec), "last_entry": table_timestamp(row.last_usec), + })).collect(); + return print_json_table(args.output, &rows); + } if !args.quiet { - println!("IDX INVOCATION ID FIRST ENTRY LAST ENTRY"); + print!("{}", colors::Colors::detect().table_header("INVOCATION ID")); } let quiet_index_width = if args.quiet { rows.iter() @@ -2249,22 +2272,28 @@ fn is_hex(b: u8) -> bool { b.is_ascii_hexdigit() } -fn apply_matches(journal: &mut SdJournal, matches: &[String]) -> Result<()> { +fn apply_matches(journal: &mut SdJournal, matches: &[OsString]) -> Result<()> { for item in matches { - if item == "+" { + if item.as_encoded_bytes() == b"+" { SdJournalAddDisjunction(journal).map_err(|err| anyhow!("add disjunction: {err}"))?; continue; } - let data = - parse_match_string(item).map_err(|err| anyhow!("invalid match {item}: {err}"))?; + let data = parse_match_bytes(item.as_encoded_bytes()) + .map_err(|err| anyhow!("invalid match {}: {err}", item.to_string_lossy()))?; SdJournalAddMatch(journal, &data).map_err(|err| anyhow!("add match: {err}"))?; } Ok(()) } -fn validate_path_match_arguments(matches: &[String]) -> Result<()> { +fn validate_path_match_arguments(matches: &[OsString]) -> Result<()> { + for (i, arg) in matches.iter().enumerate() { + if arg == "+" && (i == 0 || i + 1 == matches.len() || matches[i - 1] == "+") { + return Err(anyhow!("\"+\" can only be used between terms.")); + } + } + for item in matches { - if item != "+" && !item.contains('=') { + if item != "+" && !item.as_encoded_bytes().contains(&b'=') { return Err(portable_unsupported( "path match argument", "portable mode supports FIELD=VALUE matches and '+' disjunctions only; path matches require host filesystem metadata inspection", @@ -2320,13 +2349,6 @@ fn add_match_pair(journal: &mut SdJournal, field: &str, value: &str) -> Result<( Ok(()) } -fn add_match_group(journal: &mut SdJournal, pairs: &[(&str, &str)]) -> Result<()> { - for (field, value) in pairs { - add_match_pair(journal, field, value)?; - } - Ok(()) -} - fn add_unit_disjunction(journal: &mut SdJournal) -> Result<()> { SdJournalAddDisjunction(journal).map_err(|err| anyhow!("add unit disjunction: {err}")) } @@ -2347,11 +2369,6 @@ fn current_uid_string() -> Option { } } -fn add_impossible_match(journal: &mut SdJournal, reason: &str) -> Result<()> { - add_match_pair(journal, "__JOURNALCTL_NEVER_MATCH", reason)?; - add_unit_conjunction(journal) -} - fn effective_boot_descriptor(args: &Args) -> Option<&str> { if let Some(boot) = args.boot.as_deref() { return Some(boot); @@ -2387,57 +2404,88 @@ fn apply_boot_match( Ok(()) } -fn add_journalctl_unit_matches( - journal: &mut SdJournal, - system_units: &[String], - user_units: &[String], -) -> Result<()> { +fn add_unit_match_group(journal: &mut SdJournal, pairs: &[(&str, &[u8])]) -> Result<()> { + for (field, value) in pairs { + let mut payload = format!("{field}=").into_bytes(); + payload.extend_from_slice(value); + SdJournalAddMatch(journal, &payload)?; + } + Ok(()) +} + +fn add_journalctl_unit_matches(journal: &mut SdJournal, args: &Args) -> Result<()> { + if args.unit.is_empty() && args.user_unit.is_empty() { + return Ok(()); + } + // Follow retains the initial wildcard expansion, like native journalctl. + let (system_units, user_units) = args + .unit_matches + .get_or_init(|| { + (|| -> Result { + let (system, user) = effective_unit_specs(args)?; + let expanded_system = expand_unit_specs(journal, system, SYSTEM_UNIT_FIELDS_FULL)?; + let expanded_user = expand_unit_specs(journal, user, USER_UNIT_FIELDS_FULL)?; + if (!system.is_empty() || !user.is_empty()) + && expanded_system.is_empty() + && expanded_user.is_empty() + { + return Err(anyhow!("Failed to add filter for units: No data available")); + } + Ok((expanded_system, expanded_user)) + })() + .map_err(|err| err.to_string()) + }) + .as_ref() + .map_err(|err| anyhow!("{err}"))?; if system_units.is_empty() && user_units.is_empty() { return Ok(()); } - let mut added = false; - let system_units = expand_unit_specs(journal, system_units, SYSTEM_UNIT_FIELDS_FULL)?; - for unit in &system_units { - add_system_unit_match_groups(journal, unit)?; - added = true; + for unit in system_units { + add_system_unit_match_groups(journal, unit, true)?; } - let user_units = expand_unit_specs(journal, user_units, USER_UNIT_FIELDS_FULL)?; let uid = current_uid_string(); - for unit in &user_units { - add_user_unit_match_groups(journal, unit, uid.as_deref())?; - added = true; + for unit in user_units { + add_user_unit_match_groups(journal, unit, uid.as_deref(), true)?; } - if added { - add_unit_conjunction(journal) - } else { - add_impossible_match(journal, "unit-glob") - } + add_unit_conjunction(journal) } -fn add_system_unit_match_groups(journal: &mut SdJournal, unit: &str) -> Result<()> { - add_match_group(journal, &[("_SYSTEMD_UNIT", unit)])?; +fn add_system_unit_match_groups(journal: &mut SdJournal, unit: &[u8], full: bool) -> Result<()> { + add_unit_match_group(journal, &[("_SYSTEMD_UNIT", unit)])?; add_unit_disjunction(journal)?; - add_match_group( + add_unit_match_group( journal, - &[("_SYSTEMD_CGROUP", "/init.scope"), ("UNIT", unit)], + &[ + ("_SYSTEMD_CGROUP", b"/init.scope".as_slice()), + ("UNIT", unit), + ], )?; add_unit_disjunction(journal)?; - add_match_group(journal, &[("_UID", "0"), ("OBJECT_SYSTEMD_UNIT", unit)])?; + add_unit_match_group( + journal, + &[("_UID", b"0".as_slice()), ("OBJECT_SYSTEMD_UNIT", unit)], + )?; add_unit_disjunction(journal)?; - add_match_group( + if !full { + return Ok(()); + } + add_unit_match_group( journal, - &[("MESSAGE_ID", COREDUMP_MESSAGE_ID), ("COREDUMP_UNIT", unit)], + &[ + ("MESSAGE_ID", COREDUMP_MESSAGE_ID.as_bytes()), + ("COREDUMP_UNIT", unit), + ], )?; - if unit.ends_with(".slice") { + if unit.ends_with(b".slice") { add_unit_disjunction(journal)?; - add_match_group(journal, &[("_SYSTEMD_SLICE", unit)])?; + add_unit_match_group(journal, &[("_SYSTEMD_SLICE", unit)])?; } add_unit_disjunction(journal) @@ -2445,8 +2493,9 @@ fn add_system_unit_match_groups(journal: &mut SdJournal, unit: &str) -> Result<( fn add_user_unit_match_groups( journal: &mut SdJournal, - unit: &str, + unit: &[u8], uid: Option<&str>, + full: bool, ) -> Result<()> { add_user_unit_match_group(journal, &[("_SYSTEMD_USER_UNIT", unit)], uid, false)?; add_unit_disjunction(journal)?; @@ -2457,9 +2506,12 @@ fn add_user_unit_match_groups( add_user_unit_match_group(journal, &[("OBJECT_SYSTEMD_USER_UNIT", unit)], uid, true)?; add_unit_disjunction(journal)?; + if !full { + return Ok(()); + } add_user_unit_match_group(journal, &[("COREDUMP_USER_UNIT", unit)], uid, true)?; - if unit.ends_with(".slice") { + if unit.ends_with(b".slice") { add_unit_disjunction(journal)?; add_user_unit_match_group(journal, &[("_SYSTEMD_USER_SLICE", unit)], uid, false)?; } @@ -2469,11 +2521,11 @@ fn add_user_unit_match_groups( fn add_user_unit_match_group( journal: &mut SdJournal, - pairs: &[(&str, &str)], + pairs: &[(&str, &[u8])], uid: Option<&str>, include_root_uid: bool, ) -> Result<()> { - add_match_group(journal, pairs)?; + add_unit_match_group(journal, pairs)?; if let Some(uid) = uid { add_match_pair(journal, "_UID", uid)?; if include_root_uid { @@ -2485,16 +2537,16 @@ fn add_user_unit_match_group( fn expand_unit_specs( journal: &mut SdJournal, - specs: &[String], + specs: &[Vec], fields: &[&str], -) -> Result> { +) -> Result>> { let mut out = Vec::new(); let mut seen = std::collections::HashSet::new(); let mut patterns = Vec::new(); for spec in specs { - let unit = mangle_unit_name(spec); - if is_glob_pattern(&unit) { + let unit = spec.clone(); + if unit.iter().any(|c| b"*?[".contains(c)) { patterns.push(unit); } else if seen.insert(unit.clone()) { out.push(unit); @@ -2505,15 +2557,15 @@ fn expand_unit_specs( return Ok(out); } + let mut matcher = unit_pattern::Matcher::new(&patterns)?; for field in fields { SdJournalVisitUniqueValues(journal, field, |value| { - let value = String::from_utf8_lossy(value); - if patterns - .iter() - .any(|pattern| glob_pattern_matches(pattern, &value)) - && seen.insert(value.to_string()) - { - out.push(value.to_string()); + let value = value.split(|c| *c == 0).next().unwrap_or_default(); + if matcher.matches(value) { + let value = value.to_vec(); + if seen.insert(value.clone()) { + out.push(value); + } } Ok(()) }) @@ -2523,74 +2575,122 @@ fn expand_unit_specs( Ok(out) } -fn mangle_unit_name(value: &str) -> String { - let value = value.trim(); - if UNIT_SUFFIXES.iter().any(|suffix| value.ends_with(suffix)) { - value.to_string() - } else { - format!("{value}.service") - } -} - +#[cfg(any( + test, + not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")) +))] fn is_glob_pattern(value: &str) -> bool { value.contains(['*', '?', '[']) } -fn glob_pattern_matches(pattern: &str, value: &str) -> bool { - Regex::new(&glob_pattern_to_regex(pattern)).is_ok_and(|regex| regex.is_match(value)) -} - +#[cfg(any( + test, + not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")) +))] fn glob_pattern_to_regex(pattern: &str) -> String { - let mut out = String::from("^"); - let mut chars = pattern.chars().peekable(); - while let Some(ch) = chars.next() { - match ch { + let mut out = String::from("(?s)^"); + let chars: Vec = pattern.chars().collect(); + let mut i = 0; + while i < chars.len() { + match chars[i] { '*' => out.push_str(".*"), '?' => out.push('.'), '[' => { - let mut class = String::from("["); - let mut closed = false; - if chars - .peek() - .is_some_and(|next| *next == '!' || *next == '^') - { - chars.next(); - class.push('^'); + let mut end = i + 1; + if end < chars.len() && matches!(chars[end], '!' | '^') { + end += 1; } - while let Some(next) = chars.next() { - class.push(next); - if next == ']' { - closed = true; - break; + if end < chars.len() && chars[end] == ']' { + end += 1; + } + while end < chars.len() && chars[end] != ']' { + if chars[end] == '[' + && end + 1 < chars.len() + && matches!(chars[end + 1], ':' | '.') + { + let mut k = end + 2; + while k + 1 < chars.len() + && !(chars[k] == chars[end + 1] && chars[k + 1] == ']') + { + k += 1; + } + if k + 1 < chars.len() { + end = k + 2; + continue; + } } + end += 1; } - if closed { - out.push_str(&class); - } else { + if end == chars.len() { out.push_str("\\["); - out.push_str(®ex::escape(&class[1..])); + } else { + out.push('['); + let mut start = i + 1; + if matches!(chars[start], '!' | '^') { + out.push('^'); + start += 1; + } + let mut k = start; + while k < end { + if chars[k] == '[' && k + 1 < end && matches!(chars[k + 1], ':' | '.') { + let marker = chars[k + 1]; + let mut n = k + 2; + while n + 1 < end && !(chars[n] == marker && chars[n + 1] == ']') { + n += 1; + } + if n + 1 < end { + if marker == ':' { + out.extend(&chars[k..n + 2]); + } else { + if n != k + 3 { + return "[".into(); + } + if r"\[]^-".contains(chars[k + 2]) { + out.push('\\'); + } + out.push(chars[k + 2]); + } + k = n + 2; + continue; + } + } + let c = chars[k]; + if matches!(c, '\\' | ']' | '[') { + out.push('\\'); + } + out.push(c); + k += 1; + } + out.push(']'); + i = end; } } - _ => out.push_str(®ex::escape(&ch.to_string())), + c => out.push_str(®ex::escape(&c.to_string())), } + i += 1; } out.push('$'); out } fn apply_cli_matches(journal: &mut SdJournal, args: &Args) -> Result<()> { - let (system_units, user_units) = effective_unit_specs(args); - add_journalctl_unit_matches(journal, &system_units, &user_units)?; + add_journalctl_unit_matches(journal, args)?; + apply_non_unit_matches(journal, args) +} +fn apply_non_unit_matches(journal: &mut SdJournal, args: &Args) -> Result<()> { if args.dmesg { add_field_matches(journal, "_TRANSPORT", ["kernel"])?; } if !args.identifier.is_empty() { - add_field_matches( - journal, - "SYSLOG_IDENTIFIER", - args.identifier.iter().map(String::as_str), - )?; + for value in &args.identifier { + let mut data = b"SYSLOG_IDENTIFIER=".to_vec(); + data.extend_from_slice(value.as_encoded_bytes()); + SdJournalAddMatch(journal, &data) + .map_err(|err| anyhow!("add identifier match: {err}"))?; + } + SdJournalAddConjunction(journal) + .map_err(|err| anyhow!("add identifier conjunction: {err}"))?; } let priorities = parse_priority_filter(&args.priority)?; @@ -2609,63 +2709,123 @@ fn apply_cli_matches(journal: &mut SdJournal, args: &Args) -> Result<()> { Ok(()) } -fn effective_unit_specs(args: &Args) -> (Vec, Vec) { - let mut system_units = args.unit.clone(); - let mut user_units = args.user_unit.clone(); - if args.user && !system_units.is_empty() { - user_units.extend(std::mem::take(&mut system_units)); +fn raw_unit_specs(args: &Args) -> (Vec, Vec) { + let mut system = args.unit.clone(); + let mut user = args.user_unit.clone(); + if args.user && !args.system { + for value in system.drain(..) { + if !user.contains(&value) { + user.push(value); + } + } } - (system_units, user_units) + (system, user) +} + +fn effective_unit_specs(args: &Args) -> Result<&UnitSpecs> { + args.unit_specs + .get_or_init(|| { + let (system, user) = raw_unit_specs(args); + let normalize = |values: Vec| -> Result>> { + values + .iter() + .map(|value| { + let (name, escaped) = unit_name::normalize(value.as_encoded_bytes())?; + if let Some(escaped) = escaped.filter(|_| !args.quiet) { + let mut diagnostic = b"Invalid unit name \"".to_vec(); + diagnostic.extend_from_slice(value.as_encoded_bytes()); + diagnostic.extend_from_slice(b"\" escaped as \""); + diagnostic.extend_from_slice(escaped.as_bytes()); + diagnostic + .extend_from_slice(b"\" (maybe you should use systemd-escape?).\n"); + let _ = std::io::stderr().lock().write_all(&diagnostic); + } + Ok(name.into_bytes()) + }) + .collect() + }; + (|| Ok((normalize(system)?, normalize(user)?)))() + .map_err(|e: anyhow::Error| e.to_string()) + }) + .as_ref() + .map_err(|e| anyhow!("{e}")) } #[derive(Debug)] struct CliPostFilters { - grep: Option, - exclude_identifiers: std::collections::HashSet, + grep: Option, + shown: Cell, + exclude_identifiers: std::collections::HashSet>, + capture_highlight: Cell, + highlight: Cell, + buffered_highlights: std::cell::RefCell>, } impl CliPostFilters { fn from_args(args: &Args) -> Result { let exclude_identifiers = if output_uses_exclude_identifier(args.output) { - args.exclude_identifier.iter().cloned().collect() + args.exclude_identifier + .iter() + .map(|value| value.as_encoded_bytes().to_vec()) + .collect() } else { std::collections::HashSet::new() }; Ok(Self { grep: compile_grep_filter(args.grep.as_deref(), args.case_sensitive.as_deref())?, exclude_identifiers, + shown: Cell::new(false), + capture_highlight: Cell::new(false), + highlight: Cell::new(colors::MatchRange::default()), + buffered_highlights: std::cell::RefCell::new(VecDeque::new()), }) } - fn matches(&self, entry: &Entry) -> bool { - if let Some(regex) = &self.grep { - let mut matched = false; - for_each_entry_value(entry, "MESSAGE", |value| { - if regex.is_match(&String::from_utf8_lossy(value)) { - matched = true; - } - }); - if !matched { - return false; + fn matches(&self, entry: &Entry) -> Result { + let Some(pattern) = &self.grep else { + return Ok(true); + }; + let value = entry + .field_values + .get("MESSAGE") + .and_then(|v| v.first().map(Vec::as_slice)) + .or_else(|| entry.fields.get("MESSAGE").map(Vec::as_slice)); + match value { + Some(value) if self.capture_highlight.get() => { + let mut highlight = colors::MatchRange::default(); + let matched = pattern.match_with_highlight(value, &mut highlight)?; + self.highlight.set(highlight); + Ok(matched) + } + Some(value) => pattern.matches(value), + None => Ok(false), + } + } + + fn buffer_highlight(&self) { + if self.capture_highlight.get() && self.grep.is_some() { + self.buffered_highlights + .borrow_mut() + .push_back(self.highlight.get()); + } + } + + fn restore_highlight(&self) { + if self.capture_highlight.get() { + if let Some(highlight) = self.buffered_highlights.borrow_mut().pop_front() { + self.highlight.set(highlight); } } - true } fn excludes_entry(&self, entry: &Entry) -> bool { if self.exclude_identifiers.is_empty() { return false; } - let mut excluded = false; - for_each_entry_value(entry, "SYSLOG_IDENTIFIER", |value| { - if self - .exclude_identifiers - .contains(&String::from_utf8_lossy(value).into_owned()) - { - excluded = true; - } - }); - excluded + output::last_value_for_name(entry, "SYSLOG_IDENTIFIER").is_some_and(|value| { + self.exclude_identifiers + .contains(value.split(|&b| b == 0).next().unwrap_or_default()) + }) } } @@ -2684,37 +2844,19 @@ fn output_uses_exclude_identifier(mode: OutputModeArg) -> bool { ) } -fn for_each_entry_value(entry: &Entry, field: &str, mut visitor: F) -where - F: FnMut(&[u8]), -{ - if let Some(values) = entry.field_values.get(field) { - for value in values { - visitor(value); - } - return; - } - if let Some(value) = entry.fields.get(field) { - visitor(value); - } -} - fn compile_grep_filter( - pattern: Option<&str>, + pattern: Option<&OsStr>, case_sensitive: Option<&str>, -) -> Result> { +) -> Result> { let Some(pattern) = pattern else { return Ok(None); }; + let bytes = pattern.as_encoded_bytes(); let sensitive = match case_sensitive { Some(value) => parse_bool_option("--case-sensitive", value)?, - None => pattern.chars().any(char::is_uppercase), + None => pcre::Pattern::compile(b"[[:upper:]]", true)?.matches(bytes)?, }; - RegexBuilder::new(pattern) - .case_insensitive(!sensitive) - .build() - .map(Some) - .map_err(|err| anyhow!("Bad pattern \"{pattern}\": {err}")) + pcre::Pattern::compile(bytes, sensitive).map(Some) } fn parse_bool_option(option: &str, value: &str) -> Result { @@ -2742,17 +2884,18 @@ fn parse_priority_filter(values: &[String]) -> Result> { } fn parse_priority_level(value: &str) -> Result { - let normalized = value.trim().to_ascii_lowercase(); - let level = match normalized.as_str() { - "emerg" | "panic" => Some(0), + let level = match value { + "emerg" => Some(0), "alert" => Some(1), - "crit" | "critical" => Some(2), - "err" | "error" => Some(3), - "warning" | "warn" => Some(4), + "crit" => Some(2), + "err" => Some(3), + "warning" => Some(4), "notice" => Some(5), "info" => Some(6), "debug" => Some(7), - _ => normalized.parse::().ok().filter(|value| *value <= 7), + _ => parse_native_i32(value) + .filter(|value| (0..=7).contains(value)) + .map(|value| value as u8), }; level.ok_or_else(|| anyhow!("Unknown log level {value}")) } @@ -2780,13 +2923,6 @@ const FACILITY_NAMES: [(&str, u8); 20] = [ ("local7", 23), ]; -fn facility_help_requested(values: &[String]) -> bool { - values - .iter() - .flat_map(|value| value.split(',')) - .any(|item| item.trim() == "help") -} - fn print_facility_help(quiet: bool) { if !quiet { println!("Available facilities:"); @@ -2807,7 +2943,6 @@ fn parse_facility_filter(values: &[String]) -> Result> { let mut facilities = Vec::new(); for value in values { for item in value.split(',') { - let item = item.trim(); if item.is_empty() || item == "help" { continue; } @@ -2822,9 +2957,9 @@ fn parse_facility_filter(values: &[String]) -> Result> { } fn parse_facility(value: &str) -> Result { - if let Ok(number) = value.parse::() { - if number <= 23 { - return Ok(number); + if let Some(number) = parse_native_i32(value) { + if (0..=127).contains(&number) { + return Ok(number as u8); } } FACILITY_NAMES @@ -2836,6 +2971,7 @@ fn parse_facility(value: &str) -> Result { struct BootSeparatorState { enabled: bool, previous_boot: Option<[u8; 16]>, + colors: colors::Colors, } impl BootSeparatorState { @@ -2843,6 +2979,7 @@ impl BootSeparatorState { Self { enabled: !quiet && !suppressed && output_emits_boot_separators(mode), previous_boot: None, + colors: colors::Colors::default(), } } @@ -2854,7 +2991,7 @@ impl BootSeparatorState { .previous_boot .is_some_and(|previous| previous != entry.boot_id) { - writeln!(stdout, "-- Boot {} --", hex::encode(entry.boot_id))?; + stdout.write_all(self.colors.boot(&hex::encode(entry.boot_id)).as_bytes())?; } self.previous_boot = Some(entry.boot_id); Ok(()) @@ -2877,22 +3014,42 @@ fn output_emits_boot_separators(mode: OutputModeArg) -> bool { ) } +fn render_matching_entry( + stdout: &mut W, + renderer: &mut OutputRenderer, + boots: &mut BootSeparatorState, + post_filters: &CliPostFilters, + entry: &Entry, +) -> Result { + boots.colors = renderer.colors(); + boots.before_entry(stdout, entry)?; + if !post_filters.matches(entry)? { + return Ok(false); + } + render_entry(stdout, renderer, post_filters, entry)?; + Ok(true) +} + fn render_entry( stdout: &mut W, renderer: &mut OutputRenderer, - boot_separators: &mut BootSeparatorState, post_filters: &CliPostFilters, entry: &Entry, ) -> Result { - boot_separators.before_entry(stdout, entry)?; if post_filters.excludes_entry(entry) { + post_filters.shown.set(true); return Ok(false); } if renderer.skips_entry(entry) { + post_filters.shown.set(true); return Ok(false); } + if post_filters.capture_highlight.get() { + renderer.set_highlight(post_filters.highlight.get()); + } let output = renderer.render(entry)?; stdout.write_all(&output)?; + post_filters.shown.set(true); Ok(true) } @@ -2909,6 +3066,23 @@ fn show_head_or_all_with_reverse( output_options: &OutputOptions, ) -> Result<()> { let mut stdout = std::io::stdout().lock(); + if limit == Some(0) { + let cursor = initial_cursor( + journal, + since_usec, + until_usec, + cursor_control, + reverse, + show_cursor, + )?; + print_no_entries(&mut stdout, 0, quiet)?; + return finish_cursor_output( + &mut stdout, + show_cursor, + cursor_control.update_file.as_deref(), + cursor.as_deref(), + ); + } let mut renderer = OutputRenderer::new(output_options.clone()); let mut boot_separators = BootSeparatorState::new( output_options.mode(), @@ -2916,7 +3090,7 @@ fn show_head_or_all_with_reverse( output_options.suppress_boot_separators(), ); let mut shown = 0usize; - let mut last_cursor: Option = None; + let mut last_cursor = String::new(); if reverse { if limit.is_none() { let (count, cursor) = stream_previous_output_events( @@ -2927,6 +3101,7 @@ fn show_head_or_all_with_reverse( cursor_control.seek.as_ref(), output_options, quiet, + show_cursor || cursor_control.update_file.is_some(), &mut stdout, &mut renderer, )?; @@ -2948,22 +3123,21 @@ fn show_head_or_all_with_reverse( limit.unwrap_or(0), output_options, quiet, + show_cursor || cursor_control.update_file.is_some(), )?; - let mut disabled_boot_separators = - BootSeparatorState::new(output_options.mode(), quiet, true); for event in events { match event { ReverseOutputEvent::Boot(boot_id) => { - writeln!(stdout, "-- Boot {} --", hex::encode(boot_id))?; + stdout.write_all( + output_options + .colors() + .boot(&hex::encode(boot_id)) + .as_bytes(), + )?; } ReverseOutputEvent::Entry(entry) => { - render_entry( - &mut stdout, - &mut renderer, - &mut disabled_boot_separators, - post_filters, - &entry, - )?; + post_filters.restore_highlight(); + render_entry(&mut stdout, &mut renderer, post_filters, &entry)?; } } } @@ -2976,29 +3150,25 @@ fn show_head_or_all_with_reverse( )?; return Ok(()); } - for_each_matching_entry_with_direction( + for_each_entry_with_checkpoint( journal, since_usec, until_usec, reverse, - post_filters, cursor_control.seek.as_ref(), + (show_cursor || cursor_control.update_file.is_some()).then_some(&mut last_cursor), |entry| { - if limit.is_some_and(|limit| shown >= limit) { - return Ok(false); - } - render_entry( + if !render_matching_entry( &mut stdout, &mut renderer, &mut boot_separators, post_filters, &entry, - )?; - shown += 1; - if !entry.cursor.is_empty() { - last_cursor = Some(entry.cursor.clone()); + )? { + return Ok(true); } - Ok(true) + shown += 1; + Ok(limit.is_none_or(|limit| shown < limit)) }, )?; print_no_entries(&mut stdout, shown, quiet)?; @@ -3006,11 +3176,57 @@ fn show_head_or_all_with_reverse( &mut stdout, show_cursor, cursor_control.update_file.as_deref(), - last_cursor.as_deref(), + Some(last_cursor.as_str()), )?; Ok(()) } +fn initial_cursor( + journal: &mut SdJournal, + since: Option, + until: Option, + control: &CursorControl, + from_tail: bool, + show_cursor: bool, +) -> Result> { + if !show_cursor && control.update_file.is_none() && control.seek.is_none() { + return Ok(None); + } + let mut cursor = String::new(); + if let Some(seek) = control.seek.as_ref() { + seek_cursor_start_tracked(journal, seek, from_tail, Some(&mut cursor))?; + // A successful --after-cursor advance replaces the seeded cursor. + if let Ok(current) = SdJournalGetCursor(journal) { + cursor = current; + } + return Ok((!cursor.is_empty()).then_some(cursor)); + } + if from_tail { + if let Some(until) = until { + SdJournalSeekRealtimeUsec(journal, until) + } else { + SdJournalSeekTail(journal) + } + } else if let Some(since) = since { + SdJournalSeekRealtimeUsec(journal, since) + } else { + SdJournalSeekHead(journal) + } + .map_err(|err| anyhow!("seek: {err}"))?; + let n = if from_tail { + SdJournalPrevious(journal) + } else { + SdJournalNext(journal) + } + .map_err(|err| anyhow!("iterate: {err}"))?; + if n == 0 { + return Ok(None); + } + Ok(Some( + SdJournalGetCursor(journal).map_err(|err| anyhow!("get cursor: {err}"))?, + )) +} + fn show_tail_with_reverse( journal: &mut SdJournal, limit: usize, @@ -3024,20 +3240,27 @@ fn show_tail_with_reverse( output_options: &OutputOptions, ) -> Result<()> { if limit == 0 { - if let Some(cursor_seek) = cursor_control.seek.as_ref() { - let _ = seek_cursor_start(journal, cursor_seek, false)?; - } + let cursor = initial_cursor( + journal, + since_usec, + until_usec, + cursor_control, + true, + show_cursor, + )?; let mut stdout = std::io::stdout().lock(); print_no_entries(&mut stdout, 0, quiet)?; finish_cursor_output( &mut stdout, show_cursor, cursor_control.update_file.as_deref(), - None, + cursor.as_deref(), )?; return Ok(()); } + let mut last_cursor = String::new(); + let capture_cursor = show_cursor || cursor_control.update_file.is_some(); let mut selected = VecDeque::new(); if reverse { let (events, count, last_cursor) = previous_output_events( @@ -3049,24 +3272,23 @@ fn show_tail_with_reverse( limit, output_options, quiet, + show_cursor || cursor_control.update_file.is_some(), )?; let mut stdout = std::io::stdout().lock(); let mut renderer = OutputRenderer::new(output_options.clone()); - let mut disabled_boot_separators = - BootSeparatorState::new(output_options.mode(), quiet, true); for event in events { match event { ReverseOutputEvent::Boot(boot_id) => { - writeln!(stdout, "-- Boot {} --", hex::encode(boot_id))?; + stdout.write_all( + output_options + .colors() + .boot(&hex::encode(boot_id)) + .as_bytes(), + )?; } ReverseOutputEvent::Entry(entry) => { - render_entry( - &mut stdout, - &mut renderer, - &mut disabled_boot_separators, - post_filters, - &entry, - )?; + post_filters.restore_highlight(); + render_entry(&mut stdout, &mut renderer, post_filters, &entry)?; } } } @@ -3079,29 +3301,18 @@ fn show_tail_with_reverse( )?; return Ok(()); } else if cursor_control.seek.is_none() { - for_each_matching_entry_with_direction( - journal, - since_usec, - until_usec, - true, - post_filters, - None, - |entry| { - if selected.len() >= limit { - return Ok(false); - } - selected.push_front(entry); - Ok(true) - }, - )?; + for_each_entry_with_direction(journal, since_usec, until_usec, true, None, |entry| { + selected.push_front(entry); + Ok(selected.len() < limit) + })?; } else { - for_each_matching_entry_with_direction( + for_each_entry_with_checkpoint( journal, since_usec, until_usec, false, - post_filters, cursor_control.seek.as_ref(), + capture_cursor.then_some(&mut last_cursor), |entry| { if limit > 0 && selected.len() == limit { selected.pop_front(); @@ -3120,25 +3331,44 @@ fn show_tail_with_reverse( quiet, output_options.suppress_boot_separators(), ); - let mut last_cursor: Option = None; + let mut shown = 0; for entry in &selected { - render_entry( + if capture_cursor && cursor_control.seek.is_none() { + last_cursor.clone_from(&entry.cursor); + } + if render_matching_entry( &mut stdout, &mut renderer, &mut boot_separators, post_filters, entry, - )?; - if !entry.cursor.is_empty() { - last_cursor = Some(entry.cursor.clone()); + )? { + shown += 1; + } + } + if capture_cursor + && cursor_control.seek.is_none() + && until_usec.is_some() + && selected.len() < limit + { + if !last_cursor.is_empty() { + SdJournalSeekCursorDirection(journal, &last_cursor, journal::Direction::Forward) + .map_err(|err| anyhow!("seek cursor: {err}"))?; + } else { + SdJournalSeekRealtimeUsec(journal, until_usec.unwrap()) + .map_err(|err| anyhow!("seek realtime: {err}"))?; + } + if SdJournalNext(journal).map_err(|err| anyhow!("next: {err}"))? > 0 { + last_cursor = + SdJournalGetCursor(journal).map_err(|err| anyhow!("get cursor: {err}"))?; } } - print_no_entries(&mut stdout, selected.len(), quiet)?; + print_no_entries(&mut stdout, shown, quiet)?; finish_cursor_output( &mut stdout, show_cursor, cursor_control.update_file.as_deref(), - last_cursor.as_deref(), + Some(last_cursor.as_str()), )?; Ok(()) } @@ -3163,6 +3393,7 @@ fn stream_previous_output_events( cursor_seek: Option<&CursorSeek>, output_options: &OutputOptions, quiet: bool, + capture_cursor: bool, stdout: &mut W, renderer: &mut OutputRenderer, ) -> Result<(usize, Option)> { @@ -3171,12 +3402,15 @@ fn stream_previous_output_events( quiet, output_options.suppress_boot_separators(), ); - let mut disabled_boot_separators = BootSeparatorState::new(output_options.mode(), quiet, true); - let mut pending_boot = None; let mut count = 0usize; - let mut last_cursor = None; + let mut last_cursor: Option = None; let mut visit = |entry: Entry| -> Result { + if capture_cursor && !entry.cursor.is_empty() { + last_cursor + .get_or_insert_default() + .clone_from(&entry.cursor); + } if since_usec.is_some_and(|since| entry.realtime < since) { return Ok(false); } @@ -3185,39 +3419,38 @@ fn stream_previous_output_events( .previous_boot .is_some_and(|previous| previous != entry.boot_id) { - pending_boot = Some(entry.boot_id); + stdout.write_all( + output_options + .colors() + .boot(&hex::encode(entry.boot_id)) + .as_bytes(), + )?; } boot_context.previous_boot = Some(entry.boot_id); } if since_usec.is_none_or(|since| entry.realtime >= since) && until_usec.is_none_or(|until| entry.realtime <= until) - && post_filters.matches(&entry) + && post_filters.matches(&entry)? { - if let Some(boot_id) = pending_boot.take() { - writeln!(stdout, "-- Boot {} --", hex::encode(boot_id))?; - } - render_entry( - stdout, - renderer, - &mut disabled_boot_separators, - post_filters, - &entry, - )?; - if !entry.cursor.is_empty() { - last_cursor = Some(entry.cursor.clone()); - } + render_entry(stdout, renderer, post_filters, &entry)?; count += 1; } Ok(true) }; if let Some(cursor_seek) = cursor_seek { - if let Some(entry) = seek_cursor_start(journal, cursor_seek, true)? { + let mut initial = String::new(); + if let Some(entry) = seek_cursor_start_tracked( + journal, + cursor_seek, + true, + capture_cursor.then_some(&mut initial), + )? { if !visit(entry)? { return Ok((count, last_cursor)); } } else { - return Ok((count, last_cursor)); + return Ok((count, (!initial.is_empty()).then_some(initial))); } } else if let Some(until) = until_usec { SdJournalSeekRealtimeUsec(journal, until).map_err(|err| anyhow!("seek realtime: {err}"))?; @@ -3237,11 +3470,6 @@ fn stream_previous_output_events( } } } - if count > 0 { - if let Some(boot_id) = pending_boot { - writeln!(stdout, "-- Boot {} --", hex::encode(boot_id))?; - } - } Ok((count, last_cursor)) } @@ -3254,18 +3482,23 @@ fn previous_output_events( limit: usize, output_options: &OutputOptions, quiet: bool, + capture_cursor: bool, ) -> Result<(Vec, usize, Option)> { let mut boot_context = BootSeparatorState::new( output_options.mode(), quiet, output_options.suppress_boot_separators(), ); - let mut pending_boot = None; let mut events = Vec::new(); let mut count = 0usize; - let mut last_cursor = None; + let mut last_cursor: Option = None; let mut visit = |entry: Entry| -> Result { + if capture_cursor && !entry.cursor.is_empty() { + last_cursor + .get_or_insert_default() + .clone_from(&entry.cursor); + } if since_usec.is_some_and(|since| entry.realtime < since) { return Ok(false); } @@ -3274,21 +3507,16 @@ fn previous_output_events( .previous_boot .is_some_and(|previous| previous != entry.boot_id) { - pending_boot = Some(entry.boot_id); + events.push(ReverseOutputEvent::Boot(entry.boot_id)); } boot_context.previous_boot = Some(entry.boot_id); } if since_usec.is_none_or(|since| entry.realtime >= since) && until_usec.is_none_or(|until| entry.realtime <= until) - && post_filters.matches(&entry) + && post_filters.matches(&entry)? { - if let Some(boot_id) = pending_boot.take() { - events.push(ReverseOutputEvent::Boot(boot_id)); - } - if !entry.cursor.is_empty() { - last_cursor = Some(entry.cursor.clone()); - } events.push(ReverseOutputEvent::Entry(entry)); + post_filters.buffer_highlight(); count += 1; if limit > 0 && count >= limit { return Ok(false); @@ -3298,12 +3526,18 @@ fn previous_output_events( }; if let Some(cursor_seek) = cursor_seek { - if let Some(entry) = seek_cursor_start(journal, cursor_seek, true)? { + let mut initial = String::new(); + if let Some(entry) = seek_cursor_start_tracked( + journal, + cursor_seek, + true, + capture_cursor.then_some(&mut initial), + )? { if !visit(entry)? { return Ok((events, count, last_cursor)); } } else { - return Ok((events, count, last_cursor)); + return Ok((events, count, (!initial.is_empty()).then_some(initial))); } } else if let Some(until) = until_usec { SdJournalSeekRealtimeUsec(journal, until).map_err(|err| anyhow!("seek realtime: {err}"))?; @@ -3323,21 +3557,38 @@ fn previous_output_events( } } } - if count > 0 { - if let Some(boot_id) = pending_boot { - events.push(ReverseOutputEvent::Boot(boot_id)); - } - } Ok((events, count, last_cursor)) } -fn for_each_matching_entry_with_direction( +fn for_each_entry_with_direction( + journal: &mut SdJournal, + since_usec: Option, + until_usec: Option, + reverse: bool, + cursor_seek: Option<&CursorSeek>, + visitor: F, +) -> Result<()> +where + F: FnMut(Entry) -> Result, +{ + for_each_entry_with_checkpoint( + journal, + since_usec, + until_usec, + reverse, + cursor_seek, + None, + visitor, + ) +} + +fn for_each_entry_with_checkpoint( journal: &mut SdJournal, since_usec: Option, until_usec: Option, reverse: bool, - post_filters: &CliPostFilters, cursor_seek: Option<&CursorSeek>, + mut checkpoint: Option<&mut String>, mut visitor: F, ) -> Result<()> where @@ -3345,10 +3596,14 @@ where { if reverse { if let Some(cursor_seek) = cursor_seek { - if let Some(entry) = seek_cursor_start(journal, cursor_seek, true)? { + if let Some(entry) = + seek_cursor_start_tracked(journal, cursor_seek, true, checkpoint.as_deref_mut())? + { + if let Some(cursor) = checkpoint.as_deref_mut() { + cursor.clone_from(&entry.cursor); + } if since_usec.is_none_or(|since| entry.realtime >= since) && until_usec.is_none_or(|until| entry.realtime <= until) - && post_filters.matches(&entry) { if !visitor(entry)? { return Ok(()); @@ -3373,12 +3628,14 @@ where _ => { let entry = SdJournalGetEntry(journal).map_err(|err| anyhow!("get entry: {err}"))?; + if let Some(cursor) = checkpoint.as_deref_mut() { + cursor.clone_from(&entry.cursor); + } if since_usec.is_some_and(|since| entry.realtime < since) { break; } if since_usec.is_none_or(|since| entry.realtime >= since) && until_usec.is_none_or(|until| entry.realtime <= until) - && post_filters.matches(&entry) { if !visitor(entry)? { return Ok(()); @@ -3391,10 +3648,14 @@ where } if let Some(cursor_seek) = cursor_seek { - if let Some(entry) = seek_cursor_start(journal, cursor_seek, false)? { + if let Some(entry) = + seek_cursor_start_tracked(journal, cursor_seek, false, checkpoint.as_deref_mut())? + { + if let Some(cursor) = checkpoint.as_deref_mut() { + cursor.clone_from(&entry.cursor); + } if since_usec.is_none_or(|since| entry.realtime >= since) && until_usec.is_none_or(|until| entry.realtime <= until) - && post_filters.matches(&entry) { if !visitor(entry)? { return Ok(()); @@ -3414,12 +3675,14 @@ where _ => { let entry = SdJournalGetEntry(journal).map_err(|err| anyhow!("get entry: {err}"))?; + if let Some(cursor) = checkpoint.as_deref_mut() { + cursor.clone_from(&entry.cursor); + } if until_usec.is_some_and(|until| entry.realtime > until) { break; } if since_usec.is_none_or(|since| entry.realtime >= since) && until_usec.is_none_or(|until| entry.realtime <= until) - && post_filters.matches(&entry) { if !visitor(entry)? { return Ok(()); @@ -3431,13 +3694,29 @@ where Ok(()) } -fn seek_cursor_start( +fn seek_cursor_start_tracked( journal: &mut SdJournal, cursor_seek: &CursorSeek, reverse: bool, + checkpoint: Option<&mut String>, ) -> Result> { - SdJournalSeekCursor(journal, &cursor_seek.cursor) - .map_err(|err| anyhow!("seek cursor: {err}"))?; + SdJournalSeekCursorDirection( + journal, + &cursor_seek.cursor, + if reverse { + journal::Direction::Backward + } else { + journal::Direction::Forward + }, + ) + .map_err(|err| anyhow!("seek cursor: {err}"))?; + if let Some(checkpoint) = checkpoint { + match SdJournalGetCursor(journal) { + Ok(cursor) => *checkpoint = cursor, + Err(FacadeError::NoEntry | FacadeError::EndOfEntries) => {} + Err(err) => return Err(anyhow!("get cursor: {err}")), + } + } if cursor_seek.after { match SdJournalTestCursor(journal, &cursor_seek.cursor) { Ok(true) => { @@ -3459,7 +3738,10 @@ fn seek_cursor_start( match SdJournalGetEntry(journal) { Ok(entry) => Ok(Some(entry)), Err(FacadeError::NoEntry | FacadeError::EndOfEntries) => Ok(None), - Err(err) => Err(anyhow!("get entry: {err}")), + Err(err) => match SdJournalGetCursor(journal) { + Err(FacadeError::NoEntry | FacadeError::EndOfEntries) => Ok(None), + _ => Err(anyhow!("get entry: {err}")), + }, } } @@ -3755,21 +4037,36 @@ fn parse_boot_descriptor(descriptor: &str) -> Result<(String, isize)> { if descriptor.is_empty() { return Err(anyhow!("failed to parse boot descriptor: {descriptor}")); } - let (boot_id, rest) = if let Some(id) = parse_boot_id_prefix(descriptor) { - let consumed = if descriptor.as_bytes().get(8) == Some(&b'-') { - 36 - } else { - 32 - }; - (id, &descriptor[consumed..]) + let (boot_id, rest) = if descriptor + .as_bytes() + .get(..32) + .is_some_and(|id| id.iter().all(u8::is_ascii_hexdigit)) + { + let id = &descriptor[..32]; + ( + if id.bytes().all(|b| b == b'0') { + String::new() + } else { + id.to_ascii_lowercase() + }, + &descriptor[32..], + ) } else { (String::new(), descriptor) }; + if descriptor.len() >= 32 + && !rest.is_empty() + && !rest.starts_with('+') + && !rest.starts_with('-') + { + return Err(anyhow!("failed to parse boot descriptor: {descriptor}")); + } let offset = if rest.is_empty() { 0 } else { - rest.parse::() - .map_err(|_| anyhow!("failed to parse boot descriptor: {descriptor}"))? + parse_native_i32(rest) + .ok_or_else(|| anyhow!("failed to parse boot descriptor: {descriptor}"))? + as isize }; Ok((boot_id, offset)) } @@ -3797,195 +4094,6 @@ fn parse_invocation_descriptor(descriptor: &str) -> Result Result> { - let Some(descriptor) = effective_invocation_descriptor(args) else { - return Ok(None); - }; - let Some((id, offset)) = parse_invocation_descriptor(descriptor)? else { - return Ok(None); - }; - - let invocation_unit_option = if id.is_empty() || offset != 0 { - Some("-I/--invocation= with an offset") - } else { - None - }; - let invocations = collect_invocations_from_input(input, args, invocation_unit_option)?; - let target = if !id.is_empty() { - invocations - .iter() - .position(|entry| entry.id == id) - .map(|base| base as isize + offset) - } else if offset > 0 { - Some(offset - 1) - } else { - Some(invocations.len() as isize - 1 + offset) - }; - - let Some(target) = target else { - return Err(anyhow!( - "No journal entry found for the invocation ({}{offset:+}).", - id - )); - }; - if target < 0 || target as usize >= invocations.len() { - return Err(anyhow!( - "No journal entry found for the invocation ({}{offset:+}).", - id - )); - } - Ok(Some(invocations[target as usize].id.clone())) -} - -fn apply_single_invocation_unit( - journal: &mut SdJournal, - args: &Args, - option_name: &str, -) -> Result<()> { - let (system_units, user_units) = single_invocation_unit(journal, args, option_name)?; - add_journalctl_unit_matches(journal, &system_units, &user_units) -} - -fn single_invocation_unit( - journal: &mut SdJournal, - args: &Args, - option_name: &str, -) -> Result<(Vec, Vec)> { - let (system_specs, user_specs) = effective_unit_specs(args); - let count = system_specs.len() + user_specs.len(); - if count == 0 { - return Err(anyhow!( - "Using {option_name} requires a unit. Please specify a unit name with -u/--unit=/--user-unit=." - )); - } - if count > 1 { - return Err(anyhow!( - "Using {option_name} with multiple units is not supported." - )); - } - if system_specs.len() == 1 { - let units = expand_unit_specs(journal, &system_specs, SYSTEM_UNIT_FIELDS_FULL)?; - let query = mangle_unit_name(&system_specs[0]); - if units.is_empty() { - return Err(anyhow!("No matching unit found for '{query}' in journal.")); - } - if units.len() > 1 { - return Err(anyhow!( - "Multiple matching units found for '{query}' in journal." - )); - } - return Ok((units, Vec::new())); - } - let units = expand_unit_specs(journal, &user_specs, USER_UNIT_FIELDS_FULL)?; - let query = mangle_unit_name(&user_specs[0]); - if units.is_empty() { - return Err(anyhow!("No matching unit found for '{query}' in journal.")); - } - if units.len() > 1 { - return Err(anyhow!( - "Multiple matching units found for '{query}' in journal." - )); - } - Ok((Vec::new(), units)) -} - -fn collect_invocations_from_input( - input: &CliInput, - args: &Args, - unit_option: Option<&str>, -) -> Result> { - let mut indexed_journal = open_invocation_scope_journal(input, args, unit_option)?; - let candidate_ids = collect_invocation_candidate_ids(&mut indexed_journal)?; - - let mut out = Vec::new(); - for id in candidate_ids { - if let Some(entry) = invocation_bounds_for_id(input, args, unit_option, &id)? { - out.push(entry); - } - } - out.sort_by(|a, b| { - a.first_usec - .cmp(&b.first_usec) - .then_with(|| a.id.cmp(&b.id)) - }); - Ok(out) -} - -fn open_invocation_scope_journal( - input: &CliInput, - args: &Args, - unit_option: Option<&str>, -) -> Result { - let mut journal = input.open_journal()?; - apply_boot_match(&mut journal, args, None)?; - if let Some(option_name) = unit_option { - apply_single_invocation_unit(&mut journal, args, option_name)?; - } - Ok(journal) -} - -fn collect_invocation_candidate_ids(journal: &mut SdJournal) -> Result> { - let mut seen = HashSet::new(); - let mut out = Vec::new(); - for field in INVOCATION_ID_FIELDS { - SdJournalVisitUniqueValues(journal, field, |value| { - let Ok(text) = std::str::from_utf8(value) else { - return Ok(()); - }; - let Some(id) = parse_invocation_id_value(text) else { - return Ok(()); - }; - if !id.chars().all(|ch| ch == '0') && seen.insert(id.clone()) { - out.push(id); - } - Ok(()) - }) - .map_err(|err| anyhow!("visit unique invocation values for {field}: {err}"))?; - } - Ok(out) -} - -fn invocation_bounds_for_id( - input: &CliInput, - args: &Args, - unit_option: Option<&str>, - id: &str, -) -> Result> { - let mut first_journal = open_invocation_scope_journal(input, args, unit_option)?; - add_invocation_matches(&mut first_journal, id)?; - SdJournalSeekHead(&mut first_journal).map_err(|err| anyhow!("seek invocation head: {err}"))?; - if SdJournalNext(&mut first_journal).map_err(|err| anyhow!("next invocation: {err}"))? == 0 { - return Ok(None); - } - let first_entry = SdJournalGetEntry(&mut first_journal) - .map_err(|err| anyhow!("get invocation entry: {err}"))?; - - let mut last_journal = open_invocation_scope_journal(input, args, unit_option)?; - add_invocation_matches(&mut last_journal, id)?; - SdJournalSeekTail(&mut last_journal).map_err(|err| anyhow!("seek invocation tail: {err}"))?; - if SdJournalPrevious(&mut last_journal).map_err(|err| anyhow!("previous invocation: {err}"))? - == 0 - { - return Ok(None); - } - let last_entry = SdJournalGetEntry(&mut last_journal) - .map_err(|err| anyhow!("get invocation entry: {err}"))?; - - Ok(Some(InvocationEntry { - id: id.to_string(), - first_usec: first_entry.realtime, - last_usec: last_entry.realtime, - })) -} - -fn parse_invocation_id_value(value: &str) -> Option { - let clean = value.trim().replace('-', ""); - if clean.len() == 32 && clean.bytes().all(|b| b.is_ascii_hexdigit()) { - return Some(clean.to_ascii_lowercase()); - } - None -} - fn parse_boot_id_prefix(value: &str) -> Option { if value.len() >= 32 && value[..32].bytes().all(|b| b.is_ascii_hexdigit()) { return Some(value[..32].to_ascii_lowercase()); @@ -4006,188 +4114,8 @@ fn parse_boot_id_prefix(value: &str) -> Option { None } -fn parse_optional_timestamp(value: Option<&str>) -> Result> { - value - .filter(|v| !v.trim().is_empty()) - .map(parse_timestamp_usec) - .transpose() -} - -fn parse_timestamp_usec(value: &str) -> Result { - let value = value.trim(); - match value { - "now" => return Ok(Local::now().timestamp_micros() as u64), - "today" | "yesterday" | "tomorrow" => { - let today = Local::now().date_naive(); - let date = match value { - "yesterday" => today.pred_opt().ok_or_else(|| anyhow!("date underflow"))?, - "tomorrow" => today.succ_opt().ok_or_else(|| anyhow!("date overflow"))?, - _ => today, - }; - return local_datetime_to_usec(date.and_hms_opt(0, 0, 0).unwrap()); - } - _ => {} - } - if let Some(epoch) = value.strip_prefix('@') { - return parse_epoch_timestamp_usec(epoch); - } - if let Some(duration) = value.strip_suffix(" ago") { - let delta = parse_duration_usec(duration.trim())? as i64; - let now = Local::now().timestamp_micros(); - return Ok(now.saturating_sub(delta) as u64); - } - let bytes = value.as_bytes(); - let signed_date_prefix = bytes.len() >= 6 - && matches!(bytes[0], b'+' | b'-') - && bytes[1..5].iter().all(|b| b.is_ascii_digit()) - && bytes[5] == b'-'; - if matches!(bytes.first(), Some(b'+' | b'-')) && !signed_date_prefix { - let delta = parse_duration_usec(&value[1..])? as i64; - let now = Local::now().timestamp_micros(); - return Ok(if value.starts_with('+') { - now + delta - } else { - now - delta - } as u64); - } - parse_local_timestamp_usec(value) -} - -fn parse_epoch_timestamp_usec(value: &str) -> Result { - let (whole, frac) = value.split_once('.').unwrap_or((value, "")); - if whole.is_empty() - || !whole.bytes().all(|b| b.is_ascii_digit()) - || !frac.bytes().all(|b| b.is_ascii_digit()) - { - return Err(anyhow!("failed to parse timestamp: @{value}")); - } - let seconds = whole.parse::()?; - let mut frac_padded = frac.to_string(); - frac_padded.push_str("000000"); - let usec = frac_padded[..6].parse::()?; - Ok(seconds * 1_000_000 + usec) -} - -fn parse_local_timestamp_usec(value: &str) -> Result { - if has_compact_timezone_offset(value) { - return Err(anyhow!("failed to parse timestamp: {value}")); - } - if let Ok(dt) = chrono::DateTime::parse_from_rfc3339(value) { - return Ok(dt.timestamp_micros() as u64); - } - for fmt in [ - "%Y-%m-%dT%H:%M:%S%.f%:z", - "%Y-%m-%dT%H:%M:%S%:z", - "%Y-%m-%dT%H:%M%:z", - ] { - if let Ok(dt) = chrono::DateTime::parse_from_str(value, fmt) { - return Ok(dt.timestamp_micros() as u64); - } - } - for fmt in [ - "%Y-%m-%dT%H:%M:%S%.f", - "%Y-%m-%dT%H:%M:%S", - "%Y-%m-%dT%H:%M", - "%Y-%m-%d %H:%M:%S%.f", - "%Y-%m-%d %H:%M:%S", - "%Y-%m-%d %H:%M", - ] { - if let Ok(dt) = NaiveDateTime::parse_from_str(value, fmt) { - return local_datetime_to_usec(dt); - } - } - if let Ok(date) = NaiveDate::parse_from_str(value, "%Y-%m-%d") { - return local_datetime_to_usec(date.and_hms_opt(0, 0, 0).unwrap()); - } - for fmt in ["%H:%M:%S%.f", "%H:%M:%S", "%H:%M"] { - if let Ok(time) = NaiveTime::parse_from_str(value, fmt) { - let date = Local::now().date_naive(); - return local_datetime_to_usec(date.and_time(time)); - } - } - Err(anyhow!("failed to parse timestamp: {value}")) -} - -fn has_compact_timezone_offset(value: &str) -> bool { - let bytes = value.as_bytes(); - bytes.len() >= 5 - && value.contains('T') - && matches!(bytes[bytes.len() - 5], b'+' | b'-') - && bytes[bytes.len() - 4..] - .iter() - .all(|byte| byte.is_ascii_digit()) -} - -fn local_datetime_to_usec(dt: NaiveDateTime) -> Result { - let local = Local - .from_local_datetime(&dt) - .earliest() - .ok_or_else(|| anyhow!("failed to parse local timestamp"))?; - Ok(local.timestamp_micros() as u64) -} - -fn parse_duration_usec(value: &str) -> Result { - parse_duration_usec_mode(value, false) -} - fn parse_duration_usec_allow_zero(value: &str) -> Result { - parse_duration_usec_mode(value, true) -} - -fn parse_duration_usec_mode(value: &str, allow_zero: bool) -> Result { - let mut total = 0_f64; - let bytes = value.as_bytes(); - let mut i = 0usize; - let mut seen = false; - while i < bytes.len() { - while i < bytes.len() && bytes[i].is_ascii_whitespace() { - i += 1; - } - let start = i; - while i < bytes.len() && (bytes[i].is_ascii_digit() || bytes[i] == b'.') { - i += 1; - } - if start == i { - return Err(anyhow!("failed to parse duration: {value}")); - } - seen = true; - let number = value[start..i].parse::()?; - while i < bytes.len() && bytes[i].is_ascii_whitespace() { - i += 1; - } - let unit_start = i; - while i < bytes.len() && bytes[i].is_ascii_alphabetic() { - i += 1; - } - let unit = if unit_start == i { - "s" - } else { - &value[unit_start..i] - }; - total += number * duration_unit_multiplier(unit)?; - } - if !seen || (!allow_zero && total == 0_f64) { - return Err(anyhow!("failed to parse duration: {value}")); - } - Ok(total as u64) -} - -fn duration_unit_multiplier(unit: &str) -> Result { - if unit == "M" { - return Ok(2_629_800_000_000_f64); - } - match unit.to_ascii_lowercase().as_str() { - "us" | "usec" | "usecs" => Ok(1_f64), - "ms" | "msec" | "msecs" => Ok(1_000_f64), - "s" | "sec" | "secs" | "second" | "seconds" => Ok(1_000_000_f64), - "m" | "min" | "mins" | "minute" | "minutes" => Ok(60_000_000_f64), - "h" | "hr" | "hour" | "hours" => Ok(3_600_000_000_f64), - "d" | "day" | "days" => Ok(86_400_000_000_f64), - "w" | "week" | "weeks" => Ok(604_800_000_000_f64), - "month" | "months" => Ok(2_629_800_000_000_f64), - "y" | "year" | "years" => Ok(31_557_600_000_000_f64), - _ => Err(anyhow!("failed to parse duration unit: {unit}")), - } + timestamp::duration(value) } fn for_each_follow_entry( @@ -4196,7 +4124,6 @@ fn for_each_follow_entry( boot_descriptor_override: Option<&str>, since_usec: Option, until_usec: Option, - post_filters: &CliPostFilters, cursor_seek: Option<&CursorSeek>, mut visitor: F, ) -> Result<()> @@ -4204,12 +4131,11 @@ where F: FnMut(Entry) -> Result<()>, { let mut journal = open_filtered_journal(input, args, boot_descriptor_override)?; - for_each_matching_entry_with_direction( + for_each_entry_with_direction( &mut journal, since_usec, until_usec, false, - post_filters, cursor_seek, |entry| { visitor(entry)?; @@ -4237,11 +4163,13 @@ fn run_follow( since_usec: Option, until_usec: Option, tail: usize, + all: bool, post_filters: &CliPostFilters, cursor_control: &CursorControl, output_options: &OutputOptions, ) -> Result<()> { let mut last_seen_cursor: Option = None; + let mut last_visited_cursor: Option = None; let mut renderer = OutputRenderer::new(output_options.clone()); let mut boot_separators = BootSeparatorState::new( output_options.mode(), @@ -4252,27 +4180,29 @@ fn run_follow( let boot_descriptor_override = resolve_follow_boot_descriptor(input, args)?; let boot_descriptor_override = boot_descriptor_override.as_deref(); - if args.no_tail { + if cursor_control.seek.is_some() || all { for_each_follow_entry( input, args, boot_descriptor_override, since_usec, until_usec, - post_filters, cursor_control.seek.as_ref(), |entry| { - render_entry( + let matched = render_matching_entry( &mut stdout, &mut renderer, &mut boot_separators, post_filters, &entry, )?; - if !entry.cursor.is_empty() { + if matched && !entry.cursor.is_empty() { last_seen_cursor = Some(entry.cursor.clone()); update_cursor_file(cursor_control, last_seen_cursor.as_deref())?; } + if !entry.cursor.is_empty() { + last_visited_cursor = Some(entry.cursor); + } Ok(()) }, )?; @@ -4284,11 +4214,13 @@ fn run_follow( boot_descriptor_override, since_usec, until_usec, - post_filters, cursor_control.seek.as_ref(), |entry| { - if !entry.cursor.is_empty() { - last_seen_cursor = Some(entry.cursor.clone()); + if tail == 0 { + if !entry.cursor.is_empty() { + last_visited_cursor = Some(entry.cursor); + } + return Ok(()); } if tail > 0 && selected.len() == tail { selected.pop_front(); @@ -4299,24 +4231,34 @@ fn run_follow( Ok(()) }, )?; + if let Some(entry) = selected.back().filter(|entry| !entry.cursor.is_empty()) { + last_visited_cursor = Some(entry.cursor.clone()); + } for entry in &selected { - render_entry( + if render_matching_entry( &mut stdout, &mut renderer, &mut boot_separators, post_filters, entry, - )?; + )? && !entry.cursor.is_empty() + { + last_seen_cursor = Some(entry.cursor.clone()); + } } update_cursor_file(cursor_control, last_seen_cursor.as_deref())?; } + if cursor_control.update_file.is_some() && last_visited_cursor != last_seen_cursor { + update_cursor_file(cursor_control, last_visited_cursor.as_deref())?; + last_seen_cursor.clone_from(&last_visited_cursor); + } let min_poll_interval = Duration::from_millis(100); let max_poll_interval = Duration::from_secs(1); let mut poll_interval = min_poll_interval; loop { thread::sleep(poll_interval); - let last_cursor = last_seen_cursor.clone(); + let last_cursor = last_visited_cursor.clone(); let follow_seek = last_cursor.map(|cursor| CursorSeek { cursor, after: true, @@ -4329,24 +4271,32 @@ fn run_follow( boot_descriptor_override, since_usec, until_usec, - post_filters, cursor_seek, |entry| { - render_entry( + let matched = render_matching_entry( &mut stdout, &mut renderer, &mut boot_separators, post_filters, &entry, )?; - emitted += 1; - if !entry.cursor.is_empty() { + if matched { + emitted += 1; + } + if matched && !entry.cursor.is_empty() { last_seen_cursor = Some(entry.cursor.clone()); update_cursor_file(cursor_control, last_seen_cursor.as_deref())?; } + if !entry.cursor.is_empty() { + last_visited_cursor = Some(entry.cursor); + } Ok(()) }, )?; + if cursor_control.update_file.is_some() && last_visited_cursor != last_seen_cursor { + update_cursor_file(cursor_control, last_visited_cursor.as_deref())?; + last_seen_cursor.clone_from(&last_visited_cursor); + } poll_interval = if emitted > 0 { min_poll_interval } else { @@ -4606,7 +4556,7 @@ mod tests { // -- SOW-0121 parser parity tests ---------------------------------- // - // Every official systemd v260.1 long option must be recognized by the + // Every official systemd v262 long option must be recognized by the // parser. The set is enumerated by the shared manifest at // tests/parser-parity/v260-manifest.json. The list is duplicated // here (intentionally) so the parser contract is enforced by Rust @@ -4711,16 +4661,17 @@ mod tests { let flag = match (*opt, *takes_value) { ("output", true) => "--output=short".to_string(), ("synchronize-on-exit", true) => "--synchronize-on-exit=false".to_string(), + ("since" | "until", true) => format!("--{opt}=@0"), + ("priority" | "facility", true) => format!("--{opt}=3"), + ("invocation", true) => "--invocation=all".to_string(), + ("interval" | "vacuum-time", true) => format!("--{opt}=1s"), + ("vacuum-files", true) => "--vacuum-files=1".to_string(), + ("vacuum-size", true) => "--vacuum-size=1M".to_string(), (_, true) => format!("--{opt}=placeholder"), (_, false) => format!("--{opt}"), }; let argv: Vec = vec!["journalctl".to_string(), flag]; - let parsed = Args::try_parse_from(&argv); - if *opt == "help" { - let err = parsed.expect_err("--help must exit through clap help"); - assert_eq!(err.kind(), clap::error::ErrorKind::DisplayHelp); - continue; - } + let parsed = parser::parse_cli(argv.iter().map(OsString::from).collect()); assert!( parsed.is_ok(), "parser rejected official option --{opt}: {:?}", @@ -4733,7 +4684,7 @@ mod tests { fn every_official_output_mode_is_accepted() { for mode in OFFICIAL_OUTPUT_MODES { let argv: Vec = vec!["journalctl".to_string(), format!("--output={mode}")]; - let parsed = Args::try_parse_from(&argv); + let parsed = parser::parse_cli(argv.iter().map(OsString::from).collect()); assert!( parsed.is_ok(), "parser rejected output mode {mode}: {:?}", @@ -4795,7 +4746,7 @@ mod tests { #[test] fn oldest_lines_conflict_applies_only_to_show_action() { - let show_args = Args::parse_from([ + let show_args = cli_args(&[ "journalctl", "--file=/tmp/x.journal", "--lines=+1", @@ -4806,7 +4757,7 @@ mod tests { "show action must reject --lines=+N with --reverse" ); - let list_boots_args = Args::parse_from([ + let list_boots_args = cli_args(&[ "journalctl", "--file=/tmp/x.journal", "--list-boots", @@ -4819,55 +4770,64 @@ mod tests { #[test] fn timestamp_parser_accepts_stock_iso_t_forms() { - let local_space = parse_timestamp_usec("2023-11-15 00:00").expect("space local"); - let local_t = parse_timestamp_usec("2023-11-15T00:00").expect("T local"); + let local_space = timestamp::Parser::default() + .parse("2023-11-15 00:00") + .expect("space local"); + let local_t = timestamp::Parser::default() + .parse("2023-11-15T00:00") + .expect("T local"); assert_eq!(local_t, local_space); - parse_timestamp_usec("2023-11-15T00:00:00Z").expect("UTC RFC3339"); - parse_timestamp_usec("2023-11-15T00:00:00.000001Z").expect("UTC RFC3339 fraction"); - parse_timestamp_usec("2023-11-15T00:00:00+02:00").expect("offset RFC3339"); - parse_timestamp_usec("2023-11-15T00:00+02:00").expect("offset without seconds"); + timestamp::Parser::default() + .parse("2023-11-15T00:00:00Z") + .expect("UTC RFC3339"); + timestamp::Parser::default() + .parse("2023-11-15T00:00:00.000001Z") + .expect("UTC RFC3339 fraction"); + timestamp::Parser::default() + .parse("2023-11-15T00:00:00+02:00") + .expect("offset RFC3339"); + timestamp::Parser::default() + .parse("2023-11-15T00:00+02:00") + .expect("offset without seconds"); assert!( - parse_timestamp_usec("2023-11-15T00:00:00+0200").is_err(), + timestamp::Parser::default() + .parse("2023-11-15T00:00:00+0200") + .is_err(), "stock rejects compact timezone offsets" ); } - #[test] - fn lines_optional_argument_does_not_consume_match() { - let args = preprocess_optional_boot_args([ - "journalctl".to_string(), - "--lines".to_string(), - "TEST_ID=journalctl-query".to_string(), - ]); - assert_eq!( - args, - vec![ - "journalctl".to_string(), - "--lines=10".to_string(), - "TEST_ID=journalctl-query".to_string() - ] - ); + fn cli_args(argv: &[&str]) -> Args { + parser::parse_cli(argv.iter().map(OsString::from).collect()) + .expect("parse") + .expect("nonterminal") + } - let args = preprocess_optional_boot_args([ - "journalctl".to_string(), - "--lines".to_string(), - "+2".to_string(), - "TEST_ID=journalctl-query".to_string(), - ]); - assert_eq!( - args, - vec![ - "journalctl".to_string(), - "--lines=+2".to_string(), - "TEST_ID=journalctl-query".to_string() - ] + #[test] + fn ordered_optional_and_required_values() { + let args = cli_args(&["journalctl", "--lines", "TEST_ID=journalctl-query"]); + assert_eq!(args.lines.as_deref(), Some("10")); + assert_eq!(args.matches, vec!["TEST_ID=journalctl-query"]); + let args = cli_args(&["journalctl", "--lines", "+2", "TEST_ID=journalctl-query"]); + assert_eq!(args.lines.as_deref(), Some("+2")); + let args = cli_args(&["journalctl", "--grep", "--help", "-rn2"]); + assert_eq!(args.grep.as_deref(), Some(OsStr::new("--help"))); + assert_eq!(args.lines.as_deref(), Some("2")); + assert!(args.reverse); + assert!( + parser::parse_cli( + ["journalctl", "--since=bad", "--help"] + .map(OsString::from) + .to_vec() + ) + .is_err() ); } #[test] fn rotate_with_vacuum_is_distinct_unsupported_action() { - let args = Args::parse_from(["journalctl", "--rotate", "--vacuum-files=2"]); + let args = cli_args(&["journalctl", "--rotate", "--vacuum-files=2"]); let err = enforce_portable_unsupported(&args).expect_err("rotate+vacuum should be rejected"); let msg = err.to_string(); @@ -4879,12 +4839,92 @@ mod tests { #[test] fn boot_all_merge_is_not_rejected_as_conflict() { - let args = Args::parse_from(preprocess_optional_boot_args([ - "journalctl".to_string(), - "--file=/tmp/x.journal".to_string(), - "--boot=all".to_string(), - "--merge".to_string(), - ])); + let args = cli_args(&[ + "journalctl", + "--file=/tmp/x.journal", + "--boot=all", + "--merge", + ]); enforce_boot_merge_conflict(&args).expect("boot=all must not conflict with --merge"); } } + +#[derive(Debug)] +struct SilentNoData; +impl std::fmt::Display for SilentNoData { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "no matching entries") + } +} +impl std::error::Error for SilentNoData {} + +fn no_data_error(quiet: bool, message: &str) -> anyhow::Error { + if quiet { + SilentNoData.into() + } else { + anyhow!("{message}") + } +} + +fn enforce_field_enumeration_scope(args: &Args) -> Result<()> { + if !args.fields && args.field.is_none() { + return Ok(()); + } + let boot = args.this_boot || args.boot.as_deref().is_some_and(|b| b != "all"); + let invocation = + args.invocation_latest || args.invocation.as_deref().is_some_and(|b| b != "all"); + if boot + || invocation + || args.dmesg + || !args.unit.is_empty() + || !args.user_unit.is_empty() + || !args.identifier.is_empty() + || !args.exclude_identifier.is_empty() + || !args.priority.is_empty() + || !args.facility.is_empty() + || args.since.is_some() + || args.until.is_some() + || args.cursor.is_some() + || args.after_cursor.is_some() + || args.cursor_file.is_some() + || args.grep.is_some() + { + return Err(anyhow!( + "-F/--field= and -N/--fields cannot be combined with options that limit the journal." + )); + } + Ok(()) +} + +fn is_json_output(mode: OutputModeArg) -> bool { + matches!( + mode, + OutputModeArg::Json + | OutputModeArg::JsonPretty + | OutputModeArg::JsonSse + | OutputModeArg::JsonSeq + ) +} + +fn table_timestamp(value: u64) -> Option { + (value != u64::MAX).then_some(value) +} + +fn print_json_table(mode: OutputModeArg, rows: &[serde_json::Value]) -> Result<()> { + let mut text = if matches!(mode, OutputModeArg::JsonPretty) { + let mut bytes = Vec::new(); + output::write_systemd_pretty_json_array_at(&mut bytes, rows, 0)?; + String::from_utf8(bytes)? + } else { + serde_json::to_string(rows)? + }; + if colors::Colors::detect().enabled() { + text = String::from_utf8(colors::json(text.as_bytes()))?; + } + match mode { + OutputModeArg::JsonSse => println!("data: {text}\n"), + OutputModeArg::JsonSeq => println!("\x1e{text}"), + _ => println!("{text}"), + } + Ok(()) +} diff --git a/rust/src/cmd/journalctl/native_unit_pattern.c b/rust/src/cmd/journalctl/native_unit_pattern.c new file mode 100644 index 00000000..afc9edc0 --- /dev/null +++ b/rust/src/cmd/journalctl/native_unit_pattern.c @@ -0,0 +1,15 @@ +#define _GNU_SOURCE +#include "../../../../go/cmd/journalctl/unit_pattern_compat.h" + +void *journalctl_unit_pattern_new(const char *pattern) { + return journal_unit_pattern_new(pattern); +} +int journalctl_unit_pattern_needed(const char *pattern) { + return journal_unit_pattern_needed(pattern); +} +int journalctl_unit_pattern_match(void *pattern, const char *value) { + return journal_unit_pattern_match(pattern, value); +} +void journalctl_unit_pattern_free(void *pattern) { + journal_unit_pattern_free(pattern); +} diff --git a/rust/src/cmd/journalctl/output.rs b/rust/src/cmd/journalctl/output.rs index b942de78..6883dba5 100644 --- a/rust/src/cmd/journalctl/output.rs +++ b/rust/src/cmd/journalctl/output.rs @@ -1,40 +1,61 @@ +use super::colors::{self, Colors, MatchRange}; use super::{Args, OutputModeArg, format_journal_bytes}; use anyhow::{Result, anyhow}; -use chrono::{Local, TimeZone, Utc}; use journal::Entry; use serde_json::{Map, Value}; -#[cfg(unix)] -use std::ffi::CStr; +use std::borrow::Cow; +use std::sync::OnceLock; #[derive(Clone)] pub(crate) struct OutputOptions { mode: OutputModeArg, output_fields: Vec, output_fields_set: bool, + invalid_cat_field: Option>, utc: bool, no_hostname: bool, full_width: bool, show_all: bool, truncate_newline: bool, merge: bool, + colors: Colors, + urlify: bool, } const PRINT_CHAR_THRESHOLD: usize = 300; const JSON_THRESHOLD: usize = 4096; const DEFAULT_COLUMNS: usize = 80; +pub(super) fn valid_field_name(name: &str) -> bool { + valid_field_bytes(name.as_bytes()) +} + +fn valid_field_bytes(name: &[u8]) -> bool { + !name.is_empty() + && !name.starts_with(b"__") + && name + .iter() + .all(|&b| b.is_ascii_uppercase() || b.is_ascii_digit() || b == b'_') +} + impl OutputOptions { pub(crate) fn from_args(args: &Args, full_width: bool) -> Self { + let (output_fields, output_fields_set, invalid_cat_field) = + parse_output_fields(&args.output_fields); + let colors = Colors::detect(); Self { mode: args.output.clone(), - output_fields: parse_output_fields(args.output_fields.as_deref()), - output_fields_set: args.output_fields.is_some(), + output_fields, + output_fields_set, + invalid_cat_field, utc: args.utc, no_hostname: args.no_hostname, full_width, show_all: args.all, truncate_newline: args.truncate_newline, merge: args.merge, + colors, + urlify: super::output_links::enabled(colors), } } @@ -45,18 +66,44 @@ impl OutputOptions { pub(crate) fn suppress_boot_separators(&self) -> bool { self.merge } + + pub(super) fn colors(&self) -> Colors { + self.colors + } + + pub(super) fn urlify(&self) -> bool { + self.urlify + } } pub(crate) struct OutputRenderer { options: OutputOptions, previous_delta: Option<(u64, u64, [u8; 16])>, + columns: usize, + highlight: MatchRange, } impl OutputRenderer { pub(crate) fn new(options: OutputOptions) -> Self { + let columns = if !options.full_width + && !options.show_all + && output_skips_missing_message(options.mode) + { + std::env::var("COLUMNS") + .ok() + .and_then(|value| super::parse_native_u64(&value)) + .filter(|n| (1..=65535).contains(n)) + .map(|n| n as usize) + .or_else(super::terminal::stdout_columns) + .unwrap_or(DEFAULT_COLUMNS) + } else { + DEFAULT_COLUMNS + }; Self { options, previous_delta: None, + columns, + highlight: MatchRange::default(), } } @@ -75,7 +122,7 @@ impl OutputRenderer { OutputModeArg::ShortDelta => self.render_short_delta(entry), OutputModeArg::ShortUnix => self.render_short(entry, TimestampMode::ShortUnix), OutputModeArg::WithUnit => self.render_with_unit(entry), - OutputModeArg::Cat => Ok(self.render_cat(entry)), + OutputModeArg::Cat => self.render_cat(entry), OutputModeArg::Verbose => self.render_verbose(entry), OutputModeArg::Export => Ok(self.render_export(entry)), OutputModeArg::Json => self.render_json(entry, JsonFrame::Line), @@ -86,6 +133,14 @@ impl OutputRenderer { } } + pub(super) fn set_highlight(&mut self, highlight: MatchRange) { + self.highlight = highlight; + } + + pub(super) fn colors(&self) -> Colors { + self.options.colors + } + pub(crate) fn skips_entry(&self, entry: &Entry) -> bool { output_skips_missing_message(self.options.mode) && values_for_name(entry, "MESSAGE").is_empty() @@ -93,30 +148,41 @@ impl OutputRenderer { fn render_short(&mut self, entry: &Entry, timestamp_mode: TimestampMode) -> Result> { let timestamp = format_timestamp(entry, timestamp_mode, self.options.utc)?; - let label = entry_label(entry, &self.options); - let prefix = format!("{timestamp} {label}: "); - let mut out = prefix.clone().into_bytes(); - out.extend_from_slice(&display_message(entry, &self.options, prefix.len())); + let (header, columns) = short_header(entry, &self.options, false); + let prefix_columns = timestamp.len() + columns + 2; + let mut out = timestamp.into_bytes(); + out.extend_from_slice(&header); + out.extend_from_slice(b": "); + out.extend_from_slice(&display_message( + entry, + &self.options, + prefix_columns, + self.columns, + self.highlight, + )); out.push(b'\n'); Ok(out) } fn render_with_unit(&mut self, entry: &Entry) -> Result> { let timestamp = format_timestamp(entry, TimestampMode::ShortFull, self.options.utc)?; - let label = format_entry_label( + let (header, columns) = short_header(entry, &self.options, true); + let prefix_columns = timestamp.len() + columns + 2; + let mut out = timestamp.into_bytes(); + out.extend_from_slice(&header); + out.extend_from_slice(b": "); + out.extend_from_slice(&display_message( entry, - &unit_label(entry).unwrap_or_else(|| base_entry_label(entry)), &self.options, - ); - let prefix = format!("{timestamp} {label}: "); - let mut out = prefix.clone().into_bytes(); - out.extend_from_slice(&display_message(entry, &self.options, prefix.len())); + prefix_columns, + self.columns, + self.highlight, + )); out.push(b'\n'); Ok(out) } fn render_short_delta(&mut self, entry: &Entry) -> Result> { - let label = entry_label(entry, &self.options); let current_realtime = display_realtime_usec(entry); let current_monotonic = display_monotonic_usec(entry); let current = (current_realtime, current_monotonic, entry.boot_id); @@ -124,54 +190,141 @@ impl OutputRenderer { let delta = match self.previous_delta { Some((previous_realtime, previous_monotonic, previous_boot)) => { let (diff, marker) = if previous_boot == entry.boot_id { - (current_monotonic.abs_diff(previous_monotonic), " ") + (current_monotonic.saturating_sub(previous_monotonic), " ") } else { - (current_realtime.abs_diff(previous_realtime), "*") + (current_realtime.saturating_sub(previous_realtime), "*") }; format!(" <{}{}>", format_monotonic(diff), marker) } None => " ".to_string(), }; self.previous_delta = Some(current); - let prefix = format!("[{monotonic}{delta}] {label}: "); - let mut out = prefix.clone().into_bytes(); - out.extend_from_slice(&display_message(entry, &self.options, prefix.len())); + let timestamp = format!("[{monotonic}{delta}]"); + let (header, columns) = short_header(entry, &self.options, false); + let prefix_columns = timestamp.len() + columns + 2; + let mut out = timestamp.into_bytes(); + out.extend_from_slice(&header); + out.extend_from_slice(b": "); + out.extend_from_slice(&display_message( + entry, + &self.options, + prefix_columns, + self.columns, + self.highlight, + )); out.push(b'\n'); Ok(out) } - fn render_cat(&self, entry: &Entry) -> Vec { + fn render_cat(&mut self, entry: &Entry) -> Result> { let mut out = Vec::new(); + if let Some(name) = &self.options.invalid_cat_field { + return Err(anyhow!( + "Invalid field name: {}", + String::from_utf8_lossy(name) + )); + } if self.options.output_fields_set { for name in &self.options.output_fields { - for value in values_for_name(entry, name) { - out.extend_from_slice(&display_value(value, self.options.truncate_newline)); + if let Some(value) = values_for_name(entry, name).first() { + if self.options.colors.enabled() { + self.append_colored_cat_field(&mut out, entry, name, value); + } else { + out.extend_from_slice(value); + } out.push(b'\n'); } } - return out; - } - for value in values_for_name(entry, "MESSAGE") { - out.extend_from_slice(&display_value(value, self.options.truncate_newline)); + } else if let Some(value) = values_for_name(entry, "MESSAGE").first() { + if self.options.colors.enabled() { + self.append_colored_cat_field(&mut out, entry, "MESSAGE", value); + } else { + out.extend_from_slice(value); + } out.push(b'\n'); } - out + Ok(out) + } + + fn append_colored_cat_field(&self, out: &mut Vec, entry: &Entry, name: &str, value: &[u8]) { + let (on, off, highlight) = self.options.colors.log(entry_priority(entry, true), false); + out.extend_from_slice(on.as_bytes()); + if name == "MESSAGE" { + let span = self.highlight; + out.extend_from_slice(&value[..span.start]); + out.extend_from_slice(highlight.as_bytes()); + out.extend_from_slice(&value[span.start..span.end]); + out.extend_from_slice(on.as_bytes()); + out.extend_from_slice(&value[span.end..]); + } else { + out.extend_from_slice(value); + } + out.extend_from_slice(off.as_bytes()); } fn render_verbose(&self, entry: &Entry) -> Result> { let mut out = Vec::new(); - out.extend_from_slice( - format!( - "{} [{}]\n", - format_timestamp(entry, TimestampMode::Verbose, self.options.utc)?, - entry.cursor - ) - .as_bytes(), - ); + if self.options.colors.enabled() { + let underline = if self.options.colors.underline { + "\x1b[0;4m" + } else { + "" + }; + let timestamp = format_timestamp(entry, TimestampMode::Verbose, self.options.utc)?; + let grey = self.options.colors.grey(); + out.extend_from_slice( + format!( + "{underline}{timestamp}{} {grey}[{}]{grey}\n", + colors::NORMAL, + entry.cursor + ) + .as_bytes(), + ); + } else { + out.extend_from_slice( + format!( + "{} [{}]\n", + format_timestamp(entry, TimestampMode::Verbose, self.options.utc)?, + entry.cursor + ) + .as_bytes(), + ); + } for (name, value) in verbose_fields(entry, &self.options) { - out.extend_from_slice(format!(" {name}=").as_bytes()); - out.extend_from_slice(&display_verbose_value(&name, value, &self.options)); + let (on, off) = if !self.options.colors.enabled() { + ("", "") + } else if name == "MESSAGE" { + (colors::HIGHLIGHT, colors::NORMAL) + } else if name.starts_with('_') { + (colors::GREEN, colors::NORMAL) + } else { + ("", "") + }; + out.extend_from_slice(format!(" {on}{name}=").as_bytes()); + let mut displayed = display_verbose_value(&name, value, &self.options); + if self.options.colors.enabled() + && name == "CONFIG_FILE" + && (self.options.show_all + || (self.options.full_width + || name.len() + value.len() + 1 < PRINT_CHAR_THRESHOLD) + && journal_text_printable(value)) + { + if let Some(linked) = super::output_links::file(value, self.options.urlify) { + displayed = full_message_with_nul(&linked, 4 + name.len() + 1); + } + } + out.extend_from_slice(&displayed); + let printable = off.is_empty() + || self.options.show_all + || (self.options.full_width || name.len() + value.len() + 1 < PRINT_CHAR_THRESHOLD) + && journal_text_printable(value); + if !printable { + out.extend_from_slice(off.as_bytes()); + } out.push(b'\n'); + if printable { + out.extend_from_slice(off.as_bytes()); + } } Ok(out) } @@ -215,6 +368,16 @@ impl OutputRenderer { out.push(b'\n'); } } + if self.options.colors.enabled() { + let start = match frame { + JsonFrame::Sse => 6, + JsonFrame::Seq => 1, + _ => 0, + }; + let mut colored = out[..start].to_vec(); + colored.extend_from_slice(&colors::json(&out[start..])); + return Ok(colored); + } Ok(out) } } @@ -231,6 +394,7 @@ enum TimestampMode { Verbose, } +#[derive(Clone, Copy)] enum JsonFrame { Line, Pretty, @@ -238,14 +402,28 @@ enum JsonFrame { Seq, } -fn parse_output_fields(value: Option<&str>) -> Vec { - value - .unwrap_or("") - .split(',') - .map(str::trim) - .filter(|field| !field.is_empty()) - .map(str::to_string) - .collect() +fn parse_output_fields(values: &[std::ffi::OsString]) -> (Vec, bool, Option>) { + let mut names = Vec::new(); + let mut set = false; + let mut invalid_cat = None; + for value in values { + for field in value + .as_encoded_bytes() + .split(|&b| b == b',') + .filter(|field| !field.is_empty()) + { + set = true; + if invalid_cat.is_none() && !valid_field_bytes(field) { + invalid_cat = Some(field.to_vec()); + } + if let Ok(name) = std::str::from_utf8(field) { + if !names.iter().any(|existing| existing == name) { + names.push(name.to_string()); + } + } + } + } + (names, set, invalid_cat) } fn format_timestamp(entry: &Entry, mode: TimestampMode, utc: bool) -> Result { @@ -258,7 +436,7 @@ fn format_timestamp(entry: &Entry, mode: TimestampMode, utc: bool) -> Result Result "%b %d %H:%M:%S", TimestampMode::ShortFull => "%a %Y-%m-%d %H:%M:%S", - TimestampMode::ShortIso => "%Y-%m-%dT%H:%M:%S%:z", - TimestampMode::ShortIsoPrecise => "%Y-%m-%dT%H:%M:%S%.6f%:z", + TimestampMode::ShortIso => "%Y-%m-%dT%H:%M:%S", + TimestampMode::ShortIsoPrecise => "%Y-%m-%dT%H:%M:%S%.6f", TimestampMode::ShortPrecise => "%b %d %H:%M:%S%.6f", TimestampMode::Verbose => "%a %Y-%m-%d %H:%M:%S%.6f", TimestampMode::ShortMonotonic | TimestampMode::ShortUnix => unreachable!(), }; - if utc { - let dt = Utc - .timestamp_opt(secs, nanos) - .single() - .ok_or_else(|| anyhow!("invalid realtime timestamp"))?; - let formatted = dt.format(pattern).to_string(); - return Ok(if needs_zone_name { - format!("{formatted} UTC") - } else { - formatted - }); - } - let dt = Local - .timestamp_opt(secs, nanos) - .single() - .ok_or_else(|| anyhow!("invalid realtime timestamp"))?; - let formatted = dt.format(pattern).to_string(); - if needs_zone_name { - let zone = local_timezone_name(secs).unwrap_or_else(|| dt.format("%Z").to_string()); - return Ok(format!("{formatted} {zone}")); - } - Ok(formatted) + rendering_zone().format_display( + secs, + nanos, + utc, + pattern, + needs_zone_name, + matches!( + mode, + TimestampMode::ShortIso | TimestampMode::ShortIsoPrecise + ), + ) } fn output_skips_missing_message(mode: OutputModeArg) -> bool { @@ -327,11 +494,10 @@ fn display_monotonic_usec(entry: &Entry) -> u64 { } fn source_realtime_usec(entry: &Entry) -> Option { - values_for_name(entry, "_SOURCE_REALTIME_TIMESTAMP") - .first() - .and_then(|value| std::str::from_utf8(value).ok()) - .and_then(|value| value.parse::().ok()) - .filter(|value| *value > 0) + last_value_for_name(entry, "_SOURCE_REALTIME_TIMESTAMP") + .and_then(|value| std::str::from_utf8(c_string_bytes(value)).ok()) + .and_then(super::parse_native_u64) + .filter(|value| *value > 0 && *value < 1 << 55) } fn map_clock_usec(value: u64, from: u64, to: u64) -> u64 { @@ -345,119 +511,420 @@ pub(crate) fn format_header_timestamp(usec: u64) -> Result { } let secs = (usec / 1_000_000) as i64; let nanos = ((usec % 1_000_000) * 1000) as u32; - let dt = Local - .timestamp_opt(secs, nanos) - .single() - .ok_or_else(|| anyhow!("invalid realtime timestamp"))?; - let formatted = dt.format("%a %Y-%m-%d %H:%M:%S").to_string(); - let zone = local_timezone_name(secs).unwrap_or_else(|| dt.format("%Z").to_string()); - Ok(format!("{formatted} {zone}")) + rendering_zone().format_display(secs, nanos, false, "%a %Y-%m-%d %H:%M:%S", true, false) } fn format_monotonic(usec: u64) -> String { format!("{:>5}.{:06}", usec / 1_000_000, usec % 1_000_000) } -#[cfg(unix)] -fn local_timezone_name(secs: i64) -> Option { - let timestamp = libc::time_t::try_from(secs).ok()?; - let mut local_tm = std::mem::MaybeUninit::::uninit(); - // SAFETY: localtime_r initializes local_tm for the provided timestamp - // and returns a null pointer on failure. - let ptr = unsafe { libc::localtime_r(×tamp, local_tm.as_mut_ptr()) }; - if ptr.is_null() { - return None; - } - // SAFETY: localtime_r returned success, so local_tm is initialized. - let local_tm = unsafe { local_tm.assume_init() }; - if local_tm.tm_zone.is_null() { - return None; - } - // SAFETY: tm_zone is a NUL-terminated C string owned by the C runtime. - unsafe { CStr::from_ptr(local_tm.tm_zone) } - .to_str() - .ok() - .map(str::to_string) +fn rendering_zone() -> &'static super::timestamp_zone::Zone { + static ZONE: OnceLock = OnceLock::new(); + ZONE.get_or_init(super::timestamp_zone::Zone::local) } -#[cfg(not(unix))] -fn local_timezone_name(_secs: i64) -> Option { - None +pub(super) fn last_value_for_name<'a>(entry: &'a Entry, name: &str) -> Option<&'a [u8]> { + entry + .field_values + .get(name) + .and_then(|values| values.last()) + .or_else(|| entry.fields.get(name)) + .map(Vec::as_slice) } -fn entry_label(entry: &Entry, options: &OutputOptions) -> String { - format_entry_label(entry, &base_entry_label(entry), options) +fn short_field<'a>(entry: &'a Entry, name: &str, options: &OutputOptions) -> Option<&'a [u8]> { + last_value_for_name(entry, name).filter(|value| short_field_printable(value, options)) } -fn base_entry_label(entry: &Entry) -> String { - first_string(entry, "SYSLOG_IDENTIFIER") - .or_else(|| first_string(entry, "_COMM")) - .or_else(|| first_string(entry, "_EXE")) - .unwrap_or_else(|| "unknown".to_string()) +fn short_field_printable(value: &[u8], options: &OutputOptions) -> bool { + options.show_all || value.len() < PRINT_CHAR_THRESHOLD && journal_text_printable(value) } -fn format_entry_label(entry: &Entry, label: &str, options: &OutputOptions) -> String { - let mut parts = Vec::new(); +fn short_header(entry: &Entry, options: &OutputOptions, with_unit: bool) -> (Vec, usize) { + let mut out = Vec::new(); + let mut columns = 0; if !options.no_hostname { - if let Some(hostname) = first_string(entry, "_HOSTNAME") { - parts.push(hostname); + if let Some(value) = short_field(entry, "_HOSTNAME", options) { + out.push(b' '); + out.extend_from_slice(c_string_bytes(value)); + columns += value.len() + 1; } } - let mut label = if label.is_empty() { - "unknown".to_string() - } else { - label.to_string() - }; - if let Some(pid) = first_string(entry, "_PID") { - label.push('['); - label.push_str(&pid); - label.push(']'); - } else if let Some(pid) = first_string(entry, "SYSLOG_PID") { - label.push('['); - label.push_str(&pid); - label.push(']'); - } - parts.push(label); - parts.join(" ") -} - -fn unit_label(entry: &Entry) -> Option { - for name in [ - "_SYSTEMD_UNIT", - "_SYSTEMD_USER_UNIT", - "UNIT", - "USER_UNIT", - "OBJECT_SYSTEMD_UNIT", - "OBJECT_SYSTEMD_USER_UNIT", - ] { - if let Some(value) = first_string(entry, name) { - return Some(value); - } - } - None -} - -fn display_message(entry: &Entry, options: &OutputOptions, prefix_columns: usize) -> Vec { - values_for_name(entry, "MESSAGE") - .first() + let mut unit_shown = false; + if with_unit { + let unit = last_value_for_name(entry, "_SYSTEMD_UNIT"); + let user_unit = last_value_for_name(entry, "_SYSTEMD_USER_UNIT"); + if unit.is_some_and(|v| short_field_printable(v, options)) + || user_unit.is_some_and(|v| short_field_printable(v, options)) + { + if let Some(value) = unit { + out.push(b' '); + out.extend_from_slice(c_string_bytes(value)); + columns += value.len() + 1; + } + if let Some(value) = user_unit { + out.push(if unit.is_some() { b'/' } else { b' ' }); + out.extend_from_slice(c_string_bytes(value)); + // v262 accounts the system-unit length here as well. + columns += unit.map_or(0, |v| v.len()) + 1; + } + unit_shown = true; + } + } + if !unit_shown { + if let Some(value) = short_field(entry, "SYSLOG_IDENTIFIER", options) + .or_else(|| short_field(entry, "_COMM", options)) + { + out.push(b' '); + out.extend_from_slice(c_string_bytes(value)); + columns += value.len() + 1; + } else { + out.extend_from_slice(b" unknown"); + } + } + if let Some(value) = + short_field(entry, "_PID", options).or_else(|| short_field(entry, "SYSLOG_PID", options)) + { + out.push(b'['); + out.extend_from_slice(c_string_bytes(value)); + out.push(b']'); + columns += value.len() + 2; + } + (out, columns) +} + +fn display_message( + entry: &Entry, + options: &OutputOptions, + prefix_columns: usize, + columns: usize, + highlight: MatchRange, +) -> Vec { + if options.colors.enabled() || options.urlify { + return display_colored_message(entry, options, prefix_columns, columns, highlight); + } + last_value_for_name(entry, "MESSAGE") .map(|value| { - let value = if options.truncate_newline { - truncate_at_newline(value) + let (prepared, printable) = if options.show_all { + (Cow::Borrowed(value), true) } else { - *value + prepare_short_message(value, options.truncate_newline) }; - if !options.show_all && !journal_text_printable(value) { + let value = if options.show_all && options.truncate_newline { + truncate_at_newline(&prepared) + } else { + &prepared + }; + if !printable { return blob_data(value.len()).into_bytes(); } - let mut out = indent_continuation_lines(c_string_bytes(value), prefix_columns); + if options.show_all && value.contains(&0) { + return full_message_with_nul(value, prefix_columns); + } if !options.show_all && !options.full_width { - out = ellipsize_line(&out, prefix_columns); + return shortened_message(value, prefix_columns, columns); + } + let mut out = indent_continuation_lines(value, prefix_columns); + if out.last() == Some(&b'\n') { + out.pop(); } out }) .unwrap_or_default() } +fn entry_priority(entry: &Entry, first: bool) -> u8 { + let value = if first { + values_for_name(entry, "PRIORITY").first().copied() + } else { + last_value_for_name(entry, "PRIORITY") + }; + match value { + Some([byte @ b'0'..=b'7']) => byte - b'0', + _ => 6, + } +} + +fn advance_highlight(original: MatchRange, shift: &mut MatchRange, at: usize, size: usize) { + if at < original.start { + shift.start += size; + } + if at < original.end { + shift.end += size; + } +} + +fn display_colored_message( + entry: &Entry, + options: &OutputOptions, + prefix: usize, + columns: usize, + mut span: MatchRange, +) -> Vec { + let Some(value) = last_value_for_name(entry, "MESSAGE") else { + return Vec::new(); + }; + let prepared = if !options.show_all { + match value.iter().position(|b| b"\t\r\x1b".contains(b)) { + Some(start) => Cow::Owned(strip_short_controls_with_highlight( + value, + start, + Some(&mut span), + )), + None => Cow::Borrowed(value), + } + } else { + Cow::Borrowed(value) + }; + let value = if options.truncate_newline { + truncate_at_newline(&prepared) + } else { + &prepared + }; + let mut out = if options.urlify { + short_field(entry, "DOCUMENTATION", options) + .map(|url| super::output_links::documentation(url, ellipsis_marker() == "…")) + .unwrap_or_default() + } else { + Vec::new() + }; + if !options.show_all && !journal_text_printable(value) { + out.extend_from_slice(blob_data(value.len()).as_bytes()); + return out; + } + let mut message = Cow::Borrowed(value); + if let Some(config) = last_value_for_name(entry, "CONFIG_FILE") { + if value.starts_with(config) + && (value.len() == config.len() || matches!(value.get(config.len()), Some(b':' | b' '))) + && (span.start == 0 || span.start > config.len()) + { + if let Some(mut linked) = super::output_links::file(config, options.urlify) + .filter(|linked| linked.len() >= config.len()) + { + let shift = linked.len() - config.len(); + linked.extend_from_slice(&value[config.len()..]); + message = Cow::Owned(linked); + span.start += shift; + span.end += shift; + } + } + } + let value = message.as_ref(); + let audit = + last_value_for_name(entry, "_TRANSPORT").is_some_and(|v| c_string_bytes(v) == b"audit"); + let (on, off, highlight) = options.colors.log(entry_priority(entry, false), audit); + let (mut start, mut line) = (0, 0); + while start < value.len() { + let end = value[start..] + .iter() + .position(|&b| b == b'\n') + .map_or(value.len(), |n| start + n); + let part = &value[start..end]; + let tail = line == 2 || end + 1 >= PRINT_CHAR_THRESHOLD; + let shorten = !options.show_all + && !options.full_width + && (prefix + part.len() + 1 >= columns || tail); + if line > 0 { + out.push(b'\n'); + } + if shorten && columns.saturating_sub(prefix) < 3 { + out.extend_from_slice(b"..."); + } else { + if line > 0 { + out.extend(std::iter::repeat_n(b' ', prefix)); + } + out.extend_from_slice(on.as_bytes()); + if !shorten { + if start <= span.start && span.start < part.len() { + out.extend_from_slice(c_string_bytes(&part[..span.start])); + out.extend_from_slice(highlight.as_bytes()); + out.extend_from_slice(c_string_bytes( + &part[span.start..span.end.min(part.len())], + )); + if part.len() > span.end { + out.extend_from_slice(on.as_bytes()); + out.extend_from_slice(c_string_bytes(&part[span.end..])); + } + } else { + out.extend_from_slice(c_string_bytes(part)); + } + } else if columns - prefix > part.len() + 3 { + out.extend_from_slice(c_string_bytes(part)); + out.extend_from_slice(b"..."); + } else { + let percent = if tail { 100 } else { 90 }; + out.extend_from_slice(c_string_bytes(&ellipsize_line( + part, + columns - prefix, + percent, + ))); + } + out.extend_from_slice(off.as_bytes()); + } + if shorten && tail { + break; + } + start = end + 1; + line += 1; + } + out +} + +fn prepare_short_message(value: &[u8], truncate: bool) -> (Cow<'_, [u8]>, bool) { + if let Ok(text) = std::str::from_utf8(value) { + let mut printable = true; + for (index, ch) in text.char_indices() { + if matches!(ch, '\x1b' | '\t' | '\r') { + let mut out = strip_short_controls(value, index); + if truncate { + out.truncate(truncate_at_newline(&out).len()); + } + let printable = journal_text_printable(&out); + return (Cow::Owned(out), printable); + } + if truncate && matches!(ch, '\n' | '\0') { + return (Cow::Borrowed(&value[..index]), printable); + } + let cp = ch as u32; + printable &= (cp >= 0x20 || ch == '\n') && !(0x7f..=0x9f).contains(&cp); + } + return (Cow::Borrowed(value), printable); + } + let mut out = match value + .iter() + .position(|b| matches!(b, b'\x1b' | b'\t' | b'\r')) + { + Some(index) => Cow::Owned(strip_short_controls(value, index)), + None => Cow::Borrowed(value), + }; + if truncate { + let len = truncate_at_newline(&out).len(); + match &mut out { + Cow::Owned(value) => value.truncate(len), + Cow::Borrowed(value) => *value = &value[..len], + } + } + let printable = journal_text_printable(&out); + (out, printable) +} + +fn strip_short_controls(value: &[u8], start: usize) -> Vec { + strip_short_controls_with_highlight(value, start, None) +} + +fn strip_short_controls_with_highlight( + value: &[u8], + start: usize, + highlight: Option<&mut MatchRange>, +) -> Vec { + let original = highlight.as_deref().copied().unwrap_or_default(); + let mut shift = MatchRange::default(); + let mut out = Vec::with_capacity(value.len()); + out.extend_from_slice(&value[..start]); + let mut i = start; + while i < value.len() { + match value[i] { + b'\t' => { + out.extend_from_slice(b" "); + advance_highlight(original, &mut shift, i, 7); + i += 1; + } + b'\r' => { + let mut end = i + 1; + while end < value.len() && value[end] == b'\r' { + end += 1; + } + if end < value.len() && value[end] != b'\n' { + out.extend_from_slice(&value[i..end]); + } + i = end; + } + b'\x1b' if i + 1 < value.len() => { + let mut end = i + 2; + let mut complete = false; + let (mut invalid_at, mut size) = (i + 1, 1); + match value[i + 1] { + b'[' => { + size = 2; + while end < value.len() { + let b = value[end]; + end += 1; + if b == b'm' { + complete = true; + break; + } + if b != 0 && b != b';' && b != b':' && !b.is_ascii_digit() { + invalid_at = end - 1; + break; + } + invalid_at = end; + } + } + b']' => { + size = 2; + while end < value.len() { + let b = value[end]; + end += 1; + if b == 7 { + complete = true; + break; + } + if b == 27 { + invalid_at = end; + if value.get(end) == Some(&b'\\') { + end += 1; + complete = true; + } + break; + } + if !(32..127).contains(&b) { + invalid_at = end - 1; + break; + } + invalid_at = end; + } + } + _ => {} + } + if complete { + i = end; + } else { + // Native preserves malformed sequences and reprocesses their body. + advance_highlight(original, &mut shift, invalid_at, size); + out.extend_from_slice(&value[i..i + 2]); + i += 2; + } + } + byte => { + out.push(byte); + i += 1; + } + } + } + if let Some(highlight) = highlight { + highlight.start += shift.start; + highlight.end += shift.end; + } + out +} + +fn full_message_with_nul(value: &[u8], prefix_columns: usize) -> Vec { + let mut out = Vec::new(); + for (index, line) in value + .strip_suffix(b"\n") + .unwrap_or(value) + .split(|&b| b == b'\n') + .enumerate() + { + if index > 0 { + out.push(b'\n'); + out.extend(std::iter::repeat(b' ').take(prefix_columns)); + } + out.extend_from_slice(c_string_bytes(line)); + } + out +} + fn indent_continuation_lines(value: &[u8], prefix_columns: usize) -> Vec { let newline_count = value.iter().filter(|byte| **byte == b'\n').count(); if newline_count == 0 { @@ -473,12 +940,6 @@ fn indent_continuation_lines(value: &[u8], prefix_columns: usize) -> Vec { out } -fn first_string(entry: &Entry, name: &str) -> Option { - values_for_name(entry, name) - .first() - .map(|value| String::from_utf8_lossy(value).into_owned()) -} - fn values_for_name<'a>(entry: &'a Entry, name: &str) -> Vec<&'a [u8]> { if let Some(values) = entry.field_values.get(name) { if !values.is_empty() { @@ -492,29 +953,23 @@ fn values_for_name<'a>(entry: &'a Entry, name: &str) -> Vec<&'a [u8]> { .unwrap_or_default() } -fn display_value(value: &[u8], truncate_newline: bool) -> Vec { - if !truncate_newline { - return value.to_vec(); - } - truncate_at_newline(value).to_vec() -} - fn display_verbose_value(name: &str, value: &[u8], options: &OutputOptions) -> Vec { - if options.show_all { - return c_string_bytes(value).to_vec(); - } - if !journal_text_printable(value) - || (!options.full_width && name.len() + 1 + value.len() >= PRINT_CHAR_THRESHOLD) + if !options.show_all + && (!journal_text_printable(value) + || (!options.full_width && name.len() + 1 + value.len() >= PRINT_CHAR_THRESHOLD)) { return blob_data(value.len()).into_bytes(); } + if value.contains(&b'\n') || value.contains(&0) { + return full_message_with_nul(value, 4 + name.len() + 1); + } value.to_vec() } fn truncate_at_newline(value: &[u8]) -> &[u8] { let end = value .iter() - .position(|byte| *byte == b'\n') + .position(|byte| matches!(byte, b'\n' | b'\r' | 0)) .unwrap_or(value.len()); &value[..end] } @@ -527,22 +982,163 @@ fn c_string_bytes(value: &[u8]) -> &[u8] { &value[..end] } -fn ellipsize_line(value: &[u8], prefix_columns: usize) -> Vec { - let limit = DEFAULT_COLUMNS.saturating_sub(prefix_columns + 1); - if value.len() <= limit { +fn ellipsis_marker() -> &'static str { + static MARKER: std::sync::OnceLock<&'static str> = std::sync::OnceLock::new(); + MARKER.get_or_init(|| { + let locale = ["LC_ALL", "LC_CTYPE", "LANG"] + .iter() + .find_map(|name| std::env::var(name).ok().filter(|s| !s.is_empty())) + .unwrap_or_default(); + if locale.to_ascii_lowercase().contains("utf") { + "…" + } else { + "..." + } + }) +} + +fn shortened_message(value: &[u8], prefix: usize, columns: usize) -> Vec { + let mut out = Vec::new(); + let mut start = 0; + let mut line = 0; + while start < value.len() { + let end = value[start..] + .iter() + .position(|&b| b == b'\n') + .map(|n| start + n) + .unwrap_or(value.len()); + let part = &value[start..end]; + let tail = line == 2 || end + 1 >= PRINT_CHAR_THRESHOLD; + let shorten = prefix + part.len() + 1 >= columns || tail; + if line > 0 { + out.push(b'\n'); + } + if shorten && columns.saturating_sub(prefix) < 3 { + out.extend_from_slice(b"..."); + } else { + if line > 0 { + out.extend(std::iter::repeat(b' ').take(prefix)); + } + if !shorten { + out.extend_from_slice(part); + } else if columns - prefix > part.len() + 3 { + out.extend_from_slice(part); + out.extend_from_slice(b"..."); + } else { + out.extend_from_slice(&ellipsize_line( + part, + columns - prefix, + if tail { 100 } else { 90 }, + )); + } + } + if tail { + break; + } + start = end + 1; + line += 1; + } + out +} + +fn ellipsize_line(value: &[u8], columns: usize, percent: usize) -> Vec { + if value.is_ascii() { + if value.len() <= columns { + return value.to_vec(); + } + let marker = ellipsis_marker(); + let marker_width = if marker == "..." { 3 } else { 1 }; + let left = ((columns - marker_width) * percent + 50) / 100; + let right = columns - marker_width - left; + let mut out = value[..left].to_vec(); + out.extend_from_slice(marker.as_bytes()); + out.extend_from_slice(&value[value.len() - right..]); + return out; + } + let text = std::str::from_utf8(value).expect("printable message is UTF8"); + let (mut left, mut right, mut used) = (0, text.len(), 0); + let budget = (columns - 1) * percent / 100; + for ch in text.chars() { + let width = journal_char_width(ch); + if used + width > budget { + break; + } + used += width; + left += ch.len_utf8(); + } + for ch in text[left..].chars().rev() { + if used >= columns { + break; + } + let width = journal_char_width(ch); + if used + width > columns { + break; + } + used += width; + right -= ch.len_utf8(); + } + if left >= right { return value.to_vec(); } - let mut end = limit; - if let Ok(text) = std::str::from_utf8(value) { - while end > 0 && !text.is_char_boundary(end) { - end -= 1; + if used >= columns { + if right < text.len() { + right += text[right..].chars().next().unwrap().len_utf8(); + } else if left > 0 { + left -= text[..left].chars().next_back().unwrap().len_utf8(); } } - let mut out = value[..end].to_vec(); + let mut out = value[..left].to_vec(); out.extend_from_slice("…".as_bytes()); + out.extend_from_slice(&value[right..]); out } +// v262 fixes character width to these intervals. +fn journal_char_width(ch: char) -> usize { + if matches!(ch as u32, + 0x1100..=0x115F | + 0x2329..=0x232A | + 0x2E80..=0x2E99 | + 0x2E9B..=0x2EF3 | + 0x2F00..=0x2FD5 | + 0x2FF0..=0x2FFB | + 0x3000..=0x303E | + 0x3041..=0x3096 | + 0x3099..=0x30FF | + 0x3105..=0x312D | + 0x3131..=0x318E | + 0x3190..=0x31BA | + 0x31C0..=0x31E3 | + 0x31F0..=0x321E | + 0x3220..=0x3247 | + 0x3250..=0x32FE | + 0x3300..=0x4DBF | + 0x4E00..=0xA48C | + 0xA490..=0xA4C6 | + 0xA960..=0xA97C | + 0xAC00..=0xD7A3 | + 0xF900..=0xFAFF | + 0xFE10..=0xFE19 | + 0xFE30..=0xFE52 | + 0xFE54..=0xFE66 | + 0xFE68..=0xFE6B | + 0xFF01..=0xFF60 | + 0xFFE0..=0xFFE6 | + 0x1B000..=0x1B001 | + 0x1F200..=0x1F202 | + 0x1F210..=0x1F23A | + 0x1F240..=0x1F248 | + 0x1F250..=0x1F251 | + 0x1F300..=0x1F567 | + 0x20000..=0x2FFFD | + 0x30000..=0x3FFFD + ) { + 2 + } else { + 1 + } +} + fn journal_text_printable(value: &[u8]) -> bool { let Ok(text) = std::str::from_utf8(value) else { return false; @@ -671,7 +1267,8 @@ fn json_object(entry: &Entry, options: &OutputOptions) -> Map { fn add_json_value(map: &mut Map, name: String, value: &[u8], show_all: bool) { let value = json_value_for_bytes(&name, value, show_all); match map.remove(&name) { - Some(Value::Array(mut values)) => { + // Scalar byte arrays contain numbers; repeated values never do. + Some(Value::Array(mut values)) if values.first().is_some_and(|v| !v.is_number()) => { values.push(value); map.insert(name, Value::Array(values)); } @@ -748,24 +1345,7 @@ fn write_systemd_pretty_json_at(out: &mut Vec, value: &Value, depth: usize) } out.push(b'}'); } - Value::Array(values) => { - out.push(b'['); - if !values.is_empty() { - out.push(b'\n'); - } - for (idx, value) in values.iter().enumerate() { - write_json_tabs(out, depth + 1); - write_systemd_pretty_json_at(out, value, depth + 1)?; - if idx + 1 != values.len() { - out.push(b','); - } - out.push(b'\n'); - } - if !values.is_empty() { - write_json_tabs(out, depth); - } - out.push(b']'); - } + Value::Array(values) => write_systemd_pretty_json_array_at(out, values, depth)?, _ => { serde_json::to_writer(&mut *out, value).map_err(|err| anyhow!("json output: {err}"))?; } @@ -773,6 +1353,30 @@ fn write_systemd_pretty_json_at(out: &mut Vec, value: &Value, depth: usize) Ok(()) } +pub(super) fn write_systemd_pretty_json_array_at( + out: &mut Vec, + values: &[Value], + depth: usize, +) -> Result<()> { + out.push(b'['); + if !values.is_empty() { + out.push(b'\n'); + } + for (idx, value) in values.iter().enumerate() { + write_json_tabs(out, depth + 1); + write_systemd_pretty_json_at(out, value, depth + 1)?; + if idx + 1 != values.len() { + out.push(b','); + } + out.push(b'\n'); + } + if !values.is_empty() { + write_json_tabs(out, depth); + } + out.push(b']'); + Ok(()) +} + fn write_json_tabs(out: &mut Vec, depth: usize) { out.extend(std::iter::repeat(b'\t').take(depth)); } diff --git a/rust/src/cmd/journalctl/output_links.rs b/rust/src/cmd/journalctl/output_links.rs new file mode 100644 index 00000000..fc6fdf13 --- /dev/null +++ b/rust/src/cmd/journalctl/output_links.rs @@ -0,0 +1,180 @@ +use super::colors::Colors; + +pub(super) fn enabled(colors: Colors) -> bool { + std::env::var("SYSTEMD_URLIFY") + .ok() + .and_then(|value| super::parse_bool_option("SYSTEMD_URLIFY", &value).ok()) + .unwrap_or(colors.enabled()) +} + +fn terminal_link(url: &[u8], text: &[u8]) -> Vec { + if url.len() > 2000 || url.iter().any(|b| !(32..127).contains(b)) { + return text.to_vec(); + } + let mut out = b"\x1b]8;;".to_vec(); + out.extend_from_slice(url); + out.extend_from_slice(b"\x1b\\"); + out.extend_from_slice(text); + out.extend_from_slice(b"\x1b]8;;\x1b\\"); + out +} + +pub(super) fn file(value: &[u8], enabled: bool) -> Option> { + let text = value.split(|&b| b == 0).next().unwrap_or_default(); + if text.is_empty() { + return None; + } + if !enabled { + return Some(text.to_vec()); + } + let Ok(path) = std::str::from_utf8(text) else { + return Some(text.to_vec()); + }; + static HOSTNAME: std::sync::OnceLock>> = std::sync::OnceLock::new(); + let hostname = HOSTNAME.get_or_init(super::terminal::hostname).as_deref()?; + #[cfg(windows)] + { + let Some(path) = std::path::absolute(path) + .ok() + .and_then(|p| p.to_str().map(str::to_owned)) + else { + return Some(text.to_vec()); + }; + return Some(match windows_file_url(&path, hostname) { + Some(url) => terminal_link(&url, text), + None => text.to_vec(), + }); + } + #[cfg(not(windows))] + { + let mut url = b"file://".to_vec(); + url.extend_from_slice(hostname); + if !std::path::Path::new(path).is_absolute() { + let cwd = working_directory()?; + let bytes = cwd.as_os_str().as_encoded_bytes(); + url.extend_from_slice(bytes); + if !bytes.ends_with(&[std::path::MAIN_SEPARATOR as u8]) { + url.push(std::path::MAIN_SEPARATOR as u8); + } + } + url.extend_from_slice(text); + Some(terminal_link(&url, text)) + } +} + +#[cfg(not(windows))] +fn working_directory() -> Option { + #[cfg(unix)] + if let Some(pwd) = std::env::var_os("PWD") + .map(std::path::PathBuf::from) + .filter(|p| p.is_absolute()) + { + use std::os::unix::fs::MetadataExt; + if let (Ok(logical), Ok(current)) = (std::fs::metadata(&pwd), std::fs::metadata(".")) { + if logical.dev() == current.dev() && logical.ino() == current.ino() { + return Some(pwd); + } + } + } + std::env::current_dir().ok() +} + +pub(super) fn documentation(value: &[u8], mut unicode: bool) -> Vec { + let url = value.split(|&b| b == 0).next().unwrap_or_default(); + if !url.is_ascii() + || ![ + b"http://".as_slice(), + b"https://", + b"file:/", + b"info:", + b"man:", + ] + .iter() + .any(|prefix| url.starts_with(prefix) && url.len() > prefix.len()) + { + return Vec::new(); + } + if let Ok(value) = std::env::var("SYSTEMD_UTF8") { + if let Ok(value) = super::parse_bool_option("SYSTEMD_UTF8", &value) { + unicode = value; + } + } + let glyph = if unicode { "[🡕]" } else { "[LNK]" }; + let mut out = terminal_link(url, glyph.as_bytes()); + out.push(b' '); + out +} + +#[cfg(any(windows, test))] +fn windows_file_url(path: &str, hostname: &[u8]) -> Option> { + let path = path.replace('\\', "/"); + if path.starts_with("//?/") || path.starts_with("//./") { + return None; + } + let (authority, path) = if let Some(rest) = path.strip_prefix("//") { + let (server, share) = rest.split_once('/')?; + if server.is_empty() || share.is_empty() { + return None; + } + (server.as_bytes(), format!("/{share}")) + } else { + let bytes = path.as_bytes(); + if bytes.len() < 3 || !bytes[0].is_ascii_alphabetic() || bytes[1..3] != *b":/" { + return None; + } + (hostname, format!("/{path}")) + }; + if authority + .iter() + .any(|b| !b.is_ascii_alphanumeric() && !b".-".contains(b)) + { + return None; + } + let mut url = b"file://".to_vec(); + url.extend_from_slice(authority); + const HEX: &[u8] = b"0123456789ABCDEF"; + for b in path.bytes() { + if b.is_ascii_alphanumeric() || b"/-._~:".contains(&b) { + url.push(b); + } else { + url.extend_from_slice(&[b'%', HEX[(b >> 4) as usize], HEX[(b & 15) as usize]]); + } + } + Some(url) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn windows_file_uri_paths() { + for (path, want) in [ + (r"C:\logs\example.conf", "file://host/C:/logs/example.conf"), + ( + r"\\server\share\é #%.conf", + "file://server/share/%C3%A9%20%23%25.conf", + ), + ("C:/logs/a?b", "file://host/C:/logs/a%3Fb"), + ] { + assert_eq!(windows_file_url(path, b"host").unwrap(), want.as_bytes()); + } + for path in [r"\\?\C:\logs", r"\\.\pipe\name", "relative", "//server"] { + assert!(windows_file_url(path, b"host").is_none()); + } + } + + #[cfg(windows)] + #[test] + fn relative_windows_file_link() { + let path = "relative #.conf"; + let absolute = std::path::absolute(path).unwrap(); + let hostname = super::super::terminal::hostname().unwrap(); + let url = windows_file_url(absolute.to_str().unwrap(), &hostname).unwrap(); + assert_eq!( + file(path.as_bytes(), true).unwrap(), + terminal_link(&url, path.as_bytes()) + ); + assert_eq!(file(path.as_bytes(), false).unwrap(), path.as_bytes()); + } +} diff --git a/rust/src/cmd/journalctl/parser.rs b/rust/src/cmd/journalctl/parser.rs new file mode 100644 index 00000000..04f6dd7a --- /dev/null +++ b/rust/src/cmd/journalctl/parser.rs @@ -0,0 +1,393 @@ +use super::*; +use clap::CommandFactory; +use std::ffi::{OsStr, OsString}; + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Kind { + None, + Required, + Optional, +} + +// The option kinds follow the pinned v262 surface, including hidden options. +const OPTIONS: &[(&str, Option, Kind)] = &[ + ("system", None, Kind::None), + ("user", None, Kind::None), + ("machine", Some(b'M'), Kind::Required), + ("merge", Some(b'm'), Kind::None), + ("directory", Some(b'D'), Kind::Required), + ("file", Some(b'i'), Kind::Required), + ("root", None, Kind::Required), + ("image", None, Kind::Required), + ("image-policy", None, Kind::Required), + ("namespace", None, Kind::Required), + ("since", Some(b'S'), Kind::Required), + ("until", Some(b'U'), Kind::Required), + ("cursor", Some(b'c'), Kind::Required), + ("after-cursor", None, Kind::Required), + ("cursor-file", None, Kind::Required), + ("this-boot", None, Kind::None), + ("boot", Some(b'b'), Kind::Optional), + ("unit", Some(b'u'), Kind::Required), + ("user-unit", None, Kind::Required), + ("invocation", None, Kind::Required), + ("I", Some(b'I'), Kind::None), + ("identifier", Some(b't'), Kind::Required), + ("exclude-identifier", Some(b'T'), Kind::Required), + ("priority", Some(b'p'), Kind::Required), + ("facility", None, Kind::Required), + ("grep", Some(b'g'), Kind::Required), + ("case-sensitive", None, Kind::Optional), + ("dmesg", Some(b'k'), Kind::None), + ("output", Some(b'o'), Kind::Required), + ("output-fields", None, Kind::Required), + ("lines", Some(b'n'), Kind::Optional), + ("reverse", Some(b'r'), Kind::None), + ("show-cursor", None, Kind::None), + ("utc", None, Kind::None), + ("catalog", Some(b'x'), Kind::None), + ("no-hostname", Some(b'W'), Kind::None), + ("full", Some(b'l'), Kind::None), + ("no-full", None, Kind::None), + ("all", Some(b'a'), Kind::None), + ("follow", Some(b'f'), Kind::None), + ("no-tail", None, Kind::None), + ("truncate-newline", None, Kind::None), + ("quiet", Some(b'q'), Kind::None), + ("synchronize-on-exit", None, Kind::Required), + ("no-pager", None, Kind::None), + ("pager-end", Some(b'e'), Kind::None), + ("interval", None, Kind::Required), + ("verify-key", None, Kind::Required), + ("force", None, Kind::None), + ("help", Some(b'h'), Kind::None), + ("version", None, Kind::None), + ("fields", Some(b'N'), Kind::None), + ("field", Some(b'F'), Kind::Required), + ("list-boots", None, Kind::None), + ("list-invocations", None, Kind::None), + ("list-namespaces", None, Kind::None), + ("disk-usage", None, Kind::None), + ("vacuum-size", None, Kind::Required), + ("vacuum-files", None, Kind::Required), + ("vacuum-time", None, Kind::Required), + ("verify", None, Kind::None), + ("sync", None, Kind::None), + ("relinquish-var", None, Kind::None), + ("smart-relinquish-var", None, Kind::None), + ("flush", None, Kind::None), + ("rotate", None, Kind::None), + ("header", None, Kind::None), + ("list-catalog", None, Kind::None), + ("dump-catalog", None, Kind::None), + ("update-catalog", None, Kind::None), + ("setup-keys", None, Kind::None), + ("new-id128", None, Kind::None), + ("introspect-cli", None, Kind::None), + ("head", None, Kind::Required), + ("tail", None, Kind::Required), + ("verify-only", None, Kind::None), +]; +fn os_slice(bytes: &[u8]) -> &OsStr { + // SAFETY: callers split an encoded argv value only at ASCII option separators. + unsafe { OsStr::from_encoded_bytes_unchecked(bytes) } +} + +fn long_option(name: &str) -> Result<(&'static str, Kind)> { + if let Some(&(name, _, kind)) = OPTIONS.iter().find(|&&(n, _, _)| n != "I" && n == name) { + return Ok((name, kind)); + } + let mut matches = OPTIONS + .iter() + .filter(|&&(n, _, _)| n != "I" && n.starts_with(name)); + let Some(&(found, _, kind)) = matches.next() else { + return Err(anyhow!("unrecognized option '--{name}'")); + }; + if matches.next().is_some() { + return Err(anyhow!("option '--{name}' is ambiguous")); + } + Ok((found, kind)) +} + +fn option_token(name: &str, value: &OsStr) -> OsString { + let mut out = OsString::from(format!("--{name}=")); + out.push(value); + out +} + +pub fn parse_cli(raw: Vec) -> Result> { + let mut tokens = vec![raw.first().cloned().unwrap_or_else(|| "journalctl".into())]; + let mut operands = Vec::new(); + let mut action = "show"; + let mut field = None; + let mut output = "short".to_string(); + let mut quiet = false; + let mut full_width = true; + let mut projection = Vec::new(); + let mut timestamps = timestamp::Parser::default(); + let mut since_usec = None; + let mut until_usec = None; + let mut i = 1; + while i < raw.len() { + let arg = raw[i].as_encoded_bytes(); + if arg == b"--" { + operands.extend_from_slice(&raw[i + 1..]); + break; + } + if arg == b"-" || !arg.starts_with(b"-") { + operands.push(raw[i].clone()); + i += 1; + continue; + } + let long = arg.starts_with(b"--"); + let mut token = &arg[if long { 2 } else { 1 }..]; + while !token.is_empty() { + let name; + let kind; + let mut value: Option = None; + if long { + let (key, inline) = match token.iter().position(|&b| b == b'=') { + Some(n) => (&token[..n], Some(&token[n + 1..])), + None => (token, None), + }; + let key = std::str::from_utf8(key).map_err(|_| anyhow!("unrecognized option"))?; + (name, kind) = long_option(key)?; + value = inline.map(|bytes| os_slice(bytes).to_owned()); + token = &[]; + } else { + let Some(&(found, _, found_kind)) = + OPTIONS.iter().find(|&&(_, s, _)| s == Some(token[0])) + else { + return Err(anyhow!("invalid option -- '{}'", token[0] as char)); + }; + name = found; + kind = found_kind; + token = &token[1..]; + if kind != Kind::None && !token.is_empty() { + value = Some(os_slice(token).to_owned()); + token = &[]; + } + } + if kind == Kind::None && value.is_some() { + return Err(anyhow!("option '--{name}' doesn't allow an argument")); + } + if kind == Kind::Required && value.is_none() { + i += 1; + value = Some( + raw.get(i) + .ok_or_else(|| anyhow!("option '--{name}' requires an argument"))? + .clone(), + ); + } + if kind == Kind::Optional && value.is_none() { + let mut default = match name { + "boot" => "0", + "lines" => "10", + _ => "true", + } + .into(); + if name != "case-sensitive" { + if let Some(next) = raw.get(i + 1).and_then(|s| s.to_str()) { + let valid = if name == "boot" { + next == "all" || parse_boot_descriptor(next).is_ok() + } else { + parse_lines_limit(Some(next)).is_ok() + }; + if valid { + i += 1; + default = raw[i].clone(); + } + } + } + value = Some(default); + } + let text = value.as_ref().and_then(|v| v.to_str()).unwrap_or(""); + match name { + "help" => { + Args::command().print_long_help()?; + println!(); + return Ok(None); + } + "version" => { + println!("journalctl (systemd-journal-sdk Rust rewrite)"); + println!("CLI compatibility: systemd v262 (8cc40e0c5e92)"); + println!("portable file-backed mode; PCRE2 {}", pcre::version()); + return Ok(None); + } + "introspect-cli" => { + print_introspection(&output)?; + return Ok(None); + } + "output" => { + if text == "help" { + print_output_mode_help(); + return Ok(None); + } + if !OUTPUT_MODE_HELP_LIST.contains(&text) { + return Err(anyhow!("Unknown output format '{text}'.")); + } + output = text.into(); + if matches!(text, "cat" | "export") || text.starts_with("json") { + quiet = true; + } + } + "facility" => { + for part in text.split(',').filter(|s| !s.is_empty()) { + if part == "help" { + print_facility_help(quiet); + return Ok(None); + } + parse_facility_filter(&[part.to_owned()])?; + } + } + "priority" => { + parse_priority_filter(&[text.to_owned()])?; + } + "lines" => { + parse_lines_limit(Some(text))?; + } + "boot" => { + if text != "all" { + parse_boot_descriptor(text)?; + } + } + "invocation" => { + parse_invocation_descriptor(text)?; + } + "case-sensitive" | "synchronize-on-exit" => { + parse_bool_option(&format!("--{name}"), text)?; + } + "since" | "until" => { + let parsed = timestamps + .parse(text) + .map_err(|_| anyhow!("Failed to parse timestamp: {text}"))?; + if name == "since" { + since_usec = Some(parsed); + } else { + until_usec = Some(parsed); + } + } + "vacuum-size" => { + parse_vacuum_size(text)?; + } + "vacuum-files" => { + parse_native_u64(text) + .ok_or_else(|| anyhow!("Failed to parse vacuum files: {text}"))?; + } + "vacuum-time" | "interval" => { + let duration = parse_duration_usec_allow_zero(text)?; + if name == "interval" && duration == 0 { + return Err(anyhow!("Failed to parse interval: {text}")); + } + } + "head" | "tail" => { + text.parse::() + .map_err(|_| anyhow!("--{name} must be a non-negative integer"))?; + } + "full" => full_width = true, + "no-full" => full_width = false, + "quiet" => quiet = true, + _ => {} + } + match name { + "field" => { + action = "field"; + field = value; + } + "fields" + | "list-boots" + | "list-invocations" + | "list-namespaces" + | "disk-usage" + | "header" + | "list-catalog" + | "dump-catalog" + | "update-catalog" + | "setup-keys" + | "new-id128" + | "verify" + | "verify-only" + | "sync" + | "relinquish-var" + | "smart-relinquish-var" + | "flush" => action = name, + "rotate" => { + action = if action == "vacuum" { + "rotate-and-vacuum" + } else { + "rotate" + } + } + "vacuum-size" | "vacuum-files" | "vacuum-time" => { + action = if action == "rotate" { + "rotate-and-vacuum" + } else { + "vacuum" + }; + tokens.push(option_token(name, value.as_deref().unwrap())); + } + "verify-key" => { + action = "verify"; + tokens.retain(|t| t != "--merge"); + tokens.push(option_token(name, value.as_deref().unwrap())); + } + "root" | "image" | "namespace" if value.as_deref().is_some_and(OsStr::is_empty) => { + let prefix = format!("--{name}="); + tokens.retain(|t| !t.as_encoded_bytes().starts_with(prefix.as_bytes())); + } + "this-boot" => tokens.push(option_token("boot", OsStr::new("0"))), + "I" => tokens.push(option_token("invocation", OsStr::new("0"))), + "output-fields" => projection.push(value.as_ref().unwrap().clone()), + _ if kind == Kind::None => tokens.push(format!("--{name}").into()), + _ => tokens.push(option_token(name, value.as_deref().unwrap())), + } + } + i += 1; + } + match action { + "show" | "vacuum" => {} + "field" => tokens.push(option_token("field", field.as_deref().unwrap())), + "rotate-and-vacuum" => tokens.push("--rotate".into()), + other => tokens.push(format!("--{other}").into()), + } + if quiet { + tokens.push("--quiet".into()); + } + if !full_width { + tokens.push("--no-full".into()); + } + tokens.push("--".into()); + tokens.extend(operands); + let mut args = Args::try_parse_from(tokens).map_err(|e| anyhow!(e.to_string()))?; + args.action = Some(action); + args.since_usec = since_usec; + args.until_usec = until_usec; + args.full = full_width; + args.no_full = !full_width; + args.output_fields = projection; + if args.no_tail { + args.lines = Some("all".into()); + args.head = None; + args.tail = None; + } + Ok(Some(args)) +} + +fn print_introspection(output: &str) -> Result<()> { + let options: serde_json::Value = + serde_json::from_slice(include_bytes!("../../../../go/cmd/journalctl/options.json"))?; + let value = serde_json::json!({"mediaType":"application/vnd.io.systemd.cli-introspection-0", + "commands":[{"names":["journalctl"],"project":"systemd-journal-sdk", "version":env!("CARGO_PKG_VERSION"), + "features":[format!("PCRE2 {}", pcre::version())], "abstract":["Query file-backed journals."], "options":options}]}); + let data = if output == "json-pretty" { + serde_json::to_string_pretty(&value)? + } else { + serde_json::to_string(&value)? + }; + match output { + "json-sse" => println!("data: {data}\n"), + "json-seq" => println!("\x1e{data}"), + _ => println!("{data}"), + } + Ok(()) +} diff --git a/rust/src/cmd/journalctl/pcre.rs b/rust/src/cmd/journalctl/pcre.rs new file mode 100644 index 00000000..7e14f319 --- /dev/null +++ b/rust/src/cmd/journalctl/pcre.rs @@ -0,0 +1,137 @@ +use anyhow::{Result, anyhow}; +use std::ffi::{c_int, c_void}; +use std::ptr::{NonNull, null_mut}; + +#[link(name = "pcre2-8")] +unsafe extern "C" { + fn pcre2_compile_8( + pattern: *const u8, + length: usize, + options: u32, + error: *mut c_int, + offset: *mut usize, + context: *mut c_void, + ) -> *mut c_void; + fn pcre2_code_free_8(code: *mut c_void); + fn pcre2_match_data_create_8(count: u32, context: *mut c_void) -> *mut c_void; + fn pcre2_match_data_free_8(data: *mut c_void); + fn pcre2_get_ovector_pointer_8(data: *mut c_void) -> *mut usize; + fn pcre2_match_8( + code: *const c_void, + subject: *const u8, + length: usize, + start: usize, + options: u32, + data: *mut c_void, + context: *mut c_void, + ) -> c_int; + fn pcre2_get_error_message_8(code: c_int, buffer: *mut u8, length: usize) -> c_int; + fn pcre2_config_8(what: u32, value: *mut c_void) -> c_int; +} + +// PCRE2's 8-bit ABI uses size_t even on platforms with a 32-bit C long. +#[derive(Debug)] +pub struct Pattern { + code: NonNull, + data: NonNull, +} + +fn error_message(code: c_int) -> String { + let mut buffer = [0u8; 512]; + // SAFETY: PCRE2 writes at most the supplied buffer length. + let n = unsafe { pcre2_get_error_message_8(code, buffer.as_mut_ptr(), buffer.len()) }; + if n < 0 { + return format!("PCRE2 error {code}"); + } + String::from_utf8_lossy(&buffer[..n as usize]).into_owned() +} + +pub fn version() -> String { + let mut buffer = [0u8; 128]; + // SAFETY: PCRE2_CONFIG_VERSION requires a byte buffer large enough for its version. + if unsafe { pcre2_config_8(11, buffer.as_mut_ptr().cast()) } < 0 { + return "unknown".into(); + } + let n = buffer.iter().position(|&b| b == 0).unwrap_or(buffer.len()); + String::from_utf8_lossy(&buffer[..n]).into_owned() +} + +impl Pattern { + pub fn compile(pattern: &[u8], sensitive: bool) -> Result { + let mut error = 0; + let mut offset = 0; + let flags = if sensitive { 0 } else { 0x0000_0008 }; // PCRE2_CASELESS + // SAFETY: the pattern is borrowed only for compilation; PCRE2 owns its result. + let code = NonNull::new(unsafe { + pcre2_compile_8( + pattern.as_ptr(), + pattern.len(), + flags, + &mut error, + &mut offset, + null_mut(), + ) + }) + .ok_or_else(|| anyhow!("Bad pattern: {} (offset {offset})", error_message(error)))?; + // SAFETY: a one-pair match vector suffices for a boolean result, including rc=0. + let data = NonNull::new(unsafe { pcre2_match_data_create_8(1, null_mut()) }); + let Some(data) = data else { + // SAFETY: this compiled code has not been transferred to another owner. + unsafe { pcre2_code_free_8(code.as_ptr()) }; + return Err(anyhow!("Failed to allocate PCRE2 match data")); + }; + Ok(Self { code, data }) + } + + pub fn matches(&self, subject: &[u8]) -> Result { + // SAFETY: this CLI-private handle is not shared across threads. Matching borrows + // subject bytes for the call; no subsequent operation reads PCRE2's subject pointer. + let rc = unsafe { + pcre2_match_8( + self.code.as_ptr(), + subject.as_ptr(), + subject.len(), + 0, + 0, + self.data.as_ptr(), + null_mut(), + ) + }; + if rc == -1 { + return Ok(false); + } // PCRE2_ERROR_NOMATCH + if rc < 0 { + return Err(anyhow!("Pattern matching failed: {}", error_message(rc))); + } + Ok(true) + } + + pub fn match_with_highlight( + &self, + subject: &[u8], + highlight: &mut super::colors::MatchRange, + ) -> Result { + if !self.matches(subject)? { + return Ok(false); + } + // Read the existing match before another subject can replace its offsets. + let offsets = unsafe { pcre2_get_ovector_pointer_8(self.data.as_ptr()) }; + *highlight = unsafe { + super::colors::MatchRange { + start: *offsets, + end: *offsets.add(1), + } + }; + Ok(true) + } +} + +impl Drop for Pattern { + fn drop(&mut self) { + // SAFETY: both handles are uniquely owned and freed exactly once. + unsafe { + pcre2_match_data_free_8(self.data.as_ptr()); + pcre2_code_free_8(self.code.as_ptr()); + } + } +} diff --git a/rust/src/cmd/journalctl/terminal.rs b/rust/src/cmd/journalctl/terminal.rs new file mode 100644 index 00000000..5a2252ab --- /dev/null +++ b/rust/src/cmd/journalctl/terminal.rs @@ -0,0 +1,98 @@ +#[cfg(unix)] +pub(super) fn stdout_columns() -> Option { + let mut size = std::mem::MaybeUninit::::zeroed(); + let result = unsafe { libc::ioctl(libc::STDOUT_FILENO, libc::TIOCGWINSZ, size.as_mut_ptr()) }; + if result < 0 { + return None; + } + let width = unsafe { size.assume_init() }.ws_col; + (width > 0).then_some(width as usize) +} + +#[cfg(windows)] +pub(super) fn stdout_columns() -> Option { + use std::ffi::c_void; + + #[repr(C)] + struct Coord { + x: i16, + y: i16, + } + + #[repr(C)] + struct Rect { + left: i16, + top: i16, + right: i16, + bottom: i16, + } + + #[repr(C)] + struct ConsoleInfo { + size: Coord, + cursor: Coord, + attributes: u16, + window: Rect, + maximum_window: Coord, + } + + #[link(name = "kernel32")] + unsafe extern "system" { + fn GetStdHandle(kind: u32) -> *mut c_void; + fn GetConsoleScreenBufferInfo(handle: *mut c_void, info: *mut ConsoleInfo) -> i32; + } + + let mut info = std::mem::MaybeUninit::::uninit(); + let result = + unsafe { GetConsoleScreenBufferInfo(GetStdHandle((-11i32) as u32), info.as_mut_ptr()) }; + if result == 0 { + return None; + } + let window = unsafe { info.assume_init() }.window; + let width = i32::from(window.right) - i32::from(window.left) + 1; + (width > 0).then_some(width as usize) +} + +#[cfg(not(any(unix, windows)))] +pub(super) fn stdout_columns() -> Option { + None +} +#[cfg(unix)] +pub(super) fn hostname() -> Option> { + let mut name = std::mem::MaybeUninit::::zeroed(); + if unsafe { libc::uname(name.as_mut_ptr()) } != 0 { + return None; + } + let name = unsafe { name.assume_init() }; + Some( + unsafe { std::ffi::CStr::from_ptr(name.nodename.as_ptr()) } + .to_bytes() + .to_vec(), + ) +} + +#[cfg(windows)] +pub(super) fn hostname() -> Option> { + #[link(name = "kernel32")] + unsafe extern "system" { + fn GetComputerNameExW(format: u32, buffer: *mut u16, size: *mut u32) -> i32; + fn GetLastError() -> u32; + } + // Use the physical DNS hostname, matching Go's os.Hostname on Windows. + let mut buffer = vec![0u16; 64]; + loop { + let mut size = u32::try_from(buffer.len()).ok()?; + if unsafe { GetComputerNameExW(5, buffer.as_mut_ptr(), &mut size) } != 0 { + return Some(String::from_utf16_lossy(&buffer[..size as usize]).into_bytes()); + } + if unsafe { GetLastError() } != 234 || size as usize <= buffer.len() { + return None; + } + buffer.resize(size as usize, 0); + } +} + +#[cfg(not(any(unix, windows)))] +pub(super) fn hostname() -> Option> { + None +} diff --git a/rust/src/cmd/journalctl/tests/cli_subprocess.rs b/rust/src/cmd/journalctl/tests/cli_subprocess.rs index 6d6cc917..265da076 100644 --- a/rust/src/cmd/journalctl/tests/cli_subprocess.rs +++ b/rust/src/cmd/journalctl/tests/cli_subprocess.rs @@ -155,7 +155,7 @@ fn version_prints_baseline_metadata() { String::from_utf8_lossy(&out.stderr) ); assert!( - stdout.contains("v260.1") && stdout.contains("baseline"), + stdout.contains("v262") && stdout.contains("CLI compatibility"), "expected version banner, got: {stdout}" ); } diff --git a/rust/src/cmd/journalctl/timestamp.rs b/rust/src/cmd/journalctl/timestamp.rs new file mode 100644 index 00000000..d6892d2f --- /dev/null +++ b/rust/src/cmd/journalctl/timestamp.rs @@ -0,0 +1,549 @@ +use crate::timestamp_zone::Zone; +use anyhow::{Result, anyhow}; +use chrono::{Datelike, NaiveDate, NaiveDateTime, Timelike}; + +const MAX: u64 = 253_402_214_399_000_000; + +#[derive(Default)] +pub struct Parser { + local_guess: i64, + utc_guess: i64, + local: Option, +} + +pub fn duration(value: &str) -> Result { + let error = || anyhow!("failed to parse duration: {value}"); + let units = [ + ("seconds", 1_000_000), + ("second", 1_000_000), + ("sec", 1_000_000), + ("s", 1_000_000), + ("minutes", 60_000_000), + ("minute", 60_000_000), + ("min", 60_000_000), + ("months", 2_629_800_000_000), + ("month", 2_629_800_000_000), + ("M", 2_629_800_000_000), + ("msec", 1_000), + ("ms", 1_000), + ("m", 60_000_000), + ("hours", 3_600_000_000), + ("hour", 3_600_000_000), + ("hr", 3_600_000_000), + ("h", 3_600_000_000), + ("days", 86_400_000_000), + ("day", 86_400_000_000), + ("d", 86_400_000_000), + ("weeks", 604_800_000_000), + ("week", 604_800_000_000), + ("w", 604_800_000_000), + ("years", 31_557_600_000_000), + ("year", 31_557_600_000_000), + ("y", 31_557_600_000_000), + ("usec", 1), + ("us", 1), + ("μs", 1), + ("µs", 1), + ]; + let mut rest = value.trim_start_matches(ascii_space); + if rest.trim_end_matches(ascii_space) == "infinity" { + return Ok(u64::MAX); + } + let mut total = 0u64; + let mut seen = false; + loop { + rest = rest.trim_start_matches(ascii_space); + if rest.is_empty() { + return if seen { Ok(total) } else { Err(error()) }; + } + if rest.starts_with('-') { + return Err(error()); + } + let start = usize::from(rest.starts_with('+')); + let end = start + + rest.as_bytes()[start..] + .iter() + .take_while(|b| b.is_ascii_digit()) + .count(); + let integer = if end > start { + rest[start..end].parse::().map_err(|_| error())? as u64 + } else if start == 0 && rest.starts_with('.') { + 0 + } else { + return Err(error()); + }; + let mut n = end; + let mut fraction = ""; + if rest[n..].starts_with('.') { + n += 1; + let end = n + rest.as_bytes()[n..] + .iter() + .take_while(|b| b.is_ascii_digit()) + .count(); + if end == n { + return Err(error()); + } + fraction = &rest[n..end]; + n = end; + } + let after = &rest[n..]; + let mut suffix = after.trim_start_matches(ascii_space); + let mut multiplier = 1_000_000u64; + let mut matched = false; + for (name, unit) in units { + if let Some(next) = suffix.strip_prefix(name) { + multiplier = unit; + suffix = next; + matched = true; + break; + } + } + if !matched && after.len() == suffix.len() && !suffix.is_empty() { + return Err(error()); + } + if integer >= u64::MAX / multiplier { + return Err(error()); + } + total = total + .checked_add(integer * multiplier) + .filter(|v| *v < u64::MAX) + .ok_or_else(error)?; + let mut weight = multiplier / 10; + for digit in fraction.bytes() { + total = total + .checked_add(u64::from(digit - b'0') * weight) + .filter(|v| *v < u64::MAX) + .ok_or_else(error)?; + weight /= 10; + } + rest = suffix; + seen = true; + } +} + +fn ascii_space(c: char) -> bool { + matches!(c, ' ' | '\t' | '\r' | '\n' | '\u{b}' | '\u{c}') +} + +fn utc_offset(value: &str) -> Option { + if value == "Z" { + return Some(0); + } + if !matches!(value.len(), 3 | 5 | 6) || !matches!(value.as_bytes()[0], b'+' | b'-') { + return None; + } + let pair = |s: &[u8]| -> Option { + (s.len() == 2 && s.iter().all(u8::is_ascii_digit)) + .then(|| i64::from(s[0] - b'0') * 10 + i64::from(s[1] - b'0')) + }; + let bytes = value.as_bytes(); + let hour = pair(&bytes[1..3])?; + let minute = if bytes.len() == 3 { + 0 + } else if bytes.len() == 6 { + if bytes[3] != b':' { + return None; + } + pair(&bytes[4..])? + } else { + pair(&bytes[3..])? + }; + let offset = hour * 3600 + minute * 60; + if minute >= 60 || offset > 86400 { + return None; + } + Some(if bytes[0] == b'-' { -offset } else { offset }) +} + +impl Parser { + pub fn parse(&mut self, value: &str) -> Result { + if let Some(epoch) = value.strip_prefix('@') { + return duration(epoch); + } + let mut local = self.local.take().unwrap_or_else(Zone::local); + let result = self.parse_calendar_value(value, &mut local); + self.local = Some(local); + result + } + + fn parse_calendar_value(&mut self, value: &str, local: &mut Zone) -> Result { + if value.len() > 2 && value.ends_with('Z') { + if let Ok(result) = + self.parse_in_zone(value, &value[..value.len() - 1], local, true, -1, 0, true) + { + return Ok(result); + } + } + let bytes = value.as_bytes(); + if value.len() > 7 + && bytes[value.len() - 7] != b' ' + && matches!(bytes[value.len() - 6], b'+' | b'-') + { + if let Some(offset) = value.get(value.len() - 6..).and_then(utc_offset) { + return self.parse_in_zone( + value, + &value[..value.len() - 6], + local, + true, + -1, + offset, + true, + ); + } + } + if let Some((prefix, suffix)) = value.rsplit_once(' ') { + if suffix == "UTC" { + return self.parse_in_zone(value, prefix, local, true, -1, 0, true); + } + if let Some(offset) = utc_offset(suffix) { + return self.parse_in_zone(value, prefix, local, true, -1, offset, true); + } + if let Some(dst) = local.abbreviation(suffix) { + return self.parse_in_zone(value, prefix, local, false, dst, 0, true); + } + if let Some(mut named) = Zone::named(suffix) { + let result = self.parse_in_zone(value, prefix, &mut named, false, -1, 0, true); + local.restore_after_named(&named); + return result; + } + } + self.parse_in_zone(value, value, local, false, -1, 0, false) + } + + #[allow(clippy::too_many_arguments)] + fn parse_in_zone( + &mut self, + value: &str, + mut text: &str, + zone: &mut Zone, + utc: bool, + isdst: i32, + offset: i64, + with_zone: bool, + ) -> Result { + let error = || anyhow!("failed to parse timestamp: {value}"); + let now = chrono::Utc::now(); + if !with_zone { + if value == "now" { + return u64::try_from(now.timestamp_micros()).map_err(|_| error()); + } + let relative = if let Some(d) = value.strip_prefix('+') { + Some((d, false)) + } else if let Some(d) = value.strip_prefix('-') { + Some((d, true)) + } else if let Some(d) = value.strip_suffix(" ago") { + Some((d, true)) + } else { + value.strip_suffix(" left").map(|d| (d, false)) + }; + if let Some((d, negative)) = relative { + let delta = duration(d)?; + let base = u64::try_from(now.timestamp_micros()).map_err(|_| error())?; + let result = if negative { + base.checked_sub(delta) + } else { + base.checked_add(delta) + } + .filter(|v| *v <= MAX) + .ok_or_else(error)?; + return Ok(result); + } + } + let (seed, _, _) = zone.civil(now.timestamp(), utc)?; + let mut weekday = None; + let (civil, second, fraction) = if matches!(text, "today" | "yesterday" | "tomorrow") { + let day = seed.date().and_hms_opt(0, 0, 0).ok_or_else(error)?; + let shift = match text { + "yesterday" => -1, + "tomorrow" => 1, + _ => 0, + }; + (day + chrono::Duration::days(shift), 0, 0) + } else { + for (i, name) in [ + "Sunday", + "Monday", + "Tuesday", + "Wednesday", + "Thursday", + "Friday", + "Saturday", + ] + .iter() + .enumerate() + { + for prefix in [*name, &name[..3]] { + if text + .get(..prefix.len()) + .is_some_and(|s| s.eq_ignore_ascii_case(prefix)) + && text.as_bytes().get(prefix.len()) == Some(&b' ') + { + weekday = Some(i as u32); + text = &text[prefix.len() + 1..]; + break; + } + } + if weekday.is_some() { + break; + } + } + calendar(text, seed).ok_or_else(error)? + }; + if weekday.is_some_and(|w| w != civil.weekday().num_days_from_sunday()) { + return Err(error()); + } + let guess = if utc { + &mut self.utc_guess + } else { + &mut self.local_guess + }; + let seconds = inverse(civil, zone, utc, isdst, guess)? + i64::from(second) + - i64::from(civil.second()) + - offset; + let micros = u64::try_from(seconds) + .map_err(|_| error())? + .checked_mul(1_000_000) + .and_then(|v| v.checked_add(fraction)) + .filter(|v| *v <= MAX) + .ok_or_else(error)?; + Ok(micros) + } +} + +fn calendar(value: &str, seed: NaiveDateTime) -> Option<(NaiveDateTime, u32, u64)> { + let formats = [ + "%y-%m-%d %H:%M:%S", + "%Y-%m-%d %H:%M:%S", + "%Y-%m-%dT%H:%M:%S", + "%b %d %H:%M:%S", + "%y-%m-%d %H:%M", + "%Y-%m-%d %H:%M", + "%Y-%m-%dT%H:%M", + "%y-%m-%d", + "%Y-%m-%d", + "%H:%M:%S", + "%H:%M", + ]; + for format in formats { + let (mut year, mut month, mut day) = (seed.year(), seed.month(), seed.day()); + let (mut hour, mut minute, mut second) = if format.contains("%H") { + ( + seed.hour(), + seed.minute(), + if format.contains("%S") { + seed.second() + } else { + 0 + }, + ) + } else { + (0, 0, 0) + }; + let mut text = value; + let mut valid = true; + let mut chars = format.chars(); + while let Some(c) = chars.next() { + if c == ' ' { + text = text.trim_start_matches(ascii_space); + continue; + } + if c != '%' { + if let Some(next) = text.strip_prefix(c) { + text = next; + } else { + valid = false; + break; + } + continue; + } + let code = chars.next()?; + if code == 'b' { + let mut found = false; + for (m, name) in [ + "January", + "February", + "March", + "April", + "May", + "June", + "July", + "August", + "September", + "October", + "November", + "December", + ] + .iter() + .enumerate() + { + for prefix in [*name, &name[..3]] { + if text + .get(..prefix.len()) + .is_some_and(|s| s.eq_ignore_ascii_case(prefix)) + { + month = m as u32 + 1; + text = &text[prefix.len()..]; + found = true; + break; + } + } + if found { + break; + } + } + if !found { + valid = false; + break; + } + continue; + } + text = text.trim_start_matches(ascii_space); + let width = if code == 'Y' { 4 } else { 2 }; + let n = text + .bytes() + .take(width) + .take_while(u8::is_ascii_digit) + .count(); + if n == 0 { + valid = false; + break; + } + let number: u32 = text[..n].parse().ok()?; + text = &text[n..]; + match code { + 'y' => { + year = if number >= 69 { + 1900 + number as i32 + } else { + 2000 + number as i32 + } + } + 'Y' => year = number as i32, + 'm' => { + month = number; + valid = (1..=12).contains(&number); + } + 'd' => { + day = number; + valid = (1..=31).contains(&number); + } + 'H' => { + hour = number; + valid = number <= 23; + } + 'M' => { + minute = number; + valid = number <= 59; + } + 'S' => { + second = number; + valid = number <= 61; + } + _ => unreachable!(), + } + if !valid { + break; + } + } + let mut fraction = 0; + if valid && format.contains("%S") && text.starts_with('.') { + text = &text[1..]; + if text.is_empty() { + valid = false; + } + for (i, c) in text.bytes().enumerate() { + if !c.is_ascii_digit() { + valid = false; + break; + } + if i < 6 { + fraction = fraction * 10 + u64::from(c - b'0'); + } else if i == 6 && c >= b'5' { + fraction += 1; + } + } + if text.len() < 6 { + for _ in text.len()..6 { + fraction *= 10; + } + } + text = ""; + } + if valid && text.is_empty() { + let date = NaiveDate::from_ymd_opt(year, month, 1)? + .checked_add_signed(chrono::Duration::days(i64::from(day) - 1))?; + let civil = date.and_hms_opt(hour, minute, second.min(59))?; + return Some((civil, second, fraction)); + } + } + None +} + +// Cached UTC-to-civil inversion follows glibc's fixed-point/oscillation policy. +fn inverse( + civil: NaiveDateTime, + zone: &mut Zone, + utc: bool, + isdst: i32, + guess: &mut i64, +) -> Result { + let target = civil.and_utc().timestamp(); + let mut t = target + *guess; + let (mut t1, mut t2) = (t, t); + let mut previous_dst = -1; + let mut matched = false; + let mut oscillated = false; + for _ in 0..6 { + let (actual, dst, _) = zone.civil(t, utc)?; + let diff = target - actual.and_utc().timestamp(); + if diff == 0 { + matched = true; + break; + } + if t == t1 + && t != t2 + && (dst < 0 || isdst < 0 && previous_dst <= dst || isdst >= 0 && dst != isdst) + { + matched = true; + oscillated = true; + break; + } + t1 = t2; + t2 = t; + previous_dst = dst; + t += diff; + } + if !matched { + return Err(anyhow!("timestamp conversion did not converge")); + } + let (_, dst, _) = zone.civil(t, utc)?; + if isdst >= 0 && dst != isdst && !oscillated { + let mut found = false; + let mut delta = 601_200; + while delta < 457_243_209 / 2 + 601_200 && !found { + for direction in [-1, 1] { + let candidate = t + direction * delta; + let Ok((actual, other_dst, _)) = zone.civil(candidate, utc) else { + continue; + }; + if other_dst == isdst { + t = candidate + target - actual.and_utc().timestamp(); + found = true; + break; + } + } + delta += 601_200; + } + if !found { + t += i64::from(dst - isdst) * 3600; + } + zone.civil(t, utc)?; + } + *guess = t - target; + // Distinguish an inserted second from the following minute's zero second. + if civil.second() == 0 && zone.leap_hit(t) && zone.civil(t, utc)?.0.second() == 0 { + t += 1; + zone.civil(t, utc)?; + } + Ok(t) +} diff --git a/rust/src/cmd/journalctl/timestamp_zone.rs b/rust/src/cmd/journalctl/timestamp_zone.rs new file mode 100644 index 00000000..4e65d61f --- /dev/null +++ b/rust/src/cmd/journalctl/timestamp_zone.rs @@ -0,0 +1,725 @@ +use anyhow::{Result, anyhow}; +use chrono::{Datelike, NaiveDateTime, Timelike}; +use jiff::tz::TimeZone; +use std::{ + env, + fs::File, + io::Read, + path::{Path, PathBuf}, +}; + +struct Kind { + offset: i64, + dst: usize, + name: String, + standard: bool, + universal: bool, +} +struct Transition { + at: i64, + kind: usize, +} +struct Leap { + at: i64, + correction: i64, +} +struct Tzif { + kinds: Vec, + transitions: Vec, + leaps: Vec, + first_name: String, + footer: [String; 2], + footer_rule: String, +} +pub struct Zone { + provider: TimeZone, + data: Option, + names: [String; 2], + initial_names: [String; 2], + file: Option<(PathBuf, std::fs::Metadata)>, +} + +fn zone_dirs() -> Vec { + if let Some(dir) = env::var_os("TZDIR").filter(|d| !d.is_empty()) { + return vec![dir.into()]; + } + [ + "/usr/share/zoneinfo", + "/usr/share/lib/zoneinfo", + "/usr/lib/locale/TZ", + ] + .map(PathBuf::from) + .to_vec() +} + +impl Zone { + fn read(path: &Path) -> Option { + let mut bytes = Vec::new(); + let file = File::open(path).ok()?; + let info = file.metadata().ok()?; + if !info.is_file() { + return None; + } + file.take((10 << 20) + 1).read_to_end(&mut bytes).ok()?; + if bytes.len() > 10 << 20 { + return None; + } + let provider = TimeZone::tzif("local", &bytes).ok()?; + let data = Tzif::parse(&bytes)?; + let mut names = [String::new(), String::new()]; + for t in data.transitions.iter().rev() { + let k = &data.kinds[t.kind]; + if names[k.dst].is_empty() { + names[k.dst] = k.name.clone(); + } + } + if names[0].is_empty() { + names[0] = data.first_name.clone(); + } + if names[1].is_empty() { + names[1] = names[0].clone(); + } + Some(Self { + provider, + data: Some(data), + initial_names: names.clone(), + names, + file: Some((path.canonicalize().ok()?, info)), + }) + } + + pub fn local() -> Self { + if let Some(value) = env::var_os("TZ") { + let value = value.to_string_lossy(); + if value.is_empty() { + return Self::fallback("Universal".to_owned(), 0); + } + let value = value.strip_prefix(':').unwrap_or(&value); + if Path::new(value).is_absolute() { + if let Some(z) = Self::read(Path::new(value)) { + return z; + } + } + if !value.is_empty() { + for dir in zone_dirs() { + if let Some(z) = Self::read(&dir.join(value)) { + return z; + } + } + } + if !value.is_empty() { + if let Some(zone) = Self::default_rules(value) { + return zone; + } + if let Ok(provider) = TimeZone::posix(&normalize_rule(value)) { + return Self { + provider, + data: None, + names: Default::default(), + initial_names: Default::default(), + file: None, + }; + } + if zone_dirs().iter().any(|p| p.is_dir()) || TimeZone::get(value).is_err() { + let (name, offset) = fallback_rule(value); + return Self::fallback(name, offset); + } + } + } else if let Some(z) = Self::read(Path::new("/etc/localtime")) { + return z; + } + Self { + provider: TimeZone::system(), + data: None, + names: Default::default(), + initial_names: Default::default(), + file: None, + } + } + + // Default-file transitions and names are prepared once, never per row. + fn default_rules(rule: &str) -> Option { + let normalized = normalize_rule(rule); + let rule = normalized.strip_suffix(',').unwrap_or(&normalized); + if rule.contains(',') { + return None; + } + let (standard, _) = fallback_rule(rule); + if standard.is_empty() { + return None; + } + let end = standard.len() + if rule.starts_with('<') { 2 } else { 0 }; + let rest = rule[end..] + .trim_start_matches(|c: char| c.is_ascii_digit() || matches!(c, '+' | '-' | ':')); + if fallback_rule(rest).0.is_empty() { + return None; + } + let provider = TimeZone::posix(&format!("{rule},M3.2.0,M11.1.0")).ok()?; + let names = footer_names(rule); + let mut offsets = [0; 2]; + for (i, seconds) in [1705276800, 1721001600].into_iter().enumerate() { + offsets[i] = i64::from( + provider + .to_offset_info(jiff::Timestamp::from_second(seconds).ok()?) + .offset() + .seconds(), + ); + } + for dir in zone_dirs() { + let Some(mut z) = Self::read(&dir.join("posixrules")) else { + continue; + }; + let meta = z.data.as_mut()?; + if meta.kinds.len() < 2 { + continue; + } + // Native default-file conversion starts with zero remembered DST offset. + let mut old_offsets = [meta.kinds[0].offset, 0]; + for transition in meta.transitions.iter().rev() { + let kind = &meta.kinds[transition.kind]; + if kind.dst == 0 { + old_offsets[0] = kind.offset; + break; + } + } + let mut previous_dst = 0; + for transition in &mut meta.transitions { + let kind = &meta.kinds[transition.kind]; + let shift_kind = usize::from(previous_dst != 0 && !kind.standard); + if !kind.universal { + transition.at = transition + .at + .checked_add(offsets[shift_kind] - old_offsets[shift_kind])?; + } + transition.kind = kind.dst; + previous_dst = kind.dst; + } + meta.kinds = (0..2) + .map(|dst| Kind { + offset: offsets[dst], + dst, + name: names[dst].clone(), + standard: false, + universal: false, + }) + .collect(); + meta.first_name = names[0].clone(); + if !meta.footer[0].is_empty() { + meta.footer = names.clone(); + if let Ok(footer) = TimeZone::posix(&meta.footer_rule) { + z.provider = footer; + } + } + z.names = names.clone(); + z.initial_names = names.clone(); + z.file = None; + return Some(z); + } + Some(Self { + provider, + data: None, + names: names.clone(), + initial_names: names, + file: None, + }) + } + + fn fallback(name: String, offset: i64) -> Self { + Self { + provider: TimeZone::UTC, + data: Some(Tzif { + kinds: vec![Kind { + offset, + dst: 0, + name: name.clone(), + standard: false, + universal: false, + }], + transitions: Vec::new(), + leaps: Vec::new(), + first_name: name.clone(), + footer: Default::default(), + footer_rule: String::new(), + }), + names: [name.clone(), name.clone()], + initial_names: [name.clone(), name], + file: None, + } + } + + pub fn named(name: &str) -> Option { + if name.is_empty() + || name.len() >= 4096 + || name.starts_with('/') + || name.ends_with('/') + || name.contains("//") + || !name + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b"-_+/".contains(&b)) + { + return None; + } + let mut installed = false; + for dir in zone_dirs() { + installed |= dir.is_dir(); + if let Some(z) = Self::read(&dir.join(name)) { + return Some(z); + } + } + if installed { + return None; + } + let provider = TimeZone::get(name).ok()?; + if provider.iana_name() != Some(name) { + return None; + } + Some(Self { + provider, + data: None, + names: Default::default(), + initial_names: Default::default(), + file: None, + }) + } + + pub fn restore_after_named(&mut self, named: &Self) { + if self.data.is_none() { + return; + } + let same = match (&self.file, &named.file) { + (Some((a, am)), Some((b, bm))) => { + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + let _ = (a, b); + am.dev() == bm.dev() && am.ino() == bm.ino() && am.mtime() == bm.mtime() + } + #[cfg(not(unix))] + { + a == b && am.modified().ok() == bm.modified().ok() + } + } + _ => false, + }; + self.names = if same { + named.names.clone() + } else { + self.initial_names.clone() + }; + } + + pub fn abbreviation(&self, name: &str) -> Option { + if self.data.is_some() { + return self.names.iter().position(|n| n == name).map(|n| n as i32); + } + let year = chrono::Utc::now().year(); + for month in [1, 7] { + let seconds = chrono::NaiveDate::from_ymd_opt(year, month, 1)? + .and_hms_opt(12, 0, 0)? + .and_utc() + .timestamp(); + let info = self + .provider + .to_offset_info(jiff::Timestamp::from_second(seconds).ok()?); + if info.abbreviation() == name { + return Some(i32::from(info.dst().is_dst())); + } + } + None + } + + // Native calendar conversion retains TZif leaps and ignores a footer before + // the first transition; neither behavior is supplied by the zone provider. + pub fn civil(&mut self, seconds: i64, utc: bool) -> Result<(NaiveDateTime, i32, i64)> { + let table_kind = self.data.as_ref().and_then(|data| { + let i = data.transitions.partition_point(|t| t.at <= seconds); + if i == 0 { + Some(data.kinds.iter().position(|k| k.dst == 0).unwrap_or(0)) + } else if i == data.transitions.len() && !data.footer[0].is_empty() { + None + } else { + Some(data.transitions[i - 1].kind) + } + }); + let (mut offset, mut dst) = if utc { + (0, 0) + } else if let (Some(data), Some(kind)) = (&self.data, table_kind) { + (data.kinds[kind].offset, data.kinds[kind].dst as i32) + } else { + let timestamp = jiff::Timestamp::from_second(seconds)?; + let info = self.provider.to_offset_info(timestamp); + ( + i64::from(info.offset().seconds()), + i32::from(info.dst().is_dst()), + ) + }; + if let Some(data) = &self.data { + if !utc { + let i = data.transitions.partition_point(|t| t.at <= seconds); + if let Some(kind) = table_kind { + self.names = Default::default(); + let k = &data.kinds[kind]; + if i == 0 { + self.names[0] = k.name.clone(); + if let Some(k) = data.kinds.iter().find(|k| k.dst == 1) { + self.names[1] = k.name.clone(); + } + } else { + self.names[k.dst] = k.name.clone(); + for t in &data.transitions[i..] { + let k = &data.kinds[t.kind]; + if self.names[k.dst].is_empty() { + self.names[k.dst] = k.name.clone(); + } + } + } + if self.names[0].is_empty() { + self.names[0] = self.names[1].clone(); + } + if self.names[1].is_empty() { + self.names[1] = self.names[0].clone(); + } + offset = k.offset; + dst = k.dst as i32; + } else { + self.names = data.footer.clone(); + } + } + let i = data.leaps.partition_point(|l| l.at <= seconds); + if i > 0 { + let leap = &data.leaps[i - 1]; + offset -= leap.correction; + if leap.at == seconds + && (i == 1 && leap.correction > 0 + || i > 1 && leap.correction > data.leaps[i - 2].correction) + { + offset += 1; + let mut j = i - 1; + while j > 0 + && data.leaps[j].at == data.leaps[j - 1].at + 1 + && data.leaps[j].correction == data.leaps[j - 1].correction + 1 + { + offset += 1; + j -= 1; + } + } + } + } + let civil = chrono::DateTime::from_timestamp(seconds + offset, 0) + .ok_or_else(|| anyhow!("timestamp outside calendar range"))? + .naive_utc(); + Ok((civil, dst, offset)) + } + + // Rendering borrows names instead of updating the parser's abbreviation state. + pub fn format_display( + &self, + seconds: i64, + nanos: u32, + utc: bool, + pattern: &str, + named: bool, + numeric: bool, + ) -> Result { + let provider; + let (name, offset) = if utc { + ("UTC", 0) + } else if let Some(kind) = self.data.as_ref().and_then(|data| { + let i = data.transitions.partition_point(|t| t.at <= seconds); + if i == 0 { + Some(&data.kinds[data.kinds.iter().position(|k| k.dst == 0).unwrap_or(0)]) + } else if i < data.transitions.len() || data.footer[0].is_empty() { + Some(&data.kinds[data.transitions[i - 1].kind]) + } else { + None + } + }) { + (kind.name.as_str(), kind.offset) + } else { + provider = self + .provider + .to_offset_info(jiff::Timestamp::from_second(seconds)?); + ( + provider.abbreviation(), + i64::from(provider.offset().seconds()), + ) + }; + let (correction, leap) = self.data.as_ref().map_or((0, false), |data| { + let i = data.leaps.partition_point(|l| l.at <= seconds); + if i == 0 { + return (0, false); + } + let leap = &data.leaps[i - 1]; + ( + leap.correction, + leap.at == seconds + && (i == 1 && leap.correction > 0 + || i > 1 && leap.correction > data.leaps[i - 2].correction), + ) + }); + let dt = chrono::DateTime::from_timestamp(seconds + offset - correction, 0) + .ok_or_else(|| anyhow!("invalid realtime timestamp"))? + .naive_utc(); + let dt = dt + .with_nanosecond(nanos + if leap { 1_000_000_000 } else { 0 }) + .ok_or_else(|| anyhow!("invalid leap timestamp"))?; + let mut text = dt.format(pattern).to_string(); + if named && !name.is_empty() { + text.push(' '); + text.push_str(name); + } + if numeric { + use std::fmt::Write; + let sign = if offset < 0 { '-' } else { '+' }; + let offset = offset.abs(); + write!(text, "{sign}{:02}:{:02}", offset / 3600, offset % 3600 / 60)?; + } + Ok(text) + } + + pub fn leap_hit(&self, seconds: i64) -> bool { + let Some(data) = &self.data else { + return false; + }; + let i = data.leaps.partition_point(|l| l.at < seconds); + i < data.leaps.len() + && data.leaps[i].at == seconds + && (i == 0 && data.leaps[i].correction > 0 + || i > 0 && data.leaps[i].correction > data.leaps[i - 1].correction) + } +} + +impl Tzif { + fn parse(bytes: &[u8]) -> Option { + fn counts(b: &[u8]) -> Option<[u64; 6]> { + if b.len() < 44 || &b[..4] != b"TZif" { + return None; + } + let mut n = [0; 6]; + for (i, v) in n.iter_mut().enumerate() { + *v = u64::from(u32::from_be_bytes( + b[20 + i * 4..24 + i * 4].try_into().ok()?, + )); + } + Some(n) + } + fn size(n: [u64; 6], w: usize) -> u64 { + n[3] * (w as u64 + 1) + n[4] * 6 + n[5] + n[2] * (w as u64 + 4) + n[0] + n[1] + } + fn signed(b: &[u8], width: usize) -> Option { + Some(if width == 4 { + i64::from(i32::from_be_bytes(b.get(..4)?.try_into().ok()?)) + } else { + i64::from_be_bytes(b.get(..8)?.try_into().ok()?) + }) + } + let mut bytes = bytes; + let mut n = counts(bytes)?; + let mut width = 4; + if bytes[4] != 0 { + let skip = usize::try_from(44 + size(n, 4)).ok()?; + bytes = bytes.get(skip..)?; + n = counts(bytes)?; + width = 8; + } + let size = usize::try_from(size(n, width)).ok()?; + if size > bytes.len() - 44 || n[4] == 0 || n[4] > 256 || n[0] > n[4] || n[1] > n[4] { + return None; + } + let block = &bytes[44..44 + size]; + let mut at = 0; + let mut transitions: Vec = Vec::new(); + for _ in 0..n[3] { + let epoch = signed(&block[at..], width)?; + at += width; + if transitions.last().is_some_and(|t| epoch <= t.at) { + return None; + } + transitions.push(Transition { at: epoch, kind: 0 }); + } + for t in &mut transitions { + t.kind = usize::from(block[at]); + at += 1; + if t.kind as u64 >= n[4] { + return None; + } + } + let kinds_bytes = &block[at..at + n[4] as usize * 6]; + at += kinds_bytes.len(); + let names = &block[at..at + n[5] as usize]; + at += names.len(); + let name_at = |i: usize| -> Option { + let b = names.get(i..)?; + let end = b.iter().position(|b| *b == 0)?; + Some(std::str::from_utf8(&b[..end]).ok()?.to_owned()) + }; + let first_name = name_at(0)?; + let mut kinds = Vec::new(); + for b in kinds_bytes.chunks_exact(6) { + if b[4] > 1 { + return None; + } + kinds.push(Kind { + offset: i64::from(i32::from_be_bytes(b[..4].try_into().ok()?)), + dst: usize::from(b[4]), + name: name_at(usize::from(b[5]))?, + standard: false, + universal: false, + }); + } + let mut leaps: Vec = Vec::new(); + for _ in 0..n[2] { + let epoch = signed(&block[at..], width)?; + at += width; + let correction = i64::from(i32::from_be_bytes(block[at..at + 4].try_into().ok()?)); + at += 4; + if leaps.last().is_some_and(|l| epoch <= l.at) { + return None; + } + leaps.push(Leap { + at: epoch, + correction, + }); + } + for kind in kinds.iter_mut().take(n[1] as usize) { + kind.standard = block[at] != 0; + at += 1; + } + for kind in kinds.iter_mut().take(n[0] as usize) { + kind.universal = block[at] != 0; + at += 1; + } + let footer_rule = if width == 8 { + std::str::from_utf8(&bytes[44 + size..]) + .ok()? + .trim_matches('\n') + .to_owned() + } else { + String::new() + }; + let footer = footer_names(&footer_rule); + Some(Self { + kinds, + transitions, + leaps, + first_name, + footer, + footer_rule, + }) + } +} + +fn footer_names(mut text: &str) -> [String; 2] { + fn name(text: &mut &str) -> String { + if text.starts_with('<') { + if let Some(i) = text.find('>').filter(|i| *i > 1) { + let n = text[1..i].to_owned(); + *text = &text[i + 1..]; + return n; + } + return String::new(); + } + let i = text.bytes().take_while(u8::is_ascii_alphabetic).count(); + let n = text[..i].to_owned(); + *text = &text[i..]; + n + } + let standard = name(&mut text); + text = text.trim_start_matches([ + '+', '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', + ]); + let daylight = name(&mut text); + [ + standard.clone(), + if daylight.is_empty() { + standard + } else { + daylight + }, + ] +} + +// A parsed standard name/offset survives a malformed remainder of a TZ rule. +fn fallback_rule(rule: &str) -> (String, i64) { + let bytes = rule.as_bytes(); + let (name, mut rest) = if bytes.first() == Some(&b'<') { + let Some(end) = bytes.iter().position(|&b| b == b'>') else { + return (String::new(), 0); + }; + let name = &bytes[1..end]; + if name.len() < 3 + || !name + .iter() + .all(|b| b.is_ascii_alphanumeric() || *b == b'+' || *b == b'-') + { + return (String::new(), 0); + } + (&rule[1..end], &rule[end + 1..]) + } else { + let end = bytes.iter().take_while(|b| b.is_ascii_alphabetic()).count(); + if end < 3 { + return (String::new(), 0); + } + (&rule[..end], &rule[end..]) + }; + let sign = if rest.starts_with('-') { + rest = &rest[1..]; + 1 + } else { + if rest.starts_with('+') { + rest = &rest[1..]; + } + -1 + }; + let mut values = [0; 3]; + for (i, value) in values.iter_mut().enumerate() { + let end = rest.bytes().take_while(u8::is_ascii_digit).count(); + if end == 0 { + break; + } + let parsed = rest[..end].parse::().unwrap_or(u64::MAX) as u16; + *value = i64::from(parsed.min(if i == 0 { 24 } else { 59 })); + rest = &rest[end..]; + if !rest.starts_with(':') { + break; + } + rest = &rest[1..]; + } + ( + name.to_owned(), + sign * (values[0] * 3600 + values[1] * 60 + values[2]), + ) +} + +fn normalize_rule(mut rule: &str) -> String { + let mut result = String::with_capacity(rule.len()); + for _ in 0..2 { + let (name, _) = fallback_rule(rule); + if name.is_empty() { + break; + } + let end = name.len() + if rule.starts_with('<') { 2 } else { 0 }; + result.push_str(&rule[..end]); + rule = &rule[end..]; + if rule.starts_with(['+', '-']) { + result.push_str(&rule[..1]); + rule = &rule[1..]; + } + for part in 0..3 { + let end = rule.bytes().take_while(u8::is_ascii_digit).count(); + if end == 0 { + break; + } + let value = rule[..end].parse::().unwrap_or(u64::MAX) as u16; + result.push_str(&value.min(if part == 0 { 24 } else { 59 }).to_string()); + rule = &rule[end..]; + if part == 2 || !rule.starts_with(':') { + break; + } + result.push(':'); + rule = &rule[1..]; + } + } + result.push_str(rule); + result +} diff --git a/rust/src/cmd/journalctl/unit_name.rs b/rust/src/cmd/journalctl/unit_name.rs new file mode 100644 index 00000000..7a7354d0 --- /dev/null +++ b/rust/src/cmd/journalctl/unit_name.rs @@ -0,0 +1,105 @@ +use anyhow::{Result, anyhow}; + +const CHARS: &[u8] = b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789:-_.\\@"; + +pub(super) fn normalize(value: &[u8]) -> Result<(String, Option)> { + if value.is_empty() || value.contains(&0) { + return Err(anyhow!( + "Failed to mangle unit name '{}': Invalid argument", + String::from_utf8_lossy(value) + )); + } + let has_suffix = |name: &[u8]| { + super::UNIT_SUFFIXES + .iter() + .any(|s| name.ends_with(s.as_bytes())) + }; + let is_glob = |name: &[u8]| name.iter().any(|c| b"*?[".contains(c)); + let glob_char = |c: &u8| CHARS.contains(c) || b"[]!*?".contains(c); + let valid_name = |name: &[u8]| { + name.len() < 256 + && name.iter().rposition(|c| *c == b'.').is_some_and(|i| i > 0) + && name.first() != Some(&b'@') + && has_suffix(name) + && name.iter().all(|c| CHARS.contains(c)) + }; + if valid_name(value) || is_glob(value) && value.iter().all(glob_char) { + return Ok((String::from_utf8(value.to_vec())?, None)); + } + if value[0] == b'/' { + let mut components = Vec::new(); + for part in value.split(|c| *c == b'/') { + if part.is_empty() || part == b"." || part == b".." && components.is_empty() { + continue; + } + components.push(part); + } + let path = components.join(&b'/'); + if !components.contains(&b"..".as_slice()) + && components.iter().all(|p| p.len() <= 255) + && path.len() < 4095 + { + let suffix = if path.starts_with(b"dev/") || path.starts_with(b"sys/") { + ".device" + } else { + ".mount" + }; + let mut name = String::new(); + if path.is_empty() { + name.push('-'); + } + for (i, &c) in path.iter().enumerate() { + if c == b'/' { + name.push('-'); + } else if c == b'-' + || c == b'\\' + || c == b'@' + || c == b'.' && i == 0 + || !CHARS.contains(&c) + { + escape(&mut name, c); + } else { + name.push(c as char); + } + } + name.push_str(suffix); + if name.len() >= 256 { + let key = [ + 0xec, 0xf2, 0x37, 0xfb, 0x58, 0x32, 0x4a, 0x32, 0x84, 0x9f, 0x06, 0x9b, 0x0d, + 0x21, 0xeb, 0x9a, + ]; + let hash = hex::encode( + journal_core::file::hash::siphash24_parts([name.as_bytes(), b"\0"], &key) + .to_le_bytes(), + ); + name.truncate(256 - 1 - suffix.len() - 16 - 1); + name.push('_'); + name.push_str(&hash); + name.push_str(suffix); + } + return Ok((name, None)); + } + } + let mut name = String::new(); + for &c in value { + if c == b'/' { + name.push('-'); + } else if !glob_char(&c) { + escape(&mut name, c); + } else { + name.push(c as char); + } + } + let escaped = (name.as_bytes() != value).then(|| name.clone()); + if !is_glob(name.as_bytes()) && !valid_name(name.as_bytes()) { + name.push_str(".service"); + } + Ok((name, escaped)) +} + +fn escape(out: &mut String, c: u8) { + const HEX: &[u8] = b"0123456789abcdef"; + out.push_str("\\x"); + out.push(HEX[(c >> 4) as usize] as char); + out.push(HEX[(c & 15) as usize] as char); +} diff --git a/rust/src/cmd/journalctl/unit_pattern.rs b/rust/src/cmd/journalctl/unit_pattern.rs new file mode 100644 index 00000000..e32b9796 --- /dev/null +++ b/rust/src/cmd/journalctl/unit_pattern.rs @@ -0,0 +1,257 @@ +use anyhow::Result; + +#[cfg(any(target_os = "linux", target_os = "macos", target_os = "freebsd"))] +pub(super) struct Matcher { + _locale: super::file_glob::ThreadLocale, + patterns: Vec, + buffer: Vec, + #[cfg(target_os = "freebsd")] + compat: Vec>, +} + +#[cfg(any(target_os = "linux", target_os = "macos", target_os = "freebsd"))] +impl Matcher { + pub(super) fn new(patterns: &[Vec]) -> Result { + Ok(Self { + _locale: super::file_glob::ThreadLocale::new()?, + patterns: patterns + .iter() + .map(|s| std::ffi::CString::new(s.as_slice())) + .collect::>()?, + buffer: Vec::new(), + #[cfg(target_os = "freebsd")] + compat: if patterns + .iter() + .any(|p| p.windows(2).any(|s| s == b"[:" || s == b"[." || s == b"[=")) + { + patterns + .iter() + .map(|p| CompatibilityPattern::new(p)) + .collect::>()? + } else { + Vec::new() + }, + }) + } + + pub(super) fn matches(&mut self, value: &[u8]) -> bool { + self.buffer.clear(); + self.buffer.extend_from_slice(value); + self.buffer.push(0); + #[cfg(target_os = "freebsd")] + if !self.compat.is_empty() { + return self.patterns.iter().zip(&self.compat).any(|(p, c)| { + if let Some(c) = c { + c.matches(&self.buffer) + } else { + unsafe { + libc::fnmatch(p.as_ptr(), self.buffer.as_ptr().cast(), libc::FNM_NOESCAPE) + == 0 + } + } + }); + } + self.patterns.iter().any(|p| unsafe { + libc::fnmatch(p.as_ptr(), self.buffer.as_ptr().cast(), libc::FNM_NOESCAPE) == 0 + }) + } +} + +#[cfg(any( + test, + not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")) +))] +pub(super) struct ByteMatcher { + patterns: Vec, +} + +#[cfg(any( + test, + not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")) +))] +impl ByteMatcher { + pub(super) fn new(patterns: &[Vec]) -> Result { + Ok(Self { + patterns: patterns + .iter() + .filter_map(|p| { + regex::bytes::RegexBuilder::new(&super::glob_pattern_to_regex( + &String::from_utf8_lossy(p), + )) + .unicode(false) + .build() + .ok() + }) + .collect(), + }) + } + + pub(super) fn matches(&mut self, value: &[u8]) -> bool { + self.patterns.iter().any(|p| p.is_match(value)) + } +} + +#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")))] +pub(super) type Matcher = ByteMatcher; + +#[cfg(test)] +mod tests { + use super::ByteMatcher; + + #[test] + fn fallback_byte_grammar() { + for (pattern, value, want) in [ + ("[[.a.]]lpha.*", b"alpha.service".as_slice(), true), + ("[[]", b"[".as_slice(), true), + ("[[:alpha:]]lpha.*", b"alpha.service".as_slice(), true), + ("[!b]lpha.*", b"blpha.service".as_slice(), false), + ("[]a]*", b"]name".as_slice(), true), + ("alpha[", b"alpha[".as_slice(), true), + (r"alpha\x20*", br"alpha\x20name.service".as_slice(), true), + ("bad-?.service", b"bad-\xff.service".as_slice(), true), + ("bad-?.service", "bad-é.service".as_bytes(), false), + ("bad-??.service", "bad-é.service".as_bytes(), true), + ] { + let mut matcher = ByteMatcher::new(&[pattern.as_bytes().to_vec()]).unwrap(); + assert_eq!(matcher.matches(value), want, "{pattern}"); + } + } +} + +#[cfg(any( + target_os = "freebsd", + all(test, any(target_os = "linux", target_os = "macos")) +))] +struct CompatibilityPattern(std::ptr::NonNull); + +#[cfg(any( + target_os = "freebsd", + all(test, any(target_os = "linux", target_os = "macos")) +))] +unsafe extern "C" { + fn journalctl_unit_pattern_new(pattern: *const libc::c_char) -> *mut std::ffi::c_void; + fn journalctl_unit_pattern_needed(pattern: *const libc::c_char) -> libc::c_int; + fn journalctl_unit_pattern_match( + pattern: *mut std::ffi::c_void, + value: *const libc::c_char, + ) -> libc::c_int; + fn journalctl_unit_pattern_free(pattern: *mut std::ffi::c_void); +} + +#[cfg(any( + target_os = "freebsd", + all(test, any(target_os = "linux", target_os = "macos")) +))] +impl CompatibilityPattern { + fn new(pattern: &[u8]) -> Result> { + let pattern = std::ffi::CString::new(pattern)?; + if unsafe { journalctl_unit_pattern_needed(pattern.as_ptr()) } == 0 { + return Ok(None); + } + let ptr = unsafe { journalctl_unit_pattern_new(pattern.as_ptr()) }; + Ok(Some(Self(std::ptr::NonNull::new(ptr).ok_or_else( + || anyhow::anyhow!("failed to prepare unit pattern"), + )?))) + } + fn matches(&self, terminated_value: &[u8]) -> bool { + unsafe { + journalctl_unit_pattern_match(self.0.as_ptr(), terminated_value.as_ptr().cast()) != 0 + } + } +} + +#[cfg(any( + target_os = "freebsd", + all(test, any(target_os = "linux", target_os = "macos")) +))] +impl Drop for CompatibilityPattern { + fn drop(&mut self) { + unsafe { journalctl_unit_pattern_free(self.0.as_ptr()) } + } +} + +#[cfg(all( + test, + any(target_os = "linux", target_os = "macos", target_os = "freebsd") +))] +#[test] +fn compatibility_brackets() { + struct LocaleGuard(libc::locale_t, libc::locale_t); + impl Drop for LocaleGuard { + fn drop(&mut self) { + unsafe { + libc::uselocale(self.1); + libc::freelocale(self.0); + } + } + } + for locale in [c"C", c"en_US.UTF-8"] { + let selected = + unsafe { libc::newlocale(libc::LC_ALL_MASK, locale.as_ptr(), std::ptr::null_mut()) }; + assert!( + !selected.is_null(), + "required test locale {locale:?} unavailable" + ); + let previous = unsafe { libc::uselocale(selected) }; + assert!(!previous.is_null()); + let _guard = LocaleGuard(selected, previous); + for (pattern, value, want) in [ + ("[[:alpha:]]", "é".as_bytes(), locale != c"C"), + ("[[=a=]]", "à".as_bytes(), locale != c"C"), + ("[[=a=]]", b"A", locale != c"C"), + ("[[=a=]]?", "à".as_bytes(), locale != c"C"), + ("[[=a=]-z]", b"-", true), + ("[[:alpha:]]lpha.*", b"alpha.service".as_slice(), true), + ("[[:alpha:]]lpha.*", b"1lpha.service", false), + ("[![:alpha:]]*", b"1.service", true), + ("[![:alpha:]]*", b"alpha.service", false), + ("[[.a.]]lpha.*", b"alpha.service", true), + ("[[=a=]]lpha.*", b"alpha.service", true), + ("[[:alpha:]]*", b"a\xff.service", true), + ("[![:alpha:]]?", "é".as_bytes(), true), + ("*[[:alpha:]]*", b"\xffa\xff", true), + (r"[[:alpha:]\]", b"\\", true), + ("[[:alpha:]]alpha[", b"aalpha[", true), + ("[[:alpha:]]$", b"a$", true), + ("[[:alpha:]]?", b"a\n", true), + ("[[:alpha:]]", b"a\n", false), + ("[[:unknown:]]*", b"a", false), + ("[[:<:]]*", b"a", false), + ("[[.period.]]*", b".", false), + ("[[:alpha:]", b"a", false), + ("[[[:bogus:]]", b"[", true), + ("[a-[:alpha:]]", b"a]", true), + ("[[:alpha:]z-a]", b"a", true), + ("[[[:<:]]", b"[", false), + ("[[:-", b"[[:-", false), + ("[[[.a]", b"[", false), + ("[[[=a", b"[[[=a", false), + ] { + let p = CompatibilityPattern::new(pattern.as_bytes()) + .unwrap() + .unwrap(); + let mut terminated = value.to_vec(); + terminated.push(0); + assert_eq!( + p.matches(&terminated), + want, + "{locale:?}: {pattern:?} {value:?}" + ); + } + + let pattern = CompatibilityPattern::new(b"[[:digit:]a-z]") + .unwrap() + .unwrap(); + for value in ["a", "A", "é", "É", "à", "z"] { + let value = std::ffi::CString::new(value).unwrap(); + let expected = unsafe { + libc::fnmatch(c"[a-z]".as_ptr(), value.as_ptr(), libc::FNM_NOESCAPE) == 0 + }; + assert_eq!( + pattern.matches(value.as_bytes_with_nul()), + expected, + "native range: {value:?}" + ); + } + } +} diff --git a/rust/src/cmd/journalctl/vacuum_metadata.rs b/rust/src/cmd/journalctl/vacuum_metadata.rs new file mode 100644 index 00000000..3662355b --- /dev/null +++ b/rust/src/cmd/journalctl/vacuum_metadata.rs @@ -0,0 +1,68 @@ +use std::fs::Metadata; +use std::path::Path; + +fn earlier(current: u64, candidate: u64) -> u64 { + if candidate > 0 && candidate < current { + candidate + } else { + current + } +} + +pub(super) fn oldest_realtime(metadata: &Metadata, mut realtime: u64) -> u64 { + if let Some(stamp) = metadata + .accessed() + .ok() + .and_then(super::system_time_to_usec) + { + realtime = earlier(realtime, stamp); + } + if let Some(stamp) = metadata.created().ok().and_then(super::system_time_to_usec) { + if !cfg!(target_os = "linux") || stamp >= 1_000_000 { + realtime = earlier(realtime, stamp); + } + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + if let (Ok(seconds), Ok(nanos)) = ( + u64::try_from(metadata.ctime()), + u64::try_from(metadata.ctime_nsec()), + ) { + if let Some(stamp) = seconds + .checked_mul(1_000_000) + .and_then(|value| value.checked_add(nanos / 1_000)) + { + realtime = earlier(realtime, stamp); + } + } + } + realtime +} + +pub(super) fn creation_realtime(path: &Path, realtime: u64) -> u64 { + legacy_creation_time(path).map_or(realtime, |stamp| earlier(realtime, stamp)) +} + +#[cfg(target_os = "linux")] +fn legacy_creation_time(path: &Path) -> Option { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt; + + let path = CString::new(path.as_os_str().as_bytes()).ok()?; + let mut data = [0u8; 8]; + let size = unsafe { + libc::getxattr( + path.as_ptr(), + c"user.crtime_usec".as_ptr(), + data.as_mut_ptr().cast(), + data.len(), + ) + }; + (size == data.len() as libc::ssize_t).then(|| u64::from_le_bytes(data)) +} + +#[cfg(not(target_os = "linux"))] +fn legacy_creation_time(_path: &Path) -> Option { + None +} diff --git a/rust/src/cmd/journalctl/vacuum_order.rs b/rust/src/cmd/journalctl/vacuum_order.rs new file mode 100644 index 00000000..eba94a3f --- /dev/null +++ b/rust/src/cmd/journalctl/vacuum_order.rs @@ -0,0 +1,144 @@ +use super::VacuumCandidate; +use std::cmp::Ordering; + +struct Run { + next: usize, + end: usize, +} + +pub(super) struct VacuumOrder<'a> { + candidates: &'a [VacuumCandidate], + heads: Vec, +} + +impl<'a> VacuumOrder<'a> { + pub(super) fn new(candidates: &'a mut [VacuumCandidate]) -> Self { + // A pairwise sequence/time comparator can cycle after a clock rollback. + candidates.sort_unstable_by(|a, b| { + b.have_seqno + .cmp(&a.have_seqno) + .then_with(|| { + if a.have_seqno { + a.seqnum_id.cmp(&b.seqnum_id).then(a.seqnum.cmp(&b.seqnum)) + } else { + Ordering::Equal + } + }) + .then_with(|| a.name.cmp(&b.name)) + }); + let mut heads = Vec::new(); + let mut start = 0; + while start < candidates.len() { + let mut end = start + 1; + if candidates[start].have_seqno { + while end < candidates.len() + && candidates[end].have_seqno + && candidates[end].seqnum_id == candidates[start].seqnum_id + { + end += 1; + } + } + heads.push(Run { next: start, end }); + start = end; + } + let mut order = Self { candidates, heads }; + for i in (0..order.heads.len() / 2).rev() { + order.down(i); + } + order + } + + fn less(&self, i: usize, j: usize) -> bool { + let a = &self.candidates[self.heads[i].next]; + let b = &self.candidates[self.heads[j].next]; + a.realtime + .cmp(&b.realtime) + .then(b.have_seqno.cmp(&a.have_seqno)) + .then_with(|| { + if a.have_seqno { + a.seqnum_id.cmp(&b.seqnum_id) + } else { + Ordering::Equal + } + }) + .then_with(|| a.name.cmp(&b.name)) + .is_lt() + } + + fn down(&mut self, mut i: usize) { + loop { + let mut child = 2 * i + 1; + if child >= self.heads.len() { + return; + } + if child + 1 < self.heads.len() && self.less(child + 1, child) { + child += 1; + } + if !self.less(child, i) { + return; + } + self.heads.swap(i, child); + i = child; + } + } +} + +impl<'a> Iterator for VacuumOrder<'a> { + type Item = &'a VacuumCandidate; + + fn next(&mut self) -> Option { + let candidate = &self.candidates[self.heads.first()?.next]; + self.heads[0].next += 1; + if self.heads[0].next == self.heads[0].end { + self.heads.swap_remove(0); + } + // The newly exposed timestamp may move backward; only the root changed. + self.down(0); + Some(candidate) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + #[test] + fn history_order() { + let mut rows: Vec<_> = [ + ('A', 1, 1, 300), + ('B', 1, 2, 100), + ('C', 2, 1, 200), + ('D', 0, 0, 200), + ('E', 1, 2, 50), + ] + .into_iter() + .map(|(name, id, seqnum, realtime)| VacuumCandidate { + name: name.to_string().into(), + path: PathBuf::new(), + usage: 0, + have_seqno: id != 0, + seqnum_id: [id; 16], + seqnum, + realtime, + }) + .collect(); + fn permutations(rows: &mut [VacuumCandidate], i: usize) { + if i < rows.len() { + for j in i..rows.len() { + rows.swap(i, j); + permutations(rows, i + 1); + rows.swap(i, j); + } + return; + } + let mut copy = rows.to_vec(); + let names: Vec<_> = VacuumOrder::new(&mut copy) + .map(|c| c.name.to_str().unwrap()) + .collect(); + assert_eq!(names, ["C", "D", "A", "B", "E"]); + } + permutations(&mut rows, 0); + assert!(VacuumOrder::new(&mut []).next().is_none()); + } +} diff --git a/rust/src/crates/journal-core/src/error.rs b/rust/src/crates/journal-core/src/error.rs index e75fe82c..c5aeb7bf 100644 --- a/rust/src/crates/journal-core/src/error.rs +++ b/rust/src/crates/journal-core/src/error.rs @@ -75,6 +75,9 @@ pub enum JournalError { #[error("invalid offset")] InvalidOffset, + #[error("cyclic journal object chain")] + CyclicObjectChain, + #[error("zerocopy failure")] ZerocopyFailure, diff --git a/rust/src/crates/journal-core/src/file/array_chain.rs b/rust/src/crates/journal-core/src/file/array_chain.rs new file mode 100644 index 00000000..93c8a923 --- /dev/null +++ b/rust/src/crates/journal-core/src/file/array_chain.rs @@ -0,0 +1,27 @@ +use super::chain_guard::ChainGuard; +use crate::error::{JournalError, Result}; +use std::num::NonZeroU64; + +// Exceptional links need history, but only already visited headers matter. +#[cold] +#[inline(never)] +pub(super) fn reject_array_revisit( + head: NonZeroU64, + current: NonZeroU64, + target: NonZeroU64, + mut next_of: impl FnMut(NonZeroU64) -> Result>, +) -> Result<()> { + let mut at = head; + let mut guard = ChainGuard::default(); + loop { + if at == target { + return Err(JournalError::CyclicObjectChain); + } + if at == current { + return Ok(()); + } + let next = next_of(at)?; + guard.advance(at, next, false)?; + at = next.ok_or(JournalError::InvalidOffsetArrayOffset)?; + } +} diff --git a/rust/src/crates/journal-core/src/file/chain_guard.rs b/rust/src/crates/journal-core/src/file/chain_guard.rs new file mode 100644 index 00000000..2457274f --- /dev/null +++ b/rust/src/crates/journal-core/src/file/chain_guard.rs @@ -0,0 +1,104 @@ +use crate::error::{JournalError, Result}; +use std::num::NonZeroU64; + +/// Detects cycles without reads or allocations, retaining acyclic link orders. +#[derive(Default, Clone, Copy, Debug)] +#[cfg_attr(feature = "allocative", derive(allocative::Allocative))] +pub(super) struct ChainGuard { + checkpoint: u64, + steps: u64, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn all_small_graphs_match_a_visited_set() { + for size in 1usize..=5 { + for encoded in 0..(size + 1).pow(size as u32) { + let (mut links, mut code) = ([0usize; 6], encoded); + for link in &mut links[1..=size] { + *link = code % (size + 1); + code /= size + 1; + } + for start in 1..=size { + let (mut seen, mut at) = (0u8, start); + while at != 0 && seen & (1 << at) == 0 { + seen |= 1 << at; + at = links[at]; + } + let want_cycle = at != 0; + for descending in [false, true] { + let mut guard = ChainGuard::default(); + for _ in 0..2 { + guard.reset(); + let (mut at, mut cycle) = (start, false); + for _ in 0..4 * size + 4 { + let Some(current) = NonZeroU64::new(at as u64) else { + break; + }; + let next = links[at]; + if guard + .advance(current, NonZeroU64::new(next as u64), descending) + .is_err() + { + cycle = true; + break; + } + at = next; + } + assert_eq!(cycle, want_cycle, "{links:?}/{start}/{descending}"); + assert!(cycle || at == 0); + } + } + } + } + } + } +} + +impl ChainGuard { + pub(super) fn reset(&mut self) { + self.steps = 0; + } + + #[inline] + pub(super) fn advance( + &mut self, + current: NonZeroU64, + next: Option, + descending: bool, + ) -> Result<()> { + let (current, next) = (current.get(), next.map_or(0, NonZeroU64::get)); + let ordered = if descending { + next < current + } else { + next > current + }; + if (self.steps == 0) & ordered { + return Ok(()); + } + if next == 0 { + return Ok(()); + } + self.advance_unordered(current, next) + } + + #[cold] + #[inline(never)] + fn advance_unordered(&mut self, current: u64, next: u64) -> Result<()> { + if self.steps == 0 { + self.checkpoint = current; + self.steps = 1; + } + if next == self.checkpoint || self.steps == u64::MAX { + return Err(JournalError::CyclicObjectChain); + } + self.steps += 1; + if self.steps.is_power_of_two() { + self.checkpoint = next; + } + Ok(()) + } +} diff --git a/rust/src/crates/journal-core/src/file/cursor.rs b/rust/src/crates/journal-core/src/file/cursor.rs index be0fcbb3..124e5d86 100644 --- a/rust/src/crates/journal-core/src/file/cursor.rs +++ b/rust/src/crates/journal-core/src/file/cursor.rs @@ -40,11 +40,18 @@ impl JournalCursor { pub fn set_location(&mut self, location: Location) { self.location = location; self.array_cursor = None; + if matches!(location, Location::ResolvedEntry(_)) { + if let Some(filter) = &mut self.filter_expr { + filter.invalidate_position(); + } + } } - pub fn set_filter(&mut self, filter_expr: FilterExpr) { + pub fn set_filter(&mut self, mut filter_expr: FilterExpr) { + if matches!(self.location, Location::ResolvedEntry(_)) { + filter_expr.invalidate_position(); + } self.filter_expr = Some(filter_expr); - // FIXME: should we set cursor to None? } pub fn clear_filter(&mut self) { @@ -53,10 +60,54 @@ impl JournalCursor { self.set_location(Location::Head); } + #[inline] pub fn step( &mut self, journal_file: &JournalFile, direction: Direction, + ) -> Result { + let result = self.step_inner(journal_file, direction); + if matches!(result, Ok(true)) { + return Ok(true); + } + self.step_recovery(journal_file, direction, result) + } + + #[cold] + #[inline(never)] + fn step_recovery( + &mut self, + journal_file: &JournalFile, + direction: Direction, + result: Result, + ) -> Result { + let before = self.location; + match result { + Err(err) if journal_file.recover_reader_tail(&err)? => { + self.reset_after_recovery(journal_file, before)?; + self.step_inner(journal_file, direction) + } + other => other, + } + } + + pub(super) fn reset_after_recovery( + &mut self, + file: &JournalFile, + before: Location, + ) -> Result<()> { + self.location = before; + self.array_cursor = None; + if let Some(filter) = &mut self.filter_expr { + filter.refresh_recovery(file)?; + } + Ok(()) + } + + fn step_inner( + &mut self, + journal_file: &JournalFile, + direction: Direction, ) -> Result { let new_location = if self.filter_expr.is_some() { self.resolve_filter_location(journal_file, direction)? @@ -79,10 +130,16 @@ impl JournalCursor { } } - fn entry_list(journal_file: &JournalFile) -> Result { + fn entry_list( + journal_file: &JournalFile, + ) -> Result> { + if journal_file.reader_entry_count() == 0 { + return Ok(None); + } journal_file .entry_list() - .ok_or(JournalError::InvalidOffsetArrayOffset) + .map(Some) + .ok_or(JournalError::ObjectExceedsFileBounds) } fn store_array_cursor_value( @@ -101,7 +158,10 @@ impl JournalCursor { &mut self, journal_file: &JournalFile, ) -> Result> { - let cursor = Self::entry_list(journal_file)?.cursor_head(); + let Some(list) = Self::entry_list(journal_file)? else { + return Ok(None); + }; + let cursor = list.cursor_head(); self.store_array_cursor_value(journal_file, cursor) } @@ -109,7 +169,9 @@ impl JournalCursor { &mut self, journal_file: &JournalFile, ) -> Result> { - let entry_list = Self::entry_list(journal_file)?; + let Some(entry_list) = Self::entry_list(journal_file)? else { + return Ok(None); + }; let cursor = entry_list.cursor_tail(journal_file)?; self.store_array_cursor_value(journal_file, cursor) } @@ -117,25 +179,31 @@ impl JournalCursor { fn realtime_array_cursor( journal_file: &JournalFile, realtime: u64, - ) -> Result { - let entry_list = Self::entry_list(journal_file)?; + direction: Direction, + ) -> Result> { + let Some(entry_list) = Self::entry_list(journal_file)? else { + return Ok(None); + }; let predicate = |entry_offset| { let entry_object = journal_file.entry_ref(entry_offset)?; - Ok(entry_object.header.realtime < realtime) + Ok(match direction { + Direction::Forward => entry_object.header.realtime < realtime, + Direction::Backward => entry_object.header.realtime <= realtime, + }) }; - entry_list - .directed_partition_point(journal_file, predicate, Direction::Forward)? - .map(Ok) - .unwrap_or_else(|| entry_list.cursor_tail(journal_file)) + entry_list.directed_partition_point(journal_file, predicate, direction) } fn resolve_realtime_array( &mut self, journal_file: &JournalFile, realtime: u64, + direction: Direction, ) -> Result> { - let cursor = Self::realtime_array_cursor(journal_file, realtime)?; + let Some(cursor) = Self::realtime_array_cursor(journal_file, realtime, direction)? else { + return Ok(None); + }; self.store_array_cursor_value(journal_file, cursor) } @@ -144,6 +212,9 @@ impl JournalCursor { journal_file: &JournalFile, direction: Direction, ) -> Result> { + if self.array_cursor.is_none() { + return self.resolve_missing_array_cursor(journal_file, direction); + } let cursor = self .array_cursor .as_ref() @@ -158,6 +229,31 @@ impl JournalCursor { self.store_array_cursor_value(journal_file, cursor) } + #[cold] + #[inline(never)] + fn resolve_missing_array_cursor( + &mut self, + journal_file: &JournalFile, + direction: Direction, + ) -> Result> { + let offset = self.position()?; + let Some(list) = Self::entry_list(journal_file)? else { + return Ok(None); + }; + let predicate = |candidate| { + Ok(if direction == Direction::Forward { + candidate <= offset + } else { + candidate < offset + }) + }; + let Some(cursor) = list.directed_partition_point(journal_file, predicate, direction)? + else { + return Ok(None); + }; + self.store_array_cursor_value(journal_file, cursor) + } + fn resolve_array_cursor( &mut self, journal_file: &JournalFile, @@ -168,8 +264,8 @@ impl JournalCursor { (Location::Head, Direction::Backward) => Ok(None), (Location::Tail, Direction::Forward) => Ok(None), (Location::Tail, Direction::Backward) => self.resolve_tail_array(journal_file), - (Location::Realtime(realtime), _) => { - self.resolve_realtime_array(journal_file, realtime) + (Location::Realtime(realtime), direction) => { + self.resolve_realtime_array(journal_file, realtime, direction) } (Location::ResolvedEntry(_), direction) => { self.step_array_cursor(journal_file, direction) @@ -210,17 +306,10 @@ impl JournalCursor { direction: Direction, entry_offset: NonZeroU64, ) -> Result> { - let filter_expr = self.filter_expr_mut()?; - match direction { - Direction::Forward => Ok(filter_expr - .head() - .next(journal_file, entry_offset)? - .map(Location::ResolvedEntry)), - Direction::Backward => Ok(filter_expr - .tail(journal_file)? - .previous(journal_file, entry_offset)? - .map(Location::ResolvedEntry)), - } + Ok(self + .filter_expr_mut()? + .seek_realtime(journal_file, entry_offset, direction)? + .map(Location::ResolvedEntry)) } fn resolve_filter_realtime( @@ -229,7 +318,9 @@ impl JournalCursor { realtime: u64, direction: Direction, ) -> Result> { - let cursor = Self::realtime_array_cursor(journal_file, realtime)?; + let Some(cursor) = Self::realtime_array_cursor(journal_file, realtime, direction)? else { + return Ok(None); + }; let Some(entry_offset) = cursor.value(journal_file)? else { return Ok(None); }; @@ -268,11 +359,17 @@ impl JournalCursor { ) -> Result> { match (self.location, direction) { (Location::Head, Direction::Forward) => { + if journal_file.reader_entry_count() == 0 { + return Ok(None); + } Self::filter_head(self.filter_expr_mut()?, journal_file) } (Location::Head, Direction::Backward) => Ok(None), (Location::Tail, Direction::Forward) => Ok(None), (Location::Tail, Direction::Backward) => { + if journal_file.reader_entry_count() == 0 { + return Ok(None); + } Self::filter_tail(self.filter_expr_mut()?, journal_file) } (Location::Realtime(realtime), direction) => { diff --git a/rust/src/crates/journal-core/src/file/cursor_direction.rs b/rust/src/crates/journal-core/src/file/cursor_direction.rs new file mode 100644 index 00000000..d0ee582b --- /dev/null +++ b/rust/src/crates/journal-core/src/file/cursor_direction.rs @@ -0,0 +1,273 @@ +use super::{Direction, JournalCursor, JournalFile, Location, filter::FilterExpr, mmap::MemoryMap}; +use crate::error::{JournalError, Result}; +use std::num::NonZeroU64; + +/// Parsed cursor components for indexed directional positioning. +#[doc(hidden)] +#[derive(Clone, Copy, Debug, Default)] +pub struct CursorAnchor { + pub sequence: Option<([u8; 16], u64)>, + pub monotonic: Option<([u8; 16], u64)>, + pub realtime: Option, +} + +#[derive(Clone, Copy)] +enum Key { + Sequence, + Realtime, + Monotonic, + Offset, +} + +impl JournalFile { + fn cursor_bound( + &self, + data: Option, + key: Key, + needle: u64, + direction: Direction, + ) -> Result> { + let predicate = |offset: NonZeroU64| { + let value = if matches!(key, Key::Offset) { + offset.get() + } else { + let entry = self.entry_ref(offset)?; + match key { + Key::Sequence => entry.header.seqnum, + Key::Monotonic => entry.header.monotonic, + _ => entry.header.realtime, + } + }; + Ok(value < needle || direction == Direction::Backward && value == needle) + }; + if let Some(data) = data { + return self.data_object_directed_partition_point(data, predicate, direction); + } + if self.reader_entry_count() == 0 { + return Ok(None); + } + let list = self + .entry_list() + .ok_or(JournalError::ObjectExceedsFileBounds)?; + let Some(cursor) = list.directed_partition_point(self, predicate, direction)? else { + return Ok(None); + }; + cursor.value(self) + } + + fn cursor_boot_data(&self, boot: [u8; 16]) -> Result> { + let mut payload = [0u8; 41]; + payload[..9].copy_from_slice(b"_BOOT_ID="); + const HEX: &[u8] = b"0123456789abcdef"; + for (i, byte) in boot.into_iter().enumerate() { + payload[9 + 2 * i] = HEX[(byte >> 4) as usize]; + payload[10 + 2 * i] = HEX[(byte & 15) as usize]; + } + self.find_data_offset(self.hash(&payload), &payload) + } + + fn cursor_monotonic( + &self, + data: Option, + boot: [u8; 16], + needle: u64, + direction: Direction, + ) -> Result> { + let Some(boot_data) = self.cursor_boot_data(boot)? else { + return Ok(None); + }; + let Some(mut offset) = + self.cursor_bound(Some(boot_data), Key::Monotonic, needle, direction)? + else { + return Ok(None); + }; + let Some(data) = data else { + return Ok(Some(offset)); + }; + loop { + let Some(candidate) = + self.cursor_bound(Some(data), Key::Offset, offset.get(), direction)? + else { + return Ok(None); + }; + if candidate == offset { + return Ok(Some(offset)); + } + let Some(next) = + self.cursor_bound(Some(boot_data), Key::Offset, candidate.get(), direction)? + else { + return Ok(None); + }; + offset = next; + if offset == candidate { + return Ok(Some(offset)); + } + } + } + + fn cursor_adjacent_boot( + &self, + data: Option, + mut boot: [u8; 16], + direction: Direction, + ) -> Result> { + let mut previous = None; + loop { + let Some(boot_data) = self.cursor_boot_data(boot)? else { + return Ok(None); + }; + let (needle, opposite) = match direction { + Direction::Forward => (u64::MAX, Direction::Backward), + Direction::Backward => (0, Direction::Forward), + }; + let Some(boundary) = + self.cursor_bound(Some(boot_data), Key::Offset, needle, opposite)? + else { + return Ok(None); + }; + let needle = match direction { + Direction::Forward => boundary.get().checked_add(1), + Direction::Backward => boundary.get().checked_sub(1), + }; + let Some(needle) = needle else { + return Ok(None); + }; + let Some(offset) = self.cursor_bound(None, Key::Offset, needle, direction)? else { + return Ok(None); + }; + if data.is_none() { + return Ok(Some(offset)); + } + if previous.is_some_and(|old| match direction { + Direction::Forward => offset <= old, + Direction::Backward => offset >= old, + }) { + return Err(JournalError::InvalidQueryConfiguration); + } + previous = Some(offset); + boot = self.entry_ref(offset)?.header.boot_id; + let needle = if direction == Direction::Forward { + 0 + } else { + u64::MAX + }; + if let Some(candidate) = self.cursor_monotonic(data, boot, needle, direction)? { + return Ok(Some(candidate)); + } + } + } + + pub(super) fn cursor_initial_offset( + &self, + data: Option, + anchor: &CursorAnchor, + direction: Direction, + ) -> Result> { + if let Some((id, seqnum)) = anchor.sequence { + if id == self.journal_header_ref().seqnum_id { + return self.cursor_bound(data, Key::Sequence, seqnum, direction); + } + } + if let Some((boot, monotonic)) = anchor.monotonic { + if let Some(offset) = self.cursor_monotonic(data, boot, monotonic, direction)? { + return Ok(Some(offset)); + } + } + if let Some(realtime) = anchor.realtime { + return self.cursor_bound(data, Key::Realtime, realtime, direction); + } + if let Some((boot, _)) = anchor.monotonic { + return self.cursor_adjacent_boot(data, boot, direction); + } + let needle = if direction == Direction::Forward { + 0 + } else { + u64::MAX + }; + self.cursor_bound(data, Key::Offset, needle, direction) + } +} + +impl FilterExpr { + pub(super) fn seek_cursor_anchor( + &mut self, + file: &JournalFile, + anchor: &CursorAnchor, + direction: Direction, + ) -> Result> { + self.prepare_recovery(file, false)?; + match self { + Self::None => Ok(None), + Self::Match(data, _) => file.cursor_initial_offset(Some(*data), anchor, direction), + Self::Disjunction(children) => { + let mut best: Option = None; + for child in children { + if let Some(offset) = child.seek_cursor_anchor(file, anchor, direction)? { + best = Some(match (best, direction) { + (Some(old), Direction::Forward) => old.min(offset), + (Some(old), Direction::Backward) => old.max(offset), + (None, _) => offset, + }); + } + } + Ok(best) + } + Self::Conjunction(children) => { + let mut best: Option = None; + for child in children { + let Some(offset) = child.seek_cursor_anchor(file, anchor, direction)? else { + return Ok(None); + }; + best = Some(match (best, direction) { + (Some(old), Direction::Forward) => old.max(offset), + (Some(old), Direction::Backward) => old.min(offset), + (None, _) => offset, + }); + } + match best { + Some(offset) => self.seek_realtime(file, offset, direction), + None => Ok(None), + } + } + } + } +} + +impl JournalCursor { + pub(super) fn seek_anchor( + &mut self, + file: &JournalFile, + anchor: &CursorAnchor, + direction: Direction, + ) -> Result { + loop { + let before = file.recovery_status(); + let result = if let Some(filter) = self.filter_expr.as_mut() { + filter.seek_cursor_anchor(file, anchor, direction) + } else { + file.cursor_initial_offset(None, anchor, direction) + }; + let offset = match result { + Err(err) if file.recover_reader_tail(&err)? => { + if let Some(filter) = self.filter_expr.as_mut() { + filter.refresh_recovery(file)?; + } + continue; + } + result => result?, + }; + if before != file.recovery_status() { + if let Some(filter) = self.filter_expr.as_mut() { + filter.refresh_recovery(file)?; + } + continue; + } + self.set_location(match offset { + Some(offset) => Location::ResolvedEntry(offset), + None if direction == Direction::Forward => Location::Tail, + None => Location::Head, + }); + return Ok(offset.is_some()); + } + } +} diff --git a/rust/src/crates/journal-core/src/file/file.rs b/rust/src/crates/journal-core/src/file/file.rs index f1aeb841..bc35baf2 100644 --- a/rust/src/crates/journal-core/src/file/file.rs +++ b/rust/src/crates/journal-core/src/file/file.rs @@ -437,6 +437,12 @@ pub struct JournalFile { // Forward Secure Sealing options (consumed by JournalWriter on first use) pub seal_options: Option, + + // Keep exceptional recovery state after the frequently accessed mappings. + pub(super) recovery_enabled: bool, + pub(super) recovery: super::reader_recovery::ReaderRecoveryCell, + pub(super) recovery_changed: std::cell::Cell, + pub(super) filter_recovery_seen: std::cell::Cell, } pub(super) fn map_hash_table( @@ -516,6 +522,7 @@ impl JournalFile { let hash_table = hash_table.ok_or(JournalError::MissingHashTable)?; let bucket = hash_table.hash_item_ref(hash); let mut object_offset = bucket.head_hash_offset; + let mut chain = super::chain_guard::ChainGuard::default(); while let Some(offset) = object_offset { let object_guard = self.journal_object_ref::(offset)?; @@ -524,7 +531,9 @@ impl JournalFile { return Ok(Some(output)); } - object_offset = object_guard.next_hash_offset(); + let next = object_guard.next_hash_offset(); + chain.advance(offset, next, false)?; + object_offset = next; } Ok(None) @@ -628,7 +637,7 @@ impl JournalFile { let fd = OpenOptions::new() .read(true) .write(false) - .open(file.path())?; + .open(file.as_path())?; // Create a memory map for the header let header_size = std::mem::size_of::() as u64; @@ -663,6 +672,10 @@ impl JournalFile { let window_manager = GuardedCell::new(window_manager_builder(fd)?); let journal = JournalFile { + recovery_enabled: false, + recovery: Default::default(), + recovery_changed: std::cell::Cell::new(false), + filter_recovery_seen: std::cell::Cell::new(false), file, header_map, sanitized_header, @@ -740,7 +753,7 @@ impl JournalFile { let header = self.journal_header_ref(); header.entry_array_offset.and_then(|head_offset| { - std::num::NonZeroUsize::new(header.n_entries as usize) + std::num::NonZeroUsize::new(self.reader_entry_count() as usize) .map(|total_items| offset_array::List::new(head_offset, total_items)) }) } @@ -846,7 +859,11 @@ impl JournalFile { wm.get_slice(offset.get(), std::mem::size_of::() as u64)?; let header = ObjectHeader::ref_from_bytes(header_slice) .map_err(|_| JournalError::ZerocopyFailure)?; - header.validated_size()? + let size = header.validated_size()?; + if T::EXPECTED_TYPE.is_some_and(|expected| header.type_ != expected as u8) { + return Err(JournalError::InvalidObjectType); + } + size }; // Validate that the object doesn't exceed the journal's arena bounds @@ -909,7 +926,7 @@ impl JournalFile { where F: Fn(NonZeroU64) -> Result, { - let Some(cursor) = self.data_ref(data_offset)?.inlined_cursor() else { + let Some(cursor) = self.reader_data_cursor(data_offset)? else { return Ok(None); }; @@ -920,6 +937,61 @@ impl JournalFile { best_match.value(self) } + #[doc(hidden)] + pub fn reader_data_cursor( + &self, + offset: NonZeroU64, + ) -> Result> { + let build = || { + let result = { + let data = self.data_ref(offset)?; + self.reader_inlined_cursor(&data.header) + }; + if !matches!(&result, Ok(Some(_))) || self.recovered_last_entry().is_none() { + return result; + } + match result { + Ok(Some(cursor)) => cursor.for_reader(self), + result => result, + } + }; + let result = build(); + if let Err(err) = &result { + if self.recover_reader_tail(err)? { + return build(); + } + } + result + } + + #[doc(hidden)] + pub fn reader_inlined_cursor( + &self, + data: &DataObjectHeader, + ) -> Result> { + let cursor = data.inlined_cursor(); + if cursor.is_some() { + return Ok(cursor); + } + if data.n_entries.is_none() { + return if data.entry_offset.is_none() { + Ok(None) + } else { + Err(JournalError::InvalidOffset) + }; + } + let first = data.entry_offset.ok_or(JournalError::InvalidOffset)?; + if self.reader_recovery_active() + && self + .recovered_last_entry() + .flatten() + .is_none_or(|last| first >= last) + { + return Ok(Some(offset_array::InlinedCursor::new(first, None))); + } + Err(JournalError::ObjectExceedsFileBounds) + } + /// Creates an iterator over all field objects in the field hash table pub fn fields(&self) -> FieldIterator<'_, M> { // Get the field hash table @@ -931,6 +1003,7 @@ impl JournalFile { field_hash_table, current_bucket_index: 0, next_field_offset: None, + chain: Default::default(), }; // Find the first non-empty bucket @@ -940,13 +1013,36 @@ impl JournalFile { } pub fn field_head_data_offset(&self, field_name: &[u8]) -> Result> { - let field_hash = self.hash(field_name); - let Some(field_offset) = self.find_field_offset(field_hash, field_name)? else { - return Ok(None); - }; - - let field_guard = self.field_ref(field_offset)?; - Ok(field_guard.header.head_data_offset) + struct HeadVisitor<'a> { + name: &'a [u8], + hash: u64, + } + impl<'a> BucketVisitor<'a> for HeadVisitor<'_> { + type Object = FieldObject<&'a [u8]>; + type Output = Option; + + #[inline(always)] + fn visit( + &mut self, + object: &ValueGuard<'a, Self::Object>, + ) -> Result> { + Ok( + (object.hash() == self.hash && object.raw_payload() == self.name) + .then_some(object.header.head_data_offset), + ) + } + } + let hash = self.hash(field_name); + Ok(self + .visit_bucket( + self.field_hash_table_ref(), + hash, + HeadVisitor { + name: field_name, + hash, + }, + )? + .flatten()) } /// Creates an iterator over all DATA objects for the specified field @@ -957,6 +1053,7 @@ impl JournalFile { journal: self, head_data_offset, current_data_offset: head_data_offset, + chain: Default::default(), }) } @@ -966,20 +1063,12 @@ impl JournalFile { &'a self, field_name: &[u8], ) -> Result> { - let field_hash = self.hash(field_name); - let Some(field_offset) = self.find_field_offset(field_hash, field_name)? else { - return Ok(FieldDataOffsetIterator { - journal: self, - current_data_offset: None, - }); - }; - - let field_guard = self.field_ref(field_offset)?; - let head_data_offset = field_guard.header.head_data_offset; + let head_data_offset = self.field_head_data_offset(field_name)?; Ok(FieldDataOffsetIterator { journal: self, current_data_offset: head_data_offset, + chain: Default::default(), }) } diff --git a/rust/src/crates/journal-core/src/file/file_iterators.rs b/rust/src/crates/journal-core/src/file/file_iterators.rs index fdd19bb2..b2ae4ed6 100644 --- a/rust/src/crates/journal-core/src/file/file_iterators.rs +++ b/rust/src/crates/journal-core/src/file/file_iterators.rs @@ -1,3 +1,4 @@ +use super::chain_guard::ChainGuard; use super::file::JournalFile; use super::mmap::MemoryMap; use super::object::*; @@ -11,11 +12,13 @@ pub struct FieldIterator<'a, M: MemoryMap> { pub(super) field_hash_table: Option>, pub(super) current_bucket_index: usize, pub(super) next_field_offset: Option, + pub(super) chain: ChainGuard, } impl FieldIterator<'_, M> { /// Advances to the next non-empty bucket pub(super) fn advance_to_next_nonempty_bucket(&mut self) { + self.chain.reset(); // If we don't have a hash table, there's nothing to iterate let Some(hash_table) = &self.field_hash_table else { return; @@ -46,8 +49,13 @@ impl<'a, M: MemoryMap> Iterator for FieldIterator<'a, M> { match self.journal.field_ref(offset) { Ok(field_guard) => { + let next = field_guard.header.next_hash_offset; + if let Err(err) = self.chain.advance(offset, next, false) { + self.next_field_offset = None; + return Some(Err(err)); + } // Get the next field offset before we return the guard - self.next_field_offset = field_guard.header.next_hash_offset; + self.next_field_offset = next; // If we've reached the end of the chain, move to the next bucket if self.next_field_offset.is_none() { @@ -70,11 +78,13 @@ pub struct FieldDataIterator<'a, M: MemoryMap> { pub(super) journal: &'a JournalFile, pub(super) head_data_offset: Option, pub(super) current_data_offset: Option, + pub(super) chain: ChainGuard, } impl FieldDataIterator<'_, M> { pub(super) fn restart(&mut self) { self.current_data_offset = self.head_data_offset; + self.chain.reset(); } } @@ -86,8 +96,13 @@ impl<'a, M: MemoryMap> Iterator for FieldDataIterator<'a, M> { match self.journal.data_ref(data_offset) { Ok(data_guard) => { + let next = data_guard.header.next_field_offset; + if let Err(err) = self.chain.advance(data_offset, next, true) { + self.current_data_offset = None; + return Some(Err(err)); + } // Get the next data offset before we return the guard - self.current_data_offset = data_guard.header.next_field_offset; + self.current_data_offset = next; Some(Ok(data_guard)) } Err(e) => { @@ -103,6 +118,7 @@ impl<'a, M: MemoryMap> Iterator for FieldDataIterator<'a, M> { pub struct FieldDataOffsetIterator<'a, M: MemoryMap> { pub(super) journal: &'a JournalFile, pub(super) current_data_offset: Option, + pub(super) chain: ChainGuard, } impl<'a, M: MemoryMap> Iterator for FieldDataOffsetIterator<'a, M> { @@ -113,7 +129,12 @@ impl<'a, M: MemoryMap> Iterator for FieldDataOffsetIterator<'a, M> { match self.journal.data_ref(data_offset) { Ok(data_guard) => { - self.current_data_offset = data_guard.header.next_field_offset; + let next = data_guard.header.next_field_offset; + if let Err(err) = self.chain.advance(data_offset, next, true) { + self.current_data_offset = None; + return Some(Err(err)); + } + self.current_data_offset = next; Some(Ok((data_offset, data_guard))) } Err(e) => { diff --git a/rust/src/crates/journal-core/src/file/file_mut.rs b/rust/src/crates/journal-core/src/file/file_mut.rs index 69fbd3aa..c9901e79 100644 --- a/rust/src/crates/journal-core/src/file/file_mut.rs +++ b/rust/src/crates/journal-core/src/file/file_mut.rs @@ -37,7 +37,7 @@ impl JournalFile { let fd = OpenOptions::new() .read(true) .write(true) - .open(file.path())?; + .open(file.as_path())?; let header_size = std::mem::size_of::() as u64; let file_size = fd.metadata()?.len(); @@ -80,6 +80,10 @@ impl JournalFile { GuardedCell::new(WindowManager::new_writer_owned(fd, window_size, 32)?); let journal = JournalFile { + recovery_enabled: false, + recovery: Default::default(), + recovery_changed: std::cell::Cell::new(false), + filter_recovery_seen: std::cell::Cell::new(false), file: file.clone(), header_map, sanitized_header: None, @@ -194,6 +198,10 @@ impl JournalFile { )?); let mut jf = JournalFile { + recovery_enabled: false, + recovery: Default::default(), + recovery_changed: std::cell::Cell::new(false), + filter_recovery_seen: std::cell::Cell::new(false), file: file.clone(), header_map, sanitized_header: None, @@ -212,7 +220,7 @@ impl JournalFile { #[cfg(unix)] { use std::os::unix::fs::PermissionsExt; - if let Ok(metadata) = std::fs::metadata(self.file.path()) { + if let Ok(metadata) = std::fs::metadata(self.file.as_path()) { return metadata.permissions().mode() & 0o777; } } @@ -228,7 +236,7 @@ impl JournalFile { .write(true); #[cfg(unix)] open_options.mode(mode); - Ok(open_options.open(file.path())?) + Ok(open_options.open(file.as_path())?) } fn create_layout(options: &JournalFileOptions) -> Result { diff --git a/rust/src/crates/journal-core/src/file/file_payload.rs b/rust/src/crates/journal-core/src/file/file_payload.rs index d0efe78a..712b02ef 100644 --- a/rust/src/crates/journal-core/src/file/file_payload.rs +++ b/rust/src/crates/journal-core/src/file/file_payload.rs @@ -460,6 +460,7 @@ impl JournalFile { let context = self.data_payload_read_context(); let mut decompression_buffer = Vec::new(); let mut object_offset = hash_table.hash_item_ref(hash).head_hash_offset; + let mut chain = super::chain_guard::ChainGuard::default(); while let Some(offset) = object_offset { let result = self.data_lookup_result_at( @@ -473,6 +474,7 @@ impl JournalFile { if let Some(match_value) = result.match_value { return Ok(Some((offset, match_value))); } + chain.advance(offset, result.next_hash_offset, false)?; object_offset = result.next_hash_offset; } diff --git a/rust/src/crates/journal-core/src/file/file_tests.rs b/rust/src/crates/journal-core/src/file/file_tests.rs index 13af91d2..8dd7b4b9 100644 --- a/rust/src/crates/journal-core/src/file/file_tests.rs +++ b/rust/src/crates/journal-core/src/file/file_tests.rs @@ -5,6 +5,279 @@ use std::path::PathBuf; use std::process::Command; use tempfile::TempDir; +#[test] +fn custom_object_implementations_can_omit_the_wire_type_hint() { + struct Custom; + impl JournalObject<&[u8]> for Custom { + fn from_data(_: &[u8], _: bool) -> Option { + Some(Self) + } + } + assert!(>::EXPECTED_TYPE.is_none()); +} + +#[test] +fn writer_counted_arrays_reject_revisit_before_mutation() { + use std::num::NonZeroU64; + for kind in 0..3 { + for descending in [false, true] { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("system.journal"); + let repo = crate::repository::File::from_path(&path).unwrap(); + let mut file = JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)) + .create::(&repo) + .unwrap(); + let mut writer = JournalWriter::new(&mut file, 1, test_uuid(2)).unwrap(); + let a = writer + .allocate_new_array(&mut file, NonZeroU64::new(4).unwrap()) + .unwrap(); + let b = writer + .allocate_new_array(&mut file, NonZeroU64::new(8).unwrap()) + .unwrap(); + file.offset_array_mut(a, None) + .unwrap() + .header + .next_offset_array = Some(b); + file.offset_array_mut(b, None) + .unwrap() + .header + .next_offset_array = Some(a); + let head = if descending { b } else { a }; + let missing_tail = NonZeroU64::new(1024).unwrap(); + let failed = match kind { + 0 => writer + .append_to_data_entry_array(&mut file, head, missing_tail, 13) + .is_err(), + 1 => writer.find_entry_array_tail(&file, head, 13).is_err(), + _ => writer + .compute_entry_array_tail_entries(&file, head, missing_tail, 13) + .is_err(), + }; + assert!(failed, "required revisit selected a repeated array"); + assert!( + file.offset_array_ref(a) + .unwrap() + .get(1, 4) + .unwrap() + .is_none() + ); + } + } +} + +#[test] +fn recovered_tail_cursor_requalification_tracks_its_new_position() { + use crate::file::Mmap; + use crate::file::filter::FilterExpr; + use std::io::{Seek, SeekFrom, Write}; + + for compact in [false, true] { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("system.journal"); + let repo = crate::repository::File::from_path(&path).unwrap(); + let mut writable = JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)) + .with_compact(compact) + .create::(&repo) + .unwrap(); + let mut writer = JournalWriter::new(&mut writable, 1, test_uuid(2)).unwrap(); + for i in 1..=3 { + writer + .add_entry(&mut writable, &[b"COMMON=all"], i * 1000, i) + .unwrap(); + } + let mut entries = Vec::new(); + writable.entry_offsets(&mut entries).unwrap(); + drop(writer); + drop(writable); + let mut file = JournalFile::::open_path(&path, 64 * 1024).unwrap(); + file.set_reader_recovery(true); + let data = file + .find_data_offset(file.hash(b"COMMON=all"), b"COMMON=all") + .unwrap() + .unwrap(); + let cursor = file + .reader_data_cursor(data) + .unwrap() + .unwrap() + .tail(&file) + .unwrap(); + let mut disk = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + disk.seek(SeekFrom::Start(entries[2].get())).unwrap(); + disk.write_all(&[0; 16]).unwrap(); + assert!( + file.recover_reader_tail(&JournalError::ObjectExceedsFileBounds) + .unwrap() + ); + assert_eq!(file.recovery_status().recovered_entries, 2); + let cursor = cursor.for_reader(&file).unwrap().unwrap(); + let mut expression = FilterExpr::Match(data, cursor); + assert_eq!( + expression.previous(&file, entries[1]).unwrap(), + Some(entries[1]) + ); + } +} + +// Synthetic reconstruction exercises consistency after injected failures. It +// does not establish an upstream publication sequence or a repair contract. +fn filter_repair_retry( + compact: bool, + installed: bool, + backward: bool, + compose: bool, + tail_error: bool, + zero_data: bool, +) { + use crate::file::Mmap; + use crate::file::filter::{FilterExpr, JournalFilter, LogicalOp}; + use std::io::{Seek, SeekFrom, Write}; + + let dir = TempDir::new().unwrap(); + let path = dir.path().join("system.journal"); + let repo = crate::repository::File::from_path(&path).unwrap(); + let mut writable = JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)) + .with_compact(compact) + .with_data_hash_table_buckets(64) + .with_field_hash_table_buckets(16) + .create::(&repo) + .unwrap(); + let mut writer = JournalWriter::new(&mut writable, 1, test_uuid(2)).unwrap(); + writer + .add_entry(&mut writable, &[b"COMMON=all"], 1_000, 1) + .unwrap(); + writer + .add_entry(&mut writable, &[b"COMMON=all", b"TARGET=yes"], 2_000, 2) + .unwrap(); + let mut entries = Vec::new(); + writable.entry_offsets(&mut entries).unwrap(); + drop(writer); + drop(writable); + let original = std::fs::read(&path).unwrap(); + let mut file = JournalFile::::open_path(&path, 64 * 1024).unwrap(); + file.set_reader_recovery(true); + let target = file + .find_data_offset(file.hash(b"TARGET=yes"), b"TARGET=yes") + .unwrap() + .unwrap(); + assert!(target.get() % 4096 < 4096 - 16); + let mut filter = JournalFilter::default(); + filter.add_match(b"COMMON=all"); + filter.set_operation(&file, LogicalOp::Conjunction).unwrap(); + filter.add_match(b"TARGET=yes"); + filter.set_operation(&file, LogicalOp::Conjunction).unwrap(); + + let mut installed_expression = if installed { + Some(filter.build(&file).unwrap()) + } else { + None + }; + + let mut disk = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + disk.set_len(target.get() + 8).unwrap(); + assert!( + file.recover_reader_tail(&JournalError::ObjectExceedsFileBounds) + .unwrap() + ); + assert!(file.recovery_status().incomplete()); + if let Some(expression) = installed_expression.as_mut() { + assert!(expression.refresh_recovery(&file).is_err()); + } else { + assert!(filter.build(&file).is_err()); + } + + let mut partial = original.clone(); + let start = target.get() as usize; + partial[start..start + 16].fill(0); + let common = file + .find_data_offset(file.hash(b"COMMON=all"), b"COMMON=all") + .unwrap() + .unwrap(); + if zero_data { + let common = common.get() as usize; + partial[common + 40..common + 64].fill(0); + } + disk.seek(SeekFrom::Start(0)).unwrap(); + disk.write_all(&partial).unwrap(); + if let Some(expression) = installed_expression.as_mut() { + assert!(expression.refresh_recovery(&file).is_err()); + } else { + assert!(filter.build(&file).is_err()); + } + assert!(!file.recovery_status().incomplete()); + disk.seek(SeekFrom::Start(target.get())).unwrap(); + disk.write_all(&original[start..start + 16]).unwrap(); + if zero_data { + let start = common.get() as usize; + disk.seek(SeekFrom::Start(common.get())).unwrap(); + disk.write_all(&original[start..start + 64]).unwrap(); + } + let mut expression = installed_expression.unwrap_or_else(|| filter.build(&file).unwrap()); + if compose { + let mut clean = JournalFilter::default(); + clean.add_match(b"COMMON=all"); + expression = FilterExpr::Conjunction(vec![clean.build(&file).unwrap(), expression]); + } + if tail_error { + let common = file + .find_data_offset(file.hash(b"COMMON=all"), b"COMMON=all") + .unwrap() + .unwrap(); + let array = file + .data_ref(common) + .unwrap() + .header + .entry_array_offset + .unwrap(); + disk.seek(SeekFrom::Start(array.get() + 8)).unwrap(); + disk.write_all(&8u64.to_le_bytes()).unwrap(); + assert!(expression.previous(&file, NonZeroU64::MAX).is_err()); + let start = array.get() as usize + 8; + disk.seek(SeekFrom::Start(array.get() + 8)).unwrap(); + disk.write_all(&original[start..start + 8]).unwrap(); + let mut append = JournalFile::::open_for_append(&repo, 64 * 1024).unwrap(); + let mut writer = JournalWriter::new(&mut append, 1, test_uuid(2)).unwrap(); + writer + .add_entry(&mut append, &[b"COMMON=all", b"TARGET=yes"], 3_000, 3) + .unwrap(); + drop(writer); + drop(append); + } + let found = if backward { + expression.previous(&file, NonZeroU64::MAX) + } else { + expression.next(&file, entries[1]) + }; + assert_eq!( + found.unwrap(), + Some(entries[1]), + "compact={compact}, installed={installed}, backward={backward}, compose={compose}, tail_error={tail_error}" + ); +} + +#[test] +fn pending_filter_retries_all_branches_after_repair_clears_recovery() { + for compact in [false, true] { + for installed in [false, true] { + for backward in [false, true] { + for compose in [false, true] { + for zero_data in [false, true] { + filter_repair_retry( + compact, installed, backward, compose, false, zero_data, + ); + } + } + } + } + } +} + +#[test] +fn recovered_filter_tail_retry_preserves_captured_populations() { + for compact in [false, true] { + filter_repair_retry(compact, true, true, true, true, false); + } +} + #[derive(Clone, Copy, Debug)] struct ExpectedSanitizedHeader { header_size: u64, diff --git a/rust/src/crates/journal-core/src/file/filter.rs b/rust/src/crates/journal-core/src/file/filter.rs index 5191bdc2..6f94fd08 100644 --- a/rust/src/crates/journal-core/src/file/filter.rs +++ b/rust/src/crates/journal-core/src/file/filter.rs @@ -1,6 +1,9 @@ use super::mmap::MemoryMap; use crate::error::{JournalError, Result}; -use crate::file::{file::JournalFile, offset_array::InlinedCursor}; +use crate::file::{ + file::JournalFile, + offset_array::{Direction, InlinedCursor}, +}; use std::num::NonZeroU64; #[derive(Clone, Debug)] @@ -12,11 +15,88 @@ pub enum FilterExpr { } impl FilterExpr { + pub(super) fn invalidate_position(&mut self) { + self.visit_cursors(&mut |cursor| cursor.filter_direction = None); + } + + pub(super) fn refresh_recovery(&mut self, file: &JournalFile) -> Result<()> { + Self::refresh_recovery_batch(std::slice::from_mut(self), file) + } + + fn refresh_recovery_batch( + expressions: &mut [Self], + file: &JournalFile, + ) -> Result<()> { + file.filter_recovery_seen.set(true); + for expression in expressions.iter_mut() { + expression.visit_cursors(&mut |cursor| cursor.recovery_pending = true); + } + loop { + let before = (file.recovery_status(), file.recovered_tail_metadata()); + for expression in expressions.iter_mut() { + expression.refresh_recovery_pass(file)?; + } + if before == (file.recovery_status(), file.recovered_tail_metadata()) { + for expression in expressions.iter_mut() { + expression.visit_cursors(&mut |cursor| cursor.recovery_pending = false); + } + return Ok(()); + } + } + } + + fn visit_cursors(&mut self, visit: &mut impl FnMut(&mut InlinedCursor)) { + match self { + Self::None => {} + Self::Match(_, cursor) => visit(cursor), + Self::Conjunction(children) | Self::Disjunction(children) => { + for child in children { + child.visit_cursors(visit); + } + } + } + } + + fn any_cursor(&self, predicate: &impl Fn(&InlinedCursor) -> bool) -> bool { + match self { + Self::None => false, + Self::Match(_, cursor) => predicate(cursor), + Self::Conjunction(children) | Self::Disjunction(children) => { + children.iter().any(|child| child.any_cursor(predicate)) + } + } + } + + fn refresh_recovery_pass(&mut self, file: &JournalFile) -> Result<()> { + match self { + Self::None => {} + Self::Match(offset, current) => { + let mut cursor = match file.reader_data_cursor(*offset)? { + Some(cursor) => cursor, + None => { + let mut cursor = current.clone().empty(); + cursor.recovered_population = file.reader_recovery_active(); + cursor + } + }; + cursor.recovery_pending = true; + cursor.position_pending = true; + *current = cursor; + } + Self::Conjunction(children) | Self::Disjunction(children) => { + for child in children { + child.refresh_recovery_pass(file)?; + } + } + } + Ok(()) + } pub fn head(&mut self) -> &mut Self { match self { FilterExpr::None => (), FilterExpr::Match(_, ic) => { *ic = ic.head(); + ic.position_pending = false; } FilterExpr::Conjunction(filter_exprs) => { for filter_expr in filter_exprs.iter_mut() { @@ -34,6 +114,21 @@ impl FilterExpr { } pub fn tail(&mut self, journal_file: &JournalFile) -> Result<&mut Self> { + if !journal_file.filter_recovery_seen.get() { + self.tail_inner(journal_file)?; + return Ok(self); + } + if journal_file.filter_recovery_seen.get() + && self.any_cursor(&|cursor| cursor.recovery_pending) + { + self.refresh_recovery(journal_file)?; + } + self.tail_inner(journal_file)?; + self.visit_cursors(&mut |cursor| cursor.position_pending = false); + Ok(self) + } + + fn tail_inner(&mut self, journal_file: &JournalFile) -> Result<()> { match self { FilterExpr::None => {} FilterExpr::Match(_, ic) => { @@ -41,17 +136,98 @@ impl FilterExpr { } FilterExpr::Conjunction(filter_exprs) => { for filter_expr in filter_exprs.iter_mut() { - filter_expr.tail(journal_file)?; + filter_expr.tail_inner(journal_file)?; } } FilterExpr::Disjunction(filter_exprs) => { for filter_expr in filter_exprs.iter_mut() { - filter_expr.tail(journal_file)?; + filter_expr.tail_inner(journal_file)?; } } } - Ok(self) + Ok(()) + } + + pub(super) fn prepare_recovery( + &mut self, + file: &JournalFile, + backward: bool, + ) -> Result<()> { + if !file.filter_recovery_seen.get() { + return Ok(()); + } + if self.any_cursor(&|cursor| cursor.recovery_pending) { + self.refresh_recovery(file)?; + } + if self.any_cursor(&|cursor| cursor.position_pending) { + if backward { + self.tail_inner(file)?; + self.visit_cursors(&mut |cursor| cursor.position_pending = false); + } else { + self.head(); + } + } + Ok(()) + } + + pub(super) fn seek_realtime( + &mut self, + file: &JournalFile, + needle: NonZeroU64, + direction: Direction, + ) -> Result> { + self.prepare_recovery(file, false)?; + self.invalidate_position(); + self.seek_realtime_inner(file, needle, direction) + } + + fn seek_realtime_inner( + &mut self, + file: &JournalFile, + mut needle: NonZeroU64, + direction: Direction, + ) -> Result> { + match self { + Self::None => Ok(None), + Self::Match(_, cursor) => cursor.seek_realtime(file, needle, direction), + Self::Conjunction(children) => { + let initial = needle; + for i in 0..children.len() { + let index = if direction == Direction::Forward { + i + } else { + children.len() - 1 - i + }; + let Some(value) = + children[index].seek_realtime_inner(file, needle, direction)? + else { + return Ok(None); + }; + needle = value; + } + if needle == initial { + return Ok(Some(needle)); + } + match direction { + Direction::Forward => self.next_inner(file, needle), + Direction::Backward => self.previous_inner(file, needle), + } + } + Self::Disjunction(children) => { + let mut best = None; + for child in children { + if let Some(value) = child.seek_realtime_inner(file, needle, direction)? { + best = Some(match (best, direction) { + (Some(old), Direction::Forward) => value.min(old), + (Some(old), Direction::Backward) => value.max(old), + (None, _) => value, + }); + } + } + Ok(best) + } + } } // Returns the offset of the next matching entry, if any, with an offset @@ -60,10 +236,22 @@ impl FilterExpr { &mut self, journal_file: &JournalFile, needle_offset: NonZeroU64, + ) -> Result> { + self.prepare_recovery(journal_file, false)?; + self.next_inner(journal_file, needle_offset) + } + + fn next_inner( + &mut self, + journal_file: &JournalFile, + needle_offset: NonZeroU64, ) -> Result> { match self { FilterExpr::None => Ok(None), - FilterExpr::Match(_, ic) => ic.next_until(journal_file, needle_offset), + FilterExpr::Match(_, ic) => { + ic.prepare_filter_direction(journal_file, Direction::Forward)?; + ic.next_until(journal_file, needle_offset) + } FilterExpr::Conjunction(filter_exprs) => { let mut needle_offset = needle_offset; @@ -71,7 +259,7 @@ impl FilterExpr { let previous_offset = needle_offset; for fe in filter_exprs.iter_mut() { - if let Some(new_offset) = fe.next(journal_file, needle_offset)? { + if let Some(new_offset) = fe.next_inner(journal_file, needle_offset)? { needle_offset = new_offset; } else { return Ok(None); @@ -87,7 +275,7 @@ impl FilterExpr { let mut best_offset: Option = None; for fe in filter_exprs.iter_mut() { - if let Some(fe_offset) = fe.next(journal_file, needle_offset)? { + if let Some(fe_offset) = fe.next_inner(journal_file, needle_offset)? { best_offset = match best_offset { Some(offset) => Some(fe_offset.min(offset)), None => Some(fe_offset), @@ -106,10 +294,22 @@ impl FilterExpr { &mut self, journal_file: &JournalFile, needle_offset: NonZeroU64, + ) -> Result> { + self.prepare_recovery(journal_file, true)?; + self.previous_inner(journal_file, needle_offset) + } + + fn previous_inner( + &mut self, + journal_file: &JournalFile, + needle_offset: NonZeroU64, ) -> Result> { match self { FilterExpr::None => Ok(None), - FilterExpr::Match(_, ic) => ic.previous_until(journal_file, needle_offset), + FilterExpr::Match(_, ic) => { + ic.prepare_filter_direction(journal_file, Direction::Backward)?; + ic.previous_until(journal_file, needle_offset) + } FilterExpr::Conjunction(filter_exprs) => { let mut needle_offset = needle_offset; @@ -117,7 +317,7 @@ impl FilterExpr { let previous_offset = needle_offset; for fe in filter_exprs.iter_mut().rev() { - if let Some(new_offset) = fe.previous(journal_file, needle_offset)? { + if let Some(new_offset) = fe.previous_inner(journal_file, needle_offset)? { needle_offset = new_offset; } else { return Ok(None); @@ -133,7 +333,7 @@ impl FilterExpr { let mut best_offset: Option = None; for fe in filter_exprs.iter_mut() { - if let Some(fe_offset) = fe.previous(journal_file, needle_offset)? { + if let Some(fe_offset) = fe.previous_inner(journal_file, needle_offset)? { best_offset = match best_offset { Some(offset) => Some(fe_offset.max(offset)), None => Some(fe_offset), @@ -187,7 +387,7 @@ impl JournalFilter { let Some(offset) = journal_file.find_data_offset(hash, data)? else { return Ok(FilterExpr::None); }; - let Some(ic) = journal_file.data_ref(offset)?.inlined_cursor() else { + let Some(ic) = journal_file.reader_data_cursor(offset)? else { return Ok(FilterExpr::None); }; Ok(FilterExpr::Match(offset, ic)) @@ -303,6 +503,13 @@ impl JournalFilter { self.commit_current(journal_file)?; self.commit_level1(); + if journal_file.reader_recovery_active() + || self.level0.iter().any(|expr| { + expr.any_cursor(&|cursor| cursor.recovery_pending || cursor.recovered_population) + }) + { + FilterExpr::refresh_recovery_batch(&mut self.level0, journal_file)?; + } self.current_matches.clear(); match self.level0.len() { 0 => Err(JournalError::MalformedFilter), diff --git a/rust/src/crates/journal-core/src/file/mmap.rs b/rust/src/crates/journal-core/src/file/mmap.rs index 4cfcd33c..c83a2fb1 100644 --- a/rust/src/crates/journal-core/src/file/mmap.rs +++ b/rust/src/crates/journal-core/src/file/mmap.rs @@ -361,6 +361,21 @@ impl WindowManager { Ok(self.file_size) } + pub(super) fn cached_file_size(&self) -> u64 { + self.file_size + } + + pub(super) fn reader_extent(&mut self) -> Result { + if self.bounds_mode == BoundsMode::LiveFile { + self.refresh_file_size() + } else if self.bounds_mode == BoundsMode::Snapshot { + self.file_size = self.retained_size.min(self.file.metadata()?.len()); + Ok(self.file_size) + } else { + Ok(self.file_size.min(self.file.metadata()?.len())) + } + } + fn ensure_cached_file_contains(&mut self, end: u64) -> Result<()> { if end <= self.file_size { return Ok(()); diff --git a/rust/src/crates/journal-core/src/file/mod.rs b/rust/src/crates/journal-core/src/file/mod.rs index ca7443fb..22b2bc1c 100644 --- a/rust/src/crates/journal-core/src/file/mod.rs +++ b/rust/src/crates/journal-core/src/file/mod.rs @@ -1,5 +1,10 @@ // Modules - keep some public for advanced usage +mod array_chain; +mod chain_guard; pub mod cursor; +mod cursor_direction; +#[doc(hidden)] +pub use cursor_direction::CursorAnchor; pub mod file; mod file_iterators; mod file_mut; @@ -17,12 +22,19 @@ mod object_compression; mod object_hash; pub mod offset_array; pub mod reader; +mod reader_recovery; +pub use reader_recovery::{RecoveryReason, RecoveryStatus}; mod row_view; pub mod sigbus; mod value_guard; pub mod writer; mod writer_entry_arrays; mod writer_seal; +mod zstd_stream; +#[doc(hidden)] +pub use object_compression::decompress_zstd_payload as decompress_zstd_data; +#[doc(hidden)] +pub use zstd_stream::copy_zstd_stream; // Core functionality pub use file::{ diff --git a/rust/src/crates/journal-core/src/file/object.rs b/rust/src/crates/journal-core/src/file/object.rs index 1d925a09..482e9785 100644 --- a/rust/src/crates/journal-core/src/file/object.rs +++ b/rust/src/crates/journal-core/src/file/object.rs @@ -17,6 +17,8 @@ pub use super::object_hash::{ }; pub trait JournalObject: Sized { + /// Expected wire type for typed file reads; custom parsers may omit the hint. + const EXPECTED_TYPE: Option = None; /// Create a new journal object from a byte slice fn from_data(data: B, is_compact: bool) -> Option; } @@ -421,6 +423,7 @@ pub struct FieldObject { } impl JournalObject for FieldObject { + const EXPECTED_TYPE: Option = Some(ObjectType::Field); fn from_data(data: B, _is_compact: bool) -> Option { let (header, payload) = zerocopy::Ref::from_prefix(data).ok()?; Some(FieldObject { header, payload }) @@ -475,6 +478,7 @@ pub struct OffsetArrayObject { } impl OffsetArrayObject { + #[inline(always)] pub fn capacity(&self) -> usize { match &self.items { OffsetsType::Regular(offsets) => offsets.len(), @@ -512,14 +516,14 @@ impl OffsetArrayObject { match &self.items { OffsetsType::Regular(s) => { - offsets.extend(s[start_index..len].iter().filter_map(|&opt| opt)); + for &offset in &s[start_index..len] { + offsets.push(offset.ok_or(JournalError::InvalidOffset)?); + } } OffsetsType::Compact(s) => { - offsets.extend( - s[start_index..len] - .iter() - .filter_map(|&opt| opt.map(NonZeroU64::from)), - ); + for &offset in &s[start_index..len] { + offsets.push(NonZeroU64::from(offset.ok_or(JournalError::InvalidOffset)?)); + } } } @@ -547,6 +551,8 @@ impl std::fmt::Debug for OffsetArrayObject { } impl JournalObject for OffsetArrayObject { + const EXPECTED_TYPE: Option = Some(ObjectType::EntryArray); + #[inline(always)] fn from_data(data: B, is_compact: bool) -> Option { let (header_data, items_data) = data .split_at(std::mem::size_of::()) @@ -713,6 +719,8 @@ impl std::fmt::Debug for EntryObject { } impl JournalObject for EntryObject { + const EXPECTED_TYPE: Option = Some(ObjectType::Entry); + #[inline(always)] fn from_data(data: B, is_compact: bool) -> Option { let (header_data, items_data) = data .split_at(std::mem::size_of::()) @@ -852,6 +860,8 @@ impl std::fmt::Debug for DataObject { } impl JournalObject for DataObject { + const EXPECTED_TYPE: Option = Some(ObjectType::Data); + #[inline(always)] fn from_data(data: B, is_compact: bool) -> Option { let (header_data, remaining_data) = data .split_at(std::mem::size_of::()) @@ -1098,6 +1108,7 @@ impl std::fmt::Debug for TagObject { } impl JournalObject for TagObject { + const EXPECTED_TYPE: Option = Some(ObjectType::Tag); fn from_data(data: B, _is_compact: bool) -> Option { let header = zerocopy::Ref::from_bytes(data).ok()?; Some(TagObject { header }) diff --git a/rust/src/crates/journal-core/src/file/object_compression.rs b/rust/src/crates/journal-core/src/file/object_compression.rs index 5a76f31d..db42bb3c 100644 --- a/rust/src/crates/journal-core/src/file/object_compression.rs +++ b/rust/src/crates/journal-core/src/file/object_compression.rs @@ -131,7 +131,8 @@ fn prepare_zstd_output_buffer(buf: &mut Vec, uncompressed_size: usize) -> Re Ok(()) } -pub(super) fn decompress_zstd_payload(payload: &[u8], buf: &mut Vec) -> Result { +#[doc(hidden)] +pub fn decompress_zstd_payload(payload: &[u8], buf: &mut Vec) -> Result { if let Ok(len) = decompress_zstd_payload_native(payload, buf) { return Ok(len); } @@ -155,10 +156,33 @@ fn decompress_zstd_payload_native(payload: &[u8], buf: &mut Vec) -> Result) -> Result { - use ruzstd::decoding::StreamingDecoder; + decompress_zstd_payload_streaming_with_cap(payload, buf, MAX_UNCOMPRESSED_DATA_OBJECT_SIZE) +} - let decoder = StreamingDecoder::new(payload).map_err(|_| JournalError::DecompressorError)?; - read_limited_to_end(decoder, buf) +fn decompress_zstd_payload_streaming_with_cap( + payload: &[u8], + buf: &mut Vec, + cap: usize, +) -> Result { + struct BoundedOutput<'a> { + buf: &'a mut Vec, + cap: usize, + } + impl std::io::Write for BoundedOutput<'_> { + fn write(&mut self, data: &[u8]) -> std::io::Result { + extend_decompression_buffer(self.buf, data, self.cap) + .map_err(|_| std::io::Error::other("decompressed DATA exceeds limit"))?; + Ok(data.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + buf.clear(); + if super::zstd_stream::copy_zstd_stream(payload, &mut BoundedOutput { buf, cap }).is_err() { + return clear_decompression_error(buf); + } + Ok(buf.len()) } pub(super) fn decompress_lz4_payload(payload: &[u8], buf: &mut Vec) -> Result { @@ -188,6 +212,67 @@ pub(super) fn decompress_xz_payload(payload: &[u8], buf: &mut Vec) -> Result mod tests { use super::*; use std::io::Cursor; + use std::io::Write; + + fn frame(payload: &[u8], known_size: bool) -> Vec { + let mut encoder = zstd::stream::Encoder::new(Vec::new(), 1).unwrap(); + encoder.include_checksum(true).unwrap(); + if known_size { + encoder + .set_pledged_src_size(Some(payload.len() as u64)) + .unwrap(); + } + encoder.write_all(payload).unwrap(); + encoder.finish().unwrap() + } + + #[test] + fn zstd_complete_streams_and_failure_cleanup() { + let first = b"MESSAGE=first\0\xff"; + let second = b"|second\n"; + let expected = [first.as_slice(), second.as_slice()].concat(); + for known in [true, false] { + let mut compressed = frame(first, known); + compressed.extend_from_slice(&0x184d2a50u32.to_le_bytes()); + compressed.extend_from_slice(&3u32.to_le_bytes()); + compressed.extend_from_slice(b"pad"); + compressed.extend(frame(second, known)); + let mut out = vec![42; 7]; + let len = decompress_zstd_payload(&compressed, &mut out).unwrap(); + assert_eq!(len, expected.len()); + assert_eq!(out, expected); + for malformed in [ + compressed[..compressed.len() - 1].to_vec(), + [compressed.as_slice(), b"garbage"].concat(), + ] { + assert!(decompress_zstd_payload(&malformed, &mut out).is_err()); + assert!(out.is_empty()); + } + assert!( + decompress_zstd_payload_streaming_with_cap( + &compressed, + &mut out, + expected.len() - 1 + ) + .is_err() + ); + assert!(out.is_empty()); + assert_eq!( + decompress_zstd_payload_streaming_with_cap(&compressed, &mut out, expected.len()) + .unwrap(), + expected.len() + ); + } + let mut bad_checksum = frame(first, false); + *bad_checksum.last_mut().unwrap() ^= 1; + let mut out = vec![42]; + assert!(decompress_zstd_payload(&bad_checksum, &mut out).is_err()); + assert!(out.is_empty()); + for malformed in [&b""[..], &b"\x50\x2a\x4d\x18\x05\x00\x00\x00x"[..]] { + assert!(decompress_zstd_payload(malformed, &mut out).is_err()); + assert!(out.is_empty()); + } + } #[test] fn native_zstd_decompresses_into_reserved_vec() { diff --git a/rust/src/crates/journal-core/src/file/object_hash.rs b/rust/src/crates/journal-core/src/file/object_hash.rs index ae16eab7..44d1140e 100644 --- a/rust/src/crates/journal-core/src/file/object_hash.rs +++ b/rust/src/crates/journal-core/src/file/object_hash.rs @@ -111,6 +111,7 @@ impl HashTableMut for FieldHashTable { // Implement JournalObject for DataHashTable impl JournalObject for DataHashTable { + const EXPECTED_TYPE: Option = Some(ObjectType::DataHashTable); fn from_data(data: B, _is_compact: bool) -> Option { let (header_data, items_data) = data.split_at(std::mem::size_of::()).ok()?; @@ -135,6 +136,7 @@ impl JournalObjectMut for DataHashTable { // Implement JournalObject for FieldHashTable impl JournalObject for FieldHashTable { + const EXPECTED_TYPE: Option = Some(ObjectType::FieldHashTable); fn from_data(data: B, _is_compact: bool) -> Option { let (header_data, items_data) = data.split_at(std::mem::size_of::()).ok()?; diff --git a/rust/src/crates/journal-core/src/file/offset_array.rs b/rust/src/crates/journal-core/src/file/offset_array.rs index d1125b14..0a838406 100644 --- a/rust/src/crates/journal-core/src/file/offset_array.rs +++ b/rust/src/crates/journal-core/src/file/offset_array.rs @@ -1,3 +1,5 @@ +use super::array_chain::reject_array_revisit; +use super::file::validate_offset_alignment; use super::mmap::MemoryMap; use crate::error::{JournalError, Result}; use crate::file::JournalFile; @@ -5,6 +7,7 @@ use std::num::{NonZeroU64, NonZeroUsize}; use std::sync::Arc; #[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[cfg_attr(feature = "allocative", derive(allocative::Allocative))] pub enum Direction { Forward, Backward, @@ -15,7 +18,8 @@ pub enum Direction { #[cfg_attr(feature = "allocative", derive(allocative::Allocative))] pub struct Node { offset: NonZeroU64, - next_offset: Option, + // Array offsets are aligned; bit zero retains exceptional link ordering. + head_and_order: NonZeroU64, capacity: NonZeroUsize, // Number of items remaining in this array and subsequent arrays remaining_items: NonZeroUsize, @@ -27,14 +31,17 @@ impl Node { journal_file: &JournalFile, offset: NonZeroU64, remaining_items: NonZeroUsize, + head: NonZeroU64, + unordered: bool, ) -> Result { - let array = journal_file.offset_array_ref(offset)?; + validate_offset_alignment(head)?; + let array = journal_file.reader_offset_array_ref(offset)?; let capacity = NonZeroUsize::new(array.capacity()).ok_or(JournalError::EmptyOffsetArrayNode)?; Ok(Self { offset, - next_offset: array.header.next_offset_array, + head_and_order: NonZeroU64::new(head.get() | u64::from(unordered)).unwrap(), capacity, remaining_items, }) @@ -55,25 +62,37 @@ impl Node { self.capacity.min(self.remaining_items) } - /// Check if this array has a next array in the chain + /// Check whether the declared population requires another array. pub fn has_next(&self) -> bool { - self.next_offset.is_some() && self.remaining_items > self.len() + self.remaining_items > self.len() } /// Get the next array in the chain, if any pub fn next(&self, journal_file: &JournalFile) -> Result> { - if !self.has_next() { + if self.remaining_items <= self.len() { return Ok(None); } - let next_offset = self.next_offset.unwrap(); + let next_offset = journal_file + .reader_offset_array_ref(self.offset)? + .header + .next_offset_array + .ok_or(JournalError::ObjectExceedsFileBounds)?; let remaining_items = { let n = self.remaining_items.get().saturating_sub(self.len().get()); NonZeroUsize::new(n).ok_or(JournalError::EmptyOffsetArrayNode)? }; - let node = Self::new(journal_file, next_offset, remaining_items); - - Some(node).transpose() + let head = NonZeroU64::new(self.head_and_order.get() & !1).unwrap(); + let unordered = self.head_and_order.get() & 1 != 0 || next_offset <= self.offset; + if unordered { + reject_array_revisit(head, self.offset, next_offset, |at| { + Ok(journal_file + .reader_offset_array_ref(at)? + .header + .next_offset_array) + })?; + } + Self::new(journal_file, next_offset, remaining_items, head, unordered).map(Some) } /// Get an item at the specified index @@ -82,12 +101,22 @@ impl Node { journal_file: &JournalFile, index: usize, ) -> Result> { + self.required_offset(journal_file, index).map(Some) + } + + fn required_offset( + &self, + journal_file: &JournalFile, + index: usize, + ) -> Result { if index >= self.len().get() { return Err(JournalError::InvalidOffsetArrayIndex); } - let array = journal_file.offset_array_ref(self.offset)?; - array.get(index, self.remaining_items.get()) + let array = journal_file.reader_offset_array_ref(self.offset)?; + array + .get(index, self.remaining_items.get())? + .ok_or(JournalError::InvalidOffset) } /// Returns the first index where the predicate returns false, or array length if @@ -161,11 +190,13 @@ impl Node { impl std::fmt::Debug for Node { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let next_offset = self.next_offset.map(|x| x.get()).unwrap_or(0); - f.debug_struct("Node") .field("offset", &format!("0x{:x}", self.offset)) - .field("next_offset", &format!("0x{:x}", next_offset)) + .field( + "head_offset", + &format!("0x{:x}", self.head_and_order.get() & !1), + ) + .field("unordered", &(self.head_and_order.get() & 1 != 0)) .field("capacity", &self.capacity) .field("len", &self.len()) .field("remaining_items", &self.remaining_items) @@ -201,7 +232,13 @@ impl List { /// Get the head array of this chain pub fn head(&self, journal_file: &JournalFile) -> Result { - Node::new(journal_file, self.head_offset, self.total_items) + Node::new( + journal_file, + self.head_offset, + self.total_items, + self.head_offset, + false, + ) } /// Get the tail array of this list by traversing from head to tail @@ -335,7 +372,7 @@ impl List { loop { { - let array = journal_file.offset_array_ref(node.offset())?; + let array = journal_file.reader_offset_array_ref(node.offset())?; let remaining_items = node.remaining_items.get(); array.collect_offsets(0, remaining_items, offsets)?; } @@ -429,7 +466,13 @@ impl Cursor { debug_assert!(offset_array_list.total_items >= remaining_items); // Verify the array exists - let array = Node::new(journal_file, array_offset, remaining_items)?; + let array = Node::new( + journal_file, + array_offset, + remaining_items, + offset_array_list.head_offset, + array_offset != offset_array_list.head_offset, + )?; // Verify the index is valid if array_index >= array.len().get() { @@ -469,7 +512,13 @@ impl Cursor { if let Some(node) = self.node { return Ok(node); } - Node::new(journal_file, self.array_offset, self.remaining_items) + Node::new( + journal_file, + self.array_offset, + self.remaining_items, + self.list.head_offset, + self.array_offset != self.list.head_offset, + ) } pub fn value(&self, journal_file: &JournalFile) -> Result> { @@ -488,9 +537,7 @@ impl Cursor { } let node = self.node(journal_file)?; - let Some(value) = node.get(journal_file, self.array_index)? else { - return Ok(None); - }; + let value = node.required_offset(journal_file, self.array_index)?; self.node = Some(node); self.cached_value = Some(value); Ok(Some((self, value))) @@ -534,6 +581,16 @@ impl Cursor { })); } + self.next_array(journal_file, &array_node) + } + + #[cold] + #[inline(never)] + fn next_array( + &self, + journal_file: &JournalFile, + array_node: &Node, + ) -> Result> { if !array_node.has_next() { return Ok(None); } @@ -612,14 +669,14 @@ impl Cursor { // Copy from position in the current array { - let array = journal_file.offset_array_ref(node.offset())?; + let array = journal_file.reader_offset_array_ref(node.offset())?; let remaining_items = node.remaining_items.get(); array.collect_offsets(self.array_index, remaining_items, offsets)?; } // Copy from subsequent arrays while let Some(next_node) = node.next(journal_file)? { - let array = journal_file.offset_array_ref(next_node.offset())?; + let array = journal_file.reader_offset_array_ref(next_node.offset())?; let remaining_items = next_node.remaining_items.get(); array.collect_offsets(0, remaining_items, offsets)?; node = next_node; @@ -645,22 +702,98 @@ pub struct InlinedCursor { inlined_offset: NonZeroU64, cursor: Option, at_inlined_offset: bool, + pub(super) recovery_pending: bool, + pub(super) recovered_population: bool, + pub(super) position_pending: bool, + pub(super) filter_direction: Option, } impl InlinedCursor { + pub(super) fn prepare_filter_direction( + &mut self, + file: &JournalFile, + direction: Direction, + ) -> Result<()> { + if self.filter_direction != Some(direction) { + *self = match direction { + Direction::Forward => self.head(), + Direction::Backward => self.tail(file)?, + }; + } + Ok(()) + } + + #[doc(hidden)] + pub fn for_reader( + mut self, + journal_file: &JournalFile, + ) -> Result> { + let Some(last) = journal_file.recovered_last_entry() else { + return Ok(Some(self)); + }; + journal_file.filter_recovery_seen.set(true); + self.recovered_population = true; + let Some(last) = last else { + return Ok(Some(self.empty())); + }; + if self.inlined_offset > last { + return Ok(Some(self.empty())); + } + let cursor = if let Some(cursor) = self.cursor { + let list = cursor.list; + let count = journal_file.recovered_posting_count( + list.head_offset, + list.total_items.get(), + self.inlined_offset, + )?; + std::num::NonZeroUsize::new(count) + .map(|count| List::new(list.head_offset, count).cursor_head()) + } else { + None + }; + let mut result = Self::new(self.inlined_offset, cursor); + result.recovery_pending = self.recovery_pending; + result.recovered_population = true; + result.position_pending = self.position_pending; + Ok(Some(result)) + } + pub fn new(inlined_offset: NonZeroU64, cursor: Option) -> Self { Self { inlined_offset, cursor, at_inlined_offset: true, + recovery_pending: false, + recovered_population: false, + position_pending: false, + filter_direction: Some(Direction::Forward), } } + pub(super) fn empty(mut self) -> Self { + self.cursor = None; + self.at_inlined_offset = false; + self + } + + fn is_empty(&self) -> bool { + !self.at_inlined_offset && self.cursor.is_none() + } + pub fn head(&self) -> Self { + if self.is_empty() { + let mut result = self.clone(); + result.filter_direction = Some(Direction::Forward); + return result; + } Self { inlined_offset: self.inlined_offset, cursor: self.cursor.as_ref().map(Cursor::head), at_inlined_offset: true, + recovery_pending: self.recovery_pending, + recovered_population: self.recovered_population, + position_pending: self.position_pending, + filter_direction: Some(Direction::Forward), } } @@ -674,6 +807,7 @@ impl InlinedCursor { result.at_inlined_offset = false; } + result.filter_direction = Some(Direction::Backward); Ok(result) } @@ -685,6 +819,10 @@ impl InlinedCursor { inlined_offset: self.inlined_offset, cursor: self.cursor.clone(), at_inlined_offset: false, + recovery_pending: self.recovery_pending, + recovered_population: self.recovered_population, + position_pending: self.position_pending, + filter_direction: self.filter_direction, })); } else { return Ok(None); @@ -700,6 +838,10 @@ impl InlinedCursor { inlined_offset: self.inlined_offset, cursor: next_cursor, at_inlined_offset: false, + recovery_pending: self.recovery_pending, + recovered_population: self.recovered_population, + position_pending: self.position_pending, + filter_direction: self.filter_direction, })); } else { return Ok(None); @@ -730,7 +872,7 @@ impl InlinedCursor { } } - unreachable!(); + Ok(None) } pub fn value(&self, journal_file: &JournalFile) -> Result> { @@ -744,7 +886,58 @@ impl InlinedCursor { return cursor.value(journal_file); } - unreachable!(); + Ok(None) + } + + pub(super) fn seek_realtime( + &mut self, + file: &JournalFile, + needle: NonZeroU64, + direction: Direction, + ) -> Result> { + *self = self.head(); + self.filter_direction = Some(direction); + if self.is_empty() { + return Ok(None); + } + if direction == Direction::Forward && self.inlined_offset >= needle { + return Ok(Some(self.inlined_offset)); + } + if direction == Direction::Backward && self.inlined_offset > needle { + return Ok(None); + } + let Some(cursor) = self.cursor.as_ref() else { + return Ok((direction == Direction::Backward).then_some(self.inlined_offset)); + }; + let list = cursor.list; + let mut node = list.head(file)?; + loop { + let length = node.len().get(); + let index = node.partition_point(file, 0, length, |offset| { + Ok(offset < needle || direction == Direction::Backward && offset == needle) + })?; + if direction == Direction::Backward && index == 0 { + return self.value(file); + } + let slot = if direction == Direction::Backward || index == length { + index - 1 + } else { + index + }; + let selected = Cursor::at_cached_position(list, node, slot, None, None); + let (selected, value) = selected + .materialize_value(file)? + .ok_or(JournalError::InvalidOffset)?; + self.cursor = Some(selected); + self.at_inlined_offset = false; + if index < length { + return Ok(Some(value)); + } + let Some(next) = node.next(file)? else { + return Ok((direction == Direction::Backward).then_some(value)); + }; + node = next; + } } pub fn next_until( @@ -813,17 +1006,24 @@ impl InlinedCursor { M: MemoryMap, F: Fn(NonZeroU64) -> Result, { + if self.is_empty() { + return Ok(None); + } let inlined_match = self.inlined_partition_candidate(&predicate, direction)?; let array_match = self.array_partition_candidate(journal_file, predicate, direction)?; - match (inlined_match, array_match) { + let mut result = match (inlined_match, array_match) { (Some(best), Some(array)) => { Self::best_directed_match(journal_file, best, array, direction).map(Some) } (Some(best), None) => Ok(Some(best)), (None, Some(array)) => Ok(Some(array)), (None, None) => Ok(None), + }?; + if let Some(cursor) = &mut result { + cursor.filter_direction = Some(direction); } + Ok(result) } fn inlined_partition_candidate( @@ -867,6 +1067,10 @@ impl InlinedCursor { inlined_offset: self.inlined_offset, cursor: Some(cursor), at_inlined_offset: false, + recovery_pending: self.recovery_pending, + recovered_population: self.recovered_population, + position_pending: self.position_pending, + filter_direction: self.filter_direction, })) } diff --git a/rust/src/crates/journal-core/src/file/reader.rs b/rust/src/crates/journal-core/src/file/reader.rs index 34bde47e..b0b57a9c 100644 --- a/rust/src/crates/journal-core/src/file/reader.rs +++ b/rust/src/crates/journal-core/src/file/reader.rs @@ -14,6 +14,8 @@ use std::num::NonZeroU64; pub struct JournalReader<'a, M: MemoryMap> { cursor: JournalCursor, + recovery_eof: Option<(Direction, u64)>, + filter: Option, field_iterator: Option>, field_data_iterator: Option>, @@ -107,6 +109,7 @@ impl Default for JournalReader<'_, M> { fn default() -> Self { Self { cursor: JournalCursor::new(), + recovery_eof: None, filter: None, field_iterator: None, field_data_iterator: None, @@ -119,6 +122,37 @@ impl Default for JournalReader<'_, M> { } impl<'a, M: MemoryMap> JournalReader<'a, M> { + #[doc(hidden)] + pub fn has_filter(&self) -> bool { + self.filter.is_some() || self.cursor.filter_expr.is_some() + } + + #[doc(hidden)] + pub fn location(&self) -> Location { + self.cursor.location + } + + #[doc(hidden)] + #[cold] + #[inline(never)] + pub fn recover_after_error( + &mut self, + file: &'a JournalFile, + before: Location, + err: &crate::error::JournalError, + ) -> Result { + self.drop_guards(); + match file.recover_reader_tail(err) { + Ok(true) => { + self.cursor.reset_after_recovery(file, before)?; + Ok(true) + } + result => { + self.cursor.set_location(before); + result + } + } + } pub fn dump(&self, _journal_file: &'a JournalFile) -> Result { if let Some(_filter_expr) = self.cursor.filter_expr.as_ref() { Ok(String::from("filter expr active")) @@ -128,19 +162,73 @@ impl<'a, M: MemoryMap> JournalReader<'a, M> { } pub fn set_location(&mut self, location: Location) { + self.recovery_eof = None; self.cursor.set_location(location) } + #[doc(hidden)] + pub fn seek_cursor_direction( + &mut self, + file: &'a JournalFile, + anchor: &crate::file::CursorAnchor, + direction: Direction, + ) -> Result { + self.drop_guards(); + self.recovery_eof = None; + if let Some(filter) = self.filter.as_mut() { + let expr = filter.build(file)?; + self.cursor.set_filter(expr); + self.filter = None; + } + self.cursor.seek_anchor(file, anchor, direction) + } + pub fn step(&mut self, journal_file: &'a JournalFile, direction: Direction) -> Result { self.drop_guards(); + if let Some((last_direction, count)) = self.recovery_eof { + if direction == last_direction && count == journal_file.journal_header_ref().n_entries { + return Ok(false); + } + } + if let Some(filter) = self.filter.as_mut() { let filter_expr = filter.build(journal_file)?; self.cursor.set_filter(filter_expr); self.filter = None; } - self.cursor.step(journal_file, direction) + if self.cursor.step(journal_file, direction)? { + self.recovery_eof = None; + return Ok(true); + } + self.step_recovery_eof(journal_file, direction) + } + + #[cold] + #[inline(never)] + fn step_recovery_eof( + &mut self, + file: &'a JournalFile, + direction: Direction, + ) -> Result { + if !file.reader_recovery_active() { + return Ok(false); + } + let observed = (direction, file.journal_header_ref().n_entries); + if self.recovery_eof == Some(observed) { + return Ok(false); + } + if file.refresh_reader_recovery()? { + self.cursor + .reset_after_recovery(file, self.cursor.location)?; + if self.cursor.step(file, direction)? { + self.recovery_eof = None; + return Ok(true); + } + } + self.recovery_eof = Some(observed); + Ok(false) } /// Build the pending filter expression (if any) and return it. @@ -168,22 +256,26 @@ impl<'a, M: MemoryMap> JournalReader<'a, M> { } pub fn add_match(&mut self, data: &[u8]) { + self.recovery_eof = None; self.filter.get_or_insert_default().add_match(data); } pub fn add_conjunction(&mut self, journal_file: &'a JournalFile) -> Result<()> { + self.recovery_eof = None; self.filter .get_or_insert_default() .set_operation(journal_file, LogicalOp::Conjunction) } pub fn add_disjunction(&mut self, journal_file: &'a JournalFile) -> Result<()> { + self.recovery_eof = None; self.filter .get_or_insert_default() .set_operation(journal_file, LogicalOp::Disjunction) } pub fn flush_matches(&mut self) { + self.recovery_eof = None; self.cursor.clear_filter(); self.filter = None; } diff --git a/rust/src/crates/journal-core/src/file/reader_recovery.rs b/rust/src/crates/journal-core/src/file/reader_recovery.rs new file mode 100644 index 00000000..1e607bb1 --- /dev/null +++ b/rust/src/crates/journal-core/src/file/reader_recovery.rs @@ -0,0 +1,567 @@ +use super::array_chain::reject_array_revisit; +use super::file::{JournalFile, validate_offset_alignment}; +use super::mmap::MemoryMap; +use super::object::*; +use super::value_guard::ValueGuard; +use crate::error::{JournalError, Result}; +use std::num::NonZeroU64; +use zerocopy::FromBytes; + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)] +pub enum RecoveryReason { + #[default] + None, + MissingEntry, + MissingEntryArray, +} + +/// Observed tail loss, not a certificate of journal integrity. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)] +pub struct RecoveryStatus { + pub reason: RecoveryReason, + pub first_lost_offset: u64, + pub declared_entries: u64, + pub recovered_entries: u64, +} + +impl RecoveryStatus { + pub fn incomplete(self) -> bool { + self.reason != RecoveryReason::None + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +pub(super) struct RecoveredPrefix { + pub status: RecoveryStatus, + pub last_entry: Option, + tail_seqnum: u64, + tail_realtime: u64, + tail_monotonic: u64, + tail_boot_id: [u8; 16], +} + +#[derive(Default)] +pub(super) struct ReaderRecoveryCell( + std::cell::OnceCell>>>, +); + +impl ReaderRecoveryCell { + #[inline] + pub(super) fn get(&self) -> Option { + self.0.get().and_then(|state| state.get()) + } + + pub(super) fn set(&self, value: Option) { + if let Some(state) = self.0.get() { + state.set(value); + } else if let Some(prefix) = value { + self.initialize(prefix); + } + } + + #[cold] + #[inline(never)] + fn initialize(&self, prefix: RecoveredPrefix) { + self.0 + .get_or_init(|| Box::new(std::cell::Cell::new(None))) + .set(Some(prefix)); + } +} + +struct ArrayPrefix { + next: Option, + available: usize, + capacity: usize, + partial: bool, +} + +impl JournalFile { + fn make_reader_prefix( + &self, + reason: RecoveryReason, + lost: u64, + count: u64, + last: Option, + ) -> Result { + let mut result = RecoveredPrefix { + status: RecoveryStatus { + reason, + first_lost_offset: lost, + declared_entries: self.journal_header_ref().n_entries, + recovered_entries: count, + }, + last_entry: last, + tail_seqnum: 0, + tail_realtime: 0, + tail_monotonic: 0, + tail_boot_id: [0; 16], + }; + if let Some(last) = last { + let mut bytes = [0u8; 64]; + self.read_fresh_bytes_at(last.get(), &mut bytes)?; + let entry = EntryObjectHeader::ref_from_bytes(&bytes) + .map_err(|_| JournalError::ZerocopyFailure)?; + result.tail_seqnum = entry.seqnum; + result.tail_realtime = entry.realtime; + result.tail_monotonic = entry.monotonic; + result.tail_boot_id = entry.boot_id; + } + Ok(result) + } + + #[doc(hidden)] + pub fn recovered_tail_metadata(&self) -> Option<(u64, u64, u64, [u8; 16])> { + self.recovery.get().map(|p| { + ( + p.tail_seqnum, + p.tail_realtime, + p.tail_monotonic, + p.tail_boot_id, + ) + }) + } + #[doc(hidden)] + pub fn set_reader_recovery(&mut self, enabled: bool) { + self.recovery_enabled = enabled; + if !enabled { + self.recovery.set(None); + } + } + + pub fn recovery_status(&self) -> RecoveryStatus { + self.recovery + .get() + .map_or_else(RecoveryStatus::default, |prefix| prefix.status) + } + + #[doc(hidden)] + pub fn reader_entry_count(&self) -> u64 { + self.recovery.get().map_or_else( + || self.journal_header_ref().n_entries, + |p| p.status.recovered_entries, + ) + } + + #[doc(hidden)] + pub fn reader_recovery_active(&self) -> bool { + self.recovery.get().is_some() + } + + #[doc(hidden)] + pub fn take_reader_recovery_change(&self) -> bool { + self.recovery_changed.replace(false) + } + + fn recovery_array_prefix( + &self, + offset: NonZeroU64, + extent: u64, + ) -> Result> { + validate_offset_alignment(offset)?; + let header_size = self.journal_header_ref().header_size; + if offset.get() < header_size { + return Err(JournalError::InvalidObjectLocation); + } + let arena_end = header_size + .checked_add(self.journal_header_ref().arena_size) + .ok_or(JournalError::InvalidOffset)?; + if offset + .get() + .checked_add(24) + .is_none_or(|end| end > arena_end) + { + return Err(JournalError::InvalidObjectLocation); + } + let mut bytes = [0u8; 24]; + if offset + .get() + .checked_add(bytes.len() as u64) + .is_none_or(|end| end > extent) + { + return Ok(None); + } + self.read_fresh_bytes_at(offset.get(), &mut bytes)?; + let header = OffsetArrayObjectHeader::ref_from_bytes(&bytes) + .map_err(|_| JournalError::ZerocopyFailure)?; + if bytes.iter().all(|byte| *byte == 0) { + return Ok(None); + } + if header.object_header.type_ != ObjectType::EntryArray as u8 { + return Err(JournalError::InvalidObjectType); + } + let size = header.object_header.validated_size()?; + let item_size = if self + .journal_header_ref() + .has_incompatible_flag(HeaderIncompatibleFlags::Compact) + { + 4 + } else { + 8 + }; + if size <= 24 || (size - 24) % item_size != 0 { + return Err(JournalError::InvalidObjectSize(size)); + } + let end = offset + .get() + .checked_add(size) + .ok_or(JournalError::InvalidOffset)?; + let arena_end = self + .journal_header_ref() + .header_size + .checked_add(self.journal_header_ref().arena_size) + .ok_or(JournalError::InvalidOffset)?; + if end > arena_end { + return Err(JournalError::InvalidObjectLocation); + } + let capacity = + usize::try_from((size - 24) / item_size).map_err(|_| JournalError::InvalidOffset)?; + let available = usize::try_from((extent - offset.get() - 24) / item_size) + .unwrap_or(usize::MAX) + .min(capacity); + Ok(Some(ArrayPrefix { + next: header.next_offset_array, + capacity, + available, + partial: end > extent, + })) + } + + fn recovery_array_slot(&self, array: NonZeroU64, index: usize) -> Result> { + let compact = self + .journal_header_ref() + .has_incompatible_flag(HeaderIncompatibleFlags::Compact); + let item_size = if compact { 4 } else { 8 }; + let offset = (index as u64) + .checked_mul(item_size) + .and_then(|v| array.get().checked_add(24)?.checked_add(v)) + .ok_or(JournalError::InvalidOffset)?; + let mut bytes = [0u8; 8]; + self.read_fresh_bytes_at(offset, &mut bytes[..item_size as usize])?; + Ok(NonZeroU64::new(if compact { + u32::from_le_bytes(bytes[..4].try_into().unwrap()) as u64 + } else { + u64::from_le_bytes(bytes) + })) + } + + fn recovery_entry_present(&self, offset: NonZeroU64, extent: u64) -> Result { + validate_offset_alignment(offset)?; + let header = self.journal_header_ref(); + if offset.get() < header.header_size { + return Err(JournalError::InvalidObjectLocation); + } + let arena_end = header + .header_size + .checked_add(header.arena_size) + .ok_or(JournalError::InvalidOffset)?; + if offset + .get() + .checked_add(16) + .is_none_or(|end| end > arena_end) + { + return Err(JournalError::InvalidObjectLocation); + } + if offset.get().checked_add(16).is_none_or(|end| end > extent) { + return Ok(false); + } + let mut bytes = [0u8; 16]; + self.read_fresh_bytes_at(offset.get(), &mut bytes)?; + let object = + ObjectHeader::ref_from_bytes(&bytes).map_err(|_| JournalError::ZerocopyFailure)?; + if object.type_ == 0 && object.size == 0 { + if bytes.iter().any(|byte| *byte != 0) { + return Err(JournalError::InvalidObjectType); + } + return Ok(false); + } + if object.type_ != ObjectType::Entry as u8 { + return Err(JournalError::InvalidObjectType); + } + let size = object.validated_size()?; + let item_size = if header.has_incompatible_flag(HeaderIncompatibleFlags::Compact) { + 4 + } else { + 16 + }; + if size < 64 || (size - 64) % item_size != 0 { + return Err(JournalError::InvalidObjectSize(size)); + } + let end = offset + .get() + .checked_add(size) + .ok_or(JournalError::InvalidOffset)?; + if end + > header + .header_size + .checked_add(header.arena_size) + .ok_or(JournalError::InvalidOffset)? + { + return Err(JournalError::InvalidObjectLocation); + } + Ok(end <= extent) + } + + fn discover_reader_prefix(&self) -> Result> { + let extent = self.window_manager.borrow_mut_checked()?.reader_extent()?; + let header = self.journal_header_ref(); + let mut remaining = + usize::try_from(header.n_entries).map_err(|_| JournalError::InvalidOffset)?; + let mut offset = header.entry_array_offset; + let mut count = 0u64; + let mut last_entry = None; + let mut loss = None; + let head = offset; + let mut highest = offset; + let compact = header.has_incompatible_flag(HeaderIncompatibleFlags::Compact); + let item_size = if compact { 4 } else { 8 }; + let mut slots = [0u8; 4096]; + while remaining > 0 { + let Some(array_offset) = offset else { + let (reason, lost) = loss.unwrap_or((RecoveryReason::MissingEntryArray, 0)); + return self + .make_reader_prefix(reason, lost, count, last_entry) + .map(Some); + }; + let Some(array) = self.recovery_array_prefix(array_offset, extent)? else { + let (reason, lost) = + loss.unwrap_or((RecoveryReason::MissingEntryArray, array_offset.get())); + return self + .make_reader_prefix(reason, lost, count, last_entry) + .map(Some); + }; + let wanted = remaining.min(array.capacity); + let available = wanted.min(array.available); + for start in (0..available).step_by(slots.len() / item_size) { + let count_slots = (available - start).min(slots.len() / item_size); + let bytes = &mut slots[..count_slots * item_size]; + self.read_fresh_bytes_at( + array_offset.get() + 24 + (start * item_size) as u64, + bytes, + )?; + for slot in bytes.chunks_exact(item_size) { + let entry = NonZeroU64::new(if compact { + u32::from_le_bytes(slot.try_into().unwrap()) as u64 + } else { + u64::from_le_bytes(slot.try_into().unwrap()) + }); + let present = match entry { + Some(entry) => self.recovery_entry_present(entry, extent)?, + None => false, + }; + if present { + if loss.is_some() { + return Err(JournalError::InvalidObjectLocation); + } + if last_entry.is_some_and(|previous| entry <= Some(previous)) { + return Err(JournalError::InvalidObjectLocation); + } + count += 1; + last_entry = entry; + } else { + loss.get_or_insert(( + RecoveryReason::MissingEntry, + entry.map_or(0, NonZeroU64::get), + )); + } + } + } + if available < wanted || array.partial { + loss.get_or_insert((RecoveryReason::MissingEntryArray, array_offset.get())); + } + remaining -= wanted; + if remaining > 0 { + if let Some(next) = array.next { + if Some(next) <= highest { + reject_array_revisit(head.unwrap(), array_offset, next, |at| { + self.recovery_array_prefix(at, extent)? + .map(|array| array.next) + .ok_or(JournalError::ObjectExceedsFileBounds) + })?; + } else { + highest = Some(next); + } + } + } + offset = array.next; + } + loss.map(|(reason, lost)| self.make_reader_prefix(reason, lost, count, last_entry)) + .transpose() + } + + #[doc(hidden)] + pub fn recover_reader_tail(&self, err: &JournalError) -> Result { + if !self.recovery_enabled + || !matches!( + err, + JournalError::ObjectExceedsFileBounds + | JournalError::InvalidObjectSize(0) + | JournalError::InvalidOffset + | JournalError::Io(_) + ) + { + return Ok(false); + } + if let JournalError::Io(err) = err { + if err.kind() != std::io::ErrorKind::UnexpectedEof { + return Ok(false); + } + } + let next = self.discover_reader_prefix()?; + let changed = next != self.recovery.get(); + self.recovery.set(next); + if changed { + self.recovery_changed.set(true); + } + Ok(changed) + } + + #[doc(hidden)] + pub fn refresh_reader_recovery(&self) -> Result { + if !self.recovery_enabled || self.recovery.get().is_none() { + return Ok(false); + } + let next = self.discover_reader_prefix()?; + let changed = next != self.recovery.get(); + self.recovery.set(next); + if changed { + self.recovery_changed.set(true); + } + Ok(changed) + } + + #[inline] + pub(super) fn reader_offset_array_ref( + &self, + offset: NonZeroU64, + ) -> Result>> { + if self.recovery.get().is_none() { + return self.offset_array_ref(offset); + } + self.recovered_array_ref(offset) + } + + #[cold] + #[inline(never)] + fn recovered_array_ref( + &self, + offset: NonZeroU64, + ) -> Result>> { + validate_offset_alignment(offset)?; + let header = self.journal_header_ref(); + let arena_end = header + .header_size + .checked_add(header.arena_size) + .ok_or(JournalError::InvalidOffset)?; + if offset.get() < header.header_size + || offset + .get() + .checked_add(24) + .is_none_or(|end| end > arena_end) + { + return Err(JournalError::InvalidObjectLocation); + } + let compact = self + .journal_header_ref() + .has_incompatible_flag(HeaderIncompatibleFlags::Compact); + self.window_manager.with_guarded(offset, |wm| { + let prefix = wm.get_slice(offset.get(), 24)?; + let header = OffsetArrayObjectHeader::ref_from_bytes(prefix) + .map_err(|_| JournalError::ZerocopyFailure)?; + let size = header.object_header.validated_size()?; + if offset + .get() + .checked_add(size) + .is_none_or(|end| end > arena_end) + { + return Err(JournalError::InvalidObjectLocation); + } + if header.object_header.type_ != ObjectType::EntryArray as u8 { + return Err(JournalError::InvalidObjectType); + } + let item_size = if compact { 4 } else { 8 }; + if size <= 24 || (size - 24) % item_size != 0 { + return Err(JournalError::InvalidObjectSize(size)); + } + let available = wm.cached_file_size().saturating_sub(offset.get()); + let length = size.min(24 + (available.saturating_sub(24) / item_size) * item_size); + let bytes = wm.get_slice(offset.get(), length)?; + OffsetArrayObject::from_data(bytes, compact).ok_or(JournalError::ZerocopyFailure) + }) + } + + pub(super) fn recovered_posting_count( + &self, + head: NonZeroU64, + total: usize, + mut previous: NonZeroU64, + ) -> Result { + let Some(prefix) = self.recovery.get() else { + return Ok(total); + }; + let Some(last) = prefix.last_entry else { + return Ok(0); + }; + let extent = self.reader_file_size()?; + let (mut offset, mut remaining, mut count) = (Some(head), total, 0); + let mut highest = head; + while remaining > 0 { + let Some(at) = offset else { + if previous == last { + return Ok(count); + } + return Err(JournalError::ObjectExceedsFileBounds); + }; + let Some(array) = self.recovery_array_prefix(at, extent)? else { + if previous == last { + return Ok(count); + } + return Err(JournalError::ObjectExceedsFileBounds); + }; + let wanted = remaining.min(array.capacity); + let available = wanted.min(array.available); + for index in 0..available { + let Some(entry) = self.recovery_array_slot(at, index)? else { + if previous == last { + return Ok(count); + } + return Err(JournalError::InvalidOffset); + }; + validate_offset_alignment(entry)?; + if entry <= previous { + return Err(JournalError::InvalidOffset); + } + if entry > last { + return Ok(count); + } + previous = entry; + count += 1; + } + if available < wanted { + if previous == last { + return Ok(count); + } + return Err(JournalError::ObjectExceedsFileBounds); + } + remaining -= wanted; + if remaining > 0 { + if let Some(next) = array.next { + if next <= highest { + reject_array_revisit(head, at, next, |prior| { + self.recovery_array_prefix(prior, extent)? + .map(|array| array.next) + .ok_or(JournalError::ObjectExceedsFileBounds) + })?; + } else { + highest = next; + } + } + } + offset = array.next; + } + Ok(count) + } + + pub(super) fn recovered_last_entry(&self) -> Option> { + self.recovery.get().map(|p| p.last_entry) + } +} diff --git a/rust/src/crates/journal-core/src/file/row_view.rs b/rust/src/crates/journal-core/src/file/row_view.rs index e431d9a1..c18f0ffd 100644 --- a/rust/src/crates/journal-core/src/file/row_view.rs +++ b/rust/src/crates/journal-core/src/file/row_view.rs @@ -66,10 +66,16 @@ impl CurrentRowView { self.row_pins_active } + #[inline] pub fn clear_pins(&mut self, file: &JournalFile) -> Result<()> { if !self.row_pins_active { return Ok(()); } + self.clear_active_pins(file) + } + + #[inline(never)] + fn clear_active_pins(&mut self, file: &JournalFile) -> Result<()> { file.clear_row_payload_pins()?; self.row_pins_active = false; Ok(()) diff --git a/rust/src/crates/journal-core/src/file/writer_entry_arrays.rs b/rust/src/crates/journal-core/src/file/writer_entry_arrays.rs index fc4dd18c..42a154ae 100644 --- a/rust/src/crates/journal-core/src/file/writer_entry_arrays.rs +++ b/rust/src/crates/journal-core/src/file/writer_entry_arrays.rs @@ -65,9 +65,11 @@ impl JournalWriter { }; let mut depth = 0; let mut object_offset = hash_table.hash_item_ref(hash).head_hash_offset; + let mut chain = super::chain_guard::ChainGuard::default(); while let Some(offset) = object_offset { let object = journal_file.data_ref(offset)?; object_offset = object.header.next_hash_offset; + chain.advance(offset, object_offset, false)?; if object_offset.is_some() { depth += 1; } @@ -84,9 +86,11 @@ impl JournalWriter { }; let mut depth = 0; let mut object_offset = hash_table.hash_item_ref(hash).head_hash_offset; + let mut chain = super::chain_guard::ChainGuard::default(); while let Some(offset) = object_offset { let object = journal_file.field_ref(offset)?; object_offset = object.header.next_hash_offset; + chain.advance(offset, object_offset, false)?; if object_offset.is_some() { depth += 1; } @@ -172,17 +176,31 @@ impl JournalWriter { mut offset: NonZeroU64, mut remaining: u64, ) -> Result { + let head = offset; + let mut highest = offset; loop { let array_guard = journal_file.offset_array_ref(offset)?; let capacity = array_guard.capacity() as u64; + if capacity == 0 { + return Err(JournalError::EmptyOffsetArrayNode); + } if remaining < capacity || array_guard.header.next_offset_array.is_none() { return Ok(offset); } remaining -= capacity; - offset = array_guard + let next = array_guard .header .next_offset_array .ok_or(JournalError::InvalidOffsetArrayOffset)?; + drop(array_guard); + if next <= highest { + super::array_chain::reject_array_revisit(head, offset, next, |at| { + Ok(journal_file.offset_array_ref(at)?.header.next_offset_array) + })?; + } else { + highest = next; + } + offset = next; } } @@ -221,13 +239,30 @@ impl JournalWriter { tail_offset: NonZeroU64, mut entries: u64, ) -> Result { + let head = offset; + let mut highest = offset; while offset != tail_offset { let array_guard = journal_file.offset_array_ref(offset)?; - entries -= array_guard.capacity() as u64; - offset = array_guard + let capacity = array_guard.capacity() as u64; + if capacity == 0 { + return Err(JournalError::EmptyOffsetArrayNode); + } + entries = entries + .checked_sub(capacity) + .ok_or(JournalError::InvalidOffset)?; + let next = array_guard .header .next_offset_array .ok_or(JournalError::InvalidOffsetArrayOffset)?; + drop(array_guard); + if next <= highest { + super::array_chain::reject_array_revisit(head, offset, next, |at| { + Ok(journal_file.offset_array_ref(at)?.header.next_offset_array) + })?; + } else { + highest = next; + } + offset = next; } Ok(entries) } @@ -288,10 +323,15 @@ impl JournalWriter { let mut current_index = 0u64; #[allow(unused_assignments)] let mut tail_offset = array_offset; + let head = array_offset; + let mut highest = array_offset; loop { let array_guard = journal_file.offset_array_ref(array_offset)?; let capacity = array_guard.capacity() as u64; + if capacity == 0 { + return Err(JournalError::EmptyOffsetArrayNode); + } if current_index + capacity >= current_count { // This is the tail array @@ -306,6 +346,14 @@ impl JournalWriter { return Err(JournalError::InvalidOffsetArrayOffset); }; + drop(array_guard); + if next_offset <= highest { + super::array_chain::reject_array_revisit(head, array_offset, next_offset, |at| { + Ok(journal_file.offset_array_ref(at)?.header.next_offset_array) + })?; + } else { + highest = next_offset; + } array_offset = next_offset; } diff --git a/rust/src/crates/journal-core/src/file/zstd_stream.rs b/rust/src/crates/journal-core/src/file/zstd_stream.rs new file mode 100644 index 00000000..572c213e --- /dev/null +++ b/rust/src/crates/journal-core/src/file/zstd_stream.rs @@ -0,0 +1,169 @@ +use ruzstd::decoding::errors::FrameDecoderError; +use ruzstd::decoding::{FrameDecoder, StreamingDecoder}; +use std::error::Error; +use std::io::{self, BufReader, Read, Write}; + +fn decoder_error(error: FrameDecoderError) -> io::Error { + // Decoder wrappers retain transport errors in their source chain. + let mut source: Option<&(dyn Error + 'static)> = Some(&error); + let mut kind = io::ErrorKind::InvalidData; + while let Some(cause) = source { + if let Some(io_error) = cause.downcast_ref::() { + kind = if io_error.kind() == io::ErrorKind::UnexpectedEof { + io::ErrorKind::InvalidData + } else { + io_error.kind() + }; + } + source = cause.source(); + } + io::Error::new(kind, error) +} + +fn copy_error(error: io::Error) -> io::Error { + if error + .get_ref() + .is_some_and(|cause| cause.is::()) + { + let inner = error + .into_inner() + .unwrap() + .downcast::() + .unwrap(); + decoder_error(*inner) + } else { + error + } +} + +fn invalid_stream() -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, "invalid Zstandard stream") +} + +/// Decode all frames without buffering the whole input or output. +#[doc(hidden)] +pub fn copy_zstd_stream(input: impl Read, output: &mut impl Write) -> io::Result { + let mut input = BufReader::new(input); + let mut frame = FrameDecoder::new(); + let mut total = 0u64; + let mut seen_frame = false; + loop { + let mut prefix = [0u8; 5]; + match input.read(&mut prefix[..1]) { + Ok(0) if seen_frame => return Ok(total), + Ok(0) => return Err(invalid_stream()), + Ok(_) => {} + Err(err) if err.kind() == io::ErrorKind::Interrupted => continue, + Err(err) => return Err(err), + } + input.read_exact(&mut prefix[1..4])?; + let magic = u32::from_le_bytes(prefix[..4].try_into().unwrap()); + if magic & 0xfffffff0 == 0x184d2a50 { + let mut size = [0u8; 4]; + input.read_exact(&mut size)?; + let size = u64::from(u32::from_le_bytes(size)); + if io::copy(&mut (&mut input).take(size), &mut io::sink())? != size { + return Err(invalid_stream()); + } + seen_frame = true; + continue; + } + if magic != 0xfd2fb528 { + return Err(invalid_stream()); + } + input.read_exact(&mut prefix[4..])?; + let has_size = prefix[4] & 0xe0 != 0; + let source = prefix.as_slice().chain(&mut input); + let mut decoder = + StreamingDecoder::new_with_decoder(source, &mut frame).map_err(decoder_error)?; + let expected = decoder.decoder.content_size(); + let count = io::copy(&mut decoder, output).map_err(copy_error)?; + if (has_size && count != expected) + || decoder + .decoder + .get_checksum_from_data() + .is_some_and(|checksum| Some(checksum) != decoder.decoder.get_calculated_checksum()) + { + return Err(invalid_stream()); + } + total = total.checked_add(count).ok_or_else(invalid_stream)?; + seen_frame = true; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + struct FailingRead<'a> { + bytes: &'a [u8], + left: usize, + } + impl Read for FailingRead<'_> { + fn read(&mut self, out: &mut [u8]) -> io::Result { + if self.left == 0 { + return Err(io::Error::new( + io::ErrorKind::PermissionDenied, + "source unavailable", + )); + } + let count = out.len().min(self.left).min(self.bytes.len()); + out[..count].copy_from_slice(&self.bytes[..count]); + self.bytes = &self.bytes[count..]; + self.left -= count; + Ok(count) + } + } + struct FailingWrite; + impl Write for FailingWrite { + fn write(&mut self, _: &[u8]) -> io::Result { + Err(io::Error::new( + io::ErrorKind::StorageFull, + "destination full", + )) + } + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } + } + + #[test] + fn stream_preserves_io_failures() { + let plain = b"stream failure classification".repeat(64); + let compressed = ruzstd::encoding::compress_to_vec( + plain.as_slice(), + ruzstd::encoding::CompressionLevel::Fastest, + ); + for left in [0, 2, 5, 6, compressed.len() - 1] { + let source = FailingRead { + bytes: &compressed, + left, + }; + let error = copy_zstd_stream(source, &mut Vec::new()).unwrap_err(); + assert_eq!( + error.kind(), + io::ErrorKind::PermissionDenied, + "offset {left}: {error}" + ); + } + assert_eq!( + copy_zstd_stream(compressed.as_slice(), &mut FailingWrite) + .unwrap_err() + .kind(), + io::ErrorKind::StorageFull + ); + for end in [0, 2, 5, compressed.len() - 1] { + let error = copy_zstd_stream(&compressed[..end], &mut Vec::new()).unwrap_err(); + assert!( + matches!( + error.kind(), + io::ErrorKind::InvalidData | io::ErrorKind::UnexpectedEof + ), + "offset {end}: {error}" + ); + } + let mut out = Vec::new(); + copy_zstd_stream(compressed.as_slice(), &mut out).unwrap(); + assert_eq!(out, plain); + } +} diff --git a/rust/src/crates/journal-engine/src/cache.rs b/rust/src/crates/journal-engine/src/cache.rs index 30a6a645..b61a1126 100644 --- a/rust/src/crates/journal-engine/src/cache.rs +++ b/rust/src/crates/journal-engine/src/cache.rs @@ -13,7 +13,9 @@ use serde::{Deserialize, Serialize}; /// v3: Index semantics changed from the v2 ND_REMAPPING-specific behavior, and /// FileIndexKey gained an explicit consumer namespace. Old v2 cache entries /// are not reused across SDK versions with different index rules. -const CACHE_VERSION: u32 = 3; +/// v4: Native array traversal rejects missing required links and counted +/// revisits; older cached indexes may contain incomplete or duplicate rows. +const CACHE_VERSION: u32 = 4; /// Cache key for file indexes that includes the file, facets, source timestamp /// field, and cache version. Different facet configurations or timestamp fields @@ -97,6 +99,68 @@ mod tests { assert_ne!(default, named); } + #[test] + fn native_path_support_preserves_ordinary_file_cache_identity() { + use std::hash::{Hash, Hasher}; + + #[derive(Hash, Serialize)] + struct LegacyFile<'a> { + path: &'a str, + origin: &'a journal_registry::Origin, + status: &'a journal_registry::Status, + } + let file = test_file(); + let legacy = LegacyFile { + path: file.path(), + origin: file.origin(), + status: file.status(), + }; + assert_eq!( + serde_json::to_vec(&file).unwrap(), + serde_json::to_vec(&legacy).unwrap() + ); + let mut actual = std::collections::hash_map::DefaultHasher::new(); + let mut previous = std::collections::hash_map::DefaultHasher::new(); + file.hash(&mut actual); + legacy.hash(&mut previous); + assert_eq!(actual.finish(), previous.finish()); + let decoded: File = serde_json::from_slice(&serde_json::to_vec(&legacy).unwrap()).unwrap(); + assert_eq!(decoded, file); + assert_eq!(decoded.as_path(), Path::new(legacy.path)); + } + + #[cfg(unix)] + #[test] + fn native_path_cache_keys_round_trip_without_replacement_aliases() { + use std::os::unix::ffi::OsStringExt; + + let path = std::path::PathBuf::from(std::ffi::OsString::from_vec( + b"/example/raw-\xff.journal".to_vec(), + )); + let file = File::from_raw_path(&path).unwrap(); + let decoy = File::from_raw_path(Path::new(path.to_string_lossy().as_ref())).unwrap(); + let key = FileIndexKey::new(&file, &Facets::new(&[]), None); + let replacement = FileIndexKey::new(&decoy, &Facets::new(&[]), None); + assert_ne!(key, replacement); + let encoded = serde_json::to_vec(&key).unwrap(); + let decoded: FileIndexKey = serde_json::from_slice(&encoded).unwrap(); + assert_eq!(decoded, key); + assert_eq!(decoded.file.as_path(), path); + // An old text-only consumer must fail to open, rather than select the decoy. + assert!(decoded.file.path().as_bytes().contains(&0)); + assert_eq!( + std::fs::File::open(decoded.file.path()).unwrap_err().kind(), + std::io::ErrorKind::InvalidInput + ); + let mut cache = HashMap::new(); + cache.insert(key, "native"); + assert_eq!(cache.get(&decoded), Some(&"native")); + assert_eq!(cache.get(&replacement), None); + let mut malformed = serde_json::to_value(&decoded).unwrap(); + malformed["file"]["path"] = serde_json::json!("\0journal-path-unix-v1:2f00"); + assert!(serde_json::from_value::(malformed).is_err()); + } + #[test] fn old_serialized_cache_key_without_namespace_decodes_as_old_key() { let file = test_file(); diff --git a/rust/src/crates/journal-engine/tests/index_integrity.rs b/rust/src/crates/journal-engine/tests/index_integrity.rs new file mode 100644 index 00000000..b1788de9 --- /dev/null +++ b/rust/src/crates/journal-engine/tests/index_integrity.rs @@ -0,0 +1,111 @@ +use journal_core::file::{JournalFile, JournalFileOptions, JournalWriter, MmapMut}; +use journal_engine::{ + Facets, FileIndexCacheBuilder, FileIndexKey, IndexingLimits, QueryTimeRange, + batch_compute_file_indexes, +}; +use journal_index::FieldName; +use journal_registry::{File, Monitor, Registry}; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +fn read_u64(bytes: &[u8], offset: usize) -> u64 { + u64::from_le_bytes(bytes[offset..offset + 8].try_into().unwrap()) +} + +#[tokio::test(flavor = "current_thread")] +async fn failed_native_field_indexes_are_not_cached() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let machine = Uuid::from_u128(1); + let directory = temp.path().join(machine.simple().to_string()); + std::fs::create_dir_all(&directory).unwrap(); + let mut files = Vec::new(); + for index in 0..2 { + let sequence = Uuid::from_u128(3 + index); + let path = directory.join(format!( + "system@{}-0000000000000001-00000000000f4240.journal", + sequence.simple() + )); + let file = File::from_path(&path).unwrap(); + let mut journal = JournalFile::::create( + &file, + JournalFileOptions::new(machine, Uuid::from_u128(2), sequence) + .with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut journal, 1, Uuid::from_u128(2)).unwrap(); + for row in 1..=14 { + writer + .add_entry( + &mut journal, + &[b"MESSAGE=row", b"SOURCE_TIMESTAMP=2000000"], + row * 1_000_000, + row, + ) + .unwrap(); + } + journal.sync().unwrap(); + drop(writer); + drop(journal); + let mut source = std::fs::read(&path).unwrap(); + source[16] = 0; + std::fs::write(&path, &source).unwrap(); + files.push((file, path, source)); + } + for source_timestamp in [false, true] { + let payload: &[u8] = if source_timestamp { + b"SOURCE_TIMESTAMP=2000000" + } else { + b"MESSAGE=row" + }; + let mut damaged = files[1].2.clone(); + let mut data = read_u64(&damaged, 88) as usize; + let payload_start = if compact { 72 } else { 64 }; + loop { + let size = read_u64(&damaged, data + 8) as usize; + if damaged[data] == 1 && &damaged[data + payload_start..data + size] == payload { + break; + } + data += (size + 7) & !7; + assert!(data < damaged.len()); + } + if source_timestamp { + let head = read_u64(&damaged, data + 48) as usize; + damaged[head + 16..head + 24].fill(0); + } else { + damaged[data + 32..data + 40].copy_from_slice(&(data as u64).to_le_bytes()); + } + std::fs::write(&files[1].1, damaged).unwrap(); + let facets = Facets::new(&["MESSAGE".to_owned()]); + let source_field = + source_timestamp.then(|| FieldName::new_unchecked("SOURCE_TIMESTAMP")); + let good_key = FileIndexKey::new(&files[0].0, &facets, source_field.clone()); + let bad_key = FileIndexKey::new(&files[1].0, &facets, source_field); + let cache = FileIndexCacheBuilder::new() + .with_memory_capacity(4) + .without_disk_cache() + .build() + .await + .unwrap(); + let (monitor, _receiver) = Monitor::new().unwrap(); + let registry = Registry::new(monitor); + let responses = batch_compute_file_indexes( + &cache, + ®istry, + vec![good_key.clone(), bad_key.clone()], + &QueryTimeRange::new(0, 20).unwrap(), + CancellationToken::new(), + IndexingLimits::default(), + None, + ) + .await + .unwrap(); + assert_eq!(responses.len(), 1, "{compact}/{source_timestamp}"); + assert_eq!(responses[0].0, good_key); + assert_eq!(responses[0].1.total_entries(), 14); + assert!(cache.get(&good_key).await.unwrap().is_some()); + assert!(cache.get(&bad_key).await.unwrap().is_none()); + cache.close().await.unwrap(); + } + } +} diff --git a/rust/src/crates/journal-index/src/file_indexer.rs b/rust/src/crates/journal-index/src/file_indexer.rs index 2a105a96..a8fa2814 100644 --- a/rust/src/crates/journal-index/src/file_indexer.rs +++ b/rust/src/crates/journal-index/src/file_indexer.rs @@ -12,10 +12,11 @@ use crate::{ Seconds, }; use journal_core::collections::{HashMap, HashSet}; +use journal_core::error::JournalError; use journal_core::file::{HashableObject, JournalFile, Mmap, offset_array::InlinedCursor}; use journal_registry::File; use std::num::NonZeroU64; -use tracing::{error, trace, warn}; +use tracing::{trace, warn}; /// Default maximum number of unique values to index per field. pub const DEFAULT_MAX_UNIQUE_VALUES_PER_FIELD: usize = 500; @@ -269,6 +270,9 @@ impl FileIndexer { ) -> Result<()> { let field_data_iterator = match journal_file.field_data_objects(field_name.as_bytes()) { Ok(field_data_iterator) => field_data_iterator, + Err(JournalError::CyclicObjectChain) => { + return Err(JournalError::CyclicObjectChain.into()); + } Err(e) => { warn!( "failed to iterate field data objects for field '{}' in file {}: {:#?}", @@ -287,8 +291,12 @@ impl FileIndexer { } let (data_payload, inlined_cursor) = { - let Ok(data_object) = data_object else { - continue; + let data_object = match data_object { + Ok(data_object) => data_object, + Err(JournalError::CyclicObjectChain) => { + return Err(JournalError::CyclicObjectChain.into()); + } + Err(_) => continue, }; if data_object.raw_payload().len() >= self.limits.max_field_payload_size || data_object.is_compressed() @@ -298,7 +306,9 @@ impl FileIndexer { } let data_payload = String::from_utf8_lossy(data_object.raw_payload()).into_owned(); - let Some(inlined_cursor) = data_object.inlined_cursor() else { + let Some(inlined_cursor) = + journal_file.reader_inlined_cursor(&data_object.header)? + else { continue; }; (data_payload, inlined_cursor) @@ -325,10 +335,7 @@ impl FileIndexer { tail_object_offset: NonZeroU64, ) -> Result { self.entry_offsets.clear(); - if let Err(err) = inlined_cursor.collect_offsets(journal_file, &mut self.entry_offsets) { - warn!("failed to collect entry offsets from DATA object index: {err:?}"); - return Ok(false); - } + inlined_cursor.collect_offsets(journal_file, &mut self.entry_offsets)?; self.entry_indices.clear(); for entry_offset in self @@ -338,11 +345,7 @@ impl FileIndexer { .filter(|offset| *offset <= tail_object_offset) { let Some(entry_index) = self.entry_offset_index.get(&entry_offset) else { - panic!( - "missing entry offset {} from index (total offsets: {})", - entry_offset, - self.entry_offset_index.len() - ); + return Err(IndexError::MissingOffset); }; self.entry_indices.push(*entry_index as u32); } @@ -374,9 +377,15 @@ impl FileIndexer { // Collect all the inlined cursors of the source timestamp field self.source_timestamp_cursor_pairs.clear(); for data_object_result in field_data_iterator { - let Ok(data_object) = data_object_result else { - warn!("loading data object failed"); - continue; + let data_object = match data_object_result { + Ok(data_object) => data_object, + Err(JournalError::CyclicObjectChain) => { + return Err(JournalError::CyclicObjectChain.into()); + } + Err(_) => { + warn!("loading data object failed"); + continue; + } }; let Ok(source_timestamp) = @@ -386,7 +395,7 @@ impl FileIndexer { continue; }; - let Some(ic) = data_object.inlined_cursor() else { + let Some(ic) = journal_file.reader_inlined_cursor(&data_object.header)? else { use journal_core::file::JournalState; let file_state = JournalState::try_from(journal_file.journal_header_ref().state) @@ -411,13 +420,7 @@ impl FileIndexer { for (ts, ic) in self.source_timestamp_cursor_pairs.iter() { self.entry_offsets.clear(); - match ic.collect_offsets(journal_file, &mut self.entry_offsets) { - Ok(_) => {} - Err(e) => { - error!("failed to collect offsets from source timestamp: {}", e); - continue; - } - } + ic.collect_offsets(journal_file, &mut self.entry_offsets)?; for entry_offset in &self.entry_offsets { self.source_timestamp_entry_offset_pairs diff --git a/rust/src/crates/journal-index/tests/array_integrity.rs b/rust/src/crates/journal-index/tests/array_integrity.rs new file mode 100644 index 00000000..0a62e0c9 --- /dev/null +++ b/rust/src/crates/journal-index/tests/array_integrity.rs @@ -0,0 +1,175 @@ +use journal_common::Seconds; +use journal_core::file::{JournalFile, JournalFileOptions, JournalWriter, MmapMut}; +use journal_core::repository::File; +use journal_index::{FieldName, FileIndexer}; +use uuid::Uuid; + +fn read_u64(bytes: &[u8], offset: usize) -> u64 { + u64::from_le_bytes(bytes[offset..offset + 8].try_into().unwrap()) +} + +#[test] +fn archived_indexes_reject_required_missing_or_revisited_arrays() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let machine = Uuid::from_u128(1); + let boot = Uuid::from_u128(2); + let sequence = Uuid::from_u128(3); + let directory = temp.path().join(machine.simple().to_string()); + std::fs::create_dir_all(&directory).unwrap(); + let path = directory.join(format!( + "system@{}-0000000000000001-00000000000f4240.journal", + sequence.simple() + )); + let file = File::from_path(&path).unwrap(); + assert!(!file.is_active()); + let mut journal = JournalFile::::create( + &file, + JournalFileOptions::new(machine, boot, sequence).with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut journal, 1, boot).unwrap(); + for index in 1..=13 { + writer + .add_entry(&mut journal, &[b"MESSAGE=row"], index * 1_000_000, index) + .unwrap(); + } + journal.sync().unwrap(); + drop(writer); + drop(journal); + let mut source = std::fs::read(&path).unwrap(); + source[16] = 0; + std::fs::write(&path, &source).unwrap(); + let healthy = FileIndexer::default() + .index(&file, None, &[], Seconds(1)) + .unwrap(); + assert_eq!(healthy.total_entries(), 13); + assert!(!healthy.online()); + assert!(healthy.is_fresh()); + + let head = read_u64(&source, 176) as usize; + let next = read_u64(&source, head + 16) as usize; + assert_ne!(next, 0); + for cycle in [false, true] { + let mut damaged = source.clone(); + if cycle { + damaged[next + 16..next + 24].copy_from_slice(&(head as u64).to_le_bytes()); + } else { + damaged[head + 16..head + 24].fill(0); + } + std::fs::write(&path, damaged).unwrap(); + let result = FileIndexer::default().index(&file, None, &[], Seconds(1)); + assert!( + result.is_err(), + "compact={compact}/cycle={cycle}: incomplete archived index succeeded: {:?}", + result.as_ref().map(|index| index.total_entries()) + ); + } + } +} + +#[test] +fn archived_indexes_reject_incomplete_requested_field_indexes() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let directory = temp.path().join(Uuid::from_u128(1).simple().to_string()); + std::fs::create_dir_all(&directory).unwrap(); + let path = directory.join(format!( + "system@{}-0000000000000001-00000000000f4240.journal", + Uuid::from_u128(3).simple() + )); + let file = File::from_path(&path).unwrap(); + let mut journal = JournalFile::::create( + &file, + JournalFileOptions::new(Uuid::from_u128(1), Uuid::from_u128(2), Uuid::from_u128(3)) + .with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut journal, 1, Uuid::from_u128(2)).unwrap(); + for index in 1..=14 { + writer + .add_entry( + &mut journal, + &[b"MESSAGE=row", b"SOURCE_TIMESTAMP=2000000"], + index * 1_000_000, + index, + ) + .unwrap(); + } + journal.sync().unwrap(); + drop(writer); + drop(journal); + let mut source = std::fs::read(&path).unwrap(); + source[16] = 0; + for source_timestamp in [false, true] { + let (name, payload): (&str, &[u8]) = if source_timestamp { + ("SOURCE_TIMESTAMP", b"SOURCE_TIMESTAMP=2000000") + } else { + ("MESSAGE", b"MESSAGE=row") + }; + let field = FieldName::new_unchecked(name.to_owned()); + let fields = if source_timestamp { + Vec::new() + } else { + vec![field.clone()] + }; + let source_field = source_timestamp.then_some(&field); + std::fs::write(&path, &source).unwrap(); + assert_eq!( + FileIndexer::default() + .index(&file, source_field, &fields, Seconds(1)) + .unwrap() + .total_entries(), + 14 + ); + let payload_start = if compact { 72 } else { 64 }; + let mut data = read_u64(&source, 88) as usize; + loop { + let size = read_u64(&source, data + 8) as usize; + if source[data] == 1 && &source[data + payload_start..data + size] == payload { + break; + } + data += (size + 7) & !7; + assert!(data < source.len(), "missing requested DATA"); + } + let head = read_u64(&source, data + 48) as usize; + let next = read_u64(&source, head + 16) as usize; + assert_ne!(next, 0); + for mutation in [ + "field-cycle", + "missing-link", + "posting-cycle", + "missing-inline", + "missing-head", + "unindexed-inline", + ] { + if source_timestamp && mutation == "unindexed-inline" { + continue; + } + let mut damaged = source.clone(); + match mutation { + "field-cycle" => { + damaged[data + 32..data + 40].copy_from_slice(&(data as u64).to_le_bytes()) + } + "missing-link" => damaged[head + 16..head + 24].fill(0), + "posting-cycle" => { + damaged[next + 16..next + 24].copy_from_slice(&(head as u64).to_le_bytes()) + } + "missing-inline" => damaged[data + 40..data + 48].fill(0), + "missing-head" => damaged[data + 48..data + 56].fill(0), + "unindexed-inline" => { + damaged[data + 40..data + 48].copy_from_slice(&(data as u64).to_le_bytes()) + } + _ => unreachable!(), + } + std::fs::write(&path, damaged).unwrap(); + let result = FileIndexer::default().index(&file, source_field, &fields, Seconds(1)); + assert!( + result.is_err(), + "compact={compact}/source={source_timestamp}/{mutation}: incomplete index succeeded: {:?}", + result.as_ref().map(|index| index.total_entries()) + ); + } + } + } +} diff --git a/rust/src/crates/journal-registry/src/repository/file.rs b/rust/src/crates/journal-registry/src/repository/file.rs index fb131f18..5b54bcce 100644 --- a/rust/src/crates/journal-registry/src/repository/file.rs +++ b/rust/src/crates/journal-registry/src/repository/file.rs @@ -1,8 +1,9 @@ -use crate::repository::RepositoryError; use crate::repository::error::Result; +use crate::repository::RepositoryError; use serde::{Deserialize, Serialize}; use std::cmp::Ordering; -use std::path::Path; +use std::hash::{Hash, Hasher}; +use std::path::{Path, PathBuf}; use std::sync::Arc; use uuid::Uuid; @@ -179,12 +180,79 @@ pub struct Origin { pub source: Source, } -#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[cfg_attr(feature = "allocative", derive(allocative::Allocative))] pub(crate) struct FileInner { pub(crate) path: String, pub(crate) origin: Origin, pub(crate) status: Status, + #[serde(skip)] + pub(crate) native_path: Option>, +} + +impl Hash for FileInner { + fn hash(&self, state: &mut H) { + // Preserve existing cache hashes; the native key already encodes all bytes. + self.path.hash(state); + self.origin.hash(state); + self.status.hash(state); + } +} + +const NATIVE_PATH_KEY: &str = "\0journal-path-unix-v1:"; + +#[cfg(unix)] +fn native_path_key(path: &Path) -> String { + use std::os::unix::ffi::OsStrExt; + + let bytes = path.as_os_str().as_bytes(); + let mut key = String::with_capacity(NATIVE_PATH_KEY.len() + bytes.len() * 2); + key.push_str(NATIVE_PATH_KEY); + for byte in bytes { + key.push(char::from(b"0123456789abcdef"[(byte >> 4) as usize])); + key.push(char::from(b"0123456789abcdef"[(byte & 15) as usize])); + } + key +} + +fn decode_native_path_key(key: &str) -> std::result::Result, &'static str> { + let Some(encoded) = key.strip_prefix(NATIVE_PATH_KEY) else { + return Ok(None); + }; + #[cfg(not(unix))] + { + let _ = encoded; + Err("native Unix path keys are unsupported on this platform") + } + #[cfg(unix)] + { + use std::os::unix::ffi::OsStringExt; + + if encoded.len() % 2 != 0 { + return Err("invalid native path key length"); + } + let mut bytes = Vec::with_capacity(encoded.len() / 2); + for pair in encoded.as_bytes().chunks_exact(2) { + let high = char::from(pair[0]) + .to_digit(16) + .ok_or("invalid native path key")?; + let low = char::from(pair[1]) + .to_digit(16) + .ok_or("invalid native path key")?; + bytes.push(((high << 4) | low) as u8); + } + if bytes.contains(&0) { + return Err("NUL in native filesystem path"); + } + let path = PathBuf::from(std::ffi::OsString::from_vec(bytes)); + if !path.is_absolute() { + return Err("native path key is not absolute"); + } + if path.to_str().is_some() { + return Err("native path key must contain non-UTF8 bytes"); + } + Ok(Some(path)) + } } #[derive(Debug, Clone, PartialEq, Eq, Hash)] @@ -207,7 +275,14 @@ impl<'de> serde::Deserialize<'de> for File { where D: serde::Deserializer<'de>, { - let inner = FileInner::deserialize(deserializer)?; + let mut inner = FileInner::deserialize(deserializer)?; + if let Some(path) = decode_native_path_key(&inner.path).map_err(serde::de::Error::custom)? { + #[cfg(unix)] + { + inner.path = native_path_key(&path); + } + inner.native_path = Some(Box::new(path)); + } Ok(File { inner: Arc::new(inner), }) @@ -215,10 +290,20 @@ impl<'de> serde::Deserialize<'de> for File { } impl File { + /// Legacy text path/key. Native byte paths use an encoded, non-openable key. + /// Use `as_path` for filesystem operations. pub fn path(&self) -> &str { &self.inner.path } + /// Returns the original native filesystem path without lossy conversion. + pub fn as_path(&self) -> &Path { + self.inner + .native_path + .as_deref() + .map_or_else(|| Path::new(&self.inner.path), |path| path.as_path()) + } + pub fn origin(&self) -> &Origin { &self.inner.origin } @@ -255,20 +340,33 @@ impl File { path: path_str.to_string(), origin, status, + native_path: None, }); Some(File { inner }) } pub fn from_raw_path(path: &Path) -> Option { - let path = path.to_str()?; - let raw_path = Path::new(path); - if !raw_path.is_absolute() { + let raw_path = path; + if !raw_path.is_absolute() || raw_path.as_os_str().as_encoded_bytes().contains(&0) { return None; } + let (path, native_path) = match path.to_str() { + Some(text) => (text.to_string(), None), + None => { + #[cfg(unix)] + { + (native_path_key(path), Some(Box::new(path.to_path_buf()))) + } + #[cfg(not(unix))] + { + return None; + } + } + }; let inner = Arc::new(FileInner { - path: path.to_string(), + path, origin: Origin { machine_id: None, namespace: None, @@ -281,6 +379,7 @@ impl File { ), }, status: Status::Active, + native_path, }); Some(File { inner }) @@ -329,13 +428,14 @@ impl File { path: String::from(path), origin, status, + native_path: None, }); Some(File { inner }) } pub fn dir(&self) -> Result<&str> { - Path::new(&self.inner.path) + self.as_path() .parent() .and_then(|p| { if self.inner.origin.machine_id.is_some() { @@ -346,7 +446,7 @@ impl File { }) .and_then(|p| p.to_str()) .ok_or_else(|| RepositoryError::InvalidUtf8 { - path: Path::new(&self.inner.path).to_path_buf(), + path: self.as_path().to_path_buf(), }) } @@ -462,6 +562,19 @@ mod tests { ); assert_eq!(file.status(), &Status::Active); } + + #[cfg(unix)] + #[test] + fn from_raw_path_rejects_nul() { + use std::os::unix::ffi::OsStrExt; + for bytes in [ + b"/invalid\0.journal".as_slice(), + b"/invalid\xff\0.journal".as_slice(), + ] { + let path = std::path::Path::new(std::ffi::OsStr::from_bytes(bytes)); + assert!(File::from_raw_path(path).is_none()); + } + } } impl Ord for File { diff --git a/rust/src/crates/journal-registry/src/repository/mod.rs b/rust/src/crates/journal-registry/src/repository/mod.rs index e1fc370e..f69dd3d9 100644 --- a/rust/src/crates/journal-registry/src/repository/mod.rs +++ b/rust/src/crates/journal-registry/src/repository/mod.rs @@ -77,6 +77,7 @@ mod tests { head_seqnum: 1000 + head_realtime, head_realtime, }, + native_path: None, }; File { @@ -89,6 +90,7 @@ mod tests { path: "/var/log/journal/system.journal".to_string(), origin: origin.clone(), status: Status::Active, + native_path: None, }; File { @@ -101,6 +103,7 @@ mod tests { path: format!("/var/log/journal/system@{}-{}.journal~", timestamp, number), origin: origin.clone(), status: Status::Disposed { timestamp, number }, + native_path: None, }; File { diff --git a/rust/src/internal/testcmd/reader_core_bench/src/auxiliary.rs b/rust/src/internal/testcmd/reader_core_bench/src/auxiliary.rs new file mode 100644 index 00000000..a0adee5c --- /dev/null +++ b/rust/src/internal/testcmd/reader_core_bench/src/auxiliary.rs @@ -0,0 +1,99 @@ +use super::*; +use journal::{IndexedSnapshot, IndexedSnapshotOptions, SnapshotControl, SnapshotEntry}; + +pub(super) fn read(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { + require_single_file_input(inputs, cfg.surface, "indexed/verification modes")?; + if cfg.direction != Direction::Forward || cfg.bounds != "snapshot" { + return Err(anyhow!( + "indexed/verification modes require snapshot bounds and forward direction" + )); + } + let control = SnapshotControl::default(); + let mut counts = Counts::default(); + if cfg.mode.starts_with("verify-") { + if cfg.strategy != ExperimentalMmapStrategy::Windowed + || cfg.window_size != DEFAULT_WINDOW_SIZE + { + return Err(anyhow!( + "verification modes use public SDK defaults; select windowed access and the default window size" + )); + } + for _ in 0..cfg.catalog_repetitions { + match cfg.mode { + "verify-index" => journal::verify_index(&inputs[0], &control)?, + "verify-file" => journal::verify_file(&inputs[0])?, + other => return Err(anyhow!("invalid verification mode: {other}")), + } + counts.add_record_marker(1); + } + counts + .extra + .insert("completed_verifications".into(), json!(counts.records)); + counts + .extra + .insert("reader_options".into(), json!("public SDK defaults")); + return Ok(counts); + } + if cfg.filter_matches.len() != 1 { + return Err(anyhow!( + "indexed modes require one --match FIELD=value declaration" + )); + } + let declaration = &cfg.filter_matches[0]; + let separator = declaration + .iter() + .position(|&byte| byte == b'=') + .filter(|&index| index > 0) + .ok_or_else(|| anyhow!("indexed declaration must be FIELD=value"))?; + let (name, value) = (&declaration[..separator], &declaration[separator + 1..]); + let options = IndexedSnapshotOptions { + reader: reader_options(cfg.bounds, cfg.strategy, cfg.window_size)?, + capture_fields: vec![name.to_vec()], + capture_values: vec![(name.to_vec(), value.to_vec())], + }; + let mut captured_entries = 0; + let mut captured = journal::CapturedValue::default(); + let digest = cfg.mode.ends_with("-digest"); + for _ in 0..cfg.catalog_repetitions { + let mut snapshot = IndexedSnapshot::open(&inputs[0], options.clone(), &control)?; + captured_entries = snapshot.entry_count(); + captured = snapshot.captured_value(name, value)?; + let mut visit = |entry: &mut SnapshotEntry<'_>| { + counts.add_record_marker(entry.metadata().realtime); + entry.visit_payloads(|payload| { + if digest { + counts.add_full_payload(black_box(payload)); + } else { + counts.add_payload(black_box(payload)); + } + Ok(()) + }) + }; + match cfg.mode.strip_suffix("-digest").unwrap_or(cfg.mode) { + "indexed-open" => counts.add_record_marker(captured_entries), + "indexed-exact" => snapshot.visit_match(name, value, &control, &mut visit)?, + "indexed-field" => snapshot.visit_field( + name, + &control, + |candidate| Ok(candidate == value), + &mut visit, + )?, + "indexed-all" => snapshot.visit_entries(&control, &mut visit)?, + other => return Err(anyhow!("invalid indexed mode: {other}")), + } + drop(snapshot); + } + counts + .extra + .insert("completed_opens".into(), json!(cfg.catalog_repetitions)); + counts + .extra + .insert("captured_entries".into(), json!(captured_entries)); + counts + .extra + .insert("captured_value_entries".into(), json!(captured.entry_count)); + counts + .extra + .insert("captured_value_present".into(), json!(captured.present)); + Ok(counts) +} diff --git a/rust/src/internal/testcmd/reader_core_bench/src/filter.rs b/rust/src/internal/testcmd/reader_core_bench/src/filter.rs new file mode 100644 index 00000000..5a33f8c2 --- /dev/null +++ b/rust/src/internal/testcmd/reader_core_bench/src/filter.rs @@ -0,0 +1,183 @@ +use super::{ + Counts, ReadConfig, borrowed_input_paths, open_facade_reader, reader_options, utf8_input_paths, +}; +use anyhow::{Result, anyhow}; +use journal::{Direction, DirectoryReader, FileReader, SdJournal}; +use std::hint::black_box; +use std::path::PathBuf; + +pub(super) fn decode_matches(text: &[String], hex: &[String]) -> Result>> { + let mut matches: Vec> = text.iter().map(|value| value.as_bytes().to_vec()).collect(); + for value in hex { + if !value.is_ascii() || value.len() % 2 != 0 { + return Err(anyhow!("invalid --match-hex")); + } + let bytes = (0..value.len()) + .step_by(2) + .map(|at| u8::from_str_radix(&value[at..at + 2], 16)) + .collect::, _>>()?; + matches.push(bytes); + } + Ok(matches) +} + +trait FilterReader { + fn add(&mut self, payload: &[u8]); + fn clear(&mut self); + fn position(&mut self, cfg: &ReadConfig<'_>); + fn step(&mut self, direction: Direction) -> Result; + fn realtime(&self) -> Result; + fn payloads(&mut self, visitor: &mut dyn FnMut(&[u8]) -> journal::Result<()>) -> Result<()>; +} + +impl FilterReader for FileReader { + fn add(&mut self, payload: &[u8]) { + self.add_match(payload); + } + fn clear(&mut self) { + self.flush_matches(); + } + fn position(&mut self, cfg: &ReadConfig<'_>) { + if let Some(time) = cfg.filter_realtime { + self.seek_realtime(time); + } else if cfg.direction == Direction::Forward { + self.seek_head(); + } else { + self.seek_tail(); + } + } + fn step(&mut self, direction: Direction) -> Result { + Ok(if direction == Direction::Forward { + self.next()? + } else { + self.previous()? + }) + } + fn realtime(&self) -> Result { + Ok(self.get_realtime_usec()?) + } + fn payloads(&mut self, visitor: &mut dyn FnMut(&[u8]) -> journal::Result<()>) -> Result<()> { + Ok(self.visit_entry_payloads(visitor)?) + } +} + +impl FilterReader for DirectoryReader { + fn add(&mut self, payload: &[u8]) { + self.add_match(payload); + } + fn clear(&mut self) { + self.flush_matches(); + } + fn position(&mut self, cfg: &ReadConfig<'_>) { + if let Some(time) = cfg.filter_realtime { + self.seek_realtime(time); + } else if cfg.direction == Direction::Forward { + self.seek_head(); + } else { + self.seek_tail(); + } + } + fn step(&mut self, direction: Direction) -> Result { + Ok(if direction == Direction::Forward { + self.next()? + } else { + self.previous()? + }) + } + fn realtime(&self) -> Result { + Ok(self.get_realtime_usec()?) + } + fn payloads(&mut self, visitor: &mut dyn FnMut(&[u8]) -> journal::Result<()>) -> Result<()> { + Ok(self.visit_entry_payloads(visitor)?) + } +} + +impl FilterReader for SdJournal { + fn add(&mut self, payload: &[u8]) { + self.add_match(payload); + } + fn clear(&mut self) { + self.flush_matches(); + } + fn position(&mut self, cfg: &ReadConfig<'_>) { + if let Some(time) = cfg.filter_realtime { + self.seek_realtime_usec(time); + } else if cfg.direction == Direction::Forward { + self.seek_head(); + } else { + self.seek_tail(); + } + } + fn step(&mut self, direction: Direction) -> Result { + Ok(if direction == Direction::Forward { + self.next()? != 0 + } else { + self.previous()? != 0 + }) + } + fn realtime(&self) -> Result { + Ok(self.get_realtime_usec()?) + } + fn payloads(&mut self, visitor: &mut dyn FnMut(&[u8]) -> journal::Result<()>) -> Result<()> { + self.restart_data()?; + while let Some(payload) = self.enumerate_available_data()? { + visitor(payload)?; + } + Ok(()) + } +} + +fn scan(mut reader: R, cfg: &ReadConfig<'_>) -> Result { + let install = |reader: &mut R| { + reader.clear(); + for value in cfg.filter_matches { + reader.add(value); + } + }; + install(&mut reader); + let mut counts = Counts::default(); + for repetition in 0..cfg.filter_repetitions { + if repetition != 0 && cfg.filter_rebuild { + install(&mut reader); + } + reader.position(cfg); + let mut count = 0; + while (cfg.filter_limit == 0 || count < cfg.filter_limit) && reader.step(cfg.direction)? { + counts.add_record_marker(reader.realtime()?); + if cfg.mode.ends_with("-digest") { + reader.payloads(&mut |payload| { + counts.add_full_payload(payload); + Ok(()) + })?; + } + count += 1; + } + } + black_box(counts.checksum); + Ok(counts) +} + +pub(super) fn read(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { + let options = reader_options(cfg.bounds, cfg.strategy, cfg.window_size)?; + if cfg.mode.starts_with("facade-") { + let paths = utf8_input_paths(inputs)?; + return scan( + open_facade_reader(&borrowed_input_paths(&paths), cfg.surface, options)?, + cfg, + ); + } + match cfg.surface { + "file" if inputs.len() == 1 => { + scan(FileReader::open_with_options(&inputs[0], options)?, cfg) + } + "directory" if inputs.len() == 1 => scan( + DirectoryReader::open_with_options(&inputs[0], options)?, + cfg, + ), + "open-files" => scan( + DirectoryReader::open_files_with_options(inputs.iter(), options)?, + cfg, + ), + _ => Err(anyhow!("invalid filter benchmark input/surface")), + } +} diff --git a/rust/src/internal/testcmd/reader_core_bench/src/main.rs b/rust/src/internal/testcmd/reader_core_bench/src/main.rs index 406df3cc..0ef8563b 100644 --- a/rust/src/internal/testcmd/reader_core_bench/src/main.rs +++ b/rust/src/internal/testcmd/reader_core_bench/src/main.rs @@ -14,6 +14,9 @@ use std::num::NonZeroU64; use std::path::{Path, PathBuf}; use std::time::Instant; +mod auxiliary; +mod filter; + const DEFAULT_WINDOW_SIZE: u64 = 32 * 1024 * 1024; #[derive(Parser, Debug)] @@ -32,6 +35,10 @@ struct Args { bounds: String, #[arg(long, default_value = "windowed")] mmap_strategy: String, + #[arg(long, default_value = "HIGH_CARD_00")] + catalog_field: String, + #[arg(long, default_value_t = 1)] + catalog_repetitions: usize, #[arg(long = "explorer-facet")] explorer_facets: Vec, #[arg(long = "explorer-filter")] @@ -52,6 +59,18 @@ struct Args { explorer_after_usec: Option, #[arg(long)] explorer_before_usec: Option, + #[arg(long = "match")] + filter_matches: Vec, + #[arg(long = "match-hex")] + filter_matches_hex: Vec, + #[arg(long, default_value_t = 1)] + filter_repetitions: usize, + #[arg(long, default_value_t = 0)] + filter_limit: u64, + #[arg(long)] + filter_realtime: Option, + #[arg(long, default_value_t = false)] + filter_rebuild: bool, } #[derive(Default)] @@ -70,6 +89,15 @@ impl Counts { self.checksum = checksum_payload(self.checksum, payload); } + fn add_full_payload(&mut self, payload: &[u8]) { + self.fields = self.fields.saturating_add(1); + self.bytes = self.bytes.saturating_add(payload.len() as u64); + self.checksum = self.checksum.rotate_left(5) ^ payload.len() as u64; + for &byte in payload { + self.checksum = (self.checksum ^ u64::from(byte)).wrapping_mul(1099511628211); + } + } + fn add_record_marker(&mut self, value: u64) { self.records = self.records.saturating_add(1); self.checksum = self.checksum.rotate_left(7) ^ value; @@ -83,6 +111,8 @@ struct ReadConfig<'a> { bounds: &'a str, strategy: ExperimentalMmapStrategy, window_size: u64, + catalog_field: &'a str, + catalog_repetitions: usize, explorer_facets: &'a [String], explorer_filters: &'a [String], explorer_histogram: Option<&'a str>, @@ -93,6 +123,11 @@ struct ReadConfig<'a> { explorer_strategy: ExplorerStrategy, explorer_after_usec: Option, explorer_before_usec: Option, + filter_matches: &'a [Vec], + filter_repetitions: usize, + filter_limit: u64, + filter_realtime: Option, + filter_rebuild: bool, } #[derive(Debug, Clone, Copy)] @@ -564,13 +599,40 @@ fn read_sdk_file(path: &Path, cfg: &ReadConfig<'_>) -> Result { Direction::Backward => reader.seek_tail(), } let mut counts = Counts::default(); + if cfg.mode == "sdk-digest" { + counts.extra.insert( + "mmap_stats_before".into(), + mmap_stats_json(reader.mmap_stats()?), + ); + } while step_file_reader(&mut reader, cfg.direction)? { record_file_reader_mode(cfg.mode, &mut reader, &mut counts)?; } + if cfg.mode == "sdk-digest" { + counts.extra.insert( + "mmap_stats_after".into(), + mmap_stats_json(reader.mmap_stats()?), + ); + } black_box(counts.checksum); Ok(counts) } +fn mmap_stats_json(stats: journal_core::file::WindowManagerStats) -> Value { + json!({ + "file_size": stats.file_size, + "window_count": stats.window_count, + "row_pin_count": stats.row_pin_count, + "row_pin_limit": stats.row_pin_limit, + "row_overflow_object_count": stats.row_overflow_object_count, + "current_mapped_bytes": stats.current_mapped_bytes, + "max_mapped_bytes": stats.max_mapped_bytes, + "map_count": stats.map_count, + "remap_count": stats.remap_count, + "eviction_count": stats.eviction_count, + }) +} + fn configure_reader_filters(reader: &mut FileReader, cfg: &ReadConfig<'_>) -> Result<()> { reader.flush_matches(); for filter in parse_explorer_filters(cfg.explorer_filters)? { @@ -740,6 +802,9 @@ fn record_file_reader_mode(mode: &str, reader: &mut FileReader, counts: &mut Cou "sdk-payloads" => record_payload_visitor(reader.get_realtime_usec()?, counts, |visitor| { reader.visit_entry_payloads(visitor) }), + "sdk-digest" => record_payload_digest(reader.get_realtime_usec()?, counts, |visitor| { + reader.visit_entry_payloads(visitor) + }), other => Err(anyhow!("invalid SDK file mode: {other}")), } } @@ -792,6 +857,9 @@ fn record_directory_reader_mode( "sdk-payloads" => record_payload_visitor(reader.get_realtime_usec()?, counts, |visitor| { reader.visit_entry_payloads(visitor) }), + "sdk-digest" => record_payload_digest(reader.get_realtime_usec()?, counts, |visitor| { + reader.visit_entry_payloads(visitor) + }), other => Err(anyhow!("invalid SDK directory mode: {other}")), } } @@ -808,6 +876,18 @@ where Ok(()) } +fn record_payload_digest(realtime: u64, counts: &mut Counts, visit: F) -> Result<()> +where + F: FnOnce(&mut dyn FnMut(&[u8]) -> journal::Result<()>) -> journal::Result<()>, +{ + counts.add_record_marker(realtime); + visit(&mut |payload| { + counts.add_full_payload(black_box(payload)); + Ok(()) + })?; + Ok(()) +} + fn read_facade(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { let options = reader_options(cfg.bounds, cfg.strategy, cfg.window_size)?; let owned_paths = utf8_input_paths(inputs)?; @@ -889,11 +969,26 @@ fn record_facade_mode( } Ok(()) } + "facade-digest" => { + counts.add_record_marker(journal.get_realtime_usec()?); + SdJournalRestartData(journal)?; + while let Some(payload) = SdJournalEnumerateAvailableData(journal)? { + counts.add_full_payload(black_box(payload)); + } + Ok(()) + } other => Err(anyhow!("invalid facade mode: {other}")), } } fn run(args: &Args) -> Result<(Counts, f64, Value, Value)> { + if args.catalog_repetitions == 0 { + return Err(anyhow!("catalog-repetitions must be positive")); + } + if args.filter_repetitions == 0 { + return Err(anyhow!("filter-repetitions must be positive")); + } + let matches = filter::decode_matches(&args.filter_matches, &args.filter_matches_hex)?; let cfg = ReadConfig { mode: &args.mode, surface: &args.surface, @@ -901,6 +996,8 @@ fn run(args: &Args) -> Result<(Counts, f64, Value, Value)> { bounds: &args.bounds, strategy: parse_mmap_strategy(&args.mmap_strategy)?, window_size: args.window_size, + catalog_field: &args.catalog_field, + catalog_repetitions: args.catalog_repetitions, explorer_facets: &args.explorer_facets, explorer_filters: &args.explorer_filters, explorer_histogram: args.explorer_histogram.as_deref(), @@ -911,6 +1008,11 @@ fn run(args: &Args) -> Result<(Counts, f64, Value, Value)> { explorer_strategy: parse_explorer_strategy(&args.explorer_strategy)?, explorer_after_usec: args.explorer_after_usec, explorer_before_usec: args.explorer_before_usec, + filter_matches: &matches, + filter_repetitions: args.filter_repetitions, + filter_limit: args.filter_limit, + filter_realtime: args.filter_realtime, + filter_rebuild: args.filter_rebuild, }; let status_before = process_status_kb(); let started = Instant::now(); @@ -922,6 +1024,25 @@ fn run(args: &Args) -> Result<(Counts, f64, Value, Value)> { fn dispatch_read(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { match cfg.mode { + "indexed-open" + | "indexed-exact" + | "indexed-field" + | "indexed-all" + | "indexed-exact-digest" + | "indexed-field-digest" + | "indexed-all-digest" + | "verify-index" + | "verify-file" => auxiliary::read(inputs, cfg), + "sdk-filter" | "sdk-filter-digest" | "facade-filter" | "facade-filter-digest" => { + filter::read(inputs, cfg) + } + "sdk-fields" | "sdk-unique" | "sdk-unique-state" => { + if matches!(cfg.surface, "directory" | "open-files") { + return read_directory_catalog(inputs, cfg); + } + require_single_file_input(inputs, cfg.surface, "catalog modes")?; + read_catalog(&inputs[0], cfg) + } "core-next" | "core-offsets" | "core-payloads" => { require_single_file_input(inputs, cfg.surface, "core modes")?; read_core(&inputs[0], cfg) @@ -938,12 +1059,80 @@ fn dispatch_read(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { require_single_file_input(inputs, cfg.surface, "explorer query mode")?; read_explorer_query_file(&inputs[0], cfg) } - "sdk-entry" | "sdk-payloads" => dispatch_sdk_read(inputs, cfg), - "facade-next" | "facade-data" => read_facade(inputs, cfg), + "sdk-entry" | "sdk-payloads" | "sdk-digest" => dispatch_sdk_read(inputs, cfg), + "facade-next" | "facade-data" | "facade-digest" => read_facade(inputs, cfg), other => Err(anyhow!("invalid --mode: {other}")), } } +fn read_catalog(path: &Path, cfg: &ReadConfig<'_>) -> Result { + let options = reader_options(cfg.bounds, cfg.strategy, cfg.window_size)?; + let mut reader = FileReader::open_with_options(path, options)?; + let mut counts = Counts::default(); + if cfg.mode == "sdk-unique-state" { + reader.query_unique_state(cfg.catalog_field)?; + } + for _ in 0..cfg.catalog_repetitions { + counts.add_record_marker(0); + match cfg.mode { + "sdk-fields" => { + for field in reader.enumerate_fields()? { + counts.add_payload(field.as_bytes()); + } + } + "sdk-unique" => reader.visit_unique_values(cfg.catalog_field, |value| { + counts.add_payload(value); + Ok(()) + })?, + "sdk-unique-state" => { + reader.restart_unique_state(); + while let Some(value) = reader.enumerate_unique_payload(cfg.catalog_field)? { + counts.add_payload(&value); + } + } + _ => unreachable!(), + } + } + Ok(counts) +} + +fn read_directory_catalog(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { + let options = reader_options(cfg.bounds, cfg.strategy, cfg.window_size)?; + let mut reader = match cfg.surface { + "directory" if inputs.len() == 1 => { + DirectoryReader::open_with_options(&inputs[0], options)? + } + "open-files" => DirectoryReader::open_files_with_options(inputs, options)?, + _ => return Err(anyhow!("directory catalog requires one directory input")), + }; + let mut counts = Counts::default(); + if cfg.mode == "sdk-unique-state" { + reader.query_unique_state(cfg.catalog_field)?; + } + for _ in 0..cfg.catalog_repetitions { + counts.add_record_marker(0); + match cfg.mode { + "sdk-fields" => { + for field in reader.enumerate_fields()? { + counts.add_payload(field.as_bytes()); + } + } + "sdk-unique" => reader.visit_unique_values(cfg.catalog_field, |value| { + counts.add_payload(value); + Ok(()) + })?, + "sdk-unique-state" => { + reader.restart_unique_state(); + while let Some(value) = reader.enumerate_unique_payload()? { + counts.add_payload(&value); + } + } + _ => unreachable!(), + } + } + Ok(counts) +} + fn dispatch_sdk_read(inputs: &[PathBuf], cfg: &ReadConfig<'_>) -> Result { match cfg.surface { "file" => { @@ -976,6 +1165,11 @@ fn main() -> Result<()> { "fields": counts.fields, "bytes": counts.bytes, "checksum": counts.checksum, + "checksum_algorithm": if args.mode.ends_with("-digest") { + "full-payload-fnv64-v1" + } else { + "sampled-payload-v1" + }, "extra": counts.extra, "read_seconds": read_seconds, "read_rows_per_second": if read_seconds > 0.0 { counts.records as f64 / read_seconds } else { 0.0 }, diff --git a/rust/src/journal/src/directory.rs b/rust/src/journal/src/directory.rs index 0a7857a2..3f9e07cd 100644 --- a/rust/src/journal/src/directory.rs +++ b/rust/src/journal/src/directory.rs @@ -18,6 +18,8 @@ pub struct DirectoryReader { #[cfg(test)] unique_cache_builds: usize, pub(super) non_overlapping: bool, + recovery_changed: bool, + merging: bool, } #[derive(Debug, Clone, PartialEq, Eq, Hash)] @@ -28,6 +30,7 @@ struct DirectoryUniqueCacheKey { #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] struct DirectoryUniqueFileSignature { + recovery_reason: RecoveryReason, file_id: [u8; 16], n_objects: u64, n_entries: u64, @@ -76,6 +79,18 @@ struct DirectoryBootNewest { } impl DirectoryReader { + pub fn recovery_statuses(&self) -> Vec { + self.files + .iter() + .filter_map(|file| { + let status = file.recovery_status(); + status.incomplete().then_some(FileRecoveryStatus { + file_id: file.header_snapshot.header.file_id, + status, + }) + }) + .collect() + } pub fn open(path: impl AsRef) -> Result { Self::open_with_options(path, ReaderOptions::default()) } @@ -127,7 +142,7 @@ impl DirectoryReader { Self::from_readers(files, false) } - fn from_readers(mut files: Vec, allow_empty: bool) -> Result { + pub(super) fn from_readers(mut files: Vec, allow_empty: bool) -> Result { if files.is_empty() && !allow_empty { return Err(SdkError::InvalidPath( "no readable journal files".to_string(), @@ -137,6 +152,9 @@ impl DirectoryReader { files.sort_by_key(FileReader::header_realtime_start); let boot_newest = build_directory_boot_newest(&files); let non_overlapping = directory_files_non_overlapping(&files); + for reader in &files { + reader.take_recovery_change(); + } let candidates = vec![None; files.len()]; Ok(Self { files, @@ -153,10 +171,13 @@ impl DirectoryReader { #[cfg(test)] unique_cache_builds: 0, non_overlapping, + recovery_changed: false, + merging: false, }) } pub fn seek_head(&mut self) { + self.merging = false; self.pending_realtime_seek = None; self.realtime_seek_bound = None; self.index = usize::MAX; @@ -169,6 +190,7 @@ impl DirectoryReader { } pub fn seek_tail(&mut self) { + self.merging = false; self.pending_realtime_seek = None; self.realtime_seek_bound = None; self.index = usize::MAX; @@ -181,6 +203,7 @@ impl DirectoryReader { } pub fn seek_realtime(&mut self, usec: u64) { + self.merging = false; self.pending_realtime_seek = Some(usec); self.realtime_seek_bound = None; self.index = usize::MAX; @@ -198,42 +221,93 @@ impl DirectoryReader { } fn step_merged(&mut self, direction: Direction) -> Result { + let index = self.index; + let before = if self.current_key.is_some() { + self.files.get(index).and_then(|reader| { + reader + .inner + .with_reader(|reader| reader.get_entry_offset().ok()) + }) + } else { + None + }; + let result = self.step_merged_inner(direction); + if result.is_err() { + self.index = index; + if let Some(offset) = before { + let reader = &mut self.files[index]; + reader + .inner + .with_reader_mut(|reader| reader.release_object_guards()); + reader + .inner + .with_file(|file| reader.row.clear_current_best_effort(file)); + reader + .inner + .with_reader_mut(|reader| reader.set_location(Location::ResolvedEntry(offset))); + // Other children still own valid prefetched candidates. + self.candidates[index] = None; + } + } + result + } + + fn step_merged_inner(&mut self, direction: Direction) -> Result { if self.can_step_sequential(direction) { return self.step_sequential(direction); } - self.prepare_merge_direction(direction); + self.merging = true; - let mut best: Option = None; - for idx in 0..self.files.len() { - self.fill_candidate(idx, direction)?; - let Some(candidate) = self.candidates[idx] else { - continue; - }; - let replace = match best { - None => true, - Some(current) => { - let cmp = self.compare_entry_keys(candidate.key, current.key); - (direction == Direction::Forward && cmp < 0) - || (direction == Direction::Backward && cmp > 0) + 'retry: loop { + self.prepare_merge_direction(direction); + + let mut best: Option = None; + for idx in 0..self.files.len() { + self.fill_candidate(idx, direction)?; + if self.recovery_changed { + self.restart_recovery_merge(); + continue 'retry; + } + let Some(candidate) = self.candidates[idx] else { + continue; + }; + let replace = match best { + None => true, + Some(current) => { + let cmp = self.compare_entry_keys(candidate.key, current.key); + (direction == Direction::Forward && cmp < 0) + || (direction == Direction::Backward && cmp > 0) + } + }; + if replace { + best = Some(candidate); } - }; - if replace { - best = Some(candidate); } - } - let Some(best) = best else { - self.index = usize::MAX; + let Some(best) = best else { + self.index = usize::MAX; + self.realtime_seek_bound = None; + return Ok(false); + }; + + self.index = best.reader_index; + self.current_key = Some(best.key); + self.candidates[best.reader_index] = None; self.realtime_seek_bound = None; - return Ok(false); - }; + return Ok(true); + } + } - self.index = best.reader_index; - self.current_key = Some(best.key); - self.candidates[best.reader_index] = None; - self.realtime_seek_bound = None; - Ok(true) + fn restart_recovery_merge(&mut self) { + self.recovery_changed = false; + self.boot_newest = build_directory_boot_newest(&self.files); + self.non_overlapping = false; + if self.current_key.is_none() { + self.pending_realtime_seek = self.realtime_seek_bound.map(|(usec, _)| usec); + } + self.direction = None; + self.reset_candidates(); } fn prepare_merge_direction(&mut self, direction: Direction) { @@ -278,6 +352,9 @@ impl DirectoryReader { if !self.advance_candidate_reader(reader_index, direction)? { return Ok(()); } + if self.recovery_changed { + return Ok(()); + } let key = self.files[reader_index].current_directory_entry_key()?; if !self.candidate_matches_realtime_bound(key) { continue; @@ -296,10 +373,12 @@ impl DirectoryReader { reader_index: usize, direction: Direction, ) -> Result { - match direction { + let result = match direction { Direction::Forward => self.files[reader_index].next(), Direction::Backward => self.files[reader_index].previous(), - } + }; + self.recovery_changed |= self.files[reader_index].take_recovery_change(); + result } fn candidate_matches_realtime_bound(&self, key: DirectoryEntryKey) -> bool { @@ -469,6 +548,105 @@ impl DirectoryReader { self.files[self.index].test_cursor(cursor) } + /// Position each file by its native cursor indexes and select the merged boundary. + pub fn seek_cursor_direction(&mut self, cursor: &str, direction: Direction) -> Result<()> { + let anchor = parse::directional_cursor_anchor(cursor) + .map_err(|err| SdkError::InvalidCursor(err.to_string()))?; + loop { + self.index = self.files.len(); + self.current_key = None; + self.pending_realtime_seek = None; + self.realtime_seek_bound = None; + self.reset_candidates(); + self.direction = Some(direction); + self.merging = true; + self.recovery_changed = false; + let mut best: Option = None; + let mut exact = None; + for i in 0..self.files.len() { + let result = self.files[i].seek_cursor_anchor(&anchor, direction); + self.recovery_changed |= self.files[i].take_recovery_change(); + result?; + let key = match self.files[i].current_directory_entry_key() { + Err(SdkError::NoEntry | SdkError::Journal(JournalError::UnsetCursor)) => { + continue; + } + result => result?, + }; + self.candidates[i] = Some(DirectoryCandidate { + reader_index: i, + key, + }); + if anchor.sequence == Some((key.seqnum_id, key.seqnum)) { + exact = Some(i); + } + if let Some(old) = best { + let cmp = self.compare_entry_keys(key, self.candidates[old].unwrap().key); + if direction == Direction::Forward && cmp < 0 + || direction == Direction::Backward && cmp > 0 + { + best = Some(i); + } + } else { + best = Some(i); + } + } + if self.recovery_changed { + self.boot_newest = build_directory_boot_newest(&self.files); + continue; + } + let Some(best) = exact.or(best) else { + return Ok(()); + }; + let key = self.candidates[best].unwrap().key; + let next_anchor = journal_core::file::CursorAnchor { + sequence: Some((key.seqnum_id, key.seqnum)), + monotonic: Some((key.boot_id, key.monotonic)), + realtime: Some(key.realtime), + }; + for i in 0..self.files.len() { + if i == best { + continue; + } + if let Some(candidate) = self.candidates[i] { + let cmp = self.compare_entry_keys(candidate.key, key); + if direction == Direction::Forward && cmp > 0 + || direction == Direction::Backward && cmp < 0 + { + continue; + } + self.candidates[i] = None; + } + let result = self.files[i].seek_cursor_anchor(&next_anchor, direction); + self.recovery_changed |= self.files[i].take_recovery_change(); + result?; + let candidate = match self.files[i].current_directory_entry_key() { + Err(SdkError::NoEntry | SdkError::Journal(JournalError::UnsetCursor)) => { + continue; + } + result => result?, + }; + let cmp = self.compare_entry_keys(candidate, key); + if direction == Direction::Forward && cmp > 0 + || direction == Direction::Backward && cmp < 0 + { + self.candidates[i] = Some(DirectoryCandidate { + reader_index: i, + key: candidate, + }); + } + } + if self.recovery_changed { + self.boot_newest = build_directory_boot_newest(&self.files); + continue; + } + self.index = best; + self.current_key = Some(key); + self.candidates[best] = None; + return Ok(()); + } + } + pub fn seek_cursor(&mut self, cursor: &str) -> Result<()> { let want = parse::parse_cursor_location(cursor, true) .map_err(|err| SdkError::InvalidCursor(err.to_string()))?; @@ -644,8 +822,9 @@ impl DirectoryReader { .files .iter() .map(|reader| { - let header = reader.header(); + let header = reader.effective_header(); DirectoryUniqueFileSignature { + recovery_reason: reader.recovery_status().reason, file_id: header.file_id, n_objects: header.n_objects, n_entries: header.n_entries, @@ -671,7 +850,10 @@ impl DirectoryReader { pub fn list_boots(&self) -> Vec { let mut boots: HashMap = HashMap::new(); for reader in &self.files { - let header = reader.cached_header().header; + let header = reader.effective_header(); + if reader.recovery_status().incomplete() && header.n_entries == 0 { + continue; + } let boot_id = hex::encode(header.tail_entry_boot_id); let first = header.head_entry_realtime as i64; let last = header.tail_entry_realtime as i64; @@ -702,25 +884,25 @@ impl DirectoryReader { } pub fn add_match(&mut self, data: &[u8]) { + self.invalidate_filter_candidates(); for reader in &mut self.files { reader.add_match(data); } - self.reset_merge_state(); } pub fn add_conjunction(&mut self) -> Result<()> { + self.invalidate_filter_candidates(); for reader in &mut self.files { reader.add_conjunction()?; } - self.reset_merge_state(); Ok(()) } pub fn add_disjunction(&mut self) -> Result<()> { + self.invalidate_filter_candidates(); for reader in &mut self.files { reader.add_disjunction()?; } - self.reset_merge_state(); Ok(()) } @@ -728,10 +910,33 @@ impl DirectoryReader { for reader in &mut self.files { reader.flush_matches(); } + self.pending_realtime_seek = None; self.reset_merge_state(); } + fn invalidate_filter_candidates(&mut self) { + if self.non_overlapping + && !self.merging + && self.direction.is_some() + && self.current_key.is_some() + && self.index < self.files.len() + { + // Sequential children have no prefetched rows to reposition. + self.reset_candidates(); + return; + } + if self.current_key.is_none() && self.pending_realtime_seek.is_none() { + self.pending_realtime_seek = self.realtime_seek_bound.map(|(usec, _)| usec); + } + // Revalidate changed matches from the committed key, not prefetched rows. + self.merging = self.current_key.is_some(); + self.direction = None; + self.realtime_seek_bound = None; + self.reset_candidates(); + } + fn reset_merge_state(&mut self) { + self.merging = false; self.index = usize::MAX; self.current_key = None; self.direction = None; @@ -740,7 +945,7 @@ impl DirectoryReader { } fn can_step_sequential(&self, direction: Direction) -> bool { - if !self.non_overlapping || self.pending_realtime_seek.is_some() { + if !self.non_overlapping || self.merging || self.pending_realtime_seek.is_some() { return false; } if self.direction.is_some_and(|current| current != direction) && self.current_key.is_some() @@ -792,7 +997,12 @@ impl DirectoryReader { self.index = 0; } while self.index < self.files.len() { - if self.files[self.index].next()? { + let advanced = self.advance_candidate_reader(self.index, Direction::Forward)?; + if self.recovery_changed { + self.restart_recovery_merge(); + return self.step_merged_inner(Direction::Forward); + } + if advanced { self.current_key = Some(self.files[self.index].current_directory_entry_key()?); return Ok(true); } @@ -806,7 +1016,12 @@ impl DirectoryReader { self.index = self.files.len() - 1; } loop { - if self.files[self.index].previous()? { + let advanced = self.advance_candidate_reader(self.index, Direction::Backward)?; + if self.recovery_changed { + self.restart_recovery_merge(); + return self.step_merged_inner(Direction::Backward); + } + if advanced { self.current_key = Some(self.files[self.index].current_directory_entry_key()?); return Ok(true); } @@ -819,7 +1034,14 @@ impl DirectoryReader { } fn finish_sequential_end(&mut self) -> Result { - self.clear_current_directory_entry(); + self.index = usize::MAX; + if !self + .files + .first() + .is_some_and(|reader| reader.inner.with_reader(|reader| reader.has_filter())) + { + self.current_key = None; + } Ok(false) } @@ -831,12 +1053,12 @@ impl DirectoryReader { pub(super) fn is_journal_file_name(path: &Path) -> bool { path.file_name() - .and_then(|name| name.to_str()) + .map(|name| name.as_encoded_bytes()) .is_some_and(|name| { - name.ends_with(".journal") - || name.ends_with(".journal~") - || name.ends_with(".journal.zst") - || name.ends_with(".journal~.zst") + name.ends_with(b".journal") + || name.ends_with(b".journal~") + || name.ends_with(b".journal.zst") + || name.ends_with(b".journal~.zst") }) } @@ -920,21 +1142,21 @@ fn id128_string_valid(s: &str) -> bool { fn build_directory_boot_newest(files: &[FileReader]) -> HashMap<[u8; 16], DirectoryBootNewest> { let mut newest: HashMap<[u8; 16], DirectoryBootNewest> = HashMap::new(); for reader in files { - let header = reader.cached_header(); - if header.header.tail_entry_boot_id == [0; 16] { + let header = reader.effective_header(); + if header.tail_entry_boot_id == [0; 16] { continue; } - let replace = match newest.get(&header.header.tail_entry_boot_id) { + let replace = match newest.get(&header.tail_entry_boot_id) { None => true, - Some(current) => header.header.tail_entry_monotonic > current.monotonic, + Some(current) => header.tail_entry_monotonic > current.monotonic, }; if replace { newest.insert( - header.header.tail_entry_boot_id, + header.tail_entry_boot_id, DirectoryBootNewest { - machine_id: header.header.machine_id, - monotonic: header.header.tail_entry_monotonic, - realtime: header.header.tail_entry_realtime, + machine_id: header.machine_id, + monotonic: header.tail_entry_monotonic, + realtime: header.tail_entry_realtime, }, ); } diff --git a/rust/src/journal/src/explorer.rs b/rust/src/journal/src/explorer.rs index 419f254d..00955435 100644 --- a/rust/src/journal/src/explorer.rs +++ b/rust/src/journal/src/explorer.rs @@ -1478,9 +1478,7 @@ impl FileReader { && query.after_realtime_usec.is_none() && query.before_realtime_usec.is_none() { - let count = self - .inner - .with_file(|file| file.journal_header_ref().n_entries); + let count = self.inner.with_file(|file| file.reader_entry_count()); return Ok(IndexedCandidateSet::All { count }); } @@ -2295,7 +2293,7 @@ fn indexed_count_facet_group( for item in file.field_data_objects_with_offsets(field)? { let (_, data) = item?; let Some((value, cursor)) = - indexed_value_and_cursor(&data, field, &mut decompressed, &mut result.stats)? + indexed_value_and_cursor(file, &data, field, &mut decompressed, &mut result.stats)? else { continue; }; @@ -2344,7 +2342,7 @@ fn indexed_count_histogram( for item in file.field_data_objects_with_offsets(&field)? { let (_, data) = item?; let Some((value, cursor)) = - indexed_value_and_cursor(&data, &field, &mut decompressed, &mut result.stats)? + indexed_value_and_cursor(file, &data, &field, &mut decompressed, &mut result.stats)? else { continue; }; @@ -2375,6 +2373,7 @@ fn indexed_count_histogram( } fn indexed_value_and_cursor( + file: &JournalFile, data: &DataObject<&[u8]>, field: &[u8], decompressed: &mut Vec, @@ -2397,7 +2396,10 @@ fn indexed_value_and_cursor( if payload_field != field { return Ok(None); } - Ok(Some((value.to_vec(), data.inlined_cursor()))) + Ok(Some(( + value.to_vec(), + file.reader_inlined_cursor(&data.header)?, + ))) } fn indexed_count_facet_entries( @@ -2550,6 +2552,10 @@ fn indexed_visit_entries( where F: FnMut(NonZeroU64) -> Result<()>, { + let cursor = cursor + .map(|cursor| cursor.for_reader(file)) + .transpose()? + .flatten(); let Some(mut cursor) = cursor.map(|cursor| cursor.head()) else { return Ok(()); }; diff --git a/rust/src/journal/src/facade.rs b/rust/src/journal/src/facade.rs index ed9bd6fe..ee2273ad 100644 --- a/rust/src/journal/src/facade.rs +++ b/rust/src/journal/src/facade.rs @@ -66,12 +66,20 @@ pub struct SdJournal { pub type UniqueValue = (String, Vec); +pub fn SdJournalRecoveryStatuses(journal: &SdJournal) -> Vec { + journal.recovery_statuses() +} + pub fn SdJournalOpen(path: &str, flags: u32) -> std::result::Result { + SdJournalOpenPath(Path::new(path), flags) +} + +/// Opens a native filesystem path without requiring UTF-8 encoding. +pub fn SdJournalOpenPath(path: &Path, flags: u32) -> std::result::Result { if flags != 0 { return Err(Error::Unsupported); } - let path = Path::new(path); let reader = if path.is_dir() { ReaderKind::Directory(DirectoryReader::open(path).map_err(map_error)?) } else { @@ -89,6 +97,20 @@ pub fn SdJournalOpenFileWithOptions( path: &str, flags: u32, options: ReaderOptions, +) -> std::result::Result { + SdJournalOpenFilePathWithOptions(Path::new(path), flags, options) +} + +/// Opens a native file path with the default reader options. +pub fn SdJournalOpenFilePath(path: &Path, flags: u32) -> std::result::Result { + SdJournalOpenFilePathWithOptions(path, flags, ReaderOptions::default()) +} + +/// Opens a native file path with explicit reader options. +pub fn SdJournalOpenFilePathWithOptions( + path: &Path, + flags: u32, + options: ReaderOptions, ) -> std::result::Result { if flags != 0 { return Err(Error::Unsupported); @@ -106,6 +128,23 @@ pub fn SdJournalOpenDirectoryWithOptions( path: &str, flags: u32, options: ReaderOptions, +) -> std::result::Result { + SdJournalOpenDirectoryPathWithOptions(Path::new(path), flags, options) +} + +/// Opens a native directory path with the default reader options. +pub fn SdJournalOpenDirectoryPath( + path: &Path, + flags: u32, +) -> std::result::Result { + SdJournalOpenDirectoryPathWithOptions(path, flags, ReaderOptions::default()) +} + +/// Opens a native directory path with explicit reader options. +pub fn SdJournalOpenDirectoryPathWithOptions( + path: &Path, + flags: u32, + options: ReaderOptions, ) -> std::result::Result { if flags != 0 { return Err(Error::Unsupported); @@ -123,26 +162,47 @@ pub fn SdJournalOpenFilesWithOptions( paths: &[&str], flags: u32, options: ReaderOptions, +) -> std::result::Result { + SdJournalOpenFilePathsWithOptions(paths, flags, options) +} + +/// Opens explicit native paths, deduplicating canonical aliases. +pub fn SdJournalOpenFilePaths>( + paths: &[P], + flags: u32, +) -> std::result::Result { + SdJournalOpenFilePathsWithOptions(paths, flags, ReaderOptions::default()) +} + +/// Opens explicit native paths with reader options and canonical deduplication. +pub fn SdJournalOpenFilePathsWithOptions>( + paths: &[P], + flags: u32, + options: ReaderOptions, ) -> std::result::Result { if flags != 0 { return Err(Error::Unsupported); } + if paths.len() == 1 { + return SdJournalOpenFilePathWithOptions(paths[0].as_ref(), flags, options); + } let paths = dedupe_open_file_paths(paths); if paths.len() == 1 { - return SdJournalOpenFileWithOptions(&paths[0], flags, options); + return SdJournalOpenFilePathWithOptions(&paths[0], flags, options); } Ok(SdJournal::new(ReaderKind::Directory( DirectoryReader::open_files_with_options(&paths, options).map_err(map_error)?, ))) } -fn dedupe_open_file_paths(paths: &[&str]) -> Vec { +fn dedupe_open_file_paths>(paths: &[P]) -> Vec { let mut out = Vec::with_capacity(paths.len()); let mut seen = HashSet::with_capacity(paths.len()); for path in paths { - let key = canonical_open_file_key(Path::new(path)); + let path = path.as_ref(); + let key = canonical_open_file_key(path); if seen.insert(key) { - out.push((*path).to_string()); + out.push(path.to_path_buf()); } } out @@ -157,6 +217,12 @@ pub fn SdJournalClose(j: SdJournal) { } impl SdJournal { + pub fn recovery_statuses(&self) -> Vec { + match &self.reader { + ReaderKind::File(reader) => reader.recovery_statuses(), + ReaderKind::Directory(reader) => reader.recovery_statuses(), + } + } fn new(reader: ReaderKind) -> Self { Self { reader, @@ -261,6 +327,20 @@ impl SdJournal { } } + /// Eager directional extension; the existing facade seek contract is unchanged. + pub fn seek_cursor_direction( + &mut self, + cursor: &str, + direction: crate::Direction, + ) -> std::result::Result<(), Error> { + self.reset_iterators(); + match &mut self.reader { + ReaderKind::File(reader) => reader.seek_cursor_direction(cursor, direction), + ReaderKind::Directory(reader) => reader.seek_cursor_direction(cursor, direction), + } + .map_err(map_error) + } + pub fn seek_cursor(&mut self, cursor: &str) -> std::result::Result<(), Error> { self.reset_iterators(); match &mut self.reader { @@ -291,7 +371,10 @@ impl SdJournal { ReaderKind::File(reader) => reader.get_cursor(), ReaderKind::Directory(reader) => reader.get_cursor(), } - .map_err(map_error) + .map_err(|err| match err { + SdkError::Journal(journal_core::error::JournalError::UnsetCursor) => Error::NoEntry, + err => map_error(err), + }) } pub fn get_seqnum(&self) -> std::result::Result<(u64, [u8; 16]), Error> { @@ -651,3 +734,12 @@ fn map_error(err: SdkError) -> Error { } } } + +/// Eager directional extension to the libsystemd-compatible facade. +pub fn SdJournalSeekCursorDirection( + j: &mut SdJournal, + cursor: &str, + direction: crate::Direction, +) -> std::result::Result<(), Error> { + j.seek_cursor_direction(cursor, direction) +} diff --git a/rust/src/journal/src/lib.rs b/rust/src/journal/src/lib.rs index 5c153271..d6ba7c28 100644 --- a/rust/src/journal/src/lib.rs +++ b/rust/src/journal/src/lib.rs @@ -55,18 +55,21 @@ pub use facade::{ SdJournalEnumerateField, SdJournalEnumerateFields, SdJournalFlushMatches, SdJournalGetCursor, SdJournalGetData, SdJournalGetEntry, SdJournalGetMonotonicUsec, SdJournalGetRealtimeUsec, SdJournalGetSeqnum, SdJournalListBoots, SdJournalNext, SdJournalNextSkip, SdJournalOpen, - SdJournalOpenDirectory, SdJournalOpenDirectoryWithOptions, SdJournalOpenFile, + SdJournalOpenDirectory, SdJournalOpenDirectoryPath, SdJournalOpenDirectoryPathWithOptions, + SdJournalOpenDirectoryWithOptions, SdJournalOpenFile, SdJournalOpenFilePath, + SdJournalOpenFilePathWithOptions, SdJournalOpenFilePaths, SdJournalOpenFilePathsWithOptions, SdJournalOpenFileWithOptions, SdJournalOpenFiles, SdJournalOpenFilesWithOptions, - SdJournalPrevious, SdJournalPreviousSkip, SdJournalProcessOutput, SdJournalQueryUnique, - SdJournalQueryUniqueState, SdJournalRestartData, SdJournalRestartFields, - SdJournalRestartUnique, SdJournalSeekCursor, SdJournalSeekHead, SdJournalSeekRealtimeUsec, - SdJournalSeekTail, SdJournalSetOutputMode, SdJournalTestCursor, SdJournalVisitUniqueValues, + SdJournalOpenPath, SdJournalPrevious, SdJournalPreviousSkip, SdJournalProcessOutput, + SdJournalQueryUnique, SdJournalQueryUniqueState, SdJournalRecoveryStatuses, + SdJournalRestartData, SdJournalRestartFields, SdJournalRestartUnique, SdJournalSeekCursor, + SdJournalSeekCursorDirection, SdJournalSeekHead, SdJournalSeekRealtimeUsec, SdJournalSeekTail, + SdJournalSetOutputMode, SdJournalTestCursor, SdJournalVisitUniqueValues, }; pub use journal_core::error::JournalError; use journal_core::file::ExperimentalMmapStrategy; pub use journal_core::file::{ BucketUtilization, Compression, Direction, EntryItemsType, FieldNamePolicy, HashableObject, - JournalFile, JournalReader, Location, Mmap, WindowManagerStats, + JournalFile, JournalReader, Location, Mmap, RecoveryReason, RecoveryStatus, WindowManagerStats, }; use journal_core::file::{CurrentRowMetadata, CurrentRowView}; pub use journal_log_writer::{ @@ -394,6 +397,12 @@ pub struct FileReader { bounds: ReaderBounds, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FileRecoveryStatus { + pub file_id: [u8; 16], + pub status: RecoveryStatus, +} + fn key_from_metadata(metadata: CurrentRowMetadata) -> DirectoryEntryKey { DirectoryEntryKey { seqnum_id: metadata.seqnum_id, @@ -427,12 +436,22 @@ impl FileReader { } pub fn open_with_options(path: impl AsRef, options: ReaderOptions) -> Result { + Self::open_with_recovery(path, options, true) + } + + pub(crate) fn open_with_recovery( + path: impl AsRef, + options: ReaderOptions, + recovery: bool, + ) -> Result { let path = path.as_ref(); if is_zst_file(path) { - return Self::open_zst(path, options); + return Self::open_zst(path, options, recovery); } - let file = open_journal_file(path, options)?; + let mut file = open_journal_file(path, options)?; + file.set_reader_recovery(recovery); + Self::recover_missing_header_tail(&file)?; let header_snapshot = FileHeaderSnapshot::from_file(&file); Ok(Self { inner: ReaderCellBuilder { @@ -447,15 +466,21 @@ impl FileReader { }) } - fn open_zst(path: &Path, options: ReaderOptions) -> Result { + fn open_zst(path: &Path, options: ReaderOptions, recovery: bool) -> Result { let temp_path = decompress_zst_to_temp(path, "rust-sdk-journal")?; - let file = match open_journal_file(&temp_path, options) { + let mut file = match open_journal_file(&temp_path, options) { Ok(file) => file, Err(err) => { let _ = std::fs::remove_file(&temp_path); return Err(err); } }; + file.set_reader_recovery(recovery); + if let Err(err) = Self::recover_missing_header_tail(&file) { + drop(file); + let _ = std::fs::remove_file(&temp_path); + return Err(err); + } let header_snapshot = FileHeaderSnapshot::from_file(&file); Ok(Self { inner: ReaderCellBuilder { @@ -477,10 +502,57 @@ impl FileReader { self.live_header() } + fn recover_missing_header_tail(file: &JournalFile) -> Result<()> { + let header = file.journal_header_ref(); + let extent = file.reader_file_size()?; + if header + .tail_object_offset + .is_some_and(|offset| offset.get().checked_add(16).is_none_or(|end| end > extent)) + { + file.recover_reader_tail(&JournalError::ObjectExceedsFileBounds)?; + } + Ok(()) + } + + pub fn recovery_status(&self) -> RecoveryStatus { + self.inner.with_file(JournalFile::recovery_status) + } + + pub fn recovery_statuses(&self) -> Vec { + let status = self.recovery_status(); + if status.incomplete() { + vec![FileRecoveryStatus { + file_id: self.header_snapshot.header.file_id, + status, + }] + } else { + Vec::new() + } + } + pub(crate) fn cached_header(&self) -> FileHeaderSnapshot { self.header_snapshot } + pub(crate) fn effective_header(&self) -> FileHeader { + let mut header = self.header(); + self.inner.with_file(|file| { + if let Some((seqnum, realtime, monotonic, boot)) = file.recovered_tail_metadata() { + header.n_entries = file.reader_entry_count(); + header.tail_entry_seqnum = seqnum; + header.tail_entry_realtime = realtime; + header.tail_entry_monotonic = monotonic; + header.tail_entry_boot_id = boot; + } + }); + header + } + + pub(crate) fn take_recovery_change(&self) -> bool { + self.inner + .with_file(JournalFile::take_reader_recovery_change) + } + fn live_header(&self) -> FileHeader { self.inner .with_file(|file| FileHeaderSnapshot::from_file(file).header) @@ -521,6 +593,50 @@ impl FileReader { }); } + /// Position inclusively using native cursor indexes in the requested direction. + /// The existing `seek_cursor` method retains its original behavior. + pub fn seek_cursor_direction(&mut self, cursor: &str, direction: Direction) -> Result<()> { + let anchor = parse::directional_cursor_anchor(cursor) + .map_err(|err| SdkError::InvalidCursor(err.to_string()))?; + self.seek_cursor_anchor(&anchor, direction) + } + + fn seek_cursor_anchor( + &mut self, + anchor: &journal_core::file::CursorAnchor, + direction: Direction, + ) -> Result<()> { + self.inner.with_file(|file| self.row.clear_current(file))?; + loop { + let row = &mut self.row; + let retry = self.inner.with_mut(|fields| { + if !fields + .reader + .seek_cursor_direction(fields.file, anchor, direction)? + { + return Ok(false); + } + let offset = fields.reader.get_entry_offset()?; + match row.load_entry(fields.file, offset) { + Ok(_) => Ok(false), + Err(err) + if fields.reader.recover_after_error( + fields.file, + Location::Head, + &err, + )? => + { + Ok(true) + } + Err(err) => Err(err), + } + })?; + if !retry { + return Ok(()); + } + } + } + pub fn seek_cursor(&mut self, cursor: &str) -> Result<()> { let want = parse::parse_cursor_location(cursor, true) .map_err(|err| SdkError::InvalidCursor(err.to_string()))?; @@ -556,6 +672,12 @@ impl FileReader { loop { let row = &mut self.row; let status = self.inner.with_mut(|fields| { + let before_entry = fields.reader.get_entry_offset().ok(); + let before_anchor = if before_entry.is_none() { + fields.reader.location() + } else { + Location::Head + }; if !fields.reader.step(fields.file, direction)? { return Ok(StepStatus::End); } @@ -566,7 +688,17 @@ impl FileReader { .and_then(|offset| row.load_entry(fields.file, offset)) { Ok(_) => Ok(StepStatus::Valid), - Err(err) if recoverable_entry_error(&err) => Ok(StepStatus::Skip), + Err(err) + if fields.reader.recover_after_error( + fields.file, + before_entry + .map(Location::ResolvedEntry) + .unwrap_or(before_anchor), + &err, + )? => + { + Ok(StepStatus::Skip) + } Err(err) => Err(err), } })?; @@ -772,6 +904,8 @@ impl FileReader { } pub fn flush_matches(&mut self) { + self.inner + .with_file(|file| self.row.clear_current_best_effort(file)); self.inner.with_reader_mut(|reader| reader.flush_matches()); } } @@ -785,6 +919,7 @@ impl FileReader { self.invalidate_entry_data_state(); match self.enumerate_fields_indexed() { Ok(fields) => Ok(fields), + Err(err @ SdkError::Journal(JournalError::CyclicObjectChain)) => Err(err), Err(_) => enumerate_file_fields_by_scan(self), } } diff --git a/rust/src/journal/src/parse.rs b/rust/src/journal/src/parse.rs index 01e0a85a..c1ab6ea0 100644 --- a/rust/src/journal/src/parse.rs +++ b/rust/src/journal/src/parse.rs @@ -256,3 +256,24 @@ mod tests { ); } } + +pub(crate) fn directional_cursor_anchor( + cursor: &str, +) -> std::result::Result { + let parsed = parse_cursor_location(cursor, true)?; + let sequence_id = if parsed.seqnum_id.is_empty() { + None + } else { + Some(*uuid::Uuid::parse_str(&parsed.seqnum_id)?.as_bytes()) + }; + let boot_id = if parsed.boot_id.is_empty() { + None + } else { + Some(*uuid::Uuid::parse_str(&parsed.boot_id)?.as_bytes()) + }; + Ok(journal_core::file::CursorAnchor { + sequence: sequence_id.zip(parsed.seqnum_set.then_some(parsed.seqnum)), + monotonic: boot_id.zip(parsed.monotonic_set.then_some(parsed.monotonic)), + realtime: parsed.realtime_set.then_some(parsed.realtime), + }) +} diff --git a/rust/src/journal/src/reader_helpers.rs b/rust/src/journal/src/reader_helpers.rs index 8efd9a21..ccfd5c7a 100644 --- a/rust/src/journal/src/reader_helpers.rs +++ b/rust/src/journal/src/reader_helpers.rs @@ -241,13 +241,6 @@ pub(super) fn verify_entry_at_strict( Ok(()) } -pub(super) fn recoverable_entry_error(err: &JournalError) -> bool { - matches!( - err, - JournalError::InvalidObjectSize(0) | JournalError::ObjectExceedsFileBounds - ) -} - pub(super) fn recoverable_entry_data_error(err: &JournalError) -> bool { matches!( err, @@ -259,19 +252,24 @@ pub(super) fn recoverable_entry_data_error(err: &JournalError) -> bool { pub(super) fn is_zst_file(path: &Path) -> bool { path.file_name() - .and_then(|name| name.to_str()) - .is_some_and(|name| name.ends_with(".zst")) + .map(|name| name.as_encoded_bytes()) + .is_some_and(|name| name.ends_with(b".zst")) } pub(super) fn decompress_zst_to_temp(path: &Path, prefix: &str) -> Result { let source = File::open(path)?; - let mut decoder = ruzstd::decoding::StreamingDecoder::new(source) - .map_err(|err| SdkError::DecompressionFailed(err.to_string()))?; let mut temp_file = tempfile::Builder::new() .prefix(prefix) .suffix(".journal") .tempfile()?; - std::io::copy(&mut decoder, &mut temp_file)?; + journal_core::file::copy_zstd_stream(source, &mut temp_file).map_err(|err| { + match err.kind() { + std::io::ErrorKind::InvalidData | std::io::ErrorKind::UnexpectedEof => { + SdkError::DecompressionFailed(err.to_string()) + } + _ => err.into(), + } + })?; let (dest, temp_path) = temp_file.keep().map_err(|err| err.error)?; drop(dest); Ok(temp_path) diff --git a/rust/src/journal/src/sealed_verify.rs b/rust/src/journal/src/sealed_verify.rs index 92f740cd..7a3cc8a7 100644 --- a/rust/src/journal/src/sealed_verify.rs +++ b/rust/src/journal/src/sealed_verify.rs @@ -56,7 +56,7 @@ pub(crate) fn verify_file_with_options( options: ReaderOptions, ) -> Result<()> { let path = path.as_ref(); - let reader = FileReader::open_with_options(path, verify_reader_options(options)) + let reader = FileReader::open_with_recovery(path, verify_reader_options(options), false) .map_err(|err| SdkError::VerificationError(format!("open/decompression failed: {err}")))?; verify_reader_object_graph(&reader)?; reader.inner.with_file(verify_journal_file_strict) @@ -76,7 +76,7 @@ pub(crate) fn verify_file_with_key_options( options: ReaderOptions, ) -> Result<()> { let path = path.as_ref(); - let reader = FileReader::open_with_options(path, verify_reader_options(options)) + let reader = FileReader::open_with_recovery(path, verify_reader_options(options), false) .map_err(|err| SdkError::VerificationError(format!("open/decompression failed: {err}")))?; reader.inner.with_file(|file| { @@ -925,7 +925,7 @@ fn update_hmac_range( /// Whole-file `.journal.zst` inputs incur full-file staging cost. pub fn verify_index(path: impl AsRef, control: &SnapshotControl<'_>) -> Result<()> { control.check()?; - let reader = FileReader::open_with_options(path, ReaderOptions::snapshot())?; + let reader = FileReader::open_with_recovery(path, ReaderOptions::snapshot(), false)?; reader.inner.with_file(|file| { struct ControlledSource<'a, 'b> { source: JournalFileVerifySource<'a>, diff --git a/rust/src/journal/src/tests.rs b/rust/src/journal/src/tests.rs index 6284b135..de035246 100644 --- a/rust/src/journal/src/tests.rs +++ b/rust/src/journal/src/tests.rs @@ -412,3 +412,4 @@ mod facade; mod verification; mod indexed_snapshot; +mod reader_hardening; diff --git a/rust/src/journal/src/tests/facade.rs b/rust/src/journal/src/tests/facade.rs index aad000cf..9e035d7e 100644 --- a/rust/src/journal/src/tests/facade.rs +++ b/rust/src/journal/src/tests/facade.rs @@ -3,6 +3,96 @@ use std::collections::HashSet; use std::path::{Path, PathBuf}; use std::time::Instant; +#[cfg(unix)] +#[test] +fn facade_native_paths_preserve_unix_names_and_options() { + use std::os::unix::ffi::OsStringExt; + + let dir = tempfile::tempdir().expect("create temp dir"); + let source = dir.path().join("source/system.journal"); + write_facade_test_journal(&source); + let native_dir = dir + .path() + .join(std::ffi::OsString::from_vec(b"parent-\xff".to_vec())); + std::fs::create_dir(&native_dir).expect("create native parent"); + let path = native_dir.join(std::ffi::OsString::from_vec(b"name-\xfe.journal".to_vec())); + std::fs::copy(&source, &path).expect("copy native journal"); + let decoy = PathBuf::from(path.to_string_lossy().as_ref()); + write_facade_single_message_journal(&decoy, b"decoy", 9000); + let extra = dir.path().join("extra/system.journal"); + write_single_entry_journal(&extra, 3, 5000, 21, b"MESSAGE=extra"); + let options = ReaderOptions::snapshot(); + let mut readers = vec![ + (SdJournalOpenPath(&path, 0).expect("open native path"), 2), + ( + SdJournalOpenFilePath(&path, 0).expect("open native file"), + 2, + ), + ( + SdJournalOpenFilePathWithOptions(&path, 0, options).expect("open native file options"), + 2, + ), + ( + SdJournalOpenDirectoryPath(&native_dir, 0).expect("open native directory"), + 2, + ), + ( + SdJournalOpenDirectoryPathWithOptions(&native_dir, 0, options) + .expect("open native directory options"), + 2, + ), + ( + SdJournalOpenFilePaths(&[path.clone(), path.clone()], 0) + .expect("deduplicate native paths"), + 2, + ), + ( + SdJournalOpenFilePathsWithOptions(&[path.clone(), extra], 0, options) + .expect("merge native paths"), + 3, + ), + ]; + for (reader, expected) in &mut readers { + let mut messages = Vec::new(); + while reader.next().expect("advance native reader") != 0 { + messages.push(reader.get_entry().expect("read native entry").fields["MESSAGE"].clone()); + } + assert_eq!(messages.len(), *expected); + assert_eq!(messages[0], b"first"); + assert_eq!(messages[1], b"second"); + } + for error in [ + SdJournalOpenPath(&path, 1).err(), + SdJournalOpenFilePathWithOptions(&path, 1, options).err(), + SdJournalOpenDirectoryPathWithOptions(&native_dir, 1, options).err(), + SdJournalOpenFilePathsWithOptions(&[&path], 1, options).err(), + ] { + assert!(matches!(error, Some(FacadeError::Unsupported))); + } + + let compressed_dir = native_dir.join("compressed"); + std::fs::create_dir(&compressed_dir).expect("create compressed directory"); + let compressed = compressed_dir.join(std::ffi::OsString::from_vec( + b"name-\xff.journal.zst".to_vec(), + )); + let source = std::fs::read(&source).expect("read source"); + std::fs::write( + &compressed, + ruzstd::encoding::compress_to_vec( + source.as_slice(), + ruzstd::encoding::CompressionLevel::Fastest, + ), + ) + .expect("write native compressed journal"); + let mut reader = SdJournalOpenDirectoryPathWithOptions(&compressed_dir, 0, options) + .expect("discover native compressed journal"); + assert_eq!(reader.next().expect("advance compressed reader"), 1); + assert_eq!( + reader.get_entry().expect("read compressed row").fields["MESSAGE"], + b"first" + ); +} + #[test] fn facade_uncompressed_data_uses_mmap_payload_for_whole_file_reader() { let dir = tempfile::tempdir().expect("create temp dir"); @@ -1186,3 +1276,198 @@ fn assert_json_omits_lossy_raw_name(entry: &Entry, lossy_name: &str) { "JSON output must not invent lossy RAW field names" ); } + +#[test] +fn facade_cursor_getter_after_future_seek_reports_no_entry() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("system.journal"); + write_facade_test_journal(&path); + let readers = [ + SdJournalOpenFilePath(&path, 0).unwrap(), + SdJournalOpenDirectoryPath(dir.path(), 0).unwrap(), + ]; + for mut reader in readers { + assert_eq!(SdJournalNext(&mut reader).unwrap(), 1); + assert!(SdJournalGetCursor(&reader).is_ok()); + SdJournalSeekCursor(&mut reader, "t=ffffffffffffff").unwrap(); + assert!(matches!( + SdJournalGetCursor(&reader), + Err(crate::facade::Error::NoEntry) + )); + } +} + +#[test] +fn directional_cursor_native_boundaries() { + use journal_core::file::{EntryField, EntryWriteOptions}; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("system.journal"); + let (mut file, mut writer) = create_facade_test_writer(&path); + let sequence = uuid::Uuid::from_bytes(file.journal_header_ref().seqnum_id) + .simple() + .to_string(); + let boots = [test_uuid(10), test_uuid(11), test_uuid(12)]; + for i in 0..6u64 { + let time = if i == 3 { 1002 } else { 1000 + i }; + let boot = boots[(i / 2) as usize]; + let row = format!("ROW={i}"); + let boot_field = format!("_BOOT_ID={}", boot.simple()); + let message = if i % 2 == 1 { + b"MESSAGE=keep".as_slice() + } else { + b"MESSAGE=other".as_slice() + }; + writer + .add_entry_fields_with_options( + &mut file, + [row.as_bytes(), message, boot_field.as_bytes()] + .into_iter() + .map(EntryField::Raw), + time, + i % 2 + 1, + EntryWriteOptions::default().boot_id(boot), + ) + .unwrap(); + } + file.sync().unwrap(); + drop(file); + let cases = [ + ("t=3ea".to_string(), Direction::Forward, "2", "3"), + ("t=3ea".to_string(), Direction::Backward, "3", "3"), + ( + format!("s={sequence};i=1;t=3ed"), + Direction::Backward, + "0", + "", + ), + ( + format!("s={sequence};i=ffffffff"), + Direction::Backward, + "5", + "5", + ), + ( + format!("s={sequence};i=ffffffff"), + Direction::Forward, + "", + "", + ), + ( + "s=ffffffffffffffffffffffffffffffff;i=3".into(), + Direction::Backward, + "5", + "5", + ), + ( + "s=ffffffffffffffffffffffffffffffff;i=3;m=0".into(), + Direction::Backward, + "5", + "5", + ), + ( + format!("b={};m=0", boots[1].simple()), + Direction::Backward, + "1", + "1", + ), + ( + format!("b={};m=0", boots[1].simple()), + Direction::Forward, + "2", + "3", + ), + ( + format!("b={};m=ffffffff", boots[1].simple()), + Direction::Forward, + "4", + "5", + ), + ( + "b=ffffffffffffffffffffffffffffffff;m=0".into(), + Direction::Backward, + "", + "", + ), + ( + "b=ffffffffffffffffffffffffffffffff;m=0;t=3ea".into(), + Direction::Backward, + "3", + "3", + ), + ]; + for directory in [false, true] { + for filtered in [false, true] { + for (cursor, direction, raw, filtered_row) in &cases { + let mut reader = if directory { + SdJournalOpenDirectoryPath(dir.path(), 0).unwrap() + } else { + SdJournalOpenFilePath(&path, 0).unwrap() + }; + if filtered { + SdJournalAddMatch(&mut reader, b"MESSAGE=keep").unwrap(); + } + SdJournalSeekCursorDirection(&mut reader, cursor, *direction).unwrap(); + let want = if filtered { filtered_row } else { raw }; + if want.is_empty() { + assert!( + matches!( + SdJournalGetCursor(&reader), + Err(crate::facade::Error::NoEntry) + ), + "{cursor} {direction:?} dir={directory} filter={filtered}" + ); + } else { + let entry = SdJournalGetEntry(&mut reader).unwrap(); + assert_eq!( + entry.fields["ROW"], + want.as_bytes(), + "{cursor} {direction:?} dir={directory} filter={filtered}" + ); + } + } + } + } +} + +#[test] +fn directional_cursor_exact_sequence_overrides_boot_order() { + let dir = tempfile::tempdir().unwrap(); + for (i, times) in [[1100, 1200], [1150, 1180]].into_iter().enumerate() { + let path = dir.path().join(format!("{i}.journal")); + let repo = RepoFile::from_path(&path).unwrap(); + let boot = test_uuid(20 + i as u8); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new(test_uuid(1), boot, test_uuid(10 + i as u8)), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut file, 1, boot).unwrap(); + for (n, time) in times.into_iter().enumerate() { + let message = format!("MESSAGE={i}-{n}"); + let boot_field = format!("_BOOT_ID={}", boot.simple()); + writer + .add_entry( + &mut file, + &[message.as_bytes(), boot_field.as_bytes()], + time, + n as u64 + 1, + ) + .unwrap(); + } + file.sync().unwrap(); + } + let mut single = FileReader::open(dir.path().join("0.journal")).unwrap(); + assert!(single.next().unwrap()); + let cursor = single.get_cursor().unwrap(); + let mut reader = DirectoryReader::open(dir.path()).unwrap(); + reader + .seek_cursor_direction(&cursor, Direction::Forward) + .unwrap(); + for want in ["0-0", "0-1"] { + assert_eq!( + reader.get_entry().unwrap().fields["MESSAGE"], + want.as_bytes() + ); + assert_eq!(reader.next().unwrap(), want == "0-0"); + } +} diff --git a/rust/src/journal/src/tests/reader_hardening.rs b/rust/src/journal/src/tests/reader_hardening.rs new file mode 100644 index 00000000..36a72f12 --- /dev/null +++ b/rust/src/journal/src/tests/reader_hardening.rs @@ -0,0 +1,1962 @@ +// Source-byte restoration in this module is synthetic fault injection, not +// evidence of journald publication or a supported in-place repair lifecycle. +// Damaged-prefix, failed-position and bounds checks remain recovery invariants. + +use super::*; + +fn u64_at(bytes: &[u8], offset: usize) -> u64 { + u64::from_le_bytes(bytes[offset..offset + 8].try_into().unwrap()) +} + +fn hardening_fixture(path: &Path, compact: bool) -> Vec { + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + let repo = RepoFile::from_path(path).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)).with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut file, 1, test_uuid(4)).unwrap(); + for i in 0..1005 { + let message = format!("MESSAGE=row-{i:04}"); + let mut fields = vec![message.as_bytes(), b"COMMON=all"]; + if i == 0 || i == 500 || i == 1004 { + fields.push(b"SPARSE=some"); + } + if i >= 1003 { + fields.push(b"TAIL_PAIR=some"); + } + writer + .add_entry(&mut file, &fields, 1000 + i, 100 + i) + .unwrap(); + } + file.sync().unwrap(); + drop(writer); + drop(file); + std::fs::read(path).unwrap() +} + +fn array_graph(bytes: &[u8], compact: bool) -> Vec<(usize, Vec)> { + let item = if compact { 4 } else { 8 }; + let mut remaining = u64_at(bytes, 152) as usize; + let mut at = u64_at(bytes, 176) as usize; + let mut arrays = Vec::new(); + while remaining > 0 { + let count = remaining.min((u64_at(bytes, at + 8) as usize - 24) / item); + let offsets = (0..count) + .map(|i| { + if compact { + u32::from_le_bytes( + bytes[at + 24 + i * item..at + 28 + i * item] + .try_into() + .unwrap(), + ) as u64 + } else { + u64_at(bytes, at + 24 + i * item) + } + }) + .collect(); + arrays.push((at, offsets)); + remaining -= count; + at = u64_at(bytes, at + 16) as usize; + } + arrays +} + +fn short_array_cycle_fixture( + path: &Path, + compact: bool, + postings: bool, + descending: bool, +) -> (u64, usize) { + let mut source = hardening_fixture(path, compact); + let arrays = array_graph(&source, compact); + let (mut head, mut next, count, root_slot) = if postings { + let payload_start = if compact { 72 } else { 64 }; + let mut data = u64_at(&source, 88) as usize; + loop { + let size = u64_at(&source, data + 8) as usize; + if source[data] == 1 && &source[data + payload_start..data + size] == b"COMMON=all" { + break; + } + data += (size + 7) & !7; + } + let head = u64_at(&source, data + 48) as usize; + let next = u64_at(&source, head + 16) as usize; + let width = if compact { 4 } else { 8 }; + let count = 2 + + (u64_at(&source, head + 8) as usize - 24) / width + + (u64_at(&source, next + 8) as usize - 24) / width; + source[data + 56..data + 64].copy_from_slice(&(count as u64).to_le_bytes()); + (head, next, count - 1, data + 48) + } else { + let count = arrays[0].1.len() + arrays[1].1.len() + 1; + source[152..160].copy_from_slice(&(count as u64).to_le_bytes()); + (arrays[0].0, arrays[1].0, count, 176) + }; + if descending { + std::mem::swap(&mut head, &mut next); + source[root_slot..root_slot + 8].copy_from_slice(&(head as u64).to_le_bytes()); + } + source[head + 16..head + 24].copy_from_slice(&(next as u64).to_le_bytes()); + source[next + 16..next + 24].copy_from_slice(&(head as u64).to_le_bytes()); + std::fs::write(path, source).unwrap(); + (head as u64, count) +} + +#[test] +fn counted_array_cycles_reject_required_revisit() { + for compact in [false, true] { + for postings in [false, true] { + for descending in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let (_, count) = short_array_cycle_fixture(&path, compact, postings, descending); + let mut reader = + FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + if postings { + reader.add_match(b"COMMON=all"); + } + let mut failed = false; + for _ in 0..=count + 1 { + match reader.next() { + Err(SdkError::Journal(JournalError::CyclicObjectChain)) => { + failed = true; + break; + } + Err(error) => panic!("unexpected cycle error: {error:?}"), + Ok(false) => panic!("required revisit returned successful EOF"), + Ok(true) => {} + } + } + assert!(failed, "required revisit returned duplicate rows"); + } + } + } +} + +#[test] +fn counted_array_nodes_reject_required_revisit() { + use journal_core::file::offset_array::{Cursor, List, Node}; + use std::num::{NonZeroU64, NonZeroUsize}; + assert_eq!( + std::mem::size_of::(), + 2 * std::mem::size_of::() + 2 * std::mem::size_of::() + ); + for postings in [false, true] { + for descending in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let (head, count) = short_array_cycle_fixture(&path, false, postings, descending); + let file = JournalFile::::open_path(&path, 64 * 1024).unwrap(); + let list = List::new( + NonZeroU64::new(head).unwrap(), + NonZeroUsize::new(count).unwrap(), + ); + let head = list.head(&file).unwrap(); + let second = head.next(&file).unwrap().unwrap(); + assert!(matches!( + second.next(&file), + Err(JournalError::CyclicObjectChain) + )); + let cursor = Cursor::at_position( + &file, + list, + second.offset(), + 0, + NonZeroUsize::new(count - head.len().get()).unwrap(), + ) + .unwrap(); + assert!(matches!( + cursor.node(&file).unwrap().next(&file), + Err(JournalError::CyclicObjectChain) + )); + } + } +} + +#[test] +fn counted_array_accepts_acyclic_backward_link() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut source = hardening_fixture(&path, compact); + let arrays = array_graph(&source, compact); + let (a, b) = (&arrays[0], &arrays[1]); + let entries: Vec<_> = a.1.iter().chain(&b.1).copied().collect(); + let width = if compact { 4 } else { 8 }; + for (index, entry) in entries.iter().enumerate() { + let (array, slot) = if index < b.1.len() { + (b.0, index) + } else { + (a.0, index - b.1.len()) + }; + let start = array + 24 + slot * width; + source[start..start + width].copy_from_slice(&entry.to_le_bytes()[..width]); + } + source[152..160].copy_from_slice(&(entries.len() as u64).to_le_bytes()); + source[176..184].copy_from_slice(&(b.0 as u64).to_le_bytes()); + source[b.0 + 16..b.0 + 24].copy_from_slice(&(a.0 as u64).to_le_bytes()); + source[a.0 + 16..a.0 + 24].fill(0); + std::fs::write(&path, source).unwrap(); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + for index in 0..entries.len() { + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1000 + index as u64); + } + assert!(!reader.next().unwrap()); + } +} + +#[test] +fn counted_array_ignores_unused_terminal_link() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut source = hardening_fixture(&path, false); + let arrays = array_graph(&source, false); + let count = arrays[0].1.len() + arrays[1].1.len(); + source[152..160].copy_from_slice(&(count as u64).to_le_bytes()); + let next = arrays[1].0; + source[next + 16..next + 24].copy_from_slice(&(next as u64).to_le_bytes()); + std::fs::write(&path, source).unwrap(); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + for _ in 0..count { + assert!(reader.next().unwrap()); + } + assert!(!reader.next().unwrap()); +} + +fn readable_prefix(bytes: &[u8], arrays: &[(usize, Vec)], cut: usize, compact: bool) -> usize { + let mut count = 0; + for (array, entries) in arrays { + for (index, &offset) in entries.iter().enumerate() { + let slot_end = array + 24 + (index + 1) * if compact { 4 } else { 8 }; + if slot_end > cut || offset as usize + u64_at(bytes, offset as usize + 8) as usize > cut + { + return count; + } + count += 1; + } + } + count +} + +#[test] +fn recovered_prefix_keeps_reverse_time_and_exact_filter_reads() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let source = temp.path().join("system.journal"); + let bytes = hardening_fixture(&source, compact); + let arrays = array_graph(&bytes, compact); + let last_entry = *arrays.last().unwrap().1.last().unwrap() as usize; + let last_array = arrays.last().unwrap().0; + for (name, cut) in [ + ("entry-header", last_entry), + ("entry-body", last_entry + 32), + ("array-header", last_array), + ( + "array-slots", + last_array + 24 + 3 * if compact { 4 } else { 8 }, + ), + ] { + let path = temp.path().join(format!("{name}.journal")); + std::fs::write(&path, &bytes[..cut]).unwrap(); + let expected = readable_prefix(&bytes, &arrays, cut, compact); + let mut reader = + FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + let mut forward = Vec::new(); + while reader.next().unwrap() { + forward.push(reader.get_realtime_usec().unwrap()); + } + assert_eq!(forward.len(), expected, "{compact}/{name}: forward"); + let status = reader.recovery_status(); + assert!(status.incomplete(), "{compact}/{name}: status"); + assert_eq!(status.recovered_entries, expected as u64); + assert_eq!(reader.header().n_entries, 1005); + reader.seek_tail(); + let mut reverse = Vec::new(); + while reader.previous().unwrap() { + reverse.push(reader.get_realtime_usec().unwrap()); + } + assert_eq!( + reverse, + forward.iter().rev().copied().collect::>(), + "{compact}/{name}: reverse" + ); + reader.seek_realtime(1000 + (expected / 2) as u64); + assert!(reader.next().unwrap()); + assert_eq!( + reader.get_realtime_usec().unwrap(), + 1000 + (expected / 2) as u64 + ); + reader.flush_matches(); + reader.add_match(b"COMMON=all"); + reader.seek_head(); + let mut filtered = Vec::new(); + while reader.next().unwrap() { + filtered.push(reader.get_realtime_usec().unwrap()); + } + assert_eq!(filtered, forward, "{compact}/{name}: filter"); + assert!(verify_file(&path).is_err()); + let control = SnapshotControl::default(); + assert!(verify_index(&path, &control).is_err()); + assert!( + IndexedSnapshot::open(&path, IndexedSnapshotOptions::default(), &control).is_err() + ); + let before = std::fs::read(&path).unwrap(); + assert!( + JournalFile::::open_for_append( + &RepoFile::from_path(&path).unwrap(), + 8 * 1024 * 1024 + ) + .is_err() + ); + assert_eq!(std::fs::read(&path).unwrap(), before); + } + } +} + +#[test] +fn recovery_sparse_posting_boundary() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let source = hardening_fixture(&path, compact); + let arrays = array_graph(&source, compact); + let cut = *arrays.last().unwrap().1.last().unwrap() as usize; + let payload_start = if compact { 72 } else { 64 }; + let mut at = u64_at(&source, 88) as usize; + let sparse = loop { + let size = u64_at(&source, at + 8) as usize; + if source[at] == 1 + && size >= payload_start + && &source[at + payload_start..at + size] == b"SPARSE=some" + { + break at; + } + assert!(size >= 16 && at + size < source.len()); + at += (size + 7) & !7; + }; + for missing_array in [false, true] { + let mut bytes = source[..cut].to_vec(); + if missing_array { + bytes[sparse + 48..sparse + 56].fill(0); + } + std::fs::write(&path, bytes).unwrap(); + let mut reader = + FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + let mut population = 0; + while reader.next().unwrap() { + population += 1; + } + assert_eq!(population, 1004); + reader.add_match(b"SPARSE=some"); + reader.seek_head(); + if missing_array { + assert!(reader.next().is_err(), "{compact}: unknown posting targets"); + } else { + let mut times = Vec::new(); + while reader.next().unwrap() { + times.push(reader.get_realtime_usec().unwrap()); + } + assert_eq!(times, [1000, 1500], "{compact}: sparse prefix"); + } + let query = ExplorerQuery { + filters: vec![ExplorerFilter::new(b"SPARSE", [b"some".as_slice()])], + field_mode: ExplorerFieldMode::AllValues, + use_source_realtime: false, + ..Default::default() + }; + let result = reader.explore_with_strategy(&query, ExplorerStrategy::Index); + assert_eq!( + result.is_err(), + missing_array, + "{compact}: Explorer boundary" + ); + } + } +} + +#[test] +fn indexed_postings_reject_missing_pointers_without_prior_traversal() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let source = hardening_fixture(&path, compact); + let payload_start = if compact { 72 } else { 64 }; + let mut common = u64_at(&source, 88) as usize; + loop { + let size = u64_at(&source, common + 8) as usize; + if source[common] == 1 + && size >= payload_start + && &source[common + payload_start..common + size] == b"COMMON=all" + { + break; + } + assert!(size >= 16 && common + size < source.len()); + common += (size + 7) & !7; + } + let array = u64_at(&source, common + 48) as usize; + for pointer in [common + 40, common + 48, array + 16] { + let mut bytes = source.clone(); + bytes[pointer..pointer + 8].fill(0); + std::fs::write(&path, bytes).unwrap(); + for backward in [false, true] { + let mut reader = + FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + reader.add_match(b"COMMON=all"); + if backward { + reader.seek_tail(); + } + let mut failed = false; + for _ in 0..1006 { + match if backward { + reader.previous() + } else { + reader.next() + } { + Err(_) => { + failed = true; + break; + } + Ok(false) => break, + Ok(true) => {} + } + } + assert!(failed, "{compact}/{pointer}: incomplete exact result"); + assert!(!reader.recovery_status().incomplete()); + } + for mode in ["filter", "facet", "histogram"] { + let mut reader = + FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + let mut query = ExplorerQuery { + limit: 0, + field_mode: ExplorerFieldMode::AllValues, + use_source_realtime: false, + ..Default::default() + }; + match mode { + "filter" => { + query.limit = 1; + query + .filters + .push(ExplorerFilter::new(b"COMMON", [b"all".as_slice()])); + } + "facet" => query.facets.push(b"COMMON".to_vec()), + _ => query.histogram = Some(b"COMMON".to_vec()), + } + assert!( + reader + .explore_with_strategy(&query, ExplorerStrategy::Index) + .is_err(), + "{compact}/{pointer}/{mode}: incomplete indexed result" + ); + assert!(!reader.recovery_status().incomplete()); + } + } + } +} + +#[test] +fn first_exact_filter_recovers_after_its_data_guard_drops() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut bytes = hardening_fixture(&path, compact); + let arrays = array_graph(&bytes, compact); + let last = *arrays.last().unwrap().1.last().unwrap() as usize; + bytes[last..last + 16].fill(0); + let payload_start = if compact { 72 } else { 64 }; + let mut pair = u64_at(&bytes, 88) as usize; + loop { + let size = u64_at(&bytes, pair + 8) as usize; + if bytes[pair] == 1 + && size >= payload_start + && &bytes[pair + payload_start..pair + size] == b"TAIL_PAIR=some" + { + break; + } + assert!(size >= 16 && pair + size < bytes.len()); + pair += (size + 7) & !7; + } + bytes[pair + 48..pair + 56].fill(0); + std::fs::write(&path, bytes).unwrap(); + for combined in [false, true] { + for backward in [false, true] { + let mut reader = + FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + assert!(!reader.recovery_status().incomplete()); + if combined { + reader.add_match(b"COMMON=all"); + } + reader.add_match(b"TAIL_PAIR=some"); + if backward { + reader.seek_tail(); + } + assert!( + if backward { + reader.previous() + } else { + reader.next() + } + .unwrap() + ); + assert_eq!(reader.get_realtime_usec().unwrap(), 2003); + assert!( + !if backward { + reader.previous() + } else { + reader.next() + } + .unwrap() + ); + assert_eq!(reader.recovery_status().recovered_entries, 1004); + } + } + } +} + +#[test] +fn recovered_sparse_eof_preserves_matching_position_and_repair() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let original = hardening_fixture(&path, compact); + let last = *array_graph(&original, compact) + .last() + .unwrap() + .1 + .last() + .unwrap() as usize; + let mut damaged = original.clone(); + damaged[last..last + 16].fill(0); + std::fs::write(&path, damaged).unwrap(); + let mut reader = FileReader::open(&path).unwrap(); + reader.add_match(b"SPARSE=some"); + for time in [1000, 1500] { + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), time); + } + assert!(!reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1500); + assert!(reader.previous().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1000); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1500); + assert!(!reader.next().unwrap()); + + use std::io::{Seek, SeekFrom, Write}; + let mut repair = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + repair.seek(SeekFrom::Start(last as u64)).unwrap(); + repair.write_all(&original[last..last + 16]).unwrap(); + assert!(!reader.next().unwrap()); + reader.seek_realtime(1501); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 2004); + assert!(!reader.next().unwrap()); + assert!(!reader.recovery_status().incomplete()); + } +} + +#[test] +fn wrong_entry_type_is_an_error_without_recovery() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut bytes = hardening_fixture(&path, false); + let entry = array_graph(&bytes, false)[0].1[0] as usize; + bytes[entry] = 2; + std::fs::write(&path, &bytes).unwrap(); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + assert!(reader.next().is_err()); + assert!(!reader.recovery_status().incomplete()); +} + +#[test] +fn recovered_eof_rechecks_changed_count_and_direction() { + for filtered in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let source = hardening_fixture(&path, false); + let last = *array_graph(&source, false) + .last() + .unwrap() + .1 + .last() + .unwrap() as usize; + let mut damaged = source.clone(); + damaged[last..last + 16].fill(0); + std::fs::write(&path, &damaged).unwrap(); + let mut reader = FileReader::open(&path).unwrap(); + if filtered { + reader.add_match(b"COMMON=all"); + } + let mut count = 0; + while reader.next().unwrap() { + count += 1; + } + assert_eq!(count, 1004); + for _ in 0..10 { + assert!(!reader.next().unwrap()); + } + assert_eq!(std::fs::read(&path).unwrap(), damaged); + assert!(reader.previous().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 2002); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 2003); + assert!(!reader.next().unwrap()); + + // Synthetic restoration alone must not restart unchanged EOF polling. + // The actual append below publishes a new declared count. + std::fs::write(&path, &source).unwrap(); + assert!(!reader.next().unwrap()); + let repo = RepoFile::from_path(&path).unwrap(); + let mut file = JournalFile::::open_for_append(&repo, 8 * 1024 * 1024).unwrap(); + let mut writer = JournalWriter::new(&mut file, 1006, test_uuid(4)).unwrap(); + writer + .add_entry(&mut file, &[b"COMMON=all"], 3000, 3000) + .unwrap(); + file.sync().unwrap(); + for expected in [2004, 3000] { + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), expected); + } + assert!(!reader.next().unwrap()); + assert!(!reader.recovery_status().incomplete()); + } +} + +#[test] +fn recovery_rejects_interior_holes_and_out_of_arena_pointers() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let source = hardening_fixture(&path, compact); + let arrays = array_graph(&source, compact); + let last = *arrays.last().unwrap().1.last().unwrap() as usize; + for corruption in ["zero-entry", "zero-slot", "arena-pointer", "nonzero-header"] { + let mut bytes = source[..last].to_vec(); + let first = arrays[0].1[0] as usize; + match corruption { + "zero-entry" => bytes[first..first + 16].fill(0), + "nonzero-header" => { + bytes[first..first + 16].fill(0); + bytes[first + 1] = 1; + } + "zero-slot" => { + bytes[arrays[0].0 + 24..arrays[0].0 + 24 + if compact { 4 } else { 8 }].fill(0) + } + _ => { + let bad = u64_at(&source, 88) + u64_at(&source, 96) + 8; + let slot = arrays[0].0 + 24; + if compact { + bytes[slot..slot + 4].copy_from_slice(&(bad as u32).to_le_bytes()); + } else { + bytes[slot..slot + 8].copy_from_slice(&bad.to_le_bytes()); + } + } + } + std::fs::write(&path, bytes).unwrap(); + match FileReader::open_with_options(&path, ReaderOptions::snapshot()) { + Err(_) => (), + Ok(mut reader) => assert!(reader.next().is_err(), "{compact}/{corruption}"), + } + } + } +} + +#[test] +fn same_size_repair_restores_empty_exact_match_and_tail_once() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let bytes = hardening_fixture(&path, compact); + let arrays = array_graph(&bytes, compact); + let last = *arrays.last().unwrap().1.last().unwrap() as usize; + for zero_slot in [false, true] { + let mut damaged = bytes.clone(); + if zero_slot { + let (array, entries) = arrays.last().unwrap(); + let item = if compact { 4 } else { 8 }; + let slot = array + 24 + (entries.len() - 1) * item; + damaged[slot..slot + item].fill(0); + } else { + damaged[last..last + 16].fill(0); + } + std::fs::write(&path, damaged).unwrap(); + let mut reader = FileReader::open(&path).unwrap(); + let mut rows = 0; + while reader.next().unwrap() { + rows += 1; + } + assert_eq!(rows, 1004); + reader.add_match(b"MESSAGE=row-1004"); + reader.seek_head(); + assert!(!reader.next().unwrap(), "{compact}/{zero_slot}"); + assert!(reader.recovery_status().incomplete()); + assert!(!reader.next().unwrap()); + std::fs::write(&path, &bytes).unwrap(); + assert!(!reader.next().unwrap(), "unchanged terminal count"); + reader.seek_head(); + assert!( + reader.next().unwrap(), + "{compact}/{zero_slot}: explicit seek" + ); + assert_eq!(reader.get_realtime_usec().unwrap(), 2004); + assert!(!reader.recovery_status().incomplete()); + assert!(!reader.next().unwrap()); + } + } +} + +#[test] +fn realtime_seek_uses_requested_direction_and_range_with_recovery() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut bytes = hardening_fixture(&path, false); + let arrays = array_graph(&bytes, false); + for (index, &entry) in arrays.iter().flat_map(|(_, rows)| rows).enumerate() { + let at = entry as usize + 24; + bytes[at..at + 8].copy_from_slice(&(1000 + 2 * index as u64).to_le_bytes()); + } + for damaged in [false, true] { + let cut = if damaged { + *arrays.last().unwrap().1.last().unwrap() as usize + } else { + bytes.len() + }; + std::fs::write(&path, &bytes[..cut]).unwrap(); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + for filtered in [false, true] { + if filtered { + reader.add_match(b"COMMON=all"); + } + for (time, forward, expected) in [ + (1001, true, Some(1002)), + (1001, false, Some(1000)), + (999, false, None), + (999, true, Some(1000)), + (4000, true, None), + (4000, false, Some(if damaged { 3006 } else { 3008 })), + ] { + reader.seek_realtime(time); + let found = if forward { + reader.next().unwrap() + } else { + reader.previous().unwrap() + }; + assert_eq!( + found, + expected.is_some(), + "{damaged}/{filtered}/{time}/{forward}" + ); + if let Some(time) = expected { + assert_eq!(reader.get_realtime_usec().unwrap(), time); + } + } + } + } +} + +#[test] +fn directory_repair_preserves_pending_merge_rows() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let a = temp.path().join("a.journal"); + let b = temp.path().join("b.journal"); + let repo = RepoFile::from_path(&a).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)).with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut file, 1, test_uuid(4)).unwrap(); + writer + .add_entry(&mut file, &[b"MESSAGE=1"], 1000, 1) + .unwrap(); + writer + .add_entry(&mut file, &[b"MESSAGE=3"], 3000, 3) + .unwrap(); + file.sync().unwrap(); + drop(writer); + drop(file); + let mut bytes = std::fs::read(&a).unwrap(); + let last = array_graph(&bytes, compact)[0].1[1] as usize; + bytes[last + 16..last + 24].copy_from_slice(&3u64.to_le_bytes()); + bytes[160..168].copy_from_slice(&3u64.to_le_bytes()); + std::fs::write(&a, &bytes[..last]).unwrap(); + write_single_entry_journal(&b, 2, 2000, 2, b"MESSAGE=2"); + let ar = FileReader::open(&a).unwrap(); + let br = FileReader::open(&b).unwrap(); + let mut directory = DirectoryReader::from_readers(vec![br, ar], false).unwrap(); + assert!(!directory.non_overlapping); + assert!(directory.next().unwrap()); + assert_eq!(directory.get_realtime_usec().unwrap(), 1000); + std::fs::write(&a, bytes).unwrap(); + let mut times = Vec::new(); + while directory.next().unwrap() { + times.push(directory.get_realtime_usec().unwrap()); + } + assert_eq!(times, vec![2000, 3000]); + assert!(directory.recovery_statuses().is_empty()); + let boots = directory.list_boots(); + assert_eq!(boots.len(), 1); + assert_eq!(boots[0].last_entry, 3000); + directory.seek_tail(); + times.clear(); + while directory.previous().unwrap() { + times.push(directory.get_realtime_usec().unwrap()); + } + assert_eq!(times, vec![3000, 2000, 1000]); + } +} + +#[test] +fn recovery_preserves_historical_no_rtc_population() { + let mut count = 0; + for entry in std::fs::read_dir(repo_root().join("fixtures/systemd/test-data/no-rtc")).unwrap() { + let path = entry.unwrap().path(); + if !is_journal_file_name(&path) { + continue; + } + let mut reader = FileReader::open_with_options(path, ReaderOptions::snapshot()).unwrap(); + while reader.next().unwrap() { + reader.get_entry().unwrap(); + count += 1; + } + } + assert_eq!(count, 10757); +} + +#[test] +fn snapshot_observed_shrink_and_repair_preserve_original_extent() { + use std::io::{Seek, SeekFrom, Write}; + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut bytes = hardening_fixture(&path, compact); + let last = *array_graph(&bytes, compact) + .last() + .unwrap() + .1 + .last() + .unwrap(); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + reader.add_match(b"SPARSE=some"); + reader.seek_tail(); + let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + file.set_len(last + 8).unwrap(); + assert!(reader.previous().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1500); + assert_eq!(reader.recovery_status().recovered_entries, 1004); + let original_size = bytes.len() as u64; + bytes.extend_from_slice(&[0; 4096]); + file.seek(SeekFrom::Start(0)).unwrap(); + file.write_all(&bytes).unwrap(); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 2004); + assert!(!reader.recovery_status().incomplete()); + assert_eq!( + reader + .inner + .with_file(|file| file.reader_file_size().unwrap()), + original_size + ); + } +} + +#[test] +fn snapshot_cold_recovery_bounds_retained_entry_headers_by_physical_extent() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let bytes = hardening_fixture(&path, compact); + let last = *array_graph(&bytes, compact) + .last() + .unwrap() + .1 + .last() + .unwrap(); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + std::fs::OpenOptions::new() + .write(true) + .open(&path) + .unwrap() + .set_len(last + 64) + .unwrap(); + assert!(reader.inner.with_file(|file| { + file.recover_reader_tail(&JournalError::ObjectExceedsFileBounds) + .unwrap() + })); + assert_eq!(reader.recovery_status().recovered_entries, 1004); + reader.add_match(b"SPARSE=some"); + reader.seek_tail(); + assert!(reader.previous().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1500); + } +} + +enum PositionTestReader { + Sdk(FileReader), + Directory(DirectoryReader), + Facade(SdJournal), +} + +impl PositionTestReader { + fn open(path: &Path, facade: bool) -> Self { + Self::open_bounds(path, facade, false) + } + + fn open_bounds(path: &Path, facade: bool, live: bool) -> Self { + let options = if live { + ReaderOptions::default() + } else { + ReaderOptions::snapshot() + }; + if path.is_dir() { + return if facade { + Self::Facade( + SdJournalOpenDirectoryWithOptions(path.to_str().unwrap(), 0, options).unwrap(), + ) + } else { + Self::Directory(DirectoryReader::open_with_options(path, options).unwrap()) + }; + } + if facade { + Self::Facade(SdJournalOpenFileWithOptions(path.to_str().unwrap(), 0, options).unwrap()) + } else { + Self::Sdk(FileReader::open_with_options(path, options).unwrap()) + } + } + + fn seqnum(&mut self) -> u64 { + match self { + Self::Sdk(reader) => reader.get_entry().unwrap().seqnum, + Self::Directory(reader) => reader.get_seqnum().unwrap().0, + Self::Facade(reader) => SdJournalGetSeqnum(reader).unwrap().0, + } + } + + fn seek_head(&mut self) { + match self { + Self::Sdk(reader) => reader.seek_head(), + Self::Directory(reader) => reader.seek_head(), + Self::Facade(reader) => SdJournalSeekHead(reader).unwrap(), + } + } + + fn step(&mut self, backward: bool) -> std::result::Result { + match self { + Self::Directory(reader) => { + let result = if backward { + reader.previous() + } else { + reader.next() + }; + result.map_err(|err| format!("{err:?}")) + } + Self::Sdk(reader) => { + let result = if backward { + reader.previous() + } else { + reader.next() + }; + result.map_err(|err| format!("{err:?}")) + } + Self::Facade(reader) => { + let result = if backward { + SdJournalPrevious(reader) + } else { + SdJournalNext(reader) + }; + result + .map(|count| count != 0) + .map_err(|err| format!("{err:?}")) + } + } + } + + fn time(&self) -> u64 { + match self { + Self::Sdk(reader) => reader.get_realtime_usec().unwrap(), + Self::Directory(reader) => reader.get_realtime_usec().unwrap(), + Self::Facade(reader) => SdJournalGetRealtimeUsec(reader).unwrap(), + } + } + + fn want(&mut self, backward: bool, time: u64) { + assert!( + self.step(backward).unwrap(), + "backward={backward}, want {time}" + ); + assert_eq!(self.time(), time, "backward={backward}"); + } + + fn add_match(&mut self, value: &[u8]) { + match self { + Self::Sdk(reader) => reader.add_match(value), + Self::Directory(reader) => reader.add_match(value), + Self::Facade(reader) => SdJournalAddMatch(reader, value).unwrap(), + } + } + + fn add_disjunction(&mut self) { + match self { + Self::Sdk(reader) => reader.add_disjunction().unwrap(), + Self::Directory(reader) => reader.add_disjunction().unwrap(), + Self::Facade(reader) => SdJournalAddDisjunction(reader).unwrap(), + } + } + + fn seek_tail(&mut self) { + match self { + Self::Sdk(reader) => reader.seek_tail(), + Self::Directory(reader) => reader.seek_tail(), + Self::Facade(reader) => SdJournalSeekTail(reader).unwrap(), + } + } + + fn flush_matches(&mut self) { + match self { + Self::Sdk(reader) => reader.flush_matches(), + Self::Directory(reader) => reader.flush_matches(), + Self::Facade(reader) => SdJournalFlushMatches(reader).unwrap(), + } + } + + fn seek_realtime(&mut self, time: u64) { + match self { + Self::Sdk(reader) => reader.seek_realtime(time), + Self::Directory(reader) => reader.seek_realtime(time), + Self::Facade(reader) => SdJournalSeekRealtimeUsec(reader, time).unwrap(), + } + } + + fn cursor(&self) -> String { + match self { + Self::Sdk(reader) => reader.get_cursor().unwrap(), + Self::Directory(reader) => reader.get_cursor().unwrap(), + Self::Facade(reader) => SdJournalGetCursor(reader).unwrap(), + } + } + + fn message(&mut self) -> Vec { + match self { + Self::Sdk(reader) => reader.get_entry_payload(b"MESSAGE").unwrap().unwrap(), + Self::Directory(reader) => reader.get_entry_payload(b"MESSAGE").unwrap().unwrap(), + Self::Facade(reader) => SdJournalGetData(reader, "MESSAGE").unwrap(), + } + } +} + +#[test] +fn directory_filter_replacement_keeps_sequential_discovery() { + use std::io::{Seek, SeekFrom, Write}; + + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + for index in 0..3 { + let path = temp.path().join(format!("{index}.journal")); + let repo = RepoFile::from_path(&path).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)) + .with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new(&mut file, 3 * index + 1, test_uuid(4)).unwrap(); + for row in 1..=3 { + let time = (3 * index + row) * 1000; + writer + .add_entry(&mut file, &[b"COMMON=all"], time, time) + .unwrap(); + } + file.sync().unwrap(); + } + let mut reader = DirectoryReader::open(temp.path()).unwrap(); + assert!(reader.non_overlapping); + reader.add_match(b"COMMON=all"); + for time in [1000, 2000, 3000, 4000] { + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), time); + } + let path = temp.path().join("2.journal"); + let source = std::fs::read(&path).unwrap(); + let offset = array_graph(&source, compact)[0].1[0]; + let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[1]).unwrap(); + reader.add_match(b"COMMON=all"); + for time in [5000, 6000] { + assert!(reader.next().unwrap(), "nearer healthy step {time}"); + assert_eq!(reader.get_realtime_usec().unwrap(), time); + } + assert!(reader.next().is_err()); + assert_eq!(reader.get_realtime_usec().unwrap(), 6000); + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[3]).unwrap(); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 7000); + } +} + +fn directory_filter_boundary_fixture(path: &Path, compact: bool, merged: bool) { + let times = if merged { + [[1000, 3000, 5000], [2000, 4000, 6000]] + } else { + [[1000, 2000, 3000], [4000, 5000, 6000]] + }; + for (index, rows) in times.iter().enumerate() { + let path = path.join(format!("{index}.journal")); + let repo = RepoFile::from_path(&path).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new( + test_uuid(1), + test_uuid(2), + test_uuid(if merged { 3 + index as u8 } else { 3 }), + ) + .with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new( + &mut file, + if merged { 1 } else { 3 * index as u64 + 1 }, + test_uuid(4), + ) + .unwrap(); + for &time in rows { + let message = format!("MESSAGE=row-{time}"); + writer + .add_entry(&mut file, &[message.as_bytes(), b"COMMON=all"], time, time) + .unwrap(); + } + file.sync().unwrap(); + } +} + +#[test] +fn directory_filter_mutation_keeps_emitted_anchor() { + for compact in [false, true] { + for merged in [false, true] { + let temp = tempfile::tempdir().unwrap(); + directory_filter_boundary_fixture(temp.path(), compact, merged); + for facade in [false, true] { + for backward in [false, true] { + for mutation in ["install", "replace", "and", "or"] { + let mut reader = PositionTestReader::open(temp.path(), facade); + if mutation == "replace" { + reader.add_match(b"COMMON=all"); + } + for time in [1000, 2000, 3000] { + reader.want(false, time); + } + if mutation == "replace" { + reader.add_match(b"MESSAGE=row-1000"); + reader.add_match(b"MESSAGE=row-5000"); + } else if mutation == "or" { + reader.add_match(b"MESSAGE=row-1000"); + } else { + reader.add_match(b"COMMON=all"); + } + if mutation == "and" { + match &mut reader { + PositionTestReader::Directory(directory) => { + directory.add_conjunction().unwrap(); + } + PositionTestReader::Facade(facade) => { + SdJournalAddConjunction(facade).unwrap(); + } + PositionTestReader::Sdk(_) => unreachable!(), + } + reader.add_match(b"MESSAGE=row-1000"); + reader.add_match(b"MESSAGE=row-5000"); + } else if mutation == "or" { + reader.add_disjunction(); + reader.add_match(b"MESSAGE=row-5000"); + } + assert_eq!(reader.time(), 3000, "mutation changed selected row"); + let times: &[u64] = if mutation != "install" { + if backward { &[1000] } else { &[5000] } + } else if backward { + &[2000, 1000] + } else { + &[4000, 5000, 6000] + }; + for &time in times { + assert!( + reader.step(backward).unwrap(), + "compact={compact}/merge={merged}/facade={facade}/back={backward}/{mutation}" + ); + assert_eq!( + reader.time(), + time, + "compact={compact}/merge={merged}/facade={facade}/back={backward}/{mutation}" + ); + } + assert!(!reader.step(backward).unwrap()); + } + } + } + } + } +} + +#[test] +fn directory_filter_mutation_keeps_failed_realtime_seek() { + use std::io::{Seek, SeekFrom, Write}; + for compact in [false, true] { + for merged in [false, true] { + for facade in [false, true] { + for backward in [false, true] { + let temp = tempfile::tempdir().unwrap(); + directory_filter_boundary_fixture(temp.path(), compact, merged); + let mut reader = PositionTestReader::open(temp.path(), facade); + reader.add_match(b"COMMON=all"); + reader.seek_realtime(3500); + let path = temp + .path() + .join(if backward { "0.journal" } else { "1.journal" }); + let want = if backward { 3000 } else { 4000 }; + let source = std::fs::read(&path).unwrap(); + let offset = array_graph(&source, compact) + .into_iter() + .flat_map(|(_, entries)| entries) + .find(|&offset| u64_at(&source, offset as usize + 24) == want) + .unwrap(); + let patch = |typ| { + let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[typ]).unwrap(); + }; + patch(1); + assert!(reader.step(backward).is_err()); + reader.add_match(b"COMMON=all"); + patch(3); + assert!(reader.step(backward).unwrap()); + assert_eq!( + reader.time(), + want, + "{compact}/{merged}/{facade}/{backward}" + ); + } + } + } + } +} + +#[test] +fn directory_filter_flush_discards_pending_realtime_seek() { + for compact in [false, true] { + for merged in [false, true] { + let temp = tempfile::tempdir().unwrap(); + directory_filter_boundary_fixture(temp.path(), compact, merged); + for facade in [false, true] { + let mut reader = PositionTestReader::open(temp.path(), facade); + if let PositionTestReader::Directory(directory) = &reader { + assert_eq!(directory.non_overlapping, !merged); + } + reader.add_match(b"COMMON=all"); + reader.seek_realtime(5500); + reader.flush_matches(); + reader.want(false, 1000); + } + } + } +} + +#[test] +fn directory_filter_eof_reversal_keeps_nearest_match() { + for compact in [false, true] { + for merged in [false, true] { + let temp = tempfile::tempdir().unwrap(); + directory_filter_boundary_fixture(temp.path(), compact, merged); + for facade in [false, true] { + for backward in [false, true] { + let mut reader = PositionTestReader::open(temp.path(), facade); + if let PositionTestReader::Directory(directory) = &reader { + assert_eq!(directory.non_overlapping, !merged); + } + reader.add_match(b"COMMON=all"); + if backward { + reader.seek_tail(); + } + for i in 1..=6 { + reader.want(backward, if backward { 7000 - 1000 * i } else { 1000 * i }); + } + assert!(!reader.step(backward).unwrap()); + assert!(!reader.step(backward).unwrap()); + reader.want(!backward, if backward { 2000 } else { 5000 }); + } + } + } + } +} + +#[test] +fn flush_matches_clears_selected_row() { + let temp = tempfile::tempdir().unwrap(); + directory_filter_boundary_fixture(temp.path(), false, false); + for directory in [false, true] { + for facade in [false, true] { + let file_path = temp.path().join("0.journal"); + let path = if directory { temp.path() } else { &file_path }; + let mut reader = PositionTestReader::open(path, facade); + reader.add_match(b"COMMON=all"); + reader.want(false, 1000); + reader.message(); + reader.flush_matches(); + match &mut reader { + PositionTestReader::Sdk(reader) => { + assert!(reader.get_realtime_usec().is_err()); + assert!(reader.get_cursor().is_err()); + assert!(reader.get_entry().is_err()); + } + PositionTestReader::Directory(reader) => { + assert!(reader.get_realtime_usec().is_err()); + assert!(reader.get_cursor().is_err()); + assert!(reader.get_entry().is_err()); + } + PositionTestReader::Facade(reader) => { + assert!(SdJournalGetRealtimeUsec(reader).is_err()); + assert!(SdJournalGetCursor(reader).is_err()); + assert!(SdJournalGetData(reader, "MESSAGE").is_err()); + } + } + reader.want(false, 1000); + } + } +} + +#[test] +fn snapshot_secondary_posting_shrink_returns_prefix_eof() { + use std::io::{Seek, SeekFrom, Write}; + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let source = hardening_fixture(&path, compact); + let payload_start = if compact { 72 } else { 64 }; + let mut common = u64_at(&source, 88) as usize; + loop { + let size = u64_at(&source, common + 8) as usize; + if source[common] == 1 + && size >= payload_start + && &source[common + payload_start..common + size] == b"COMMON=all" + { + break; + } + assert!(size >= 16 && common + size < source.len()); + common += (size + 7) & !7; + } + let array = u64_at(&source, common + 48) as usize; + let cut = array + 24 + if compact { 4 } else { 8 }; + assert_eq!( + readable_prefix(&source, &array_graph(&source, compact), cut, compact), + 2 + ); + let mut reader = FileReader::open_with_options(&path, ReaderOptions::snapshot()).unwrap(); + reader.add_match(b"COMMON=all"); + let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + file.set_len(cut as u64).unwrap(); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1000); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1001); + assert!(!reader.next().unwrap()); + assert_eq!(reader.recovery_status().recovered_entries, 2); + file.seek(SeekFrom::Start(0)).unwrap(); + file.write_all(&source).unwrap(); + assert!(!reader.next().unwrap()); + reader.seek_realtime(1002); + assert!(reader.next().unwrap()); + assert_eq!(reader.get_realtime_usec().unwrap(), 1002); + assert!(!reader.recovery_status().incomplete()); + } +} + +#[test] +fn directory_recovery_merge_error_retains_prefetched_row() { + use std::io::{Seek, SeekFrom, Write}; + + for compact in [false, true] { + for facade in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let mut entries = Vec::new(); + for index in 0..3 { + let path = temp.path().join(format!("{index}.journal")); + let repo = RepoFile::from_path(&path).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)) + .with_compact(compact), + ) + .unwrap(); + let mut writer = + JournalWriter::new(&mut file, 3 * index + 1, test_uuid(4)).unwrap(); + for row in 1..=3 { + let time = (3 * index + row) * 1000; + let message = format!("MESSAGE=row-{time}"); + writer + .add_entry(&mut file, &[message.as_bytes(), b"COMMON=all"], time, time) + .unwrap(); + } + file.sync().unwrap(); + drop(writer); + drop(file); + entries.push( + array_graph(&std::fs::read(path).unwrap(), compact) + .into_iter() + .flat_map(|(_, rows)| rows) + .collect::>(), + ); + } + let mut reader = PositionTestReader::open(temp.path(), facade); + if let PositionTestReader::Directory(directory) = &reader { + assert!(directory.non_overlapping); + } + reader.add_match(b"COMMON=all"); + reader.seek_tail(); + for time in (4..=9).rev() { + reader.want(true, time * 1000); + } + let cursor = reader.cursor(); + for (index, row, damage) in [(0, 2, vec![0; 16]), (2, 1, vec![2])] { + let path = temp.path().join(format!("{index}.journal")); + let mut file = std::fs::OpenOptions::new().write(true).open(path).unwrap(); + file.seek(SeekFrom::Start(entries[index][row])).unwrap(); + file.write_all(&damage).unwrap(); + } + assert!(reader.step(true).is_err(), "merge probe must fail"); + assert_eq!(reader.time(), 4000, "selected row survives error"); + assert_eq!(reader.cursor(), cursor); + let path = temp.path().join("2.journal"); + let mut file = std::fs::OpenOptions::new().write(true).open(path).unwrap(); + file.seek(SeekFrom::Start(entries[2][1])).unwrap(); + file.write_all(&[3]).unwrap(); + reader.want(true, 2000); + reader.want(true, 1000); + assert!(!reader.step(true).unwrap()); + } + } +} + +#[test] +fn directory_position_after_entry_load_error_retries_in_order() { + use std::io::{Seek, SeekFrom, Write}; + + for compact in [false, true] { + for facade in [false, true] { + for filtered in [false, true] { + for backward in [false, true] { + for merged in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let times = if merged { + [[1000, 3000, 5000], [2000, 4000, 6000]] + } else { + [[1000, 2000, 3000], [4000, 5000, 6000]] + }; + let mut entries = Vec::new(); + for (index, rows) in times.iter().enumerate() { + let path = temp.path().join(format!("{index}.journal")); + let repo = RepoFile::from_path(&path).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new( + test_uuid(1), + test_uuid(2), + test_uuid(if merged { 3 + index as u8 } else { 3 }), + ) + .with_compact(compact), + ) + .unwrap(); + let mut writer = JournalWriter::new( + &mut file, + if merged { 1 } else { 3 * index as u64 + 1 }, + test_uuid(4), + ) + .unwrap(); + for &time in rows { + let message = format!("MESSAGE=row-{time}"); + writer + .add_entry( + &mut file, + &[message.as_bytes(), b"COMMON=all"], + time, + time, + ) + .unwrap(); + } + file.sync().unwrap(); + drop(writer); + drop(file); + entries.push( + array_graph(&std::fs::read(path).unwrap(), compact) + .into_iter() + .flat_map(|(_, rows)| rows) + .collect::>(), + ); + } + let mut reader = PositionTestReader::open(temp.path(), facade); + if let PositionTestReader::Directory(directory) = &reader { + assert_eq!(directory.non_overlapping, !merged); + } + if filtered { + reader.add_match(b"COMMON=all"); + } + let (before, target_file, target_row) = if merged { + (3000, 1, if backward { 0 } else { 1 }) + } else if backward { + (4000, 0, 2) + } else { + (3000, 1, 0) + }; + let initial_backward = if merged { !backward } else { backward }; + if initial_backward { + reader.seek_tail(); + } + loop { + assert!(reader.step(initial_backward).unwrap()); + if reader.time() == before { + break; + } + } + let cursor = reader.cursor(); + let message = format!("MESSAGE=row-{before}").into_bytes(); + assert_eq!(reader.message(), message); + let path = temp.path().join(format!("{target_file}.journal")); + let mut file = std::fs::OpenOptions::new().write(true).open(path).unwrap(); + let offset = entries[target_file][target_row]; + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[2]).unwrap(); + assert!( + reader.step(backward).is_err(), + "{compact}/{facade}/{filtered}/{backward}/{merged}" + ); + assert_eq!(reader.time(), before, "position while damaged"); + assert_eq!(reader.cursor(), cursor); + assert_eq!(reader.message(), message); + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[3]).unwrap(); + assert_eq!(reader.time(), before, "position after repair"); + assert_eq!(reader.cursor(), cursor); + assert_eq!(reader.message(), message); + for delta in [1000, 2000] { + reader.want( + backward, + if backward { + before - delta + } else { + before + delta + }, + ); + } + } + } + } + } + } +} + +#[test] +fn native_filter_reverse_anchor_keeps_nearest_match() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + hardening_fixture(&path, compact); + for facade in [false, true] { + for conjunction in [false, true] { + for scenario in ["install", "install-eof", "filtered-eof"] { + let mut reader = PositionTestReader::open(&path, facade); + if scenario != "filtered-eof" { + for i in 0..1005 { + reader.want(false, 1000 + i); + } + } + if conjunction { + reader.add_match(b"COMMON=all"); + } + reader.add_match(b"SPARSE=some"); + if scenario == "filtered-eof" { + for time in [1000, 1500, 2004] { + reader.want(false, time); + } + } + if scenario != "install" { + assert!( + !reader.step(false).unwrap(), + "{compact}/{facade}/{conjunction}/{scenario}" + ); + } + reader.want(true, 1500); + reader.want(false, 2004); + reader.want(true, 1500); + reader.want(true, 1000); + assert!(!reader.step(true).unwrap()); + reader.want(false, 1500); + } + } + } + } +} + +#[test] +fn directory_initial_entry_error_retries_pending_time_and_boundary() { + use std::io::{Seek, SeekFrom, Write}; + + for compact in [false, true] { + for facade in [false, true] { + for filtered in [false, true] { + for scenario in ["head", "tail", "time-forward", "time-backward"] { + if !filtered && scenario.starts_with("time-") { + continue; + } + let temp = tempfile::tempdir().unwrap(); + let mut entries = Vec::new(); + for index in 0..2 { + let path = temp.path().join(format!("{index}.journal")); + let repo = RepoFile::from_path(&path).unwrap(); + let mut file = JournalFile::::create( + &repo, + JournalFileOptions::new(test_uuid(1), test_uuid(2), test_uuid(3)) + .with_compact(compact), + ) + .unwrap(); + let mut writer = + JournalWriter::new(&mut file, 3 * index + 1, test_uuid(4)).unwrap(); + for row in 0..3 { + let time = 1000 * (3 * index + row + 1); + writer + .add_entry(&mut file, &[b"COMMON=all"], time, time) + .unwrap(); + } + file.sync().unwrap(); + drop(writer); + drop(file); + entries.push( + array_graph(&std::fs::read(path).unwrap(), compact) + .into_iter() + .flat_map(|(_, rows)| rows) + .collect::>(), + ); + } + let mut reader = PositionTestReader::open(temp.path(), facade); + if filtered { + reader.add_match(b"COMMON=all"); + } + let backward = matches!(scenario, "tail" | "time-backward"); + let (file_index, row_index, want): (usize, usize, Vec) = match scenario { + "tail" => { + reader.seek_tail(); + (1, 2, vec![6000, 5000, 4000, 3000, 2000, 1000]) + } + "time-forward" => { + reader.seek_realtime(1500); + (1, 0, vec![2000, 3000, 4000, 5000, 6000]) + } + "time-backward" => { + reader.seek_realtime(5500); + (1, 1, vec![5000, 4000, 3000, 2000, 1000]) + } + _ => (0, 0, vec![1000, 2000, 3000, 4000, 5000, 6000]), + }; + let mut file = std::fs::OpenOptions::new() + .write(true) + .open(temp.path().join(format!("{file_index}.journal"))) + .unwrap(); + let offset = entries[file_index][row_index]; + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[2]).unwrap(); + assert!( + reader.step(backward).is_err(), + "{compact}/{facade}/{filtered}/{scenario}" + ); + assert!(match &reader { + PositionTestReader::Directory(reader) => reader.get_cursor().is_err(), + PositionTestReader::Facade(reader) => SdJournalGetCursor(reader).is_err(), + _ => unreachable!(), + }); + file.seek(SeekFrom::Start(offset)).unwrap(); + file.write_all(&[3]).unwrap(); + for time in want { + reader.want(backward, time); + } + assert!(!reader.step(backward).unwrap()); + } + } + } + } +} + +#[test] +fn reader_position_after_entry_load_error_retries_same_candidate() { + use std::io::{Seek, SeekFrom, Write}; + + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let source_path = temp.path().join("source.journal"); + let source = hardening_fixture(&source_path, compact); + let offset = array_graph(&source, compact) + .iter() + .flat_map(|(_, entries)| entries.iter()) + .nth(500) + .copied() + .unwrap() as usize; + for facade in [false, true] { + for filtered in [false, true] { + for backward in [false, true] { + for damage in ["type", "size"] { + let path = temp + .path() + .join(format!("{facade}-{filtered}-{backward}-{damage}.journal")); + std::fs::write(&path, &source).unwrap(); + let mut reader = PositionTestReader::open(&path, facade); + let before = if filtered { + reader.add_match(b"SPARSE=some"); + if backward { + reader.seek_tail(); + 2004 + } else { + 1000 + } + } else { + let time = if backward { 1501 } else { 1499 }; + reader.seek_realtime(time); + time + }; + reader.want(backward, before); + let (at, damaged) = if damage == "type" { + (offset, vec![2]) + } else { + (offset + 8, 8u64.to_le_bytes().to_vec()) + }; + let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + file.seek(SeekFrom::Start(at as u64)).unwrap(); + file.write_all(&damaged).unwrap(); + assert!( + reader.step(backward).is_err(), + "{compact}/{facade}/{filtered}/{backward}/{damage}" + ); + assert_eq!(reader.time(), before, "position while damaged"); + file.seek(SeekFrom::Start(at as u64)).unwrap(); + file.write_all(&source[at..at + damaged.len()]).unwrap(); + assert_eq!( + reader.time(), + before, + "{compact}/{facade}/{filtered}/{backward}/{damage}" + ); + reader.want(backward, 1500); + } + } + } + } + } +} + +#[test] +fn native_filter_nested_direction_change_keeps_unvisited_branches() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + hardening_fixture(&path, compact); + for facade in [false, true] { + let mut reader = PositionTestReader::open(&path, facade); + reader.seek_realtime(1500); + reader.want(false, 1500); + reader.add_match(b"MESSAGE=row-0500"); + reader.add_match(b"SPARSE=some"); + reader.add_disjunction(); + reader.add_match(b"TAIL_PAIR=some"); + reader.want(false, 2003); + reader.want(true, 1500); + reader.want(false, 2003); + reader.want(false, 2004); + reader.want(true, 2003); + reader.want(true, 1500); + } + } +} + +#[test] +fn resolved_backward_filter_skips_unvisited_leaf() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut source = hardening_fixture(&path, false); + let mut common = u64_at(&source, 88) as usize; + loop { + let size = u64_at(&source, common + 8) as usize; + if source[common] == 1 && &source[common + 64..common + size] == b"COMMON=all" { + break; + } + common += (size + 7) & !7; + } + let array = u64_at(&source, common + 48) as usize; + source[array] = 1; + std::fs::write(&path, source).unwrap(); + for facade in [false, true] { + let mut reader = PositionTestReader::open(&path, facade); + reader.seek_realtime(1500); + reader.want(false, 1500); + reader.add_match(b"COMMON=all"); + reader.add_match(b"ZZ_ABSENT=x"); + assert!(!reader.step(true).unwrap()); + reader.seek_tail(); + assert!(reader.step(true).is_err()); + } +} + +#[test] +fn explicit_filtered_time_seek_checks_only_visited_slots() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let source_path = temp.path().join("source.journal"); + let source = hardening_fixture(&source_path, compact); + let payload_start = if compact { 72 } else { 64 }; + let mut data = u64_at(&source, 88) as usize; + loop { + let size = u64_at(&source, data + 8) as usize; + if source[data] == 1 && &source[data + payload_start..data + size] == b"COMMON=all" { + break; + } + data += (size + 7) & !7; + } + let width = if compact { 4 } else { 8 }; + let mut array = u64_at(&source, data + 48) as usize; + let first_slot = array + 24; + let middle_slot = + first_slot + ((u64_at(&source, array + 8) as usize - 24) / width / 2) * width; + let mut remaining = u64_at(&source, data + 56) as usize - 1; + let last_slot = loop { + let count = remaining.min((u64_at(&source, array + 8) as usize - 24) / width); + remaining -= count; + if remaining == 0 { + break array + 24 + (count - 1) * width; + } + array = u64_at(&source, array + 16) as usize; + }; + for facade in [false, true] { + for backward in [false, true] { + for probed in [false, true] { + let path = temp + .path() + .join(format!("{facade}-{backward}-{probed}.journal")); + let mut damaged = source.clone(); + let at = if probed { + middle_slot + } else if backward { + last_slot + } else { + first_slot + }; + damaged[at..at + width].fill(0); + std::fs::write(&path, damaged).unwrap(); + let mut reader = PositionTestReader::open(&path, facade); + reader.add_match(b"COMMON=all"); + reader.seek_realtime(1500); + if probed { + use std::io::{Seek, SeekFrom, Write}; + assert!(reader.step(backward).is_err(), "visited zero slot"); + let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + file.seek(SeekFrom::Start(at as u64)).unwrap(); + file.write_all(&source[at..at + width]).unwrap(); + } + reader.want(backward, 1500); + if !probed { + if backward { + reader.seek_tail(); + } else { + reader.seek_head(); + reader.want(false, 1000); + } + assert!( + reader.step(backward).is_err(), + "ordinary traversal must visit the damaged slot" + ); + } + } + } + } + } +} + +#[test] +fn native_filter_realtime_duplicate_bounds_choose_exact_entry() { + for compact in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("system.journal"); + let mut source = hardening_fixture(&path, compact); + let entries: Vec<_> = array_graph(&source, compact) + .into_iter() + .flat_map(|(_, entries)| entries) + .collect(); + for (index, offset) in entries.iter().copied().enumerate() { + let at = offset as usize + 24; + source[at..at + 8].copy_from_slice(&(1000 + 2 * (index as u64 / 7)).to_le_bytes()); + } + source[192..200].copy_from_slice(&1286u64.to_le_bytes()); + std::fs::write(&path, source).unwrap(); + for live in [false, true] { + for facade in [false, true] { + let mut reader = PositionTestReader::open_bounds(&path, facade, live); + reader.add_match(b"COMMON=all"); + for (time, backward, seqnum) in [ + (0, false, 1), + (0, true, 0), + (999, true, 0), + (1000, false, 1), + (1000, true, 7), + (1001, false, 8), + (1001, true, 7), + (1142, false, 498), + (1142, true, 504), + (1143, false, 505), + (1143, true, 504), + (1286, false, 1002), + (1286, true, 1005), + (1287, false, 0), + (u64::MAX, false, 0), + (u64::MAX, true, 1005), + ] { + reader.seek_realtime(time); + assert_eq!( + reader.step(backward).unwrap(), + seqnum != 0, + "{compact}/{live}/{facade}/{time}/{backward}" + ); + if seqnum != 0 { + assert_eq!( + reader.seqnum(), + seqnum, + "{compact}/{live}/{facade}/{time}/{backward}" + ); + } + } + } + } + } +} + +#[test] +fn directional_cursor_uses_recovered_prefix_without_mutation() { + for compact in [false, true] { + let directory = tempfile::tempdir().unwrap(); + let source_path = directory.path().join("source.journal"); + let source = hardening_fixture(&source_path, compact); + let arrays = array_graph(&source, compact); + let cut = *arrays.last().unwrap().1.last().unwrap() as usize; + let path = directory.path().join("cut.journal"); + std::fs::write(&path, &source[..cut]).unwrap(); + let mut reader = FileReader::open(&path).unwrap(); + reader.add_match(b"COMMON=all"); + reader + .seek_cursor_direction("t=ffffffffffffffff", Direction::Backward) + .unwrap(); + assert_eq!(reader.get_realtime_usec().unwrap(), 2003); + assert!(reader.recovery_status().incomplete()); + assert_eq!(std::fs::read(&path).unwrap(), source[..cut]); + let current = reader.get_cursor().unwrap(); + reader + .seek_cursor_direction(¤t, Direction::Forward) + .unwrap(); + assert_eq!(reader.get_realtime_usec().unwrap(), 2003); + assert!(!reader.next().unwrap()); + } +} diff --git a/rust/src/journal/src/verify_graph/io.rs b/rust/src/journal/src/verify_graph/io.rs index 1ebe921b..2eacca3e 100644 --- a/rust/src/journal/src/verify_graph/io.rs +++ b/rust/src/journal/src/verify_graph/io.rs @@ -11,9 +11,10 @@ pub(crate) trait VerifyByteSource { pub(super) fn decompress_payload(flags: u8, payload: &[u8]) -> Result, String> { if flags & OBJECT_COMPRESSED_ZSTD != 0 { - let mut decoder = - ruzstd::decoding::StreamingDecoder::new(payload).map_err(|err| err.to_string())?; - return read_limited_to_end(&mut decoder); + let mut out = Vec::new(); + journal_core::file::decompress_zstd_data(payload, &mut out) + .map_err(|err| err.to_string())?; + return Ok(out); } if flags & OBJECT_COMPRESSED_XZ != 0 { let mut decoder = lzma_rust2::XzReader::new(payload, false); diff --git a/tests/benchmarks/README.md b/tests/benchmarks/README.md index 8042f8c1..b9a579f7 100644 --- a/tests/benchmarks/README.md +++ b/tests/benchmarks/README.md @@ -41,6 +41,29 @@ model: it refreshes cached file size only when a read would exceed the cached end of file. `snapshot` fixes file size at open time for polling/query consumers that do not need to observe appends during the current scan. +Both reader helpers provide separate `sdk-digest` and `facade-digest` preflight +modes for full ordered payload checks. Their `full-payload-fnv64-v1` checksum +consumes every byte, payload length and row/time marker. Normal scan timing modes +retain `sampled-payload-v1`, which examines length and endpoint bytes. Compare +the diagnostic counters across the retained implementations before timing; +diagnostic hashing time is not a normal scan measurement. + +The file SDK diagnostic mode also records final access/mmap counters before +closing. Go's existing `reader_access_stats` value is captured before scanning; +`reader_access_stats_after` covers the completed diagnostic read. Directory and +facade handles do not expose these counters. When Go `--loops` is used, `extra` +describes the final reader instance, not a cumulative total across loops. + +Ordinary native filtering has separate `sdk-filter` and `facade-filter` modes. +They return ordered timestamp markers without requesting DATA values. Repeated +`--match` arguments use the SDK's same-field OR / different-field AND rules; +`--match-hex` accepts binary match bytes. `--filter-repetitions` repeats seeks on +one reader, `--filter-limit 1` isolates first-result work, `--filter-rebuild` +installs a fresh expression before each repetition, and `--filter-realtime` +sets a time anchor. The corresponding `*-filter-digest` modes consume all +selected payload bytes as an independent preflight. They belong outside timing +comparisons of metadata-only movement. + Go reader results record `mmap_strategy`. `mmap` is the default SDK reader access mode on Unix and is the production hot path. `read-at` remains in the harness as an explicit comparison and diagnostic mode. @@ -51,6 +74,183 @@ profile outputs under `.local/benchmarks/`; `--loops` repeats the selected read case inside one process to collect enough samples and is not used by the shared checksum-comparison harness. +`run_reader_regression_comparison.py` compares retained reader helper binaries +on immutable existing fixtures. Pass both languages' `--baseline-*` and +`--candidate-*` binaries, one or more `--fixture` paths, and a new `--out` +directory under `.local/`. Build each pair with matching benchmark-helper +source and compiler settings; retain the original SDK implementation in the +baseline. The runner records binary/fixture SHA-256 hashes and rejects changes +during measurement. + +The default is two warmups and ten measured baseline/candidate pairs. Order +alternates between pairs. Every run must return the same complete record, +field, byte and checksum counters. `--case` selects case-name substrings; +`--cpu` pins the harness and children to an available CPU; `--batch` repeats +child measurements within each phase and compares their mean read time. +Catalog cases exercise native FIELD/DATA enumeration, including stateful +restarts, without row scans. `--surface directory` applies the supported SDK, +facade and catalog cases to directory fixtures; the directory fingerprint +includes relative file names and content hashes. + +`--surface open-files` treats a directory fixture as an explicit list of its +journal files, fingerprinting the complete directory. The ordinary-filter +`--filter-suite` uses the deterministic mixed-cardinality corpus: dense, sparse, +absent, same-field OR and different-field AND selections, fresh first results +and repeated boundary restarts. It covers both SDK/facade surfaces, bounds and +directions. `--filter-time-usec` adds repeated realtime seeks at an explicit +corpus timestamp that must produce a result in both directions. An independent +native-byte parser computes ordered timestamp checksums before timing and saves +`filter-goldens.json`; every language and build must match them. This preflight +requires native uncompressed mixed-cardinality fixtures. Multi-file inputs must +have distinct realtime values and sequence/boot order consistent with realtime; +validate other merge orders separately. The AND +control intersects overlapping same-field alternatives, with an outcome that +differs from either group and their union. + +Metadata-only filter timings must be preceded by equivalent full-payload digest +correctness controls. `fresh-first` rebuilds the filter on one open reader; +`restart-first` repositions the installed filter. Neither repeats reader opening. +Each helper timer includes one open and close, and reports returned rows rather +than the number of filter operations. Complete filter arguments are retained in +each `.command.json`. Scenario names remain separate in comparisons and summaries. +The normal matrix also includes facade metadata stepping and Rust core modes +under both bounds, directions and mapping strategies. + +`--auxiliary-suite` adds file-only indexed snapshot and verification lifecycles. +Use `--case indexed` or `--case verify` to select those cases. Snapshot capture +declares `LOW_CARD_00=low-00-00`; exact and FIELD equality visitors select the +same population, and every visitor consumes every payload. `indexed-open` +repeats 1,000 complete opens and reports completed operations in `records`; +captured entry/value populations are separate `extra` counters. Visitor cases +report visited rows. All modes include SDK opening, work and cleanup in the +timer. Indexed snapshots require forward traversal and snapshot bounds. + +`verify-index` and `verify-file` report completed successful verifications. +Their public SDK APIs select their own reader defaults; helpers reject custom +window/access labels they cannot honor. Go uses `auto`, Rust uses `windowed`, +and both use the default window size. Use bounded fixtures spanning multiple +ENTRY arrays for compatibility verification, whose repeated-value work can be +quadratic. Keep capture, graph verification and ordinary snapshot-bounds reader +scans separate when interpreting results. + +The auxiliary runner saves independent native-byte expectations in +`auxiliary-goldens.json`, including capture metadata. Before timing, also run +`indexed-exact-digest`, `indexed-field-digest` and `indexed-all-digest` against +full-byte expectations on every measured layout and access strategy. Those +digest companions are deliberately absent from normal timing cases. For one +compressed/staged fixture, `--auxiliary-golden-source` identifies its native, +uncompressed original and freezes its hash. This supplies expected contents; +the full-byte preflight must establish the measured fixture's equivalence. +Single-frame writer-produced Zstd DATA and whole-file streams need distinct +controls; skippable-padding fixtures do not establish the native single-frame +fast path. Fixture creation and external fixture verification stay outside +the timer. + +Raw samples, process statistics, paired timing ratios and bootstrap intervals +are retained. A reproducible slower case is a regression; an inconclusive +interval is not evidence of equivalence. Investigate noise or repeated +slowdowns before accepting a candidate. Select representative cases from the +actual changed work with `--case`; the full matrix is not a mandatory gate for +every edit. Careful code review and high-level controls can establish that +cold-only checks or unchanged hot work do not negatively affect performance. +Use targeted measurements for credible added costs and adverse observations; +an uncertain interval alone does not require expanding the measurement suite. +Standard `summary.json` and +`manifest.json` files are emitted under each phase and direction. Feed the +direction-specific directories to `report_benchmarks.py` so forward/backward +results are compared separately. + +Go indexed aggregation has focused lifetime and allocation controls in +`go/journal/explorer_window_lifetime_test.go`. From `go/`, run: + +```sh +go test -count=1 ./journal -run '^TestExplorerIndexedWindow' +go test ./journal -run '^$' -bench '^BenchmarkExplorerIndexedValueOwnership$' -benchmem +``` + +The first command forces bounded-window eviction and temporary-buffer reuse, +checks both journal layouts and all codecs, and compares complete indexed +results and counters. The raw benchmark covers dense/sparse, low/high-cardinality +facets and histograms. Reader opening, fixture construction and reader close are +outside its timer. + +`BenchmarkExplorerIndexedCompressedValueOwnership` reads a prebuilt corpus +selected by `JOURNAL_SDK_EXPLORER_COMPRESSED_CORPUS`; it skips when that variable +is unset. The directory contains +`{regular,compact}/{zstd,xz,lz4}/{low,high,singleton}.journal`. Each file has 2,048 +rows using the test helper's synthetic `VALUE`/`SELECT` patterns. Low cardinality +uses the dense query with 2,197 updates; high cardinality uses the sparse query +with 34 updates; true singletons use the sparse query with 32 updates. The +benchmark checks indexed/traversal output, actual decompression, update totals, +and a window large enough to avoid eviction or temporary copies before timing. +Generate and validate the corpus once, then freeze its hashes and the complete +matching benchmark helper in both builds. Record per-case timing and allocation +samples; a successful forced-window repair must still pass the valid-control +performance gate. + +Pilot one CLI-helper iteration before increasing `--loops`: a sequential full +scan can cost much more than an indexed query. Each Go helper loop opens and +closes a fresh reader and computes result checksums/sorted summaries; these costs +are inside `read_seconds`. Reported ExplorerStats describe the final loop. +CPU profiling also covers post-timer work; `--memprofile` forces GC after reader +close, so cumulative allocation and live-after-GC memory answer different +questions. These helper timings differ from the aggregation-only Go benchmarks +above. + +Go ENTRY-slot header reads have additional controls in +`go/journal/reader_header_access_test.go` and +`go/journal/reader_header_bench_test.go`. From `go/`, run: + +```sh +go test -count=1 ./journal -run '^(TestEntrySlotHeader|TestReaderAllocationShape|TestNativeReaderFile|TestReaderAccessorInitial)' +go test ./journal -run '^$' -bench '^BenchmarkReader(Allocation|Open|EntryHeader)$' -benchmem +``` + +The access tests cover default, tiny and non-power-of-two windows, fitting and +crossing headers, cached work, partial failures, closed readers, row pins, +eviction and live ENTRY-array growth. Header acquisition preserves raw bytes; +the ENTRY validator retains its separate format/error policy. + +The opening benchmark times fresh open/count checks/close for valid empty and +five-row journals in both layouts and backends. Its optional large cases use +`JOURNAL_SDK_READER_OPEN_CORPUS/{regular,compact}/system.journal`, each containing +100,000 rows. Freeze those files before comparing builds. Fixture generation, +preflight and teardown are outside the benchmark timer. The header benchmark +also measures constrained eviction and pinned-window paths. + +To compare small opens against the exact same files, optionally set +`JOURNAL_SDK_READER_OPEN_SMALL_CORPUS` to a directory containing +`{regular,compact}/system.journal`, each with five entries. The normal generated +fixtures remain the default. Freeze the shared files' hashes and retain a +generated-file control with the same binaries. Extending a benchmark helper +also changes test linking; a timing change after rebuilding does not by itself +establish that fixture generation caused an earlier difference. + +Measure allocation class as well as raw struct size: `TestReaderAllocationShape` +retains 4,096 separately allocated readers and reports runtime size-class counts; +`BenchmarkReaderAllocation` records the corresponding allocation cost. A small +field can cross an allocator class. Reader storage reuse must be checked for +each architecture; identical 64-bit offsets do not establish identical 32-bit +layout. Keep this complete benchmark helper identical in both timing builds. +The direct-helper access, native-file opener and initial-statistics tests cover +candidate contracts; exclude those correctness-only files from both timing builds, record +the exclusions/hashes, and run them separately with the candidate. Preserve +matching pure-build flags and record per-case allocation counts/bytes alongside +complete query measurements. + +Linux native-journal opening avoids regular-file poller setup through a blocking +descriptor. Its tests cover read-only/close-on-exec flags, path errors, symlinks, +descriptor closure and sparse offsets above 2 GiB; run these on amd64 and 386. +The direct opener bypasses Go's internal test-cache input observation. Run +fixture-dependent validation uncached with `-count=1` and retain file hashes. +Compare complete reader operations as well as opener primitives; fewer syscalls +alone do not establish whole-reader performance. Whole-file `.zst` sources use +the existing opener and retain their streaming/staging checks. + +Constructor-statistics controls check empty/nonempty backends and initial mmap +fallback. They compare explicit counters, modes and fallback reason with a fresh +statistics query when constructor work reuses an existing snapshot. + The writer-core harness aligns initial hash table sizing across systemd and SDK drivers with the systemd v260.1 formula: `data=max(max_size_bytes*4/768/3,2047)` and `field=1023`. The default diff --git a/tests/benchmarks/reader_filter_golden.py b/tests/benchmarks/reader_filter_golden.py new file mode 100644 index 00000000..2bf3de53 --- /dev/null +++ b/tests/benchmarks/reader_filter_golden.py @@ -0,0 +1,193 @@ +"""Independent, untimed filter expectations for uncompressed benchmark journals.""" + +from __future__ import annotations + +import re +import struct +from pathlib import Path + + +MASK64 = (1 << 64) - 1 +JOURNAL_SUFFIXES = (".journal", ".journal~", ".journal.zst", ".journal~.zst") + + +def journal_inputs(path: Path, surface: str) -> list[Path]: + if path.is_file(): + return [path] + if surface == "open-files": + return sorted(p for p in path.rglob("*") if p.is_file() and p.name.endswith(JOURNAL_SUFFIXES)) + if surface != "directory": + raise ValueError("file benchmarks require a journal file") + paths = [p for p in path.iterdir() if p.is_file() and p.name.endswith(JOURNAL_SUFFIXES)] + for child in path.iterdir(): + if child.is_dir() and re.fullmatch(r"[0-9a-fA-F]{32}|[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}", child.name): + paths.extend(p for p in child.iterdir() if p.is_file() and p.name.endswith(JOURNAL_SUFFIXES)) + return sorted(paths) + + +def _rows(path: Path, payload_bits: dict[bytes, int]) -> list[tuple]: + data = path.read_bytes() + if data[:8] != b"LPKSHHRH" or len(data) < 208: + raise ValueError(f"filter goldens require native uncompressed journals: {path}") + compact = bool(struct.unpack_from(" dict[str, tuple[int, int, int, int]]: + payload_bits = {payload: 1 << index for index, payload in enumerate(sorted({match.encode() for case in cases for match in case["matches"]}))} + rows = [row for path in paths for row in _rows(path, payload_bits)] + if not rows: + raise ValueError("filter benchmarks require a nonempty mixed-cardinality corpus") + if len(paths) > 1: + rows.sort(key=lambda row: row[0]) + if any(left[0] == right[0] for left, right in zip(rows, rows[1:])): + raise ValueError("multi-file filter goldens require distinct realtime values; validate tied merge keys separately") + seqnums, monotonic = {}, {} + for _, _, seqnum_id, seqnum, boot_id, usec in rows: + if seqnum <= seqnums.get(seqnum_id, -1) or usec < monotonic.get(boot_id, -1): + raise ValueError("multi-file filter goldens require sequence/boot order consistent with realtime; validate other merge orders separately") + seqnums[seqnum_id], monotonic[boot_id] = seqnum, usec + profile_bit = payload_bits[b"PERF_PROFILE=mixed-cardinality-32-fields"] + if any(not row[1] & profile_bit for row in rows): + raise ValueError("--filter-suite requires the mixed-cardinality-32-fields corpus") + goldens = {} + for case in cases: + groups = {} + for match in case["matches"]: + payload = match.encode() + field = payload.split(b"=", 1)[0] + groups[field] = groups.get(field, 0) | payload_bits[payload] + selected = [row[0] for row in rows if all(row[1] & group for group in groups.values())] + if case["direction"] == "backward": + selected.reverse() + realtime = case["filter_realtime"] + if realtime is not None: + selected = [time for time in selected if (time >= realtime if case["direction"] == "forward" else time <= realtime)] + if case["filter_limit"]: + selected = selected[:case["filter_limit"]] + if case["scenario"] != "absent" and not selected: + raise ValueError(f"filter scenario {case['scenario']} has no results; choose an applicable corpus/time anchor") + checksum = 0 + for _ in range(case["filter_repetitions"]): + for time in selected: + checksum = (((checksum << 7) | (checksum >> 57)) & MASK64) ^ time + key = "/".join(str(case[key]) for key in ("mode", "bounds", "direction", "scenario")) + goldens[key] = (len(selected) * case["filter_repetitions"], 0, 0, checksum) + return goldens + + +def auxiliary_goldens(path: Path, cases: list[dict]) -> dict[str, dict]: + """Derive snapshot output from native rows, independently of SDK indexes.""" + matches = {case["snapshot_match"].encode() for case in cases if "snapshot_match" in case} + if len(matches) != 1: + raise ValueError("auxiliary goldens require one shared snapshot declaration") + match = matches.pop() + data = path.read_bytes() + if data[:8] != b"LPKSHHRH" or len(data) < 208: + raise ValueError("auxiliary goldens require a native journal") + compact = bool(struct.unpack_from(" int: + return ((value << amount) | (value >> (64 - amount))) & MASK64 + + def add(name: str, realtime: int, payloads: list[memoryview], digest: bool) -> None: + counters = totals[name] + counters[0] += 1 + counters[3] = rotate(counters[3], 7) ^ realtime + for payload in payloads: + counters[1] += 1 + counters[2] += len(payload) + checksum = rotate(counters[3], 5) ^ len(payload) + if digest: + for byte in payload: + checksum = ((checksum ^ byte) * 1099511628211) & MASK64 + elif payload: + checksum ^= payload[0] << 8 + checksum ^= payload[-1] + counters[3] = checksum + + while remaining: + if not array or array in seen or data[array] != 6: + raise ValueError("invalid auxiliary ENTRY_ARRAY chain") + seen.add(array) + count = min(remaining, (u64(data, array + 8)[0] - 24) // width) + if count <= 0: + raise ValueError("empty required auxiliary ENTRY_ARRAY") + for slot in range(count): + entry = integer(data, array + 24 + slot * width)[0] + if not entry or data[entry] != 3: + raise ValueError("invalid auxiliary ENTRY") + payloads = [] + for at in range(entry + 64, entry + u64(data, entry + 8)[0], item_width): + target = integer(data, at)[0] + if not target or data[target] != 1 or data[target + 1] != 0: + raise ValueError("auxiliary goldens require uncompressed DATA") + payloads.append(memoryview(data)[target + (72 if compact else 64):target + u64(data, target + 8)[0]]) + realtime = u64(data, entry + 24)[0] + selected = any(payload == match for payload in payloads) + for name in ("all", "selected") if selected else ("all",): + add(name, realtime, payloads, False) + if digest_modes: + add(name + "-digest", realtime, payloads, True) + remaining -= count + array = u64(data, array + 16)[0] + if totals["all"][0] != population or not totals["selected"][0]: + raise ValueError("auxiliary corpus must contain the declared snapshot value") + goldens = {} + for case in cases: + mode, repetitions = case["mode"], case.get("repetitions", 1) + if mode == "indexed-open" or mode.startswith("verify-"): + marker = population if mode == "indexed-open" else 1 + checksum = 0 + for _ in range(repetitions): + checksum = rotate(checksum, 7) ^ marker + counts = (repetitions, 0, 0, checksum) + else: + if repetitions != 1: + raise ValueError("snapshot visitor goldens require one lifecycle per child") + name = "all" if mode.startswith("indexed-all") else "selected" + counts = tuple(totals[name + ("-digest" if mode.endswith("-digest") else "")]) + extra = {"completed_verifications": repetitions, "reader_options": "public SDK defaults"} if mode.startswith("verify-") else { + "completed_opens": repetitions, "captured_entries": population, + "captured_value_entries": totals["selected"][0], "captured_value_present": True, + } + goldens[mode] = dict(counts=counts, extra=extra) + return goldens diff --git a/tests/benchmarks/run_reader_regression_comparison.py b/tests/benchmarks/run_reader_regression_comparison.py new file mode 100755 index 00000000..fcb84d45 --- /dev/null +++ b/tests/benchmarks/run_reader_regression_comparison.py @@ -0,0 +1,479 @@ +#!/usr/bin/env python3 +"""Compare retained reader binaries on the same immutable journal fixtures. + +Alternates baseline/candidate order, validates complete semantic counters, and +retains individual measurements plus standard reader benchmark summaries. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import random +import shlex +import statistics +from datetime import datetime, timezone +from pathlib import Path + +import run_reader_core_benchmarks as reader +from reader_filter_golden import auxiliary_goldens, filter_goldens, journal_inputs + + +def cases(language: str) -> list[dict]: + strategies = ( + ("windowed", "whole-file") if language == "rust" else ("mmap", "read-at") + ) + out = [] + for strategy in strategies: + for bounds in ("live", "snapshot"): + for mode in ("sdk-payloads", "facade-data", "sdk-entry", "facade-next"): + for direction in ("forward", "backward"): + out.append( + { + "mode": mode, + "bounds": bounds, + "strategy": strategy, + "direction": direction, + } + ) + if language == "rust": + for mode in ("core-next", "core-offsets", "core-payloads"): + for bounds in ("live", "snapshot"): + for strategy in strategies: + for direction in ("forward", "backward"): + out.append(dict(mode=mode, bounds=bounds, strategy=strategy, direction=direction)) + for mode, field, repetitions in ( + ("sdk-fields", "HIGH_CARD_00", 10000), + ("sdk-unique", "LOW_CARD_00", 10000), + ("sdk-unique-state", "LOW_CARD_00", 10000), + ("sdk-unique", "HIGH_CARD_00", 3), + ("sdk-unique-state", "HIGH_CARD_00", 3), + ): + out.append( + { + "mode": mode, + "field": field, + "repetitions": repetitions, + "bounds": "snapshot", + "strategy": strategies[0], + "direction": "forward", + } + ) + out.append( + { + "mode": "explorer-query", + "bounds": "snapshot", + "strategy": strategies[0], + "direction": "forward", + } + ) + return out + + +def filter_cases(language: str, realtime: int | None) -> list[dict]: + strategy = "windowed" if language == "rust" else "mmap" + dense = ["PERF_PROFILE=mixed-cardinality-32-fields"] + scenarios = [ + ("unfiltered", [], 1, 0, False, None), + ("dense", dense, 1, 0, False, None), + ("sparse", ["HIGH_CARD_00=high-00-000000"], 1, 0, False, None), + ("absent", ["LOW_CARD_00=absent"], 1, 0, False, None), + ("same-field-or", ["LOW_CARD_00=low-00-00", "LOW_CARD_00=low-00-01"], 1, 0, False, None), + ("different-field-and", ["LOW_CARD_00=low-00-00", "LOW_CARD_00=low-00-01", "LOW_CARD_01=low-01-01", "LOW_CARD_01=low-01-02"], 1, 0, False, None), + ("fresh-first", dense, 1000, 1, True, None), + ("restart-first", dense, 10000, 1, False, None), + ] + if realtime is not None: + scenarios.append(("realtime-first", dense, 1000, 1, False, realtime)) + out = [] + for mode in ("sdk-filter", "facade-filter"): + for bounds in ("live", "snapshot"): + for direction in ("forward", "backward"): + for name, matches, repetitions, limit, rebuild, time in scenarios: + out.append(dict( + mode=mode, scenario=name, matches=matches, + filter_repetitions=repetitions, filter_limit=limit, + filter_rebuild=rebuild, filter_realtime=time, + bounds=bounds, strategy=strategy, direction=direction, + )) + return out + + +def case_id(case: dict) -> str: + return "/".join( + str(case.get(key, "")) + for key in ("mode", "field", "bounds", "strategy", "direction", "scenario") + ) + + +def auxiliary_cases(language: str) -> list[dict]: + strategies = ("windowed", "whole-file") if language == "rust" else ("mmap", "read-at") + out = [dict(mode=mode, field="LOW_CARD_00", repetitions=1000 if mode == "indexed-open" else 1, + snapshot_match="LOW_CARD_00=low-00-00", bounds="snapshot", strategy=strategy, + direction="forward") + for mode in ("indexed-open", "indexed-exact", "indexed-field", "indexed-all") + for strategy in strategies] + out += [dict(mode=mode, bounds="snapshot", strategy="windowed" if language == "rust" else "auto", + direction="forward") for mode in ("verify-index", "verify-file")] + return out + + +def sha256(path: Path) -> str: + if path.is_file(): + with path.open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + digest = hashlib.sha256() + inodes = {} + for file in sorted(path.rglob("*")): + if file.is_file(): + stat = file.stat() + key = (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns) + if key not in inodes: + inodes[key] = sha256(file) + digest.update(file.relative_to(path).as_posix().encode() + b"\0") + digest.update(inodes[key].encode() + b"\n") + return digest.hexdigest() + + +def compare(pairs: list[tuple[float, float]]) -> dict: + ratios = [after / before for before, after in pairs] + rng = random.Random(159) # nosec B311 - reproducible bootstrap sampling. + sampled = sorted( + statistics.median(rng.choices(ratios, k=len(ratios))) for _ in range(10000) + ) + low, high = sampled[249], sampled[9749] + return { + "paired_median_time_ratio": statistics.median(ratios), + "paired_bootstrap_95_percent_interval": [low, high], + "baseline_median_seconds": statistics.median(p[0] for p in pairs), + "candidate_median_seconds": statistics.median(p[1] for p in pairs), + "baseline_coefficient_of_variation": statistics.stdev(p[0] for p in pairs) + / statistics.mean(p[0] for p in pairs), + "candidate_coefficient_of_variation": statistics.stdev(p[1] for p in pairs) + / statistics.mean(p[1] for p in pairs), + "assessment": "regression" + if low > 1 + else "improvement" + if high < 1 + else "inconclusive", + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + for phase in ("baseline", "candidate"): + for language in ("rust", "go"): + parser.add_argument(f"--{phase}-{language}", type=Path, required=True) + parser.add_argument("--fixture", type=Path, action="append", required=True) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument("--surface", choices=("file", "directory", "open-files"), default="file") + parser.add_argument("--filter-suite", action="store_true", help="Include ordinary filtering on the mixed-cardinality corpus") + parser.add_argument("--auxiliary-suite", action="store_true", help="Include indexed snapshot lifecycles and verification on a bounded uncompressed corpus") + parser.add_argument("--auxiliary-golden-source", type=Path, help="Uncompressed original for one compressed/staged auxiliary fixture") + parser.add_argument("--filter-time-usec", type=int, help="Add repeated filtered realtime-seek cases at this timestamp") + parser.add_argument("--repetitions", type=int, default=10) + parser.add_argument("--warmups", type=int, default=2) + parser.add_argument( + "--batch", + type=int, + default=1, + help="Child runs per phase; compare paired mean read times", + ) + parser.add_argument( + "--cpu", type=int, help="Pin this harness and its children to one allowed CPU" + ) + parser.add_argument( + "--case", + action="append", + default=[], + help="Include cases containing any supplied substring", + ) + args = parser.parse_args() + if args.repetitions < 3 or args.warmups < 1 or args.batch < 1: + parser.error("at least three measured repetitions and one warmup are required") + if args.filter_time_usec is not None and not 0 <= args.filter_time_usec <= (1 << 64) - 1: + parser.error("--filter-time-usec must fit an unsigned 64-bit timestamp") + if args.auxiliary_suite and args.surface != "file": + parser.error("--auxiliary-suite supports file input only") + if args.auxiliary_golden_source and (not args.auxiliary_suite or len(args.fixture) != 1): + parser.error("--auxiliary-golden-source requires --auxiliary-suite and one fixture") + if args.cpu is not None: + if not hasattr(os, "sched_getaffinity") or args.cpu not in os.sched_getaffinity( + 0 + ): + parser.error("--cpu must name a CPU available to this process") + os.sched_setaffinity(0, {args.cpu}) + root = reader.ROOT.resolve() + out = args.out.resolve() + if not out.is_relative_to(root / ".local") or out.exists(): + parser.error("--out must be a new directory under the repository's .local/") + fixtures = [path.resolve(strict=True) for path in args.fixture] + binaries = { + (phase, language): getattr(args, f"{phase}_{language}").resolve(strict=True) + for phase in ("baseline", "candidate") + for language in ("rust", "go") + } + binary_hashes = {key: sha256(path) for key, path in binaries.items()} + out.mkdir(parents=True) + (out / "tmp").mkdir() + env = {"TMPDIR": str(out / "tmp")} + comparisons = [] + for fixture_index, fixture in enumerate(fixtures): + fixture_hash = sha256(fixture) + inputs = [fixture] + if args.surface == "open-files" and fixture.is_dir(): + inputs = journal_inputs(fixture, args.surface) + if not inputs: + raise RuntimeError("open-files fixture contains no journals") + fixture_out = out / f"fixture-{fixture_index}" + goldens = filter_goldens(journal_inputs(fixture, args.surface), filter_cases("go", args.filter_time_usec)) if args.filter_suite else {} + auxiliary_source = args.auxiliary_golden_source.resolve(strict=True) if args.auxiliary_golden_source else fixture + auxiliary_source_hash = sha256(auxiliary_source) if args.auxiliary_suite else None + auxiliary = auxiliary_goldens(auxiliary_source, auxiliary_cases("go")) if args.auxiliary_suite else {} + if auxiliary: + fixture_out.mkdir(parents=True) + (fixture_out / "auxiliary-goldens.json").write_text(json.dumps(auxiliary, indent=2) + "\n") + retained = {"baseline": [], "candidate": []} + for phase in retained: + phase_out = fixture_out / phase + phase_out.mkdir(parents=True) + (phase_out / "manifest.json").write_text( + json.dumps( + { + "created_at": datetime.now(timezone.utc).isoformat(), + "benchmark": "reader-core", + "binaries": { + language: { + "path": str(binaries[phase, language]), + "sha256": binary_hashes[phase, language], + } + for language in ("rust", "go") + }, + "parameters": { + "repetitions": args.repetitions, + "warmups": args.warmups, + "batch": args.batch, + "cpu": args.cpu, + "surface": args.surface, + "fixture_sha256": fixture_hash, + }, + "status": "in progress; inspect per-process output on failure", + }, + indent=2, + ) + + "\n" + ) + if goldens: + (fixture_out / "filter-goldens.json").write_text(json.dumps(goldens, indent=2) + "\n") + for language in ("rust", "go"): + selected_cases = cases(language) + if args.filter_suite: + selected_cases += filter_cases(language, args.filter_time_usec) + if args.auxiliary_suite: + selected_cases += auxiliary_cases(language) + for case in selected_cases: + if args.surface != "file" and ( + case["mode"].startswith("core-") or case["mode"] == "explorer-query" + ): + continue + identity = f"{language}/{case_id(case)}" + if args.case and not any( + fragment in identity for fragment in args.case + ): + continue + pairs = [] + golden_key = "/".join(str(case[key]) for key in ("mode", "bounds", "direction", "scenario")) if "scenario" in case else "" + expected = goldens.get(golden_key) + auxiliary_expected = auxiliary.get(case["mode"]) + if auxiliary_expected: + expected = tuple(auxiliary_expected["counts"]) + for iteration in range(-args.warmups, args.repetitions): + order = ( + ("baseline", "candidate") + if iteration % 2 == 0 + else ("candidate", "baseline") + ) + times = {} + for phase in order: + command = [ + str(binaries[phase, language]), + "--surface", + args.surface, + "--mode", + case["mode"], + "--bounds", + case["bounds"], + "--mmap-strategy", + case["strategy"], + "--direction", + case["direction"], + ] + for input_path in inputs: + command += ["--input", str(input_path)] + if "field" in case: + command += [ + "--catalog-field", + case["field"], + "--catalog-repetitions", + str(case["repetitions"]), + ] + if "snapshot_match" in case: + command += ["--match", case["snapshot_match"]] + if case["mode"] == "explorer-query": + command += [ + "--explorer-strategy", + "index", + "--explorer-field-mode", + "all-values", + "--explorer-filter", + "LOW_CARD_00=low-00-00", + "--explorer-facet", + "LOW_CARD_01", + "--explorer-limit", + "100", + ] + if "scenario" in case: + for match in case["matches"]: + command += ["--match", match] + command += ["--filter-repetitions", str(case["filter_repetitions"]), "--filter-limit", str(case["filter_limit"])] + if case["filter_rebuild"]: + command += ["--filter-rebuild"] + if case["filter_realtime"] is not None: + command += ["--filter-realtime", str(case["filter_realtime"])] + times[phase] = 0.0 + for sample in range(args.batch): + print(shlex.join(command), flush=True) + stats_path = ( + fixture_out + / phase + / f"process-{len(retained[phase])}.json" + ) + result = reader.timed_run(command, stats_path, env) + stats_path.with_suffix(".stdout").write_text(result.stdout) + stats_path.with_suffix(".stderr").write_text(result.stderr) + stats_path.with_suffix(".command.json").write_text( + json.dumps( + { + "command": command, + "returncode": result.returncode, + }, + indent=2, + ) + + "\n" + ) + reader.require_ok(result, identity) + parsed = reader.parse_json_result(result.stdout) + semantic = tuple( + parsed.get(key) for key in reader.CHECKSUM_FIELDS + ) + if expected is None: + expected = semantic + if semantic != expected: + raise RuntimeError( + f"semantic mismatch {identity}/{phase}: {semantic} != {expected}" + ) + if auxiliary_expected: + extra = parsed.get("extra", {}) + if any(extra.get(key) != value for key, value in auxiliary_expected["extra"].items()): + raise RuntimeError(f"capture/verification metadata mismatch {identity}/{phase}: {extra}") + seconds = float(parsed["read_seconds"]) + if not math.isfinite(seconds) or seconds <= 0: + raise RuntimeError( + f"invalid timing {identity}: {seconds}" + ) + times[phase] += seconds / args.batch + if "field" in case: + parsed["mode"] += f"[{case['field']}]" + if "scenario" in case: + parsed["mode"] += f"[{case['scenario']}]" + retained[phase].append( + { + "warmup": iteration < 0, + "iteration": iteration, + "sample": sample, + "command": command, + "result": parsed, + "process_stats": reader.parse_time_stats( + stats_path + ), + } + ) + with (fixture_out / phase / "runs.jsonl").open( + "a" + ) as stream: + stream.write(json.dumps(retained[phase][-1]) + "\n") + if iteration >= 0: + pairs.append((times["baseline"], times["candidate"])) + comparisons.append( + dict(fixture=fixture_index, case=identity, **compare(pairs)) + ) + print(json.dumps(comparisons[-1]), flush=True) + (out / "comparisons.json").write_text( + json.dumps(comparisons, indent=2) + "\n" + ) + for phase, runs in retained.items(): + for name, value in ( + ("runs.json", runs), + ("summary.json", reader.summarize(runs)), + ): + (fixture_out / phase / name).write_text( + json.dumps(value, indent=2) + "\n" + ) + if sha256(fixture) != fixture_hash: + raise RuntimeError(f"fixture changed during measurement: {fixture}") + if args.auxiliary_suite and sha256(auxiliary_source) != auxiliary_source_hash: + raise RuntimeError(f"auxiliary golden source changed during measurement: {auxiliary_source}") + for key, binary in binaries.items(): + if sha256(binary) != binary_hashes[key]: + raise RuntimeError(f"binary changed during measurement: {binary}") + for phase, runs in retained.items(): + phase_out = fixture_out / phase + manifest = { + "created_at": datetime.now(timezone.utc).isoformat(), + "benchmark": "reader-core", + "binaries": { + language: { + "path": str(binaries[phase, language]), + "sha256": binary_hashes[phase, language], + } + for language in ("rust", "go") + }, + "parameters": { + "repetitions": args.repetitions, + "warmups": args.warmups, + "batch": args.batch, + "cpu": args.cpu, + "surface": args.surface, + "fixture_sha256": fixture_hash, + "auxiliary_golden_source_sha256": auxiliary_source_hash, + }, + "status": "measured; correctness and performance acceptance require review", + } + for name, value in ( + ("runs.json", runs), + ("summary.json", reader.summarize(runs)), + ("manifest.json", manifest), + ): + (phase_out / name).write_text(json.dumps(value, indent=2) + "\n") + for direction in ("forward", "backward"): + directional = [ + run for run in runs if run["result"]["direction"] == direction + ] + if directional: + destination = phase_out / direction + destination.mkdir() + (destination / "summary.json").write_text( + json.dumps(reader.summarize(directional), indent=2) + "\n" + ) + manifest["parameters"]["direction"] = direction + (destination / "manifest.json").write_text( + json.dumps(manifest, indent=2) + "\n" + ) + (out / "comparisons.json").write_text(json.dumps(comparisons, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/tests/coverage/README.md b/tests/coverage/README.md index 4646100c..8a5ac506 100644 --- a/tests/coverage/README.md +++ b/tests/coverage/README.md @@ -3,6 +3,24 @@ This directory contains local coverage helpers used by the GitHub Actions coverage workflow. +Coverage includes both commands. Install a native C compiler, `pkg-config` and +PCRE2-8 development files (>=10.42) before running either language's coverage. +The Go helper first checks the pure SDK with CGO disabled, then enables CGO for +full coverage so the command cannot be silently omitted by its build tags. +Both helpers also run the existing portable CLI controls against instrumented +binaries, including unit naming and indexed invocation selection. Their hits +are combined with language-test coverage before filtering and upload. Python 3 +and the repository's Go toolchain are required for synthetic fixture generation +in both jobs; set `PYTHON` to select an existing Python interpreter. + +Go subprocess profiles use a fresh per-run directory and are merged by source +block with the atomic unit-test profile. Rust runs workspace tests and CLI +subprocesses in the same cargo-llvm-cov collection before generating LCOV. +Fixture generation is not instrumented. These test-only builds do not change +release binaries or runtime performance. The full pinned-systemd oracle matrix +remains a separate compatibility gate; the portable subset needs no systemd +installation. + Reports are written under `.local/coverage/` by default: ```bash diff --git a/tests/coverage/run_go_coverage.sh b/tests/coverage/run_go_coverage.sh index 21fe0113..556eb987 100755 --- a/tests/coverage/run_go_coverage.sh +++ b/tests/coverage/run_go_coverage.sh @@ -14,9 +14,32 @@ RAW_REPORT="${OUT_DIR}/coverage.raw.out" NORMALIZED_REPORT="${OUT_DIR}/coverage.normalized.out" SUMMARY_REPORT="${OUT_DIR}/coverage.summary.out" SUMMARY="${OUT_DIR}/coverage.txt" +UNIT_REPORT="${OUT_DIR}/coverage.unit.out" +CLI_DIR=$(mktemp -d "${OUT_DIR}/cli.XXXXXX") +mkdir "${CLI_DIR}/profiles" cd "${ROOT}/go" -run go test -covermode=atomic -coverpkg=./... -coverprofile="${RAW_REPORT}" ./... +run env CGO_ENABLED=0 go test -count=1 ./... +run env CGO_ENABLED=1 go test -count=1 -covermode=atomic -coverpkg=./... -coverprofile="${UNIT_REPORT}" ./... +run env CGO_ENABLED=1 go build -cover -covermode=atomic -coverpkg=./... -o "${CLI_DIR}/journalctl" ./cmd/journalctl +run env GOCOVERDIR="${CLI_DIR}/profiles" "${PYTHON:-python3}" \ + "${ROOT}/tests/interoperability/run_journalctl_portable_smoke.py" --go "${CLI_DIR}/journalctl" +run go tool covdata textfmt -i="${CLI_DIR}/profiles" -o="${CLI_DIR}/coverage.out" +# Merge by source block so subprocess hits supplement rather than replace unit tests. +run awk ' + FNR == 1 { + if ($0 != "mode: atomic") { print "ERROR: expected atomic profile: " FILENAME > "/dev/stderr"; exit 1 } + next + } + $1 in statements && statements[$1] != $2 { + print "ERROR: inconsistent coverage block: " $1 > "/dev/stderr"; exit 1 + } + { statements[$1] = $2; hits[$1] += $3 } + END { + print "mode: atomic" + for (block in statements) printf "%s %d %.0f\n", block, statements[block], hits[block] + } +' "${UNIT_REPORT}" "${CLI_DIR}/coverage.out" > "${RAW_REPORT}" normalize_coverprofile_prefix \ "${RAW_REPORT}" \ "${NORMALIZED_REPORT}" \ diff --git a/tests/coverage/run_rust_coverage.sh b/tests/coverage/run_rust_coverage.sh index f217225f..4c603726 100755 --- a/tests/coverage/run_rust_coverage.sh +++ b/tests/coverage/run_rust_coverage.sh @@ -19,8 +19,22 @@ if ! cargo llvm-cov --version >/dev/null 2>&1; then fi cd "${ROOT}/rust" +# Keep show-env and the workspace test run on the same profile/object directory. +PROFILE_DIR="${CARGO_LLVM_COV_TARGET_DIR:-${CARGO_TARGET_DIR:-${ROOT}/rust/target}/llvm-cov-target}" +export CARGO_LLVM_COV_TARGET_DIR="${PROFILE_DIR}" run cargo llvm-cov clean --workspace -run cargo llvm-cov --workspace --lcov --output-path "${RAW_REPORT}" +run cargo llvm-cov --workspace --no-report +run cargo llvm-cov show-env --sh > "${OUT_DIR}/cli-env.sh" +( + # This environment is emitted by the installed coverage tool, not user input. + # shellcheck disable=SC1091 + source "${OUT_DIR}/cli-env.sh" + # Cargo integration tests already build this instrumented command. + run test -x "${PROFILE_DIR}/debug/journalctl" + run "${PYTHON:-python3}" "${ROOT}/tests/interoperability/run_journalctl_portable_smoke.py" \ + --rust "${PROFILE_DIR}/debug/journalctl" +) +run cargo llvm-cov report --lcov --output-path "${RAW_REPORT}" run normalize_lcov_sf_prefix "${RAW_REPORT}" "${NORMALIZED_REPORT}" "${ROOT}/rust/" "rust/" validate_lcov_records "${NORMALIZED_REPORT}" run filter_lcov_sf_paths "${NORMALIZED_REPORT}" "${REPORT}" diff --git a/tests/interoperability/README.md b/tests/interoperability/README.md index 76af65ab..858db1d1 100644 --- a/tests/interoperability/README.md +++ b/tests/interoperability/README.md @@ -1,5 +1,14 @@ # Interoperability Matrix +CLI-inclusive matrices require PCRE2-8 development files (>=10.42), +`pkg-config` and a native C compiler. Their Go command builds explicitly enable +CGO; SDK writers/adapters remain pure. Query tests use the pinned full-feature +systemd v262 oracle, including ordered parsing, binary PCRE2, timestamp/TZif +bounds, no-match status and complete cursor/follow history. Follow capture uses +files so large backlogs cannot block on an unread pipe; actual child exit status +is retained alongside intentional harness termination. Native portability smoke +checks are maintained by `.github/workflows/journalctl.yml`. + This directory contains repo-local matrix runners for writer, reader, live-concurrency, compression, lock, file-backed query, and byte-identity validation. @@ -362,7 +371,7 @@ realtime ranges, then compares stock journalctl with the Rust and Go file-backed rewrites. ```bash -python3 tests/interoperability/run_journalctl_query_matrix.py +python3 tests/interoperability/run_journalctl_query_matrix.py --journalctl /path/to/systemd-v262/journalctl ``` The matrix validates: diff --git a/tests/interoperability/go_fixture_writer.py b/tests/interoperability/go_fixture_writer.py index c3064fc1..176eb94d 100644 --- a/tests/interoperability/go_fixture_writer.py +++ b/tests/interoperability/go_fixture_writer.py @@ -59,7 +59,7 @@ def _render_go_mod() -> str: "require (\n" f"{requirements}\n" ")\n\n" - f"replace github.com/netdata/systemd-journal-sdk/go => {GO_MODULE_DIR}\n" + f"replace github.com/netdata/systemd-journal-sdk/go => {json.dumps(GO_MODULE_DIR.as_posix())}\n" ) @@ -96,6 +96,7 @@ def _render_main_go() -> str: } type spec struct { + HeadSeqnum uint64 `json:"head_seqnum"` Path string `json:"path"` MachineID string `json:"machine_id"` BootID string `json:"boot_id"` @@ -167,6 +168,7 @@ def _render_main_go() -> str: MachineID: machineID, BootID: bootID, SeqnumID: seqnumID, + HeadSeqnum: s.HeadSeqnum, FileID: fileID, Compact: s.Compact, Compression: compression, @@ -359,18 +361,6 @@ def _render_live_main_go() -> str: return w.Append(fields, opts) } -func appendAndSync(w *journal.Writer, entries []entrySpec) error { - for _, entry := range entries { - if err := appendEntry(w, entry); err != nil { - return err - } - if err := w.Sync(); err != nil { - return err - } - } - return nil -} - func main() { if len(os.Args) != 2 { fmt.Fprintln(os.Stderr, "usage: livefixturewriter ") @@ -405,9 +395,12 @@ def _render_live_main_go() -> str: _ = w.Close() } }() - if err := appendAndSync(w, s.Initial); err != nil { - fmt.Fprintln(os.Stderr, err) - os.Exit(1) + // Initial history is complete before readers start; sync the batch once. + for _, entry := range s.Initial { + if err := appendEntry(w, entry); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } } if err := w.Sync(); err != nil { fmt.Fprintln(os.Stderr, err) @@ -483,6 +476,7 @@ def _env(extra: dict[str, str] | None = None) -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) if extra: env.update(extra) + env["CGO_ENABLED"] = "0" return env @@ -510,6 +504,7 @@ def write_journal_file( boot_id: str, seqnum_id: str, entries: list[dict[str, Any]], + head_seqnum: int = 0, file_id: str = "", compact: bool = False, compression: str = "none", @@ -543,6 +538,7 @@ def write_journal_file( "machine_id": machine_id, "boot_id": boot_id, "seqnum_id": seqnum_id, + "head_seqnum": head_seqnum, "file_id": file_id, "compact": compact, "compression": compression, diff --git a/tests/interoperability/journalctl_color_cases.py b/tests/interoperability/journalctl_color_cases.py new file mode 100644 index 00000000..8c4b23ec --- /dev/null +++ b/tests/interoperability/journalctl_color_cases.py @@ -0,0 +1,217 @@ +"""Terminal color controls against the pinned native journalctl.""" + +from __future__ import annotations + +import collections +import json +import os +import re +import subprocess # nosec B404 +import tempfile +from pathlib import Path + +from go_fixture_writer import write_journal_file + + +ANSI = re.compile(rb"\x1b\[[0-9;:]*m") +JSON_TOKEN = re.compile(rb'"(?:\\.|[^"\\])*"|(?:\x1b\[[0-9;:]*m)*(?:-?\d+|null|true|false)(?:\x1b\[[0-9;:]*m)*') + + +def normalized(data: bytes, mode: str): + if mode == "cat-projection": + # Native output-fields is an unordered set; this control uses one-line values. + return sorted(data.splitlines(keepends=True)) + if not mode.startswith("json"): + return data + if mode == "json-sse": + bodies = [v.removeprefix(b"data: ") for v in data.split(b"\n\n") if v] + elif mode == "json-seq": + bodies = [v for v in data.split(b"\x1e") if v.strip()] + elif mode == "json-pretty": + text = ANSI.sub(b"", data).decode() + decoder, offset, parsed = json.JSONDecoder(), 0, [] + while offset < len(text): + if text[offset:].isspace(): + break + item, count = decoder.raw_decode(text[offset:]) + parsed.append(item) + offset += count + while offset < len(text) and text[offset].isspace(): + offset += 1 + return parsed, collections.Counter(JSON_TOKEN.findall(data)) + else: + bodies = data.splitlines() + return ([json.loads(ANSI.sub(b"", value)) for value in bodies], + collections.Counter(JSON_TOKEN.findall(b"\n".join(bodies)))) + + +def run_color_cases(commands: dict[str, str], directory: Path, + env: dict[str, str]) -> list[dict[str, object]]: + source = directory / "v262-colors.journal" + boot = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + entries = [] + values = [b"alpha hit beta"] * 8 + [ + b"hit\nsecond", b"head\nlong second hit tail", b"\talpha hit", b"\x1b[31malpha hit\x1b[0m", + b"\x1b[2J alpha hit", b"hit\x00tail", b"invalid\xff hit", b"", b"hit\n", b"hit\n\n", + b"x" * 160 + b" hit tail", b"one\ntwo\nthree hit\nfour", b"alpha hit\r\nsecond", + ] + for index, value in enumerate(values): + fields = [("MESSAGE", value), ("ROW", str(index)), ("PRIORITY", str(index % 8)), + ("_BOOT_ID", boot), ("BINARY", b"\xff\0"), ("NULL_VALUE", b"x" * 4100)] + if index == 6: + fields.append(("_TRANSPORT", "audit")) + entries.append({"realtime_usec": 1_700_005_000_000_000 + index, + "monotonic_usec": index + 1, "fields": fields}) + # First MESSAGE supplies grep offsets, while short formats display the last. + entries.append({"realtime_usec": 1_700_005_000_000_100, "monotonic_usec": 101, + "fields": [("MESSAGE", "hit first"), ("MESSAGE", "last message"), + ("PRIORITY", "3"), ("ROW", "duplicate"), ("_BOOT_ID", boot)]}) + entries.append({"realtime_usec": 1_700_005_000_000_101, "monotonic_usec": 102, + "fields": [("MESSAGE", "/var/log/journal-sdk/example.conf: hit configuration"), + ("CONFIG_FILE", "/var/log/journal-sdk/example.conf"), ("DOCUMENTATION", "https://example.org/manual"), + ("PRIORITY", "4"), ("ROW", "links"), ("_BOOT_ID", boot)]}) + entries.append({"realtime_usec": 1_700_005_000_000_102, "monotonic_usec": 103, + "fields": [("MESSAGE", "example.conf: hit configuration"), ("CONFIG_FILE", "example.conf"), + ("PRIORITY", "4"), ("ROW", "relative-links"), ("_BOOT_ID", boot)]}) + write_journal_file(source, machine_id="00112233445566778899aabbccddeeff", + boot_id=boot, seqnum_id="12121212121212121212121212121212", entries=entries) + boots = directory / "v262-color-boots.journal" + # No catalog message IDs: implicit host catalog lookup is outside portable mode. + write_journal_file(boots, machine_id="00112233445566778899aabbccddeeff", boot_id=boot, + seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000 + index, + "monotonic_usec": 1, "boot_id": ch * 32, + "fields": [("MESSAGE", "boot-" + ch), ("PRIORITY", str(priority)), + ("_BOOT_ID", ch * 32)]} + for index, (ch, priority) in enumerate([("a", 3), ("b", 7)])]) + modes = ["short", "short-full", "short-iso", "short-iso-precise", "short-precise", + "short-monotonic", "short-delta", "short-unix", "with-unit", "cat", "verbose", + "json", "json-pretty", "json-seq", "json-sse", "export"] + cases = [] + for mode in modes: + for extra in ([], ["--grep=hit"], ["--grep=hit", "-n3", "-r"], ["--all", "--grep=hit"]): + cases.append((mode + "-" + repr(extra), mode, ["-q", "-o", mode, *extra], {"SYSTEMD_COLORS": "16"})) + for policy in ["256", "24bit", "true", "false", "auto-16", "invalid"]: + cases.append(("policy-" + policy, "short", ["-q", "-oshort", "ROW=4", "ROW=7"], + {"SYSTEMD_COLORS": policy})) + cases += [("plain-verbose-multiline", "verbose", ["-q", "-overbose", "ROW=8", "ROW=13", "--all"], {"SYSTEMD_COLORS": "0"}), + ("plain-reverse-delta", "short-delta", ["-q", "-oshort-delta", "--grep=hit", "-n3", "-r"], {"SYSTEMD_COLORS": "0"})] + for mode in ("short", "verbose"): + for link in ("0", "1"): + for grep in ([], ["--grep=hit"], ["--grep=example"]): + cases.append((f"links-{mode}-{link}-{grep}", mode, ["-q", "-o", mode, "ROW=links", *grep], + {"SYSTEMD_COLORS": "16", "SYSTEMD_URLIFY": link})) + for mode in ("short", "cat", "verbose", "json"): + for policy in ("16", "true", "auto-16"): + cases.append(("no-color-" + mode + "-" + policy, mode, ["-q", "-o", mode, "ROW=4"], + {"SYSTEMD_COLORS": policy, "NO_COLOR": ""})) + cases += [("linux-underline", "verbose", ["-q", "-overbose", "ROW=0"], {"SYSTEMD_COLORS": "16", "TERM": "linux"}), + ("cat-fields", "cat-projection", ["-q", "-ocat", "--output-fields=ROW,MESSAGE", "--grep=hit", "ROW=4"], {"SYSTEMD_COLORS": "16"}), + ("truncated", "short", ["-q", "-oshort", "--truncate-newline", "--grep=hit"], {"SYSTEMD_COLORS": "16"}), + ("no-entries", "short", ["-oshort", "ROW=missing"], {"SYSTEMD_COLORS": "16"})] + for action in ("--list-boots", "--list-invocations"): + # The shared multi-boot fixture has invocation IDs as well. + if not (directory / "multi-boot-file.journal").is_file(): + continue + for mode in ("short", "json", "json-pretty", "json-seq", "json-sse"): + cases.append((action + "-" + mode, mode, [action, "-o", mode], {"SYSTEMD_COLORS": "16"})) + cases += [("boots-short", "short", ["-oshort"], {"SYSTEMD_COLORS": "16"}), + ("boots-reverse", "short", ["-oshort", "--grep=boot", "-r", "-n3"], {"SYSTEMD_COLORS": "16"}), + ("--list-boots-linux", "short", ["--list-boots"], {"SYSTEMD_COLORS": "16", "TERM": "linux"}), + ("links-utf8", "short", ["-q", "-oshort", "ROW=links"], {"SYSTEMD_COLORS": "16", "LC_ALL": "C.UTF-8"}), + ("links-plain", "short", ["-q", "-oshort", "ROW=links", "--grep=hit"], {"SYSTEMD_COLORS": "0", "SYSTEMD_URLIFY": "1"})] + logical_cwd = None + if os.name == "posix": + work = Path(tempfile.mkdtemp(prefix="v262-color-cwd-", dir=directory)) + real, logical_cwd = work / "real", work / "alias" + real.mkdir() + logical_cwd.symlink_to(real, target_is_directory=True) + cases.append(("links-logical-cwd", "short", ["-q", "-oshort", "ROW=relative-links"], + {"SYSTEMD_COLORS": "16", "PWD": str(logical_cwd) + "//"})) + clean_env = {key: value for key, value in env.items() if key not in ("NO_COLOR", "COLORTERM", "SYSTEMD_URLIFY")} + results = [] + for name, mode, args, changes in cases: + path = (directory / "multi-boot-file.journal" if name.startswith("--list-") + else boots if name.startswith("boots-") else source) + observed = {} + for reader, command in commands.items(): + observed[reader] = subprocess.run( # nosec B603 - fixed executables and synthetic input. + [command, "--file", str(path), "--no-pager", *args], + env={**clean_env, "TERM": "xterm", **changes}, capture_output=True, timeout=30, check=False, + cwd=logical_cwd if name == "links-logical-cwd" else None, + ) + expected = observed["stock"] + for reader in ("go", "rust"): + actual = observed[reader] + try: + same_output = normalized(actual.stdout, mode) == normalized(expected.stdout, mode) + except (ValueError, UnicodeError): + same_output = False + same = (actual.returncode == expected.returncode and same_output + and bool(actual.stderr) == bool(expected.stderr)) + results.append({"test": "v262-color-" + name, "reader": reader, "args": args, + "status": "PASS" if same else "FAIL", "returncode": actual.returncode, + "expected_returncode": expected.returncode, + "actual": actual.stdout.hex(), "expected": expected.stdout.hex(), + "stderr": actual.stderr.decode(errors="replace")[-1000:], + "expected_stderr": expected.stderr.decode(errors="replace")[-1000:]}) + if os.name == "posix": + results.extend(run_terminal_color_cases(commands, source, clean_env)) + return results + + +def run_terminal_color_cases(commands: dict[str, str], fixture: Path, + env: dict[str, str]) -> list[dict[str, object]]: + import errno + import pty + import termios + + results = [] + for both, mode, term in [(True, "auto-16", "xterm"), (False, "auto-16", "xterm"), + (True, "auto-16", ""), (True, "auto-16", None), + (True, "auto-16", "dumb"), (False, "true", "dumb")]: + terminal_env = {**env, "SYSTEMD_COLORS": mode, "SYSTEMD_URLIFY": "0"} + if term is None: + terminal_env.pop("TERM", None) + else: + terminal_env["TERM"] = term + observed = {} + for reader, command in commands.items(): + master, slave = pty.openpty() + output = bytearray() + with os.fdopen(master, "rb", buffering=0) as terminal: + try: + attributes = termios.tcgetattr(slave) + attributes[1] &= ~termios.ONLCR + termios.tcsetattr(slave, termios.TCSANOW, attributes) + # One bounded message fits before draining; no follow child is terminated. + result = subprocess.run( # nosec B603 - fixed binaries and synthetic input. + [command, "--file", str(fixture), "--no-pager", "-q", "-ocat", "ROW=4"], + env=terminal_env, stdout=slave, stderr=slave if both else subprocess.PIPE, + timeout=30, check=False, + ) + finally: + os.close(slave) + while True: + try: + chunk = terminal.read(4096) + except OSError as error: + if error.errno == errno.EIO: + break + raise + if not chunk: + break + output.extend(chunk) + observed[reader] = (result.returncode, bytes(output), result.stderr or b"") + expected = observed["stock"] + if expected[0] != 0 or expected[2] or not expected[1]: + raise AssertionError("native terminal color control failed") + for reader in ("go", "rust"): + actual = observed[reader] + results.append({"test": f"v262-color-terminal-{both}-{mode}-{term!r}", "reader": reader, + "status": "PASS" if actual == expected else "FAIL", + "returncode": actual[0], "expected_returncode": expected[0], + "actual": actual[1].hex(), "expected": expected[1].hex(), + "stderr": actual[2].decode(errors="replace"), + "expected_stderr": expected[2].decode(errors="replace")}) + return results diff --git a/tests/interoperability/journalctl_path_cases.py b/tests/interoperability/journalctl_path_cases.py new file mode 100644 index 00000000..1452c5a9 --- /dev/null +++ b/tests/interoperability/journalctl_path_cases.py @@ -0,0 +1,81 @@ +"""Native filenames and file-pattern controls for the pinned journalctl oracle.""" + +from __future__ import annotations + +import os +import shutil +import subprocess # nosec B404 +import tempfile +from pathlib import Path + +from go_fixture_writer import write_journal_file + + +def run_file_path_cases(commands: dict[str, str], source: Path, + env: dict[str, str]) -> list[dict[str, object]]: + if os.name != "posix": + return [] + work = Path(tempfile.mkdtemp(prefix="v262-native-paths-", dir=source.parent)) + patterns = work / "patterns" + patterns.mkdir() + names = [b"brace-a.journal", b"brace-b.journal", "unicode-é.journal".encode(), + b"raw-\xff.journal", b"raw-\xef.journal", "raw-�.journal".encode(), + b".hidden.journal", b"literal-[a].journal", b"literal-*.journal"] + for index, name in enumerate(names): + seed = work / f"seed-{index}.journal" + write_journal_file(seed, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_007_000_000_000 + index, + "monotonic_usec": index + 1, + "fields": [("MESSAGE", f"path-{index}")]}]) + shutil.copyfile(seed, patterns / os.fsdecode(name)) + parent = work / os.fsdecode(b"parent-\xff") + parent.mkdir() + shutil.copyfile(source, parent / "system.journal") + decoy = work / "parent-�" + decoy.mkdir() + shutil.copyfile(work / "seed-0.journal", decoy / "system.journal") + raw_directory = work / "raw-names" + raw_directory.mkdir() + for name in (b"raw-\xff.journal", b"raw-\xef.journal"): + shutil.copyfile(patterns / os.fsdecode(name), raw_directory / os.fsdecode(name)) + + cases = [ + ("literal-parent", ["--file", str(parent / "system.journal")], "C"), + ("directory-parent", ["--directory", str(parent)], "C"), + ("literal-byte-name", ["--file", str(raw_directory / os.fsdecode(b"raw-\xff.journal"))], "C"), + ("directory-byte-names", ["--directory", str(raw_directory)], "C"), + ("multiple-byte-names", ["--file", str(raw_directory / os.fsdecode(b"raw-\xff.journal")), + "--file", str(raw_directory / os.fsdecode(b"raw-\xef.journal"))], "C"), + ("header-byte-name", ["--file", str(raw_directory / os.fsdecode(b"raw-\xff.journal")), "--header"], "C"), + ] + for locale in ("C", "C.UTF-8"): + for pattern in ["brace-{a,b}.journal", "unicode-?.journal", "unicode-??.journal", + os.fsdecode(b"raw-[\xff].journal"), "raw-[�].journal", "raw-?.journal", + "raw-???.journal", "*.journal", ".*.journal", r"literal-\[a\].journal", + r"literal-\*.journal", "missing-*.journal", "{missing,brace-a}.journal", + "brace-{a,{b,missing}}.journal"]: + cases.append(("pattern-" + repr(pattern), ["--file", str(patterns / pattern)], locale)) + results = [] + for name, args, locale in cases: + observed = {} + for reader, command in commands.items(): + observed[reader] = subprocess.run( # nosec B603 - fixed binaries and private fixtures. + [command, *args, "--no-pager", "--boot=all", "--merge", "-q", "-ocat"], + env={**env, "LC_ALL": locale}, capture_output=True, timeout=30, check=False, + ) + expected = observed["stock"] + if not name.startswith("pattern-") and expected.returncode != 0: + raise AssertionError(f"native filename control {name} failed: {expected.stderr!r}") + for reader in ("go", "rust"): + actual = observed[reader] + same = (actual.returncode == expected.returncode and actual.stdout == expected.stdout + and bool(actual.stderr) == bool(expected.stderr)) + results.append({"test": "v262-path-" + name, "locale": locale, "reader": reader, + "status": "PASS" if same else "FAIL", "args": args, + "returncode": actual.returncode, "expected_returncode": expected.returncode, + "actual": actual.stdout.hex(), "expected": expected.stdout.hex(), + "stderr": actual.stderr.decode(errors="replace")[-1000:], + "expected_stderr": expected.stderr.decode(errors="replace")[-1000:]}) + return results diff --git a/tests/interoperability/journalctl_timestamp_cases.py b/tests/interoperability/journalctl_timestamp_cases.py new file mode 100644 index 00000000..c78ac97a --- /dev/null +++ b/tests/interoperability/journalctl_timestamp_cases.py @@ -0,0 +1,218 @@ +"""Timestamp grammar, bounds and calendar rendering for the v262 CLI target.""" + +from __future__ import annotations + +import json +import os +import subprocess # nosec B404 +import struct +from datetime import datetime, timezone +from pathlib import Path + +from go_fixture_writer import write_journal_file + + +def write_timestamp_zone(path, transitions, kinds, leaps=(), footer=None, standard=(), universal=()): + """Write a tiny valid TZif; all identities and clock rules are synthetic.""" + abbreviations = b"" + records = [] + for offset, dst, name in kinds: + records.append(struct.pack(">iBB", offset, dst, len(abbreviations))) + abbreviations += name.encode() + b"\0" + def block(width): + fmt = ">i" if width == 4 else ">q" + return (b"".join(struct.pack(fmt, t) for t, _ in transitions) + + bytes(k for _, k in transitions) + b"".join(records) + abbreviations + + b"".join(struct.pack(fmt, t) + struct.pack(">i", c) for t, c in leaps) + + bytes(standard) + bytes(universal)) + version = b"2" if footer is not None else b"\0" + header = struct.pack(">4sc15x6I", b"TZif", version, len(universal), len(standard), len(leaps), + len(transitions), len(kinds), len(abbreviations)) + data = header + block(4) + if footer is not None: + data += header + block(8) + b"\n" + footer.encode() + b"\n" + path.write_bytes(data) + return str(path.resolve()) + + +def run_timestamp_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[dict[str, object]]: + path = fixtures["file"].parent / "timestamp-bounds.journal" + # Place rows on either side of calendar, epoch-fraction, and DST boundaries. + # Compare complete returned rows/cursors; accepting a spelling is insufficient. + points = {0, 1, 999999, 1000000, 1000001, 1500000, 3600000000, 5400000000, + 253402214399000000, 253402214399000001} + for year, month, day, hour, minute, second in [ + (1970,1,1,0,0,0), (2000,1,1,0,0,0), (2012,9,22,16,34,22), + (2023,3,3,0,0,0), (2023,11,14,0,0,0), (2023,11,14,23,20,0), + (2023,11,14,23,20,50), (2024,3,10,6,30,0), (2024,3,10,7,30,0), + (2024,3,31,0,30,0), (2024,3,31,1,30,0), (2024,10,27,0,30,0), + (2024,10,27,1,30,0), (2024,11,3,5,30,0), (2024,11,3,6,30,0), + (2068,1,1,0,0,0), (datetime.now(timezone.utc).year,11,14,23,20,50), + ]: + base = int(datetime(year,month,day,hour,minute,second,tzinfo=timezone.utc).timestamp())*1_000_000 + points.update(base+d for d in [-1,0,1,123455,123456,123457,999999,1000000] if base+d >= 0) + for base in [1483228800, 1704110400]: + for offset in [-10800,-7200,-3600,0,3600,7200,10800]: + points.update((base+offset+second)*1_000_000 for second in [-2,-1,0,1,2]) + boot = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + write_journal_file(path, machine_id="00112233445566778899aabbccddeeff", boot_id=boot, + seqnum_id="12121212121212121212121212121212", entries=[ + {"realtime_usec": value, "monotonic_usec": i+1, + "fields": [("MESSAGE",str(value)),("_BOOT_ID",boot), + ("_SYSTEMD_UNIT","clock.service"), + ("_SYSTEMD_INVOCATION_ID","34343434343434343434343434343434")]} + for i,value in enumerate(sorted(value for value in points if 0 < value < (1 << 55)))]) + commands = {"stock": tools["stock_journalctl"], + "go": tools["go_journalctl"], "rust": tools["rust_journalctl"]} + check = subprocess.run( # nosec B603 - fixture validation with the selected oracle. + [commands["stock"], "--file", str(path), "--verify"], capture_output=True, timeout=30, check=False) + if check.returncode != 0: + raise AssertionError("stock rejected timestamp fixture: " + check.stderr.decode(errors="replace")) + spellings = [ + "@0", "@.5", "@1.0000009", "@1h30m", "@1 2", "@infinity", "@18446744073709.551614", + "2012-09-22 16:34:22", "2012-09-22T16:34:22", "2012-9-22T16:34:22.1234567Z", + "12-09-22 16:34:22", "12-09-22 16:34", "2012-09-22 16:34", "2012-09-22T16:34", + "12-09-22", "2012-09-22", "Nov 14 23:20:50", "November 14 23:20:50", + "00:00:00", "00:00", "today", "yesterday", "tomorrow", "now", "+0", "-0", "0 ago", "0 left", + "2023-11-14 00:00:00 UTC", "2023-11-14T00:00:00Z", "2023-11-14 00:00:00 +00", + "2023-11-14 00:00:00 +0000", "2023-11-14 00:00:00 +00:00", "2023-11-14T00:00:00.0000005Z", + "2023-11-14 00:00:00.9999999 UTC", "2023-11-14 00:00:60 UTC", "2023-11-14 00:00:61 UTC", + "2023-02-31", "Tue 2023-11-14 00:00:00 UTC", "tuesday 2023-11-14", "2023-11-14 00:00:00 Z", + "1970-01-01 00:00:00 +24:00", "1969-12-31 23:00:00 -06", "68-01-01", "69-12-31", + "2023-11-14 00:00:00 GMT", "2023-11-14 00:00:00 Etc/UTC", "2023-11-14 00:00:00 UCT", "2023-11-14 00:00:00 Universal", + "2023-11-14 00:00:00 Europe/Berlin", "2023-11-14 00:00:00 europe/berlin", + "2024-01-01 12:00:00 NZ", "2024-01-01 12:00:00 Europe//Berlin", + "2024-01-01 12:00:00 Europe/./Berlin", "2024-01-01 12:00:00 Europe/Berlin/", + "", "infinity", "@-0", "@18446744073710", "@9223372036854775808us", "@1secs", "@1msecs", + "@1H", "@1µs", "@1μs", "@0.1234567min", "@1.2.3", "@1.2 .3", "@1.2s.3", "@1.", + "2023-11-14T00:00:00. UTC", "2023-13-01", "2023-01-00", "2023-01-01 24:00:00", + "2023-01-01 00:60:00", "2023-01-01 00:00:62", "Mon 2023-11-14", "9999-12-31", + ] + cases = [("grammar-"+str(i), ["--since="+value], "UTC") for i,value in enumerate(spellings)] + for zone, fold, summer, winter in [ + ("Europe/Berlin", "2024-10-27 02:30:00", "2024-07-01 12:00:00", "2024-01-01 12:00:00"), + ("America/New_York", "2024-11-03 01:30:00", "2024-07-01 12:00:00", "2024-01-01 12:00:00"), + ]: + cases.extend([ + (zone+"-fresh-fold", ["--since="+fold], zone), + (zone+"-summer-seeded-fold", ["--since="+summer,"--since="+fold], zone), + (zone+"-winter-seeded-fold", ["--since="+winter,"--since="+fold], zone), + (zone+"-ordered-bounds", ["--until="+summer,"--since="+fold,"--until=@infinity"], zone), + ]) + cases.extend([ + ("berlin-gap", ["--since=2024-03-31 02:30:00"], "Europe/Berlin"), + ("new-york-gap", ["--since=2024-03-10 02:30:00"], "America/New_York"), + ("forced-standard-summer", ["--since=2024-07-01 12:00:00 CET"], "Europe/Berlin"), + ("forced-daylight-winter", ["--since=2024-01-01 12:00:00 CEST"], "Europe/Berlin"), + ("posix-rule-gap",["--since=2024-03-10 02:30:00"],"EST5EDT,M3.2.0/2,M11.1.0/2"), + ("posix-rule-forced-standard",["--since=2024-07-01 12:00:00 EST"],"EST5EDT,M3.2.0/2,M11.1.0/2"), + ("posix-rule-forced-daylight",["--since=2024-01-01 12:00:00 EDT"],"EST5EDT,M3.2.0/2,M11.1.0/2"), + ]) + leap = write_timestamp_zone(path.parent / "timestamp-leap.tzif", [], [(0,0,"UTC")], [(1483228800,1)]) + historical = write_timestamp_zone(path.parent / "timestamp-historical.tzif", + [(946684800,1),(1585443600,2),(1603587600,1),(1893456000,3)], + [(0,0,"OLD"),(3600,0,"NEW"),(7200,1,"DAY"),(10800,0,"FUT")]) + footer = write_timestamp_zone(path.parent / "timestamp-footer.tzif", [], + [(0,0,"OLD"),(3600,1,"DAY")], footer="XST-2XDT,M3.5.0/2,M10.5.0/3") + for suffix in ["", " UTC"]: + cases.append(("tzif-leap"+suffix, ["--since=2024-01-01 12:00:00"+suffix], leap)) + for stamp in ["2016-12-31 23:59:59 UTC", "2016-12-31 23:59:60 UTC", "2017-01-01 00:00:00 UTC"]: + cases.append(("tzif-leap-hit-"+stamp,["--since="+stamp],leap)) + for suffix in ["", " UTC", " FUT", " OLD", " DAY"]: + cases.append(("tzif-abbreviation"+suffix,["--since=2024-01-01 12:00:00"+suffix],historical)) + for suffix in [" FUT", " NEW", " DAY"]: + cases.append(("tzif-abbreviation-seeded"+suffix, + ["--since=2024-01-01 12:00:00","--since=2024-01-01 12:00:00"+suffix],historical)) + cases.extend([ + ("tzif-utc-preserves-names", ["--since=2024-01-01 12:00:00 UTC","--since=2024-01-01 12:00:00 FUT"],historical), + ("tzif-forced-dst-updates-names", ["--since=2024-01-01 12:00:00 DAY","--since=2024-01-01 12:00:00 DAY"],historical), + ("tzif-named-restores-fresh-pair", ["--since=2024-01-01 12:00:00","--since=2024-01-01 12:00:00 Etc/UTC","--since=2024-01-01 12:00:00 FUT"],historical), + ("tzif-no-transition-footer",["--since=2024-01-01 12:00:00"],footer), + ("tzif-no-transition-utc",["--since=2024-01-01 12:00:00 UTC"],footer), + ]) + for rule in ["ABC-99", "ABC-2junk", "-99:99:99DEF,M3.2.0,M11.1.0", + "XYZ5ABC", "ABC-2DEF", "ABC65536", "ABC0:65536"]: + cases.append(("tz-fallback-"+rule, ["--since=2023-11-15 00:00:00"], rule)) + results = [] + for name,args,tz in cases: + env = {**os.environ,"LC_ALL":"C","TZ":tz,"SYSTEMD_COLORS":"0"} + def execute(reader): + return subprocess.run( # nosec B603 - fixed binaries and generated argument vectors. + [commands[reader],"--file",str(path),"--no-pager","--output=json",*args], + capture_output=True,env=env,timeout=30,check=False) + expected=execute("stock") + def rows(data): + return [json.loads(line) for line in data.splitlines() if line.strip()] + for reader in ("go","rust"): + actual=execute(reader) + try: same=rows(actual.stdout)==rows(expected.stdout) + except (ValueError,UnicodeDecodeError):same=False + ok=actual.returncode==expected.returncode and same and bool(actual.stderr)==bool(expected.stderr) + results.append({"test":"timestamp-"+name,"reader":reader,"status":"PASS" if ok else "FAIL", + "args":args,"tz":tz,"expected_returncode":expected.returncode,"returncode":actual.returncode, + "expected_rows":len(expected.stdout.splitlines()),"actual_rows":len(actual.stdout.splitlines()), + "expected_stderr":expected.stderr.decode(errors="replace")[-600:], + "stderr":actual.stderr.decode(errors="replace")[-600:]}) + # Parsing and rendering must share TZ/TZDIR/TZif calendar semantics. + zone_dir = path.parent / "timestamp-zones" + zone_dir.mkdir(exist_ok=True) + write_timestamp_zone(zone_dir / "ReviewZone", [], [(-18000,0,"REV")]) + default_dir = path.parent / "timestamp-default-rules" + default_dir.mkdir(exist_ok=True) + write_timestamp_zone(default_dir / "posixrules", + [(1672531200,1),(1701388800,0),(1704067200,1),(1733011200,0)], + [(-18000,0,"STD"),(-14400,1,"DST")]) + flagged_dir = path.parent / "timestamp-flagged-rules" + flagged_dir.mkdir(exist_ok=True) + write_timestamp_zone(flagged_dir / "posixrules", + [(1672531200,1),(1701388800,0),(1704067200,1),(1733011200,0)], + [(-18000,0,"STD"),(-14400,1,"DST")], + standard=[1,1], universal=[0,1]) + rendering_zones = [ + ("posix", {"TZ":"EST5EDT,M3.2.0,M11.1.0"}), + ("fixed-posix", {"TZ":"ABC-02"}), + ("named", {"TZ":"America/New_York"}), + ("directory", {"TZ":"ReviewZone", "TZDIR":str(zone_dir.resolve())}), + ("leap", {"TZ":leap}), ("footer", {"TZ":footer}), + ("historical", {"TZ":historical}), + ("clamped-offset", {"TZ":"ABC-99"}), + ("invalid-name", {"TZ":"Missing/Zone"}), + ("empty-name", {"TZ":""}), + ("empty-setting", {"TZ":""}), + ("default-dst", {"TZ":"XYZ5ABC"}), + ("default-rebase", {"TZ":"ABC-2DEF"}), + ("default-file", {"TZ":"ABC-2DEF", "TZDIR":str(default_dir.resolve())}), + ("default-flags", {"TZ":"ABC-2DEF,", "TZDIR":str(flagged_dir.resolve())}), + ("default-missing", {"TZ":"XYZ5ABC", "TZDIR":str(zone_dir.resolve())}), + ("default-explicit", {"TZ":"XYZ5ABC,M3.2.0,M11.1.0", "TZDIR":str(default_dir.resolve())}), + ] + rendering_args = [["--output="+mode, "--no-hostname"] for mode in + ("short", "short-full", "short-iso", "short-iso-precise", + "short-precise", "short-unix", "verbose")] + rendering_args += [["--list-boots"], ["--header"], ["--list-invocations", "-uclock.service"], + ["--since=2024-03-30 20:30:00", "--output=cat"], + ["--since=2024-03-31 00:30:00 UTC", + "--since=2024-03-30 20:30:00", "--output=cat"]] + for name, settings in rendering_zones: + for args in rendering_args: + for utc in ([], ["--utc"]): + env = {**os.environ, "LC_ALL":"C", "SYSTEMD_COLORS":"0", + "SYSTEMD_URLIFY":"0", **settings} + def execute_render(reader): + return subprocess.run( # nosec B603 - fixed binaries and synthetic inputs. + [commands[reader], "--file", str(path), "--no-pager", "-q", + "--all", "--lines=all", *args, *utc], capture_output=True, + env=env, timeout=30, check=False) + expected = execute_render("stock") + for reader in ("go", "rust"): + actual = execute_render(reader) + ok = (actual.returncode == expected.returncode + and actual.stdout == expected.stdout + and bool(actual.stderr) == bool(expected.stderr)) + results.append({"test":"timestamp-render-"+name, "reader":reader, + "status":"PASS" if ok else "FAIL", "args":args+utc, + "expected_returncode":expected.returncode, + "returncode":actual.returncode, + "expected_output":expected.stdout.decode(errors="replace")[-600:] if not ok else "", + "output":actual.stdout.decode(errors="replace")[-600:] if not ok else ""}) + return results diff --git a/tests/interoperability/journalctl_unit_cases.py b/tests/interoperability/journalctl_unit_cases.py new file mode 100644 index 00000000..c1ea5b95 --- /dev/null +++ b/tests/interoperability/journalctl_unit_cases.py @@ -0,0 +1,133 @@ +"""Unit naming and invocation traversal controls for the v262 oracle matrix.""" + +from __future__ import annotations + +import os +from pathlib import Path + +from go_fixture_writer import write_journal_file + + +def unit_invocation_cases(directory: Path) -> list[tuple[str, Path, list[str], str]]: + boot = "a" * 32 + iid = "1" * 32 + uid = str(os.getuid()) if hasattr(os, "getuid") else "0" + cases = [] + + def fixture(name, rows): + path = directory / ("v262-unit-" + name + ".journal") + write_journal_file(path, machine_id="00112233445566778899aabbccddeeff", + boot_id=boot, seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_007_000_000_000 + (0 if name == "same-time" else i), + "monotonic_usec": i + 1, + "fields": [("MESSAGE", str(i)), ("_BOOT_ID", boot), *fields]} + for i, fields in enumerate(rows)]) + return path + + names = [ + (".service", ".service.service"), ("@instance.service", "@instance.service.service"), + ("a" * 256 + ".service", "a" * 256 + ".service.service"), + ("[[.a.]]lpha.*", "alpha.service"), ("[[]", "["), + ("alpha", "alpha.service"), ("alpha.*", "alpha.service"), + ("app-*", "app-alpha.slice"), ("[[:alpha:]]lpha.service", "alpha.service"), + ("[!b]lpha.service", "alpha.service"), ("[a-c]lpha.service", "alpha.service"), + ("alpha[", "alpha["), (r"alpha\x20name", r"alpha\x20name.service"), + ("/dev/sda", "dev-sda.device"), ("//sys//devices/./example", "sys-devices-example.device"), + ("/mnt/data", "mnt-data.mount"), ("/", "-.mount"), + ("/../mnt//data/.", "mnt-data.mount"), ("/.dot-dir", r"\x2edot\x2ddir.mount"), + ("/mnt/part-one", r"mnt-part\x2done.mount"), + ("/mnt/../data", "-mnt-..-data.service"), + ("alpha name", r"alpha\x20name.service"), (" alpha ", r"\x20alpha\x20.service"), + ("café", r"caf\xc3\xa9.service"), ("/café", r"caf\xc3\xa9.mount"), + ("@instance", "@instance.service"), ("alpha@.service", "alpha@.service"), + ("/waldo" + "a" * 246, "waldo" + "a" * 227 + "_7736d9ed33c2ec55.mount"), + ] + if os.name != "nt": + names += [("bad-\udcff", r"bad-\xff.service"), ("/bad-\udcff", r"bad\x2d\xff.mount")] + path = fixture("names", [[("_SYSTEMD_UNIT", target), ("_SYSTEMD_USER_UNIT", target), + ("_UID", uid), ("_SYSTEMD_INVOCATION_ID", iid)] for _, target in names]) + for i, (argument, _) in enumerate(names): + for option in ("--unit", "--user-unit"): + cases.append((f"unit-name-{i}-{option}", path, [option + "=" + argument, "-ocat"], "bytes")) + for argument in ["alpha name", "café", *(["bad-\udcff"] if os.name != "nt" else [])]: + for quiet in ([], ["-q"]): + cases.append(("unit-diagnostic-" + argument + str(quiet), path, + ["--unit=" + argument, "--utc", "--no-hostname", *quiet], "diagnostic-bytes")) + for option in ("--unit", "--user-unit"): + cases.append(("unit-empty-" + option, path, [option + "=", "-ocat"], "error:invalid argument")) + cases.append(("unit-empty-invocation-" + option, path, + [option + "=", "--list-invocations", "-ojson"], "error:invalid argument")) + cases.append(("unit-converted-user-dedup", path, + ["--user", "-ualpha", "-ualpha", "--list-invocations", "-ojson"], "json")) + cases.append(("unit-explicit-user-duplicates", path, + ["--user-unit=alpha", "--user-unit=alpha", "--list-invocations", "-ojson"], + "error:with multiple units")) + for scope, field, extra in [ + ("system-coredump", "COREDUMP_UNIT", [("MESSAGE_ID", "fc2e22bc6ee647b6b90729ab34a250b1")]), + ("system-slice", "_SYSTEMD_SLICE", []), + ("user-coredump", "COREDUMP_USER_UNIT", [("_UID", uid)]), + ("user-slice", "_SYSTEMD_USER_SLICE", [("_UID", uid)]), + ]: + unit = "crash-alpha.service" if "coredump" in scope else "app-alpha.slice" + path = fixture(scope, [[(field, unit), ("_SYSTEMD_INVOCATION_ID", iid), *extra]]) + option = "--user-unit=" if scope.startswith("user") else "--unit=" + cases.append(("unit-full-show-" + scope, path, [option + unit, "-ocat"], "bytes")) + for pattern in (unit, "crash-*" if "coredump" in scope else "*.slice"): + cases.append(("invocation-base-" + scope + pattern, path, + [option + pattern, "--list-invocations", "-ojson"], + "error:no matching unit" if "*" in pattern else "bytes")) + + bracket_invocations = fixture("bracket-invocations", [ + [("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_USER_UNIT", "alpha.service"), + ("_UID", uid), ("_SYSTEMD_INVOCATION_ID", digit * 32)] for digit in "12" + ]) + for option in ("--unit", "--user-unit"): + cases.append(("unit-bracket-invocations-" + option, bracket_invocations, + [option + "=[[:alpha:]]lpha.service", "--list-invocations", "-ojson"], "json")) + cases.append(("unit-bracket-select-" + option, bracket_invocations, + [option + "=[[.a.]]lpha.service", "--invocation=0", "-ocat"], "bytes")) + + raw_unit = fixture("raw-value", [[("_SYSTEMD_UNIT", b"bad-\xff.service")]]) + cases.append(("unit-wildcard-raw-value", raw_unit, ["--unit=bad-*", "-ocat"], "bytes")) + for name, rows in [ + ("no-id", [[("_SYSTEMD_UNIT", "alpha.service")] for _ in range(16)]), + ("same-time", [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", c * 32)] + for c in "312"]), + ("bounds", [[("_SYSTEMD_INVOCATION_ID", iid)], + [("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", iid), + ("SYSLOG_IDENTIFIER", "selected"), ("PRIORITY", "6"), ("SYSLOG_FACILITY", "3")], + [("_SYSTEMD_INVOCATION_ID", iid)]]), + ("precedence", [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", iid), + ("OBJECT_SYSTEMD_INVOCATION_ID", "2" * 32)], + [("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", "3" * 32)]]), + ("overlap", [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", value * 32)] + for value in "121343"]), + ("nested", [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", value * 32)] + for value in "12321"]), + ("broken-before-unit", [[("_SYSTEMD_UNIT", "other.service"), ("_SYSTEMD_INVOCATION_ID", "2" * 32)], + [("_SYSTEMD_UNIT", "alpha.service")], + [("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", "A" * 32)]]), + ("broken-after-unit", [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", "A" * 32)], + [("_SYSTEMD_UNIT", "alpha.service")], + [("_SYSTEMD_UNIT", "other.service"), ("_SYSTEMD_INVOCATION_ID", "2" * 32)]]), + ("invalid", [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", value), + ("OBJECT_SYSTEMD_INVOCATION_ID", iid)] for value in + ["0" * 32, "not-an-id", iid + "\0", iid + "\0tail", " " + iid, + "12345678-1234-1234-1234-123456789012", "A" * 32]]), + ]: + path = fixture(name, rows) + for args in ([], ["-n1"], ["-n+1"], ["-n2"], ["-n+2"], ["-n0"], ["-r"]): + cases.append(("invocation-" + name + str(args), path, + ["-ualpha", "--list-invocations", "-ojson", *args], "json")) + for descriptor in ("0", "1", "-1", iid, "2" * 32, "2" * 32 + "+1", "3" * 32 + "-1"): + cases.append(("invocation-select-" + name + descriptor, path, + ["-ualpha", "--invocation=" + descriptor, "-ocat"], "bytes")) + if name == "bounds": + for filters in (["-tselected"], ["-tabsent"], ["-p6"], ["-p3"], + ["--facility=3"], ["--facility=1"], ["--dmesg"]): + cases.append(("invocation-other-filter-" + str(filters), path, + ["--invocation=" + iid, "--boot=all", "-ocat", *filters], "bytes")) + for action, error in [("--list-invocations", "invalid argument"), ("-I", "no journal boot entry")]: + cases.append(("invocation-error-order-" + action, path, + [action, "--unit=", "--boot=" + "9" * 32], "error:" + error)) + return cases diff --git a/tests/interoperability/journalctl_v262_cases.py b/tests/interoperability/journalctl_v262_cases.py new file mode 100644 index 00000000..92e775bc --- /dev/null +++ b/tests/interoperability/journalctl_v262_cases.py @@ -0,0 +1,518 @@ +"""Semantic v262 controls shared by both journalctl implementations.""" + +from __future__ import annotations + +import json +import os +import re +import subprocess # nosec B404 +from pathlib import Path + +from go_fixture_writer import write_journal_file +from journalctl_path_cases import run_file_path_cases +from journalctl_color_cases import run_color_cases +from journalctl_unit_cases import unit_invocation_cases + + +def write_v262_fixture(binary: Path, values: list[bytes | None] | None = None) -> Path: + boot = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + repeated_message = values is None + if values is None: + values = [b"alpha alpha", b"first\nsecond", b"nul\x00tail", b"invalid\xffbyte", b"", None, "ÉABC".encode()] + entries = [] + for i, value in enumerate(values): + fields = [("ROW", str(i)), ("_BOOT_ID", boot)] + if value is not None: + fields.insert(0, ("MESSAGE", value)) + if i == 0 and repeated_message: + fields.append(("MESSAGE", b"only-second-match")) + entries.append({"realtime_usec": 1_700_005_000_000_000 + i, + "monotonic_usec": i + 1, "fields": fields}) + write_journal_file(binary, machine_id="00112233445566778899aabbccddeeff", + boot_id=boot, seqnum_id="12121212121212121212121212121212", entries=entries) + + return binary + + +def run_v262_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[dict[str, object]]: + """Compare status, complete bytes/JSON, and error classes with a pinned oracle.""" + env = {**os.environ, "LC_ALL": "C", "TZ": "UTC", "SYSTEMD_COLORS": "0", "COLUMNS": "80"} + commands = {"stock": tools["stock_journalctl"], + "go": tools["go_journalctl"], "rust": tools["rust_journalctl"]} + + def execute(reader: str, path: Path, args: list[str]): + checkpoint = path.parent / ("v262-checkpoint-" + reader) + has_checkpoint = "--cursor-file=@CHECKPOINT@" in args + if has_checkpoint: + checkpoint.write_bytes(b"") + args = ["--cursor-file=" + str(checkpoint) if arg == "--cursor-file=@CHECKPOINT@" else arg + for arg in args] + result = subprocess.run( # nosec B603 - fixed binaries and argument vectors. + [commands[reader], "--file", str(path), "--no-pager", *args], + env=env, capture_output=True, timeout=30, check=False, + ) + if has_checkpoint: + result.checkpoint = checkpoint.read_bytes() + return result + + binary = write_v262_fixture(fixtures["file"].parent / "v262-bytes.journal") + shortened = write_v262_fixture(fixtures["file"].parent / "v262-shortened.journal", [ + b"a" * 40 + b"\n" + b"b" * 40, b"a" * 140 + b"TAIL012345", + ("界" * 50 + "TAIL012345").encode(), b"one\ntwo\nthree\nfour", + ("e\u0301" * 70 + "TAIL012345").encode(), + b"trailing newline\n", + b"two trailing newlines\n\n", b"nul\0tail\nNEXT", + ]) + boots = fixtures["file"].parent / "v262-grep-boots.journal" + write_journal_file(boots, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000 + i, + "monotonic_usec": 1, "boot_id": ch * 32, + "fields": [("MESSAGE", message), ("_BOOT_ID", ch * 32)]} + for i, (ch, message) in enumerate(zip("abc", ["hit-a", "miss", "hit-c"]))]) + labels = fixtures["file"].parent / "v262-labels.journal" + prepared_values = [b"\x1b[31mred\x1b[0m", b"a\tb", b"line\r\r", b"a\r\nb", + b"a\rb", b"\x1b]8;;https://example.org\x07link\x1b]8;;\x1b\\", + b"\x1b[31", b"\x1b[2J", b"a\t\nsecond", b"one\r\nnext\r", b"nul\0tail"] + prepared = write_v262_fixture(fixtures["file"].parent / "v262-prepared.journal", prepared_values) + duplicate_binary = fixtures["file"].parent / "v262-duplicate-binary.journal" + write_journal_file(duplicate_binary, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000 + i, "monotonic_usec": i + 1, + "fields": [("MESSAGE", value) for value in values]} + for i, values in enumerate([[b"\xff", b"second"], [b"\xff", b"\xfe"], + [b"first", b"\xff"], [b"\xff", b"middle", b"last"]])]) + label_fields = [ + [("_EXE", "/bin/example")], + [("SYSLOG_IDENTIFIER", ""), ("_COMM", "worker")], + [("SYSLOG_IDENTIFIER", "first"), ("SYSLOG_IDENTIFIER", "second")], + [("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_USER_UNIT", "beta.service")], + [("SYSLOG_IDENTIFIER", b"bad\xff"), ("_COMM", "worker")], + [("SYSLOG_IDENTIFIER", "x" * 300), ("_COMM", "worker")], + ] + write_journal_file(labels, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000 + i, "monotonic_usec": i + 1, + "fields": [("MESSAGE", "line"), ("ROW", str(i)), *fields]} + for i, fields in enumerate(label_fields)]) + + source_cases = [ + ("duplicate", [b"1700000000000000", b"1700000001000000"]), + ("u64-max", [b"18446744073709551615"]), + ("limit-valid", [str((1 << 55) - 1).encode()]), + ("limit-invalid", [str(1 << 55).encode()]), + ("zero", [b"0"]), ("invalid-last", [b"1700000000000000", b"bad"]), + ("nul", [b"1700000000000000\0trailer"]), + ("hex", [b"0x60a24181e4000"]), ("padding", [b" +1700000000000000"]), + ] + source = fixtures["file"].parent / "v262-source-realtime.journal" + write_journal_file(source, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000 + i, + "monotonic_usec": 1_000_001 + i, + "fields": [("MESSAGE", "source-" + label), ("ROW", label), + *[("_SOURCE_REALTIME_TIMESTAMP", value) for value in values]]} + for i, (label, values) in enumerate(source_cases)]) + verified = execute("stock", source, ["--verify"]) + if verified.returncode != 0: + raise AssertionError("stock source-realtime fixture verification failed: " + + verified.stderr.decode(errors="replace")) + + # mode is exact bytes, parsed JSON (key order is unspecified), or an error class. + cases: list[tuple[str, Path, list[str], str]] = [] + cases.extend(unit_invocation_cases(fixtures["file"].parent)) + path = fixtures["file"] + unit_glob = path.parent / "v262-unit-glob.journal" + write_journal_file(unit_glob, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000, "monotonic_usec": 1, + "fields": [("MESSAGE", "unit-glob-marker"), ("_SYSTEMD_UNIT", "alpha.service"), + ("__JOURNALCTL_NEVER_MATCH", "unit-glob")]}]) + for name, args in [ + ("system-empty", ["--unit=no-such-*"]), + ("user-empty", ["--user-unit=no-such-*"]), + ("both-empty", ["--unit=no-such-*", "--user-unit=no-such-*"]), + ("zero-lines", ["--unit=no-such-*", "-n0"]), + ]: + cases.append(("unit-glob-" + name, unit_glob, args, + "error:failed to add filter for units: no data available")) + for name, args in [ + ("system-exact-missing", ["--unit=absent.service"]), + ("user-exact-missing", ["--user-unit=absent.service"]), + ("mixed-system-exact", ["--unit=absent.service", "--unit=no-such-*"]), + ("system-exact-user-empty", ["--unit=alpha.service", "--user-unit=no-such-*"]), + ("user-exact-system-empty", ["--user-unit=absent.service", "--unit=no-such-*"]), + ]: + cases.append(("unit-glob-" + name, unit_glob, [*args, "-ocat"], "bytes")) + for label, _ in source_cases: + for mode in ("short-iso-precise", "short-monotonic", "verbose"): + cases.append((f"source-realtime-{label}-{mode}", source, + ["--utc", "-q", "ROW=" + label, "-o", mode], "bytes")) + for i in range(len(prepared_values)): + for extra in ([], ["--all"], ["--truncate-newline"]): + cases.append((f"short-prepare-{i}-{extra}", prepared, + ["--utc", "-q", f"ROW={i}", *extra], "bytes")) + for mode in ("json", "json-pretty", "json-sse", "json-seq"): + cases.append(("duplicate-binary-" + mode, duplicate_binary, ["-o", mode], mode)) + for value, valid in [("1G512M", True), ("1.5G512M", True), ("10.M", True), + ("+1K", True), ("1 K 2 B", True), ("1K1K", False), + ("1M1G", False), ("1KiB", False), ("1g", False), + ("1KB", False), ("-0", False), (".5K", False), + ("18446744073709551615", True), ("18446744073709551616", False), + ("16E", False), ("15E", True), ("1B2", True)]: + cases.append(("vacuum-size-native-" + value, path, ["--vacuum-size=" + value, "--help"], + "terminal" if valid else "bytes")) + for grep in ("hit", "unmatched"): + for extra in ([], ["-r"], ["-n1"], ["-n+1"], ["-n0"], ["-n+0"], + ["--until=@1700005000.000001"], ["-r", "--since=@1700005000.000001"]): + cases.append((f"cursor-position-{grep}-{extra}", boots, + ["--grep=" + grep, "-ocat", "--show-cursor", *extra], "bytes")) + for extra in (["-n0"], ["-n+0"]): + cases.append(("cursor-zero-position-" + str(extra), path, + ["-ocat", "--show-cursor", *extra], "bytes")) + for extra in ([], ["-r"], ["-n0"], ["-n+0"], ["-n10", "--until=@1700005000.000001"]): + for grep in ([], ["--grep=hit"], ["--grep=unmatched"]): + cases.append((f"cursor-file-position-{extra}-{grep}", boots, + ["-ocat", "--cursor-file=@CHECKPOINT@", *grep, *extra], "checkpoint")) + for i in range(len(label_fields)): + for extra in ([], ["--all"]): + cases.append((f"short-label-{i}-{extra}", labels, + ["--utc", "-q", "-owith-unit" if i == 3 else "-oshort", f"ROW={i}", *extra], "bytes")) + for excluded in ("first", "second"): + cases.append((f"exclude-last-identifier-{excluded}", labels, + ["--utc", "-q", "-oshort", "ROW=2", "-T" + excluded], "bytes")) + for i in range(8): + cases.append((f"shortened-line-{i}", shortened, + ["--no-full", "--utc", "--no-hostname", "-q", f"ROW={i}"], "bytes")) + for extra in ([], ["-r"], ["-q"], ["--merge"], ["--grep=hit-a"]): + cases.append((f"grep-boot-events-{extra}", boots, + ["--boot=all", "--grep=hit", "--utc", "-oshort", *extra], "bytes")) + for value in ("ERROR", "panic", "critical", "error", "warn", "err", "warning", "0x3"): + cases.append((f"priority-native-{value}", path, [f"--priority={value}", "--help"], + "terminal" if value in ("err", "warning", "0x3") else "bytes")) + for value in ("24", "127", "128", "0x10"): + cases.append((f"facility-native-{value}", path, [f"--facility={value}", "--help"], + "bytes" if value == "128" else "terminal")) + for value in ("daemon, cron", " daemon", "daemon ", "daemon, 9", "daemon, help", + "daemon,,cron", "daemon,help"): + cases.append((f"facility-spacing-{value!r}", path, [f"--facility={value}", "--help"], + "terminal" if value in ("daemon, 9", "daemon,,cron", "daemon,help") else "bytes")) + for option, value, valid in [ + ("lines", "0x2", True), ("lines", "0b10", True), ("lines", "0o10", True), + ("lines", " 2", True), ("lines", "2 ", False), ("lines", "1_0", False), + ("boot", "0x1", True), ("boot", "2147483648", False), + ("boot", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa1", False), + ("boot", "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", False), + ("invocation", "2147483648", False), ("vacuum-files", "0x1", True), + ("vacuum-files", "-0", True), + ]: + cases.append((f"integer-{option}-{value!r}", path, [f"--{option}={value}", "--help"], + "terminal" if valid else "bytes")) + cases.extend([ + ("short-repeated-message", binary, ["--grep=alpha", "--utc", "-oshort"], "bytes"), + ("short-full-trailing-newline", shortened, ["ROW=5", "--utc", "-q"], "bytes"), + ("short-full-trailing-blank", shortened, ["ROW=6", "--utc", "-q"], "bytes"), + ("short-all-nul-lines", shortened, ["ROW=7", "--utc", "-q", "--all"], "bytes"), + ("lines-octal-count", fixtures["pager"], ["-n010", "-ocat"], "bytes"), + ("pager-explicit-all", fixtures["pager"], ["-e", "-nall", "-ocat"], "bytes"), + ("pager-grep-direction", fixtures["pager"], ["-e", "--grep=pager-100[234]", "-ocat"], "bytes"), + ("cursor-file-empty-path", path, ["--cursor-file=", "-ocat"], "bytes"), + ("cursor-file-empty-before-help", path, ["--cursor-file=", "--help"], "terminal"), + ("verify-quiet", path, ["--verify", "-q"], "bytes"), + ("verify-implied-quiet", path, ["--verify", "-ocat"], "bytes"), + ]) + for args in (["-F", "PRIORITY", "-n1"], ["-F", "PRIORITY", "-n0"], + ["-N", "-n0"], ["-F", "PRIORITY", "-N"], + ["-N", "-F", "PRIORITY"], ["--sync", "-N"], + ["--no-tail", "-n1", "-o", "cat"], + ["-qq", "-o", "json", "-o", "cat"], + ["--output-fields=PRIORITY", "--output-fields=MESSAGE", "-ojson"], + ["--output-fields=", "-ocat"], + ["--output-fields=MESSAGE", "--output-fields=", "-ocat"], + ["--system", "--user", "--unit=alpha.service", "-ocat"], + ["-p3", "-p6", "-ocat"], ["--grep", "-n1", "-ocat"]): + cases.append(("options-" + str(len(cases)), path, list(args), "json" if "-ojson" in args else "fields" if "-N" in args and ("-F" not in args or args.index("-N") > args.index("-F")) else "bytes")) + for mode in ("json", "json-pretty", "json-sse", "json-seq"): + for action in ("--list-boots", "--list-invocations"): + extra = ["--unit=alpha.service"] if action == "--list-invocations" else [] + cases.append((f"table-{action}-{mode}", path, [action, *extra, "-o", mode], mode)) + for action in ("--list-boots", "--list-invocations"): + extra = ["--unit=alpha.service"] if action == "--list-invocations" else [] + for output in ([], ["-q"], ["-ojson"]): + cases.append((f"table-zero-{action}-{output}", path, + [action, *extra, "-n0", *output], "bytes")) + for option, value in (("case-sensitive", "true"), ("synchronize-on-exit", "false")): + for padded in (" " + value, value + " "): + cases.append((f"boolean-space-{option}-{padded!r}", path, + [f"--{option}={padded}", "--help"], "bytes")) + for field, limit in (("", []), ("", ["-n0"]), (" MESSAGE ", []), ("__CURSOR", [])): + cases.append((f"invalid-field-{field!r}-{limit}", path, + ["-F", field, *limit], "bytes")) + for args, error in [ + (["-F", "PRIORITY", "-S", "@0"], "cannot be combined"), + (["-N", "-U", "@infinity"], "cannot be combined"), + (["-N", "--grep="], "cannot be combined"), + (["-N", "--cursor="], "cannot be combined"), + (["-S", "", "--help"], "Failed to parse timestamp"), + (["-pBAD", "--help"], "Unknown log level"), + (["-oBAD", "-ocat"], "Unknown output format"), + (["--case-sensitive=TRUE", "--case-sensitive="], "Bad --case-sensitive"), + (["--quiet=true"], "doesn't allow an argument"), + (["+", "MESSAGE=file-a"], "can only be used between terms"), + (["MESSAGE=file-a", "+"], "can only be used between terms"), + (["--grep=(", "-N"], "cannot be combined"), + (["--grep=(", "--list-boots"], "pattern"), + ]: + cases.append(("error-" + str(len(cases)), path, args, "error:" + error)) + for args in (["--help", "-pBAD"], ["-ohelp", "--bad-option"], + ["--facility=help", "--bad-option"], ["--grep=(", "--grep=file", "-ocat"], + ["-N", "--boot=all"], ["-N", "--invocation=all"], + ["--since=2023-11-14 00:00:00 UTC", "-ocat"]): + # Help text has implementation-specific descriptions; status and later-input + # rejection are the contract, while the shared output-mode/facility lists are exact. + mode = "terminal" if args[0] == "--help" else "fields" if "-N" in args else "bytes" + cases.append(("order-" + str(len(cases)), path, list(args), mode)) + for pattern in (r"(?=alpha)alpha", r"(alpha) \1", r"nul\x00tail", r"\xff", "Éabc", "", + "only-second-match", "unmatched-pattern", "(*UTF)."): + cases.append(("grep-" + str(len(cases)), binary, ["--grep=" + pattern, "-ocat"], "bytes")) + if os.name != "nt": + for option in ("root", "image"): + cases.append(("raw-source-presence-" + option, path, + ["--" + option + "=\udcff", "-ocat"], "bytes")) + cases.append(("grep-raw-argv", binary, ["--grep=\udcff", "-ocat"], "bytes")) + cases.append(("match-raw-argv", binary, ["MESSAGE=invalid\udcffbyte", "-ocat"], "bytes")) + cases.append(("match-raw-disjunction", binary, + ["ROW=0", "+", "MESSAGE=invalid\udcffbyte", "-ocat"], "bytes")) + for args in (["--identifier=bad\udcff", "-ocat"], + ["--exclude-identifier=bad\udcff", "-oshort", "ROW=4"], + ["--output-fields=MES\udcff", "-ojson", "ROW=4"], + ["--output-fields=MES\udcff", "-ocat", "ROW=4"]): + cases.append(("opaque-raw-" + str(len(cases)), labels, args, + "json" if "-ojson" in args else "bytes")) + for output_mode in ("json", "json-pretty", "json-sse", "json-seq"): + cases.append(("introspection-"+output_mode, path, ["-o",output_mode,"--introspect-cli","--bad-option"], "introspection:"+output_mode)) + cases.extend([ + ("quiet-sticky", path, ["-ojson","-oshort","--grep=unmatched-pattern"], "bytes"), + ("binary-cat", binary, ["-ocat"], "bytes"), + ("binary-cat-truncate", binary, ["-ocat", "--truncate-newline"], "bytes"), + ("grep-no-lines", binary, ["--grep=alpha", "-n0", "-ocat"], "bytes"), + ("grep-head-stops-at-limit", binary, ["--grep=(*UTF).", "-n+3", "-ocat"], "bytes"), + ("grep-head-zero-skips-match", binary, + ["--grep=(*UTF).", "-n+0", "--since=@1700005000.000003", "-ocat"], "bytes"), + ("grep-no-match-short", binary, ["--grep=unmatched-pattern"], "bytes"), + ("grep-render-exclusion", path, ["--grep=file", "-Tapp-a", "-Tapp-b", "-nall"], "bytes"), + ("projection-spaces-json", binary, ["-ojson", "--output-fields= MESSAGE ", "-n1"], "json"), + ("projection-spaces-cat", binary, ["-ocat", "--output-fields= MESSAGE "], "bytes"), + ("projection-invalid-no-rows", binary, ["-ocat", "--output-fields= MESSAGE ", "-n0"], "bytes"), + ("lines-oldest-zero", path, ["-n+0"], "bytes"), + ("lines-negative-zero", path, ["-n-0"], "bytes"), + ("cursor-empty", path, ["--cursor=", "-n1"], "bytes"), + ("after-cursor-empty", path, ["--after-cursor=", "-n1"], "bytes"), + ("directory-empty-source-conflict", path, ["--directory=", "--new-id128"], "bytes"), + ("machine-empty-source-conflict", path, ["--machine=", "--new-id128"], "bytes"), + ("root-empty-clear", path, ["--root=", "-n0"], "bytes"), + ("image-empty-clear", path, ["--image=", "-n0"], "bytes"), + ("namespace-empty-clear", path, ["--namespace=", "-n0"], "bytes"), + ("vacuum-action-replaced", path, ["--vacuum-size=1", "--new-id128"], "utility-status"), + ("vacuum-zero-action-replaced", path, ["--vacuum-time=0", "--new-id128"], "utility-status"), + ]) + probe = execute("stock", path, ["-ojson"]) + if probe.returncode != 0: + raise AssertionError("stock cursor probe failed: " + probe.stderr.decode(errors="replace")) + rows = [json.loads(row) for row in probe.stdout.splitlines()] + for reverse, row in ((False, rows[-1]), (True, rows[0])): + for extra in ([], ["-n0"]): + cases.append((f"cursor-after-terminal-{reverse}-{extra}", path, + ["--after-cursor=" + row["__CURSOR"], "--show-cursor", "-ocat", + *(["-r"] if reverse else []), *extra], "bytes")) + for cursor_option in ("--cursor", "--after-cursor"): + for reverse in (False, True): + extra = ["--reverse"] if reverse else [] + cases.append((f"{cursor_option}-one-{reverse}", path, + [cursor_option, rows[1]["__CURSOR"], "-n1", "-ojson", *extra], "json")) + directional = fixtures["file"].parent / "v262-directional.journal" + write_journal_file(directional, machine_id="00112233445566778899aabbccddeeff", + boot_id="a" * 32, seqnum_id="12" * 16, + entries=[{"realtime_usec": 1700005000000000 + i // 2, + "monotonic_usec": i % 2 + 1, "boot_id": "abc"[i // 2] * 32, + "fields": [("MESSAGE", "keep" if i % 2 else "other"), + ("ROW", str(i)), ("_BOOT_ID", "abc"[i // 2] * 32)]} + for i in range(6)]) + directional_probe = execute("stock", directional, ["-ojson"]) + if directional_probe.returncode: + raise AssertionError("directional cursor oracle fixture failed") + cursor_rows = [json.loads(line) for line in directional_probe.stdout.splitlines()] + anchors = [] + for row in cursor_rows: + parts = dict(part.split("=", 1) for part in row["__CURSOR"].split(";")) + anchors.extend([row["__CURSOR"], "t=" + parts["t"], + "s=" + parts["s"] + ";i=" + parts["i"], + "b=" + parts["b"] + ";m=" + parts["m"]]) + anchors.extend(["s=" + "f" * 32 + ";i=3;m=0", "b=" + "f" * 32 + ";m=0", + "s=" + "12" * 16 + ";i=ffffffff", "b=" + "b" * 32 + ";m=0", + "b=" + "b" * 32 + ";m=ffffffff", + "s=" + "12" * 16 + ";i=1;t=ffffffff"]) + for index, anchor in enumerate(anchors): + for reverse in (False, True): + for option in ("--cursor", "--after-cursor"): + for filtered in (False, True): + cases.append((f"directional-{index}-{reverse}-{option}-{filtered}", directional, + [option + "=" + anchor, "-ojson", *(["-r"] if reverse else []), + *(["MESSAGE=keep"] if filtered else [])], "json")) + results = [] + for name, fixture, args, mode in cases: + expected = execute("stock", fixture, args) + if mode.startswith("error:"): + needle = mode.split(":", 1)[1].lower() + if expected.returncode == 0 or needle not in expected.stderr.decode(errors="replace").lower(): + raise AssertionError(f"stock {name} did not produce {needle!r}: {expected}") + for reader in ("go", "rust"): + actual = execute(reader, fixture, args) + if mode.startswith("error:"): + same = (mode.split(":", 1)[1].lower() in actual.stderr.decode(errors="replace").lower() + and actual.stdout == expected.stdout) + elif mode.startswith("introspection:"): + frame = mode.split(":",1)[1] + a = parse_json(frame, actual.stdout) + e = parse_json(frame, expected.stdout) + if isinstance(a,list) and isinstance(e,list): + a = a[0] if len(a)==1 else None + e = e[0] if len(e)==1 else None + def surface(value): + if not isinstance(value,dict) or value.get("mediaType") != "application/vnd.io.systemd.cli-introspection-0": + return None + commands = value.get("commands",[]) + if len(commands)!=1 or commands[0].get("names") != ["journalctl"]: + return None + command = commands[0] + if not command.get("project") or not command.get("version") or not isinstance(command.get("features"),list): + return None + return [{k:opt[k] for k in ("names","argument","metavar","group") if k in opt} + for opt in command.get("options",[])] + same = surface(a) is not None and surface(a) == surface(e) + elif mode == "diagnostic-bytes": + same = actual.stdout == expected.stdout and actual.stderr == expected.stderr + elif mode == "fields": + same = sorted(actual.stdout.splitlines()) == sorted(expected.stdout.splitlines()) + elif mode == "terminal": + same = bool(actual.stdout) and not actual.stderr + elif mode == "utility-status": + same = bool(actual.stdout) == bool(expected.stdout) + elif mode == "checkpoint": + same = actual.stdout == expected.stdout and actual.checkpoint == expected.checkpoint + elif mode.startswith("json"): + same = parse_json(mode, actual.stdout) == parse_json(mode, expected.stdout) + if mode == "json-pretty": + separators = rb'"(?:\\.|[^"\\])*"([ \t]*:[ \t]*)' + same = same and re.findall(separators, actual.stdout) == re.findall(separators, expected.stdout) + else: + same = actual.stdout == expected.stdout + # A nonzero grep result must preserve empty stderr; actual match errors + # must remain errors rather than being mistaken for a no-match result. + stderr_ok = bool(actual.stderr) == bool(expected.stderr) + ok = actual.returncode == expected.returncode and same and stderr_ok + results.append({"test": "v262-" + name, "reader": reader, + "status": "PASS" if ok else "FAIL", "args": args, + "expected_returncode": expected.returncode, "returncode": actual.returncode, + "expected": expected.stdout.hex(), "actual": actual.stdout.hex(), + "expected_stderr": expected.stderr.decode(errors="replace")[-1000:], + "stderr": actual.stderr.decode(errors="replace")[-1000:]}) + if mode == "checkpoint": + results[-1].update(checkpoint=actual.checkpoint.hex(), expected_checkpoint=expected.checkpoint.hex()) + if os.name != "nt": + args = ["--namespace=\udcff", "-ocat"] + native = execute("stock", path, args) + if native.returncode != 0: + raise AssertionError("native namespace control with explicit file changed") + for reader in ("go", "rust"): + actual = execute(reader, path, args) + diagnostic = actual.stderr.decode(errors="replace") + ok = actual.returncode == 1 and not actual.stdout and "portable mode does not support --namespace" in diagnostic + results.append({"test": "v262-raw-namespace-portable-qualification", "reader": reader, + "status": "PASS" if ok else "FAIL", "args": args, + "contract": "explicit portable unsupported; native ignores namespace with --file", + "native_returncode": native.returncode, "expected_returncode": 1, + "returncode": actual.returncode, "stderr": diagnostic}) + if os.name == "posix": + results.extend(run_terminal_width_cases(commands, shortened, env)) + results.extend(run_file_path_cases(commands, path, env)) + results.extend(run_color_cases(commands, path.parent, env)) + return results + + +def run_terminal_width_cases(commands: dict[str, str], fixture: Path, + env: dict[str, str]) -> list[dict[str, object]]: + """A real stdout terminal must supply the width when COLUMNS is unusable.""" + import errno + import fcntl + import pty + import struct + import termios + + results = [] + for columns in (None, "invalid", "100", "65536"): + terminal_env = {**env} + if columns is None: + terminal_env.pop("COLUMNS", None) + else: + terminal_env["COLUMNS"] = columns + observed = {} + for reader, command in commands.items(): + master, slave = pty.openpty() + output = bytearray() + with os.fdopen(master, "rb", buffering=0) as terminal: + try: + fcntl.ioctl(slave, termios.TIOCSWINSZ, struct.pack("HHHH", 24, 120, 0, 0)) + attrs = termios.tcgetattr(slave) + attrs[1] &= ~termios.ONLCR + termios.tcsetattr(slave, termios.TCSANOW, attrs) + # One bounded row fits in the PTY buffer before it is drained. + result = subprocess.run( # nosec B603 - fixed binaries and fixture arguments. + [command, "--file", str(fixture), "--no-pager", "--utc", "--no-hostname", + "--no-full", "-q", "ROW=1"], env=terminal_env, + stdout=slave, stderr=subprocess.PIPE, timeout=30, check=False, + ) + finally: + os.close(slave) + while True: + try: + chunk = terminal.read(4096) + except OSError as err: + if err.errno == errno.EIO: + break + raise + if not chunk: + break + output.extend(chunk) + observed[reader] = (result.returncode, bytes(output), result.stderr) + expected = observed["stock"] + if expected[0] != 0 or expected[2] or not expected[1]: + raise AssertionError(f"stock terminal-width control failed: {expected!r}") + for reader in ("go", "rust"): + actual = observed[reader] + results.append({"test": "v262-tty-columns-" + str(columns), "reader": reader, + "status": "PASS" if actual == expected else "FAIL", + "returncode": actual[0], "expected_returncode": expected[0], + "actual": actual[1].hex(), "expected": expected[1].hex(), + "stderr": actual[2].decode(errors="replace"), + "expected_stderr": expected[2].decode(errors="replace")}) + return results + + +def parse_json(mode: str, data: bytes): + try: + if mode == "json-sse": + return [json.loads(line[6:]) for line in data.splitlines() if line.startswith(b"data: ")] + if mode == "json-seq": + return [json.loads(part) for part in data.split(b"\x1e") if part.strip()] + if mode == "json-pretty": + decoder = json.JSONDecoder() + text, values = data.decode().lstrip(), [] + while text: + value, end = decoder.raw_decode(text) + values.append(value) + text = text[end:].lstrip() + return values + return [json.loads(line) for line in data.splitlines() if line.strip()] + except (ValueError, UnicodeDecodeError): + return {"invalid_json": data.hex()} diff --git a/tests/interoperability/journalctl_vacuum_cases.py b/tests/interoperability/journalctl_vacuum_cases.py new file mode 100644 index 00000000..493fe2a7 --- /dev/null +++ b/tests/interoperability/journalctl_vacuum_cases.py @@ -0,0 +1,103 @@ +"""Approved deterministic archive ordering for clock-rollback histories.""" +from __future__ import annotations + +import os +import shutil +import subprocess # nosec B404 +import time +from pathlib import Path + +from go_fixture_writer import write_journal_file + +# All 21 originals were verified by the pinned v262 oracle. +ARCHIVES = [(2, 1, 1600000000000776), + (2, 2, 1600000000000041), + (2, 3, 1600000000000988), + (3, 4, 1600000000000497), + (2, 5, 1600000000000940), + (2, 6, 1600000000000991), + (2, 7, 1600000000000366), + (3, 8, 1600000000000913), + (1, 9, 1600000000000516), + (1, 10, 1600000000000288), + (1, 11, 1600000000000773), + (1, 12, 1600000000000633), + (2, 13, 1600000000000931), + (3, 14, 1600000000000722), + (3, 15, 1600000000000923), + (1, 16, 1600000000000317), + (1, 17, 1600000000000747), + (1, 18, 1600000000000920), + (3, 19, 1600000000000338), + (2, 20, 1600000000000573), + (1, 21, 1600000000000362)] + + +def run_vacuum_cases(binary: Path, workspace: Path) -> list[bool]: + source = workspace / "vacuum-source" + source.mkdir() + rows = [] + for group, sequence, realtime in ARCHIVES: + name = f"system@{group:032x}-{sequence:016x}-{realtime:016x}.journal" + write_journal_file(source / name, machine_id="a" * 32, boot_id="b" * 32, + seqnum_id=f"{group:032x}", head_seqnum=sequence, + entries=[{"realtime_usec": realtime, "monotonic_usec": sequence, + "fields": [("MESSAGE", "archive rollback")]}]) + rows.append((name, group, sequence, realtime)) + # Independent specification: a file is eligible only after every earlier + # sequence from its history has been removed from the ordering model. + remaining = rows.copy() + order = [] + while remaining: + available = [r for r in remaining if not any( + other[1] == r[1] and (other[2], other[0]) < (r[2], r[0]) + for other in remaining)] + chosen = min(available, key=lambda r: (r[3], r[1], r[0])) + order.append(chosen[0]) + remaining.remove(chosen) + results = [] + cases = [("count-two", ["--vacuum-files=2"], set(order[-2:])), + ("count-ten", ["--vacuum-files=10"], set(order[-10:])), + ("age", ["--vacuum-time=1d"], set()), + ("size", ["--vacuum-size=1"], set()), + ("combined", ["--vacuum-files=2", "--vacuum-time=1d"], set())] + for reverse in (False, True): + for label, args, expected in cases: + directory = workspace / f"vacuum-{label}-{reverse}" + directory.mkdir() + for name, *_ in (reversed(rows) if reverse else rows): + shutil.copyfile(source / name, directory / name) + result = subprocess.run( # nosec B603 - explicit test binary and private fixture directory. + [str(binary), "--directory", str(directory), "--quiet", *args], + capture_output=True, timeout=30, check=False, + env={**os.environ, "LC_ALL": "C", "TZ": "UTC"}) + observed = {p.name for p in directory.iterdir()} + passed = result.returncode == 0 and observed == expected + results.append(passed) + print("PASS" if passed else "FAIL", "vacuum", label, "reverse", reverse) + if not passed: + print("status", result.returncode, "stderr", result.stderr, + "expected", sorted(expected), "observed", sorted(observed)) + # An old successor cannot bypass its recent earlier sequence for age-only vacuum. + recent = int(time.time() * 1_000_000) + old = recent - 3 * 86_400_000_000 + for reverse in (False, True): + directory = workspace / f"vacuum-age-sequence-{reverse}" + directory.mkdir() + entries = [(1, 1, recent), (1, 2, old), (2, 1, old)] + keep = set() + for group, sequence, realtime in (reversed(entries) if reverse else entries): + name = f"system@{group:032x}-{sequence:016x}-{realtime:016x}.journal" + write_journal_file(directory / name, machine_id="a" * 32, boot_id="b" * 32, + seqnum_id=f"{group:032x}", head_seqnum=sequence, + entries=[{"realtime_usec": realtime, "monotonic_usec": sequence, + "fields": [("MESSAGE", "age sequence")]}]) + if group == 1: + keep.add(name) + result = subprocess.run( # nosec B603 - explicit test binary and private fixture directory. + [str(binary), "--directory", str(directory), "--quiet", "--vacuum-time=1d"], + capture_output=True, timeout=30, check=False) + passed = result.returncode == 0 and {p.name for p in directory.iterdir()} == keep + results.append(passed) + print("PASS" if passed else "FAIL", "vacuum age sequence", reverse) + return results diff --git a/tests/interoperability/run_binary_matrix.py b/tests/interoperability/run_binary_matrix.py index ca247aa8..531fe30b 100644 --- a/tests/interoperability/run_binary_matrix.py +++ b/tests/interoperability/run_binary_matrix.py @@ -19,6 +19,7 @@ import argparse import json +import os import shutil import subprocess # nosec B404 import sys @@ -62,12 +63,13 @@ class ReaderSpec: } -def run(cmd: list[str], *, cwd: Path = REPO_ROOT, timeout: int = 120, binary: bool = False) -> subprocess.CompletedProcess: +def run(cmd: list[str], *, cwd: Path = REPO_ROOT, timeout: int = 120, binary: bool = False, env: dict[str, str] | None = None) -> subprocess.CompletedProcess: # nosemgrep # subprocess is required by this harness; commands are shell=False vectors. return subprocess.run( # nosec B603 cmd, # nosemgrep cwd=str(cwd), + env=env, text=None if binary else True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, @@ -102,7 +104,7 @@ def build_tools() -> dict[str, str]: "build go livewriter", ) require_ok( - run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go"), + run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env={**os.environ, "CGO_ENABLED": "1"}), "build go journalctl", ) require_ok( @@ -114,12 +116,14 @@ def build_tools() -> dict[str, str]: "build rust journalctl", ) + cargo_target = (REPO_ROOT / os.environ.get("CARGO_TARGET_DIR", "rust/target")).resolve() for src, dst in [ - (REPO_ROOT / "rust/target/debug/livewriter", BIN_DIR / "rust-livewriter"), - (REPO_ROOT / "rust/target/debug/journalctl", BIN_DIR / "rust-journalctl"), + (cargo_target / "debug/livewriter", BIN_DIR / "rust-livewriter"), + (cargo_target / "debug/journalctl", BIN_DIR / "rust-journalctl"), ]: - if src.exists(): - shutil.copy2(src, dst) + if not src.is_file(): + raise RuntimeError(f"Cargo did not produce expected binary: {src}") + shutil.copy2(src, dst) for name in ["go-livewriter", "go-journalctl", "rust-livewriter", "rust-journalctl"]: path = BIN_DIR / name diff --git a/tests/interoperability/run_compact_matrix.py b/tests/interoperability/run_compact_matrix.py index 8d48c345..d8feac94 100644 --- a/tests/interoperability/run_compact_matrix.py +++ b/tests/interoperability/run_compact_matrix.py @@ -73,6 +73,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -127,7 +128,7 @@ def build_tools() -> dict[str, str]: run( ["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", - env=env, + env={**env, "CGO_ENABLED": "1"}, ), "build go journalctl", ) @@ -153,8 +154,9 @@ def build_tools() -> dict[str, str]: (cargo_target / "debug" / "livewriter", BIN_DIR / "rust-livewriter"), (cargo_target / "debug" / "journalctl", BIN_DIR / "rust-journalctl"), ]: - if src.exists(): - shutil.copy2(src, dst) + if not src.is_file(): + raise RuntimeError(f"expected Rust build output not found: {src}") + shutil.copy2(src, dst) for name in ["go-livewriter", "go-journalctl", "rust-livewriter", "rust-journalctl"]: path = BIN_DIR / name diff --git a/tests/interoperability/run_compression_matrix.py b/tests/interoperability/run_compression_matrix.py index 7613566d..39e3ff8c 100644 --- a/tests/interoperability/run_compression_matrix.py +++ b/tests/interoperability/run_compression_matrix.py @@ -115,6 +115,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -136,7 +137,7 @@ def build_tools() -> dict[str, str]: run( ["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", - env=env, + env={**env, "CGO_ENABLED": "1"}, ), "build go journalctl", ) @@ -162,8 +163,9 @@ def build_tools() -> dict[str, str]: (cargo_target / "debug" / "livewriter", BIN_DIR / "rust-livewriter"), (cargo_target / "debug" / "journalctl", BIN_DIR / "rust-journalctl"), ]: - if src.exists(): - shutil.copy2(src, dst) + if not src.is_file(): + raise RuntimeError(f"expected Rust build output not found: {src}") + shutil.copy2(src, dst) for name in ["go-livewriter", "go-journalctl", "rust-livewriter", "rust-journalctl"]: path = BIN_DIR / name diff --git a/tests/interoperability/run_directory_matrix.py b/tests/interoperability/run_directory_matrix.py index da2d5eca..a9bf2ad1 100644 --- a/tests/interoperability/run_directory_matrix.py +++ b/tests/interoperability/run_directory_matrix.py @@ -93,6 +93,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -102,7 +103,7 @@ def build_tools() -> dict[str, str]: go_journalctl = BIN_DIR / "go-journalctl" require_ok( - run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env=env), + run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env={**env, "CGO_ENABLED": "1"}), "build go journalctl", ) require_ok( @@ -116,8 +117,9 @@ def build_tools() -> dict[str, str]: rust_src = Path(env["CARGO_TARGET_DIR"]) / "debug" / "journalctl" rust_journalctl = BIN_DIR / "rust-journalctl" - if rust_src.exists(): - shutil.copy2(rust_src, rust_journalctl) + if not rust_src.is_file(): + raise RuntimeError(f"expected Rust build output not found: {rust_src}") + shutil.copy2(rust_src, rust_journalctl) for path in (go_journalctl, rust_journalctl): if not path.exists(): diff --git a/tests/interoperability/run_journalctl_portable_smoke.py b/tests/interoperability/run_journalctl_portable_smoke.py new file mode 100644 index 00000000..c8343333 --- /dev/null +++ b/tests/interoperability/run_journalctl_portable_smoke.py @@ -0,0 +1,209 @@ +#!/usr/bin/env python3 +"""Replay a small v262-qualified CLI contract on native non-systemd platforms.""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess # nosec B404 +import tempfile +from pathlib import Path + +import go_fixture_writer +from journalctl_v262_cases import write_v262_fixture +from journalctl_unit_cases import unit_invocation_cases +from journalctl_timestamp_cases import write_timestamp_zone +from go_fixture_writer import write_journal_file +from journalctl_vacuum_cases import run_vacuum_cases + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--go", type=Path) + parser.add_argument("--rust", type=Path) + args = parser.parse_args() + commands = [(name, binary) for name, binary in [("go", args.go), ("rust", args.rust)] if binary] + if not commands: + parser.error("provide --go, --rust, or both") + root = Path(__file__).resolve().parents[2] + scratch = root / ".local" / "interoperability" + scratch.mkdir(parents=True, exist_ok=True) + workspace = Path(tempfile.mkdtemp(prefix="portable-smoke-", dir=scratch)) + go_fixture_writer.BUILDER_DIR = workspace / "builder" + go_fixture_writer.SPEC_DIR = workspace / "specs" + fixture = write_v262_fixture(workspace / "fixture.journal") + paths = Path(tempfile.mkdtemp(prefix="portable-paths-", dir=fixture.parent)) + unit_directory = paths / "units" + unit_directory.mkdir() + unit_cases = {name: (path, options, mode) + for name, path, options, mode in unit_invocation_cases(unit_directory)} + # Expected results are qualified against the pinned v262 oracle matrix. + def invocation(index, digit, first, last): + return {"index": index, "invocation_id": digit * 32, + "first_entry": 1_700_007_000_000_000 + first, + "last_entry": 1_700_007_000_000_000 + last} + unit_expected = { + "unit-name-8---unit": (0, b"3\n5\n6\n8\n9\n10\n", False), + "unit-name-8---user-unit": (0, b"3\n5\n6\n8\n9\n10\n", False), + "unit-name-3---unit": (0, b"3\n5\n6\n8\n9\n10\n", False), + "unit-name-3---user-unit": (0, b"3\n5\n6\n8\n9\n10\n", False), + "unit-name-9---unit": (0, b"3\n5\n6\n8\n9\n10\n", False), + "unit-name-10---unit": (0, b"3\n5\n6\n8\n9\n10\n", False), + "unit-name-14---unit": (0, b"14\n", False), + "unit-name-20---unit": (0, b"20\n", False), + "unit-name-26---unit": (0, b"26\n", False), + "unit-empty---unit": (1, b"", True), + "invocation-no-id[]": (1, b"", False), + "invocation-bounds[]": (0, [invocation(0, "1", 0, 2)], False), + "invocation-precedence[]": (0, [invocation(-1, "1", 0, 0), invocation(0, "3", 1, 1)], False), + "invocation-overlap['-n1']": (0, [invocation(0, "3", 3, 5)], False), + "invocation-select-bounds0": (0, b"0\n1\n2\n", False), + "invocation-select-precedence" + "2" * 32: (0, b"0\n", False), + "invocation-select-overlap-1": (0, b"0\n2\n", False), + "invocation-other-filter-['-tabsent']": (0, b"", False), + } + for option in ("--unit", "--user-unit"): + unit_expected["unit-bracket-invocations-" + option] = ( + 0, [invocation(-1, "1", 0, 0), invocation(0, "2", 1, 1)], False) + unit_expected["unit-bracket-select-" + option] = (0, b"1\n", False) + unicode_file = write_v262_fixture(paths / "file-é.journal", [b"unicode-file"]) + write_v262_fixture(paths / ".hidden.journal", [b"hidden-file"]) + brackets = paths / "brackets" + brackets.mkdir() + literal_bracket = write_v262_fixture(brackets / "literal-[x.journal", [b"literal-bracket"]) + colored = fixture.parent / "portable-colors.journal" + write_journal_file(colored, machine_id="00112233445566778899aabbccddeeff", + boot_id="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", seqnum_id="12121212121212121212121212121212", + entries=[{"realtime_usec": 1_700_005_000_000_000, "monotonic_usec": 1, + "fields": [("MESSAGE", "alpha hit beta"), ("PRIORITY", "3")]}]) + zone_dir = workspace / "zones" + zone_dir.mkdir() + write_timestamp_zone(zone_dir / "ReviewZone", [], [(-18000,0,"REV")]) + leap_zone = write_timestamp_zone(zone_dir / "leap", [], [(0,0,"UTC")], [(1483228800,1)]) + write_timestamp_zone(zone_dir / "posixrules", [(1672531200,1),(1701388800,0)], + [(-18000,0,"STD"),(-14400,1,"DST")]) + time_cases = [ + ({"TZ":"EST5EDT,M3.2.0,M11.1.0"}, b"2023-11-14T18:36:40-05:00"), + ({"TZ":"ReviewZone", "TZDIR":str(zone_dir.resolve())}, b"2023-11-14T18:36:40-05:00"), + ({"TZ":leap_zone}, b"2023-11-14T23:36:39+00:00"), + ({"TZ":"XYZ5ABC", "TZDIR":str(zone_dir.resolve())}, b"2023-11-14T19:36:40-04:00"), + ({"TZ":"XYZ5ABC", "TZDIR":str(workspace.resolve())}, b"2023-11-14T18:36:40-05:00"), + ] + env = {**os.environ, "TZ": "UTC", "LC_ALL": "C", "SYSTEMD_COLORS": "0"} + failures = 0 + for language, binary in commands: + def execute(options, path=fixture, changes=None): + return subprocess.run( # nosec B603 - explicit binary and synthetic fixture. + [str(binary.resolve()), "--file", str(path), "--no-pager", *options], + capture_output=True, env={**env, **(changes or {})}, timeout=30, check=False) + cases = [ + (["--cursor=t=ffffffffffffff", "--boot=all", "-ojson"], 0, b"", False), + (["--after-cursor=t=ffffffffffffff", "--boot=all", "-ojson"], 0, b"", False), + (["--grep=(?=alpha)alpha", "-ocat"], 0, b"alpha alpha\n", False), + ([r"--grep=(alpha) \1", "-ocat"], 0, b"alpha alpha\n", False), + ([r"--grep=nul\x00tail", "-ocat"], 0, b"nul\x00tail\n", False), + ([r"--grep=\xff", "-ocat"], 0, b"invalid\xffbyte\n", False), + (["--grep=only-second-match", "-ocat"], 1, b"", False), + (["--grep=unmatched-pattern", "-ocat"], 1, b"", False), + (["--grep=(", "-ocat"], 1, b"", True), + (["-F", "ROW", "--since=@0"], 1, b"", True), + (["--unit=missing-*", "-ocat"], 1, b"", True), + (["--user-unit=missing-*", "-ocat"], 1, b"", True), + (["--unit=absent.service", "-ocat"], 0, b"", False), + (["--since=bad", "--help"], 1, b"", True), + (["--grep=--help", "--help"], 0, None, False), + ] + for options, code, output, diagnostic in cases: + result = execute(options) + ok = (result.returncode == code and (output is None or result.stdout == output) + and bool(result.stderr) == diagnostic) + print(language, "PASS" if ok else "FAIL", options) + failures += not ok + result = execute(["--version"]) + ok = result.returncode == 0 and b"v262" in result.stdout and b"PCRE2" in result.stdout + failures += not ok + print(language, "PASS" if ok else "FAIL", "version") + result = execute(["--introspect-cli"]) + try: + info = json.loads(result.stdout) + ok = result.returncode == 0 and info["commands"][0]["names"] == ["journalctl"] + except (ValueError, KeyError, IndexError): + ok = False + failures += not ok + print(language, "PASS" if ok else "FAIL", "introspection") + result = execute(["-ojson"]) + rows = [json.loads(line) for line in result.stdout.splitlines()] + cursor = rows[1]["__CURSOR"] if result.returncode == 0 and len(rows) == 7 else None + for option, expected in [("--cursor",b"first\nsecond\n"),("--after-cursor",b"nul\x00tail\n")]: + result = execute([option, cursor or "invalid", "-n1", "-ocat"]) + ok = cursor is not None and result.returncode == 0 and result.stdout == expected and not result.stderr + failures += not ok + print(language, "PASS" if ok else "FAIL", option) + if cursor is not None: + parts = dict(part.split("=", 1) for part in cursor.split(";")) + for anchor in (cursor, "t=" + parts["t"], "s=" + parts["s"] + ";i=" + parts["i"]): + for option, expected in [("--cursor", b"first\nsecond\n"), + ("--after-cursor", b"alpha alpha\n")]: + result = execute([option, anchor, "-r", "-n1", "-ocat"]) + ok = result.returncode == 0 and result.stdout == expected and not result.stderr + failures += not ok + print(language, "PASS" if ok else "FAIL", "reverse", option, anchor) + for path, expected in [(unicode_file, b"unicode-file\n"), + (paths / "file-{é,missing}.journal", b"unicode-file\n"), + (paths / "file-*.journal", b"unicode-file\n"), + (paths / "file-[!a]*.journal", b"unicode-file\n"), + (literal_bracket, b"literal-bracket\n"), + (paths / "*.journal", b"unicode-file\n"), + (paths / ".*.journal", b"hidden-file\n")]: + result = execute(["-ocat"], path) + ok = result.returncode == 0 and result.stdout == expected and not result.stderr + failures += not ok + print(language, "PASS" if ok else "FAIL", "file selection", path.name) + result = execute(["-ocat", "--grep=hit"], colored, {"SYSTEMD_COLORS": "16"}) + expected = b"\x1b[0;1;31malpha \x1b[0;1;39mhit\x1b[0;1;31m beta\x1b[0m\n" + ok = result.returncode == 0 and result.stdout == expected and not result.stderr + failures += not ok + print(language, "PASS" if ok else "FAIL", "colored grep") + result = execute(["-ojson"], colored, {"SYSTEMD_COLORS": "16"}) + try: + row = json.loads(re.sub(rb"\x1b\[[0-9;:]*m", b"", result.stdout)) + ok = (result.returncode == 0 and not result.stderr and row["MESSAGE"] == "alpha hit beta" + and b'"\x1b[0;32mMESSAGE\x1b[0m"' in result.stdout) + except (ValueError, KeyError): + ok = False + failures += not ok + print(language, "PASS" if ok else "FAIL", "colored JSON") + for settings, timestamp in time_cases: + result = execute(["-q", "-oshort-iso", "--no-hostname"], colored, settings) + ok = (result.returncode == 0 and not result.stderr + and result.stdout == timestamp + b" unknown: alpha hit beta\n") + failures += not ok + print(language, "PASS" if ok else "FAIL", "calendar timezone", settings) + if not ok: + print(f" observed: {result.stdout!r} {result.stderr!r}") + for name, (code, expected, diagnostic) in unit_expected.items(): + path, options, _ = unit_cases[name] + result = execute(["-q", *options], path) + try: + output = json.loads(result.stdout) if isinstance(expected, list) else result.stdout + ok = result.returncode == code and output == expected and bool(result.stderr) == diagnostic + except ValueError: + ok = False + failures += not ok + print(language, "PASS" if ok else "FAIL", name) + if not ok: + print(f" expected: status={code} stdout={expected!r} diagnostic={diagnostic}") + print(f" observed: status={result.returncode} stdout={result.stdout!r} stderr={result.stderr!r}") + for language, binary in commands: + vacuum_workspace = workspace / (language + "-vacuum") + vacuum_workspace.mkdir() + failures += sum(not passed for passed in run_vacuum_cases(binary, vacuum_workspace)) + total = (46 + len(unit_expected) + len(time_cases)) * len(commands) + print(f"portable CLI smoke: {total-failures}/{total} passed") + return int(failures != 0) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/interoperability/run_journalctl_query_matrix.py b/tests/interoperability/run_journalctl_query_matrix.py index 09b69d88..9c0bb1a8 100755 --- a/tests/interoperability/run_journalctl_query_matrix.py +++ b/tests/interoperability/run_journalctl_query_matrix.py @@ -16,12 +16,15 @@ import re import shutil import subprocess # nosec B404 +import sys import time from dataclasses import dataclass from datetime import datetime, timezone from pathlib import Path from go_fixture_writer import start_live_journal_writer, write_journal_file +from journalctl_v262_cases import run_v262_cases +from journalctl_timestamp_cases import run_timestamp_cases REPO_ROOT = Path(__file__).resolve().parents[2] @@ -207,6 +210,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -216,7 +220,7 @@ def build_tools() -> dict[str, str]: go_journalctl = BIN_DIR / "go-journalctl" require_ok( - run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env=env), + run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env={**env, "CGO_ENABLED": "1"}), "build go journalctl", ) require_ok( @@ -230,8 +234,9 @@ def build_tools() -> dict[str, str]: rust_src = Path(env["CARGO_TARGET_DIR"]) / "debug" / "journalctl" rust_journalctl = BIN_DIR / "rust-journalctl" - if rust_src.exists(): - shutil.copy2(rust_src, rust_journalctl) + if not rust_src.is_file(): + raise RuntimeError(f"expected Rust build output not found: {rust_src}") + shutil.copy2(rust_src, rust_journalctl) for path in (go_journalctl, rust_journalctl): if not path.exists(): @@ -243,11 +248,17 @@ def build_tools() -> dict[str, str]: } -def systemd_version() -> str: - result = run(["journalctl", "--version"], timeout=10) - if result.returncode == 0 and result.stdout.strip(): - return result.stdout.splitlines()[0] - return "unavailable" +def systemd_version(journalctl: str) -> str: + result = run([journalctl, "--version"], timeout=10) + require_ok(result, "selected journalctl --version") + lines = result.stdout.splitlines() + if not lines or not re.match(r"^systemd 262(?:\s|$)", lines[0]): + raise RuntimeError("query matrix requires the pinned systemd v262 oracle") + features = set(" ".join(lines[1:]).split()) + missing = {"+PCRE2", "+ZSTD", "+XZ", "+LZ4"} - features + if missing: + raise RuntimeError("selected v262 oracle lacks required features: " + ", ".join(sorted(missing))) + return lines[0] def make_fixtures() -> dict[str, Path]: @@ -536,7 +547,7 @@ def row_entry(message: str, realtime: int, monotonic: int, test_id: str, boot_id def reader_command(reader: str, tools: dict[str, str], mode: str, path: Path, args: list[str]) -> list[str]: if reader == "stock": - base = ["journalctl", f"--{mode}", str(path), "--output=json", "--no-pager", "--quiet"] + base = [tools["stock_journalctl"], f"--{mode}", str(path), "--output=json", "--no-pager", "--quiet"] elif reader == "go": base = [tools["go_journalctl"], f"--{mode}", str(path), "--output=json"] elif reader == "rust": @@ -548,7 +559,7 @@ def reader_command(reader: str, tools: dict[str, str], mode: str, path: Path, ar def raw_reader_command(reader: str, tools: dict[str, str], mode: str, path: Path, args: list[str]) -> list[str]: if reader == "stock": - base = ["journalctl", f"--{mode}", str(path), "--no-pager"] + base = [tools["stock_journalctl"], f"--{mode}", str(path), "--no-pager"] elif reader == "go": base = [tools["go_journalctl"], f"--{mode}", str(path)] elif reader == "rust": @@ -560,7 +571,7 @@ def raw_reader_command(reader: str, tools: dict[str, str], mode: str, path: Path def action_command(reader: str, tools: dict[str, str], args: list[str]) -> list[str]: if reader == "stock": - return ["journalctl", "--no-pager", "--quiet", *args] + return [tools["stock_journalctl"], "--no-pager", "--quiet", *args] if reader == "go": return [tools["go_journalctl"], "--quiet", *args] if reader == "rust": @@ -692,6 +703,7 @@ def run_static_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[d mismatched_x_cursor = replace_cursor_x(file_cursors[1], "ffffffffffffffff") cases = [ + ("directory-all", "directory", fixtures["directory"], ["TEST_ID=journalctl-query"]), ("directory-reverse", "directory", fixtures["directory"], ["--reverse", "TEST_ID=journalctl-query"]), ("directory-since", "directory", fixtures["directory"], ["--since", "@1700004000.000001", "TEST_ID=journalctl-query"]), @@ -1649,6 +1661,96 @@ def run_utility_action_cases(tools: dict[str, str], fixtures: dict[str, Path]) - } ) + if sys.platform == "linux": + results.extend(run_vacuum_metadata_cases(tools, fixtures["file"], vacuum_root)) + return results + + +def run_vacuum_metadata_cases(tools: dict[str, str], source: Path, + vacuum_root: Path) -> list[dict[str, object]]: + """Recent names and mtimes must not hide an older native timestamp.""" + now = time.time_ns() // 1000 + old = now - 30 * 86400 * 1_000_000 + fresh_ns = (now + 3600 * 1_000_000) * 1000 + results = [] + for name, retention, xattr, deleted in [ + ("atime", "1day", None, True), + ("creation-xattr", "1day", old.to_bytes(8, "little"), True), + ("creation-zero", "1day", bytes(8), False), + ("creation-infinity", "1day", ((1 << 64) - 1).to_bytes(8, "little"), False), + ("creation-short", "1day", b"\x01", False), + ("preserve-atime", "90days", None, False), + ]: + observed = {} + for reader in READERS: + directory = vacuum_root / ("metadata-" + name) / reader + directory.mkdir(parents=True, exist_ok=True) + path = directory / archived_journal_name(1, fresh_ns // 1000) + shutil.copyfile(source, path) + atime_ns = old * 1000 if xattr is None else fresh_ns + os.utime(path, ns=(atime_ns, fresh_ns)) + if xattr is not None: + os.setxattr(path, "user.crtime_usec", xattr) + result = run(action_command(reader, tools, + ["--directory", str(directory), "--vacuum-time=" + retention]), timeout=30) + exists = path.exists() + observed[reader] = (result.returncode, exists, path.stat().st_atime_ns if exists else None) + expected = observed["stock"] + if expected[0] != 0 or expected[1] == deleted: + raise AssertionError(f"stock vacuum metadata-{name} control changed: {expected!r}") + for reader in ("go", "rust"): + actual = observed[reader] + results.append({"test": "vacuum-metadata-" + name, "reader": reader, + "status": "PASS" if actual == expected else "FAIL", + "expected": expected, "actual": actual, + "returncode": actual[0]}) + for same_sequence in (True, False): + name = "shared-sequence" if same_sequence else "mixed-sequence" + first_name = archived_journal_name(1, fresh_ns // 1000) + second_name = archived_journal_name(2, fresh_ns // 1000) + if not same_sequence: + second_name = second_name.replace(VACUUM_SEQNUM_ID, "34343434343434343434343434343434") + observed = {} + for reader in READERS: + directory = vacuum_root / ("metadata-order-" + name) / reader + directory.mkdir(parents=True, exist_ok=True) + for filename in (first_name, second_name): + path = directory / filename + shutil.copyfile(source, path) + os.utime(path, ns=(fresh_ns, fresh_ns)) + os.setxattr(directory / second_name, "user.crtime_usec", old.to_bytes(8, "little")) + result = run(action_command(reader, tools, + ["--directory", str(directory), "--vacuum-files=1"]), timeout=30) + observed[reader] = (result.returncode, remaining_names(directory)) + expected = observed["stock"] + survivor = second_name if same_sequence else first_name + if expected != (0, [survivor]): + raise AssertionError(f"stock vacuum metadata order-{name} control changed: {expected!r}") + for reader in ("go", "rust"): + actual = observed[reader] + results.append({"test": "vacuum-metadata-order-" + name, "reader": reader, + "status": "PASS" if actual == expected else "FAIL", + "expected": expected, "actual": actual, + "returncode": actual[0]}) + suffix = f"@{old:016x}-0000000000000001.journal~" + raw_names = [os.fsdecode(prefix) + suffix for prefix in (b"raw-\xff", b"raw-\xef")] + observed = {} + for reader in READERS: + directory = vacuum_root / "metadata-order-raw-names" / reader + directory.mkdir(parents=True, exist_ok=True) + for filename in raw_names: + shutil.copyfile(source, directory / filename) + result = run(action_command(reader, tools, + ["--directory", str(directory), "--vacuum-files=1"]), timeout=30) + observed[reader] = (result.returncode, remaining_names(directory)) + expected = observed["stock"] + if expected != (0, [raw_names[0]]): + raise AssertionError(f"stock raw vacuum filename order changed: {expected!r}") + for reader in ("go", "rust"): + actual = observed[reader] + results.append({"test": "vacuum-metadata-order-raw-names", "reader": reader, + "status": "PASS" if actual == expected else "FAIL", + "expected": expected, "actual": actual, "returncode": actual[0]}) return results @@ -2013,7 +2115,7 @@ def run_empty_result_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> path = fixtures["file"] for case_name, extra_args, _expected_hint in cases: args = [*extra_args, "--boot=all", "TEST_ID=journalctl-empty-result"] - stock_cmd = ["journalctl", "--file", str(path), "--no-pager", *args] + stock_cmd = [tools["stock_journalctl"], "--file", str(path), "--no-pager", *args] stock = run(stock_cmd, timeout=30) require_ok(stock, f"stock {case_name}") expected = stock.stdout @@ -2044,6 +2146,77 @@ def run_empty_result_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> def run_follow_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[dict[str, object]]: results = [] cases = [ + { + "name": "follow-unit-glob-frozen", + "test_id": "journalctl-follow-unit-glob", + "args": ["--follow", "--no-tail", "--boot=all", "--merge", "-uapp-*"], + "initial": [("initial-alpha", 1_700_006_180_000_000)], + "appends": [("new-beta", 1_700_006_180_000_001), + ("late-alpha", 1_700_006_180_000_002)], + "initial_fields": [[("_SYSTEMD_UNIT", "app-alpha.service")]], + "append_fields": [[("_SYSTEMD_UNIT", unit + ".service")] for unit in ("app-beta", "app-alpha")], + "expected": ["initial-alpha", "late-alpha"], + }, + { + "name": "follow-invocation-frozen", + "test_id": "journalctl-follow-invocation", + "args": ["--follow", "--no-tail", "--boot=all", "--merge", "-ualpha", "-I"], + "initial": [("initial-one", 1_700_006_190_000_000), + ("initial-two", 1_700_006_190_000_001)], + "appends": [("new-three", 1_700_006_190_000_002), + ("late-two", 1_700_006_190_000_003)], + "initial_fields": [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", c * 32)] + for c in "12"], + "append_fields": [[("_SYSTEMD_UNIT", "alpha.service"), ("_SYSTEMD_INVOCATION_ID", c * 32)] + for c in "32"], + "expected": ["initial-two", "late-two"], + }, + { + "name": "follow-cursor-skipped-suffix", + "test_id": "journalctl-follow-cursor-suffix", + "args": ["--follow", "--no-tail", "--boot=all", "--merge", "--grep=hit"], + "initial": [("hit", 1_700_006_200_000_000)], + "appends": [("miss", 1_700_006_200_000_001)], + "expected": ["hit"], + "cursor_file": True, + "checkpoint_final": True, + }, + { + "name": "follow-cursor-no-grep-matches", + "test_id": "journalctl-follow-cursor-no-matches", + "args": ["--follow", "--no-tail", "--boot=all", "--merge", "--grep=hit"], + "initial": [("miss-initial", 1_700_006_210_000_000)], + "appends": [("miss-final", 1_700_006_210_000_001)], + "expected": [], + "cursor_file": True, + "checkpoint_final": True, + }, + { + "name": "follow-grep-native-tail", + "test_id": "journalctl-follow-grep-tail", + "args": ["--follow", "--lines=2", "--boot=all", "--merge", "--grep=hit"], + "initial": [("hit-a", 1_700_006_300_000_000, BOOT_A), + ("miss-b", 1_700_006_300_000_001, BOOT_B), + ("hit-c", 1_700_006_300_000_002, BOOT_C)], + "appends": [("hit-new", 1_700_006_300_001_000, BOOT_C)], + "expected": ["hit-c", "hit-new"], + }, + { + "name": "follow-grep-boot-events", + "test_id": "journalctl-follow-grep-boots", + "args": ["--follow", "--no-tail", "--boot=all", "--grep=hit", + "--utc", "--no-hostname", "-oshort"], + "initial": [("hit-a", 1_700_006_300_000_000, BOOT_A), + ("miss-b", 1_700_006_300_000_001, BOOT_B)], + "appends": [("miss-c", 1_700_006_300_000_002, BOOT_C), + ("hit-d", 1_700_006_300_000_003, "d" * 32)], + "raw_output": True, + "expected": [ + datetime.fromtimestamp(1_700_006_300, tz=timezone.utc).strftime("%b %d %H:%M:%S") + " unknown: hit-a", + f"-- Boot {BOOT_B} --", f"-- Boot {BOOT_C} --", f"-- Boot {'d' * 32} --", + datetime.fromtimestamp(1_700_006_300, tz=timezone.utc).strftime("%b %d %H:%M:%S") + " unknown: hit-d", + ], + }, { "name": "follow-live-append-no-tail", "test_id": "journalctl-follow", @@ -2107,6 +2280,39 @@ def run_follow_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[d "writer_boot_id": BOOT_C, }, ] + backlog = [(f"history-{i:04d}", 1_700_006_000_000_000 + i) for i in range(1005)] + appended = [("history-new-0", 1_700_006_000_002_000), ("history-new-1", 1_700_006_000_002_001)] + for name, extra, all_history in [ + ("since-default", ["--since=@0"], True), + ("explicit-all", ["--lines=all"], True), + ("no-tail-overrides-one", ["--no-tail", "--lines=1"], True), + ("explicit-zero", ["--lines=0"], False), + ]: + cases.append({"name": "follow-history-" + name, "test_id": "journalctl-history-" + name, + "args": ["--follow", "--boot=all", "--merge", *extra], "initial": backlog, + "appends": appended, "expected": [row[0] for row in (backlog if all_history else []) + appended]}) + for name, extra, count in [ + ("pager-default", [], 10), + ("pager-since", ["--since=@0"], 1000), + ]: + cases.append({"name": "follow-history-" + name, "test_id": "journalctl-history-" + name, + "args": ["--follow", "--pager-end", "--boot=all", "--merge", *extra], + "initial": backlog, "appends": appended, + "expected": [row[0] for row in backlog[-count:] + appended]}) + initial = [(f"cursor-history-{i}",1_700_006_100_000_000+i) for i in range(12)] + appends = [("cursor-new",1_700_006_100_001_000)] + for name, cursor_mode, limit, start in [ + ("cursor-one","--cursor",1,1), ("after-cursor-one","--after-cursor",1,2), + ("cursor-zero","--cursor",0,1), ("cursor-file-one","--cursor-file",1,2), + ]: + cases.append({"name":"follow-"+name,"test_id":"journalctl-"+name, + "args":["--follow","--boot=all","--merge",f"-n{limit}"], + "initial":initial,"appends":appends,"cursor_mode":cursor_mode,"cursor_index":1, + "cursor_file":cursor_mode=="--cursor-file", + "expected":[row[0] for row in initial[start:]+appends]}) + cases.append({"name":"follow-empty-cursor-file","test_id":"journalctl-empty-cursor-file", + "args":["--follow","--boot=all","--merge","-n1"],"initial":initial,"appends":appends, + "cursor_file":True,"empty_cursor_file":True,"expected":[initial[-1][0],appends[0][0]]}) for case in cases: mode = str(case.get("mode", "file")) for reader in READERS: @@ -2117,7 +2323,7 @@ def run_follow_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[d else: read_path = fixtures["follow"] / case["name"] / f"{reader}.journal" write_path = read_path - actual, returncode, stderr, cmd, cursor_written = run_follow_reader(reader, tools, mode, read_path, write_path, case) + actual, returncode, stderr, cmd, cursor_written, child_returncode, stopped_by_harness = run_follow_reader(reader, tools, mode, read_path, write_path, case) expected = case["expected"] ok = returncode == 0 and actual == expected and (not case.get("cursor_file") or cursor_written) results.append( @@ -2129,6 +2335,8 @@ def run_follow_cases(tools: dict[str, str], fixtures: dict[str, Path]) -> list[d "expected": expected, "actual": actual, "cursor_written": cursor_written, + "child_returncode": child_returncode, + "stopped_by_harness": stopped_by_harness, "returncode": returncode, "stderr": stderr[-1000:], } @@ -2143,23 +2351,29 @@ def run_follow_reader( read_path: Path, write_path: Path, case: dict[str, object], -) -> tuple[list[str], int, str, list[str], bool]: +) -> tuple[list[str], int, str, list[str], bool, int | None, bool]: if mode == "directory": read_path.mkdir(parents=True, exist_ok=True) write_path.parent.mkdir(parents=True, exist_ok=True) ready_file = write_path.with_suffix(write_path.suffix + ".ready") seq = 1 initial_entries = [] - for row in case["initial"]: + for i, row in enumerate(case["initial"]): message, realtime, *boot_override = row boot_id = str(boot_override[0]) if boot_override else BOOT_A - initial_entries.append(row_entry(message, realtime, seq, str(case["test_id"]), boot_id=boot_id)) + entry = row_entry(message, realtime, seq, str(case["test_id"]), boot_id=boot_id) + if case.get("initial_fields"): + entry["fields"].extend(case["initial_fields"][i]) + initial_entries.append(entry) seq += 1 append_entries = [] - for row in case["appends"]: + for i, row in enumerate(case["appends"]): message, realtime, *boot_override = row boot_id = str(boot_override[0]) if boot_override else BOOT_A - append_entries.append(row_entry(message, realtime, seq, str(case["test_id"]), boot_id=boot_id)) + entry = row_entry(message, realtime, seq, str(case["test_id"]), boot_id=boot_id) + if case.get("append_fields"): + entry["fields"].extend(case["append_fields"][i]) + append_entries.append(entry) seq += 1 writer_proc = start_live_journal_writer( write_path, @@ -2172,14 +2386,28 @@ def run_follow_reader( ) ready_error = wait_for_writer_ready(writer_proc, ready_file) if ready_error is not None: - return [], ready_error[0], ready_error[1], [], False + return [], ready_error[0], ready_error[1], [], False, None, False cursor_file = write_path.with_suffix(write_path.suffix + ".cursor") args = [*case["args"], f"TEST_ID={case['test_id']}"] if case.get("cursor_file"): if cursor_file.exists(): cursor_file.unlink() args = ["--cursor-file", str(cursor_file), *args] - cmd = reader_command( + if case.get("empty_cursor_file"): + cursor_file.write_text("") + if case.get("cursor_mode"): + probe = subprocess.run( # nosec B603 - private writer fixture and selected oracle. + [tools["stock_journalctl"],"--file",str(write_path),"--output=json","--boot=all"], + capture_output=True, timeout=30, check=False) + if probe.returncode != 0: + raise AssertionError("follow cursor probe failed: " + probe.stderr.decode(errors="replace")) + cursor = json.loads(probe.stdout.splitlines()[int(case["cursor_index"])])["__CURSOR"] + if case["cursor_mode"] == "--cursor-file": + cursor_file.write_text(cursor+"\n") + else: + args = [str(case["cursor_mode"]),cursor,*args] + command = raw_reader_command if case.get("raw_output") else reader_command + cmd = command( reader, tools, mode, @@ -2189,28 +2417,51 @@ def run_follow_reader( proc: subprocess.Popen[str] | None = None # nosemgrep # subprocess is required by this harness; commands are shell=False vectors. + # Follow history can exceed pipe capacity before the append window ends. + # File-backed capture keeps every reader draining without timing-dependent truncation. + stdout_path = write_path.with_suffix(".reader.stdout.log") + stderr_path = write_path.with_suffix(".reader.stderr.log") + captured_stdout = stdout_path.open("w", encoding="utf-8") + captured_stderr = stderr_path.open("w", encoding="utf-8") proc = subprocess.Popen( # nosec B603 cmd, # nosemgrep cwd=str(REPO_ROOT), text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, + stdout=captured_stdout, + stderr=captured_stderr, ) try: writer_stdout, writer_stderr = communicate_or_stop(writer_proc, timeout=10, stop_timeout=2) time.sleep(0.7) - proc.terminate() - stdout, stderr = communicate_or_kill(proc, timeout=5) + stopped_by_harness = proc.poll() is None + if stopped_by_harness: + proc.terminate() + communicate_or_kill(proc, timeout=5) finally: stop_process_if_running(writer_proc) stop_process_if_running(proc) + captured_stdout.close() + captured_stderr.close() + stdout = stdout_path.read_text(encoding="utf-8") + stderr = stderr_path.read_text(encoding="utf-8") - actual = parse_messages(stdout) - expected = case["expected"] + actual = stdout.splitlines() if case.get("raw_output") else parse_messages(stdout) cursor_written = cursor_file.exists() and bool(cursor_file.read_text(encoding="utf-8").strip()) + if case.get("checkpoint_final"): + probe = subprocess.run( # nosec B603 - private fixture and explicitly selected oracle. + [tools["stock_journalctl"], "--file", str(write_path), "-n1", "-ojson"], + capture_output=True, timeout=30, check=False, + ) + require_ok(probe, "follow final cursor probe") + expected_cursor = json.loads(probe.stdout)["__CURSOR"] + cursor_written = cursor_written and cursor_file.read_text(encoding="utf-8").strip() == expected_cursor if writer_proc.returncode != 0: - return actual, writer_proc.returncode or 1, (writer_stdout + writer_stderr + stderr)[-1000:], cmd, cursor_written - return actual, 0 if actual == expected else proc.returncode or 1, stderr, cmd, cursor_written + return actual, writer_proc.returncode or 1, (writer_stdout + writer_stderr + stderr)[-1000:], cmd, cursor_written, proc.returncode, stopped_by_harness + # The harness sends SIGTERM only to its own child after the append window. + # A real exit 1 remains a failure even if its partial output happens to match. + normal_stop = stopped_by_harness and proc.returncode in (-15, 143) + result_code = 0 if proc.returncode == 0 or normal_stop else proc.returncode or 1 + return actual, result_code, stderr, cmd, cursor_written, proc.returncode, stopped_by_harness def wait_for_writer_ready(writer_proc: subprocess.Popen[str], ready_file: Path) -> tuple[int, str] | None: @@ -2254,9 +2505,14 @@ def stop_process_if_running(proc: subprocess.Popen[str] | None) -> None: def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--skip-follow", action="store_true", help="skip live follow checks") + parser.add_argument("--journalctl", type=Path, required=True, + help="path to the full-feature systemd v262 journalctl oracle") args = parser.parse_args() + oracle = str(args.journalctl.resolve(strict=True)) + version = systemd_version(oracle) tools = build_tools() + tools["stock_journalctl"] = oracle fixtures = make_fixtures() results = run_static_cases(tools, fixtures) results.extend(run_cursor_file_cases(tools, fixtures)) @@ -2264,13 +2520,16 @@ def main() -> int: results.extend(run_utility_action_cases(tools, fixtures)) results.extend(run_output_mode_cases(tools, fixtures)) results.extend(run_empty_result_cases(tools, fixtures)) + results.extend(run_v262_cases(tools, fixtures)) + results.extend(run_timestamp_cases(tools, fixtures)) if not args.skip_follow: results.extend(run_follow_cases(tools, fixtures)) failures = [r for r in results if r["status"] != "PASS"] report = { "status": "PASS" if not failures else "FAIL", - "systemd": systemd_version(), + "systemd": version, + "stock_journalctl": oracle, "fixture_dir": str(FIXTURE_DIR), "results": results, "failures": failures, diff --git a/tests/interoperability/run_live_matrix.py b/tests/interoperability/run_live_matrix.py index d47162c7..2ee0d457 100755 --- a/tests/interoperability/run_live_matrix.py +++ b/tests/interoperability/run_live_matrix.py @@ -202,6 +202,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -221,7 +222,7 @@ def build_tools() -> dict[str, str]: run( ["go", "build", "-o", str(BIN_DIR / "go-journalctl"), "./cmd/journalctl"], cwd=REPO_ROOT / "go", - env=env, + env={**env, "CGO_ENABLED": "1"}, ), "build go journalctl", ) @@ -260,7 +261,7 @@ def build_tools() -> dict[str, str]: (cargo_target / "debug/livewriter", BIN_DIR / "rust-livewriter"), (cargo_target / "debug/journalctl", BIN_DIR / "rust-journalctl"), ]: - if not src.exists(): + if not src.is_file(): raise RuntimeError(f"expected Rust build output not found: {src}") shutil.copy2(src, dst) diff --git a/tests/interoperability/run_matrix.py b/tests/interoperability/run_matrix.py index d1d3f65f..b0d59a1d 100755 --- a/tests/interoperability/run_matrix.py +++ b/tests/interoperability/run_matrix.py @@ -10,6 +10,7 @@ import argparse import json +import os import shutil import subprocess # nosec B404 import sys @@ -49,12 +50,13 @@ class ReaderSpec: } -def run(cmd: list[str], *, cwd: Path = REPO_ROOT, timeout: int = 60) -> subprocess.CompletedProcess[str]: +def run(cmd: list[str], *, cwd: Path = REPO_ROOT, timeout: int = 60, env: dict[str, str] | None = None) -> subprocess.CompletedProcess[str]: # nosemgrep # subprocess is required by this harness; commands are shell=False vectors. return subprocess.run( # nosec B603 cmd, # nosemgrep cwd=str(cwd), + env=env, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, @@ -89,7 +91,7 @@ def build_tools() -> dict[str, str]: "build go livewriter", ) require_ok( - run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go"), + run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env={**os.environ, "CGO_ENABLED": "1"}), "build go journalctl", ) @@ -102,8 +104,9 @@ def build_tools() -> dict[str, str]: "build rust journalctl", ) - rust_livewriter = REPO_ROOT / "rust/target/debug/livewriter" - rust_journalctl = REPO_ROOT / "rust/target/debug/journalctl" + cargo_target = (REPO_ROOT / os.environ.get("CARGO_TARGET_DIR", "rust/target")).resolve() + rust_livewriter = cargo_target / "debug/livewriter" + rust_journalctl = cargo_target / "debug/journalctl" for path in (go_livewriter, go_journalctl, rust_livewriter, rust_journalctl): if not path.exists(): raise RuntimeError(f"expected tool not found after build: {path}") diff --git a/tests/interoperability/run_mixed_directory_matrix.py b/tests/interoperability/run_mixed_directory_matrix.py index 10cf8dae..cb24a213 100644 --- a/tests/interoperability/run_mixed_directory_matrix.py +++ b/tests/interoperability/run_mixed_directory_matrix.py @@ -160,6 +160,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -169,7 +170,7 @@ def build_tools() -> dict[str, str]: go_journalctl = BIN_DIR / "go-journalctl" require_ok( - run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env=env), + run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env={**env, "CGO_ENABLED": "1"}), "build go journalctl", ) require_ok( @@ -183,8 +184,9 @@ def build_tools() -> dict[str, str]: rust_src = Path(env["CARGO_TARGET_DIR"]) / "debug" / "journalctl" rust_journalctl = BIN_DIR / "rust-journalctl" - if rust_src.exists(): - shutil.copy2(rust_src, rust_journalctl) + if not rust_src.is_file(): + raise RuntimeError(f"expected Rust build output not found: {rust_src}") + shutil.copy2(rust_src, rust_journalctl) for path in (go_journalctl, rust_journalctl): if not path.exists(): diff --git a/tests/interoperability/run_reader_hardening.py b/tests/interoperability/run_reader_hardening.py new file mode 100755 index 00000000..eaeff1ea --- /dev/null +++ b/tests/interoperability/run_reader_hardening.py @@ -0,0 +1,695 @@ +#!/usr/bin/env python3 +"""Shared public-SDK chain, complete-stream and crash-prefix checks. + +All journals and subprocess artifacts stay in a new repository-local directory. +Potentially looping SDK operations run in bounded child processes. +""" + +from __future__ import annotations + +import argparse +import ctypes +import ctypes.util +import json +import os +import shlex +import subprocess # nosec B404 +from pathlib import Path + +from go_fixture_writer import write_journal_file +from run_verify_matrix import oversized_zstd_frame, parse_journal, read_u64, write_u64 + +ROOT = Path(__file__).resolve().parents[2] +IDENTITY = "00112233445566778899aabbccddeeff" + + +def posting_prefix_known( + data: bytes, at: int, last: int, extent: int, compact: bool +) -> bool: + """Check whether the fixture exposes every posting up to the recovered tail.""" + previous = read_u64(data, at + 40) + remaining = read_u64(data, at + 56) + if not remaining or previous > last: + return True + remaining -= 1 + array = read_u64(data, at + 48) + width = 4 if compact else 8 + seen = set() + while remaining: + if not array or array + 24 > extent: + return previous == last + if array in seen: + raise RuntimeError("cyclic source fixture posting arrays") + seen.add(array) + wanted = min(remaining, (read_u64(data, array + 8) - 24) // width) + available = min(wanted, (extent - array - 24) // width) + for index in range(available): + slot = array + 24 + index * width + value = int.from_bytes(data[slot : slot + width], "little") + if not value: + return previous == last + if value <= previous: + raise RuntimeError("unordered source fixture postings") + if value > last: + return True + previous = value + if available < wanted: + return previous == last + remaining -= wanted + array = read_u64(data, array + 16) + return True + + +class Zstd: + def __init__(self) -> None: + library = ctypes.util.find_library("zstd") + if library is None: + raise RuntimeError("an existing libzstd is required for fixture encoding") + self.lib = ctypes.CDLL(library) + for name, args, result in ( + ("ZSTD_createCCtx", [], ctypes.c_void_p), + ("ZSTD_freeCCtx", [ctypes.c_void_p], ctypes.c_size_t), + ( + "ZSTD_CCtx_setParameter", + [ctypes.c_void_p, ctypes.c_int, ctypes.c_int], + ctypes.c_size_t, + ), + ("ZSTD_compressBound", [ctypes.c_size_t], ctypes.c_size_t), + ( + "ZSTD_compress2", + [ + ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_size_t, + ctypes.c_void_p, + ctypes.c_size_t, + ], + ctypes.c_size_t, + ), + ("ZSTD_isError", [ctypes.c_size_t], ctypes.c_uint), + ): + function = getattr(self.lib, name) + function.argtypes, function.restype = args, result + + def checked(self, value: int) -> int: + if self.lib.ZSTD_isError(value): + raise RuntimeError("libzstd fixture encoding failed") + return value + + def frame( + self, payload: bytes, *, known: bool = True, checksum: bool = False + ) -> bytes: + context = self.lib.ZSTD_createCCtx() + if not context: + raise RuntimeError("libzstd fixture context allocation failed") + try: + self.checked(self.lib.ZSTD_CCtx_setParameter(context, 200, int(known))) + self.checked(self.lib.ZSTD_CCtx_setParameter(context, 201, int(checksum))) + output = ctypes.create_string_buffer( + self.lib.ZSTD_compressBound(len(payload)) + ) + size = self.checked( + self.lib.ZSTD_compress2( + context, output, len(output), payload, len(payload) + ) + ) + return output.raw[:size] + finally: + self.checked(self.lib.ZSTD_freeCCtx(context)) + + +def skippable(length: int = 0) -> bytes: + return b"\x50\x2a\x4d\x18" + length.to_bytes(4, "little") + bytes(length) + + +def expected_rows(entries: list[dict]) -> list[dict]: + return [ + { + "realtime": entry["realtime_usec"], + "payloads": sorted( + ( + name.encode() + + b"=" + + (value.encode() if isinstance(value, str) else value) + ).hex() + for name, value in entry["fields"] + ), + } + for entry in entries + ] + + +def fixture( + path: Path, count: int, compact: bool, env: dict +) -> tuple[bytes, dict, list[dict]]: + entries = [ + { + "realtime_usec": 1000 + i * 1000, + "monotonic_usec": 100 + i, + "fields": [ + ("MESSAGE", b"a" * 1000 + b"\x00\xff\n" + str(i).encode()), + ("COMMON", "all"), + ("TAG", "first"), + ("TAG", "second"), + ] + + ([("SPARSE", "some")] if i in (0, count // 2, count - 1) else []), + } + for i in range(count) + ] + write_journal_file( + path, + machine_id=IDENTITY, + boot_id=IDENTITY, + seqnum_id=IDENTITY, + file_id=IDENTITY, + entries=entries, + compact=compact, + env=env, + ) + data = path.read_bytes() + return data, parse_journal(data), expected_rows(entries) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--go-adapter", type=Path, required=True) + parser.add_argument("--rust-adapter", type=Path, required=True) + parser.add_argument("--journalctl", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument( + "--only", choices=("chains", "streams", "recovery"), action="append" + ) + args = parser.parse_args() + out = args.out.resolve() + if not out.is_relative_to(ROOT / ".local") or out.exists(): + parser.error("--out must be a new directory inside the repository's .local/") + adapters = { + name: path.resolve(strict=True) + for name, path in (("go", args.go_adapter), ("rust", args.rust_adapter)) + } + oracle = args.journalctl.resolve(strict=True) + out.mkdir(parents=True) + (out / "tmp").mkdir() + env = os.environ.copy() + env.update(TMPDIR=str(out / "tmp"), PYTHONDONTWRITEBYTECODE="1") + selected = set(args.only or ("chains", "streams", "recovery")) + observations = [] + + def probe( + path: Path, + operation: str, + *, + expected: list | None = None, + error: bool = False, + recovery: bool = False, + declared: int = 0, + recovered_count: int | None = None, + **options, + ) -> None: + request = dict(path=str(path), operation=operation, snapshot=True, **options) + for language, binary in adapters.items(): + command = [str(binary), "inspect-reader"] + print(f"+ {shlex.join(command)}: {path.name}/{operation}", flush=True) + process = subprocess.run( + command, + input=json.dumps(request), + text=True, + capture_output=True, + env=env, + timeout=8, + check=False, + ) # nosec B603 + if process.returncode: + raise RuntimeError( + f"{language}/{path.name}/{operation}: exit {process.returncode}: {process.stderr[-2000:]}" + ) + result = json.loads(process.stdout) + if result["error"] != error: + evidence = out / f"unexpected-result-{language}-{len(observations)}.json" + evidence.write_text(json.dumps(result, indent=2) + "\n") + raise RuntimeError( + f"{language}/{path.name}/{operation}: expected error={error}, " + f"got {result['error']}; inspect {evidence}" + ) + if expected is not None: + actual = ( + result["unique"] + if operation.removeprefix("facade-").startswith("unique") + else result["rows"] + ) + if actual != expected: + raise RuntimeError( + f"{language}/{path.name}/{operation}: incomplete or changed content: {len(actual)} != {len(expected)}" + ) + if recovery: + statuses = result["recovery_statuses"] + if len(statuses) != 1 or statuses[0]["status"]["reason"] == 0: + raise RuntimeError( + f"{language}/{path.name}: missing recovery status" + ) + status = statuses[0]["status"] + if status["declared_entries"] != declared or status[ + "recovered_entries" + ] != ( + recovered_count if recovered_count is not None else len(expected) + ): + raise RuntimeError( + f"{language}/{path.name}: incorrect recovery population: {status}" + ) + observations.append( + { + "language": language, + "fixture": path.name, + "operation": operation, + "expected_error": error, + "rows": len(result["rows"]), + "recovery_statuses": result["recovery_statuses"], + } + ) + + def stock_verify(path: Path, expected_error: bool = False) -> None: + command = [str(oracle), "--verify", "--file", str(path)] + print(f"+ {shlex.join(command)}", flush=True) + result = subprocess.run( + command, text=True, capture_output=True, env=env, timeout=15, check=False + ) # nosec B603 + if (result.returncode != 0) != expected_error: + raise RuntimeError( + f"stock verification disagreement: {path.name}: {result.stderr[-2000:]}" + ) + + zstd = Zstd() if "streams" in selected else None + for compact in (False, True): + folder = out / ("compact" if compact else "regular") + folder.mkdir() + base = folder / "base.journal" + data, parsed, rows = fixture(base, 4, compact, env) + stock_verify(base) + probe(base, "forward", expected=rows) + payload_header = 72 if compact else 64 + messages = [ + offset + for offset in parsed["data_offsets"] + if data[offset + payload_header : offset + payload_header + 8] + == b"MESSAGE=" + ] + if "chains" in selected: + field_size = read_u64(data, 88) + tail = read_u64(data, 136) + fields = [] + at = field_size + while at <= tail: + size = read_u64(data, at + 8) + if data[at] == 2: + fields.append(at) + at += (size + 7) & ~7 + message_field = next( + at + for at in fields + if data[at + 40 : at + read_u64(data, at + 8)] == b"MESSAGE" + ) + for name in ( + "field-self", + "data-self", + "field-data-multi", + "data-hash-self", + "array-self", + "ascending", + ): + mutated = bytearray(data) + operation, options, error = "unique", {"field": "MESSAGE"}, True + if name == "field-self": + write_u64(mutated, message_field + 24, message_field) + operation, options = "fields", {} + elif name == "data-self": + head = read_u64(data, message_field + 32) + write_u64(mutated, head + 32, head) + elif name == "field-data-multi": + write_u64(mutated, messages[0] + 32, messages[-1]) + elif name == "data-hash-self": + first = parsed["data_offsets"][0] + write_u64(mutated, first + 24, first) + start, size = ( + parsed["data_hash_table_offset"], + parsed["data_hash_table_size"], + ) + mutated[start : start + size] = ( + first.to_bytes(8, "little") * 2 + ) * (size // 16) + operation, options = "explorer", {"field": "MESSAGE"} + elif name == "array-self": + write_u64( + mutated, + parsed["entry_array_offset"] + 16, + parsed["entry_array_offset"], + ) + write_u64(mutated, 152, 1_000_000) + operation, options = "forward", {} + else: + write_u64(mutated, message_field + 32, messages[0]) + for index, at in enumerate(messages): + write_u64( + mutated, + at + 32, + messages[index + 1] if index + 1 < len(messages) else 0, + ) + error = False + path = folder / f"{name}.journal" + path.write_bytes(mutated) + values = [ + next( + bytes.fromhex(payload) + for payload in row["payloads"] + if bytes.fromhex(payload).startswith(b"MESSAGE=") + ) + for row in rows + ] + probe( + path, + operation, + error=error, + expected=[value[8:].hex() for value in values] + if not error + else None, + **options, + ) + if operation == "unique": + probe( + path, + "unique-state", + error=error, + expected=[value.hex() for value in values] + if not error + else None, + **options, + ) + if operation != "explorer": + probe( + path, + f"facade-{operation}", + error=error, + expected=[value[8:].hex() for value in values] + if not error + else None, + **options, + ) + if operation in ("fields", "unique"): + directory = folder / f"{name}-directory" + directory.mkdir() + (directory / "system.journal").write_bytes(mutated) + probe( + directory, + operation, + error=error, + directory=True, + expected=[value[8:].hex() for value in values] + if not error + else None, + **options, + ) + if operation == "unique": + probe( + directory, + "unique-state", + error=error, + directory=True, + expected=[value.hex() for value in values] + if not error + else None, + **options, + ) + probe( + path, + "facade-unique-state", + error=error, + expected=[value.hex() for value in values] + if not error + else None, + **options, + ) + common = next( + at + for at in parsed["data_offsets"] + if data[at + payload_header : at + read_u64(data, at + 8)] + == b"COMMON=all" + ) + array = read_u64(data, common + 48) + for name, pointer in ( + ("posting-inline", common + 40), + ("posting-head", common + 48), + ("posting-next", array + 16), + ): + mutated = bytearray(data) + write_u64(mutated, pointer, 0) + # Force a second posting array only in the missing-next case. + if name == "posting-next": + write_u64(mutated, common + 56, 1_000_000) + path = folder / f"{name}.journal" + path.write_bytes(mutated) + for operation in ( + "explorer-filter", + "explorer-facet", + "explorer-histogram", + ): + probe(path, operation, field="COMMON", error=True) + for operation in ( + "forward", "reverse", "facade-forward", "facade-reverse" + ): + probe( + path, operation, match_hex=b"COMMON=all".hex(), error=True + ) + mutated = bytearray(data) + write_u64(mutated, parsed["entry_array_offset"] + 16, 0) + write_u64(mutated, 152, 1_000_000) + path = folder / "missing-global-next.journal" + path.write_bytes(mutated) + for operation in ("forward", "reverse", "facade-forward", "facade-reverse"): + probe( + path, + operation, + expected=list(reversed(rows)) if "reverse" in operation else rows, + recovery=True, + declared=1_000_000, + ) + probe(path, "verify", error=True) + if "streams" in selected: + at = messages[0] + start, end = at + payload_header, at + read_u64(data, at + 8) + payload = data[start:end] + half = len(payload) // 2 + frame = zstd.frame(payload, checksum=True) + bad = bytearray(zstd.frame(payload[half:], checksum=True)) + bad[-1] ^= 1 + streams = { + "known": (frame, False), + "unknown": (zstd.frame(payload, known=False), False), + "concatenated": ( + zstd.frame(payload[:half]) + zstd.frame(payload[half:]), + False, + ), + "mixed-skippable": ( + skippable(7) + + zstd.frame(payload[:half], known=False) + + skippable() + + zstd.frame(payload[half:]), + False, + ), + "empty-first": (zstd.frame(b"") + frame, False), + "bad-later-checksum": (zstd.frame(payload[:half]) + bytes(bad), True), + "truncated-later-frame": ( + zstd.frame(payload[:half]) + zstd.frame(payload[half:])[:-3], + True, + ), + "oversized": (oversized_zstd_frame(768 * 1024 * 1024 + 1), True), + "trailing-garbage": (frame + b"garbage", True), + } + for name, (stream, error) in streams.items(): + padding = end - start - len(stream) - 8 + if padding < 0: + raise RuntimeError("fixture lacks padding capacity") + mutated = bytearray(data) + mutated[at + 1] = 4 + mutated[12:16] = (int.from_bytes(data[12:16], "little") | 8).to_bytes( + 4, "little" + ) + mutated[start:end] = stream + skippable(padding) + path = folder / f"zstd-{name}.journal" + path.write_bytes(mutated) + probe(path, "forward", expected=None if error else rows, error=error) + probe(path, "verify", error=error) + stock_verify(path, error) + split = len(data) // 2 + whole = ( + zstd.frame(data[:split], known=False) + + skippable(9) + + zstd.frame(data[split:]) + ) + path = folder / "whole.journal.zst" + path.write_bytes(whole) + probe(path, "forward", expected=rows) + path = folder / "truncated-whole.journal.zst" + path.write_bytes(whole[:-3]) + probe(path, "forward", error=True) + if "recovery" in selected: + source = folder / "recovery-base.journal" + original, graph, expected = fixture(source, 1005, compact, env) + clear_directory = folder / "clear-matches-directory" + clear_directory.mkdir() + (clear_directory / "system.journal").write_bytes(original) + for path, is_directory in ((source, False), (clear_directory, True)): + for operation in ("flush", "facade-flush"): + probe(path, operation, expected=expected, directory=is_directory, + match_hex=b"SPARSE=some".hex()) + item = 4 if compact else 8 + last_array = graph["entry_array_offset"] + while read_u64(original, last_array + 16): + last_array = read_u64(original, last_array + 16) + for name, cut in ( + ("lost-entry", graph["entry_offsets"][-1]), + ("lost-array", last_array), + ("partial-array", last_array + 24 + 3 * item), + ): + prefix = [] + remaining, array = 1005, graph["entry_array_offset"] + while remaining and array + 24 <= cut: + capacity = (read_u64(original, array + 8) - 24) // item + used = min(remaining, capacity) + for i in range(used): + slot = array + 24 + i * item + entry = int.from_bytes(original[slot : slot + item], "little") + if ( + slot + item > cut + or entry + read_u64(original, entry + 8) > cut + ): + break + prefix.append(expected[len(prefix)]) + else: + remaining -= used + array = read_u64(original, array + 16) + continue + break + path = folder / f"{name}.journal" + path.write_bytes(original[:cut]) + probe(path, "forward", expected=prefix, recovery=True, declared=1005) + probe( + path, + "facade-forward", + expected=prefix, + recovery=True, + declared=1005, + ) + probe( + path, + "reverse", + expected=list(reversed(prefix)), + recovery=True, + declared=1005, + ) + probe( + path, + "facade-reverse", + expected=list(reversed(prefix)), + recovery=True, + declared=1005, + ) + probe( + path, + "forward", + expected=prefix, + recovery=True, + declared=1005, + match_hex=b"COMMON=all".hex(), + ) + sparse_rows = [ + row for row in prefix if b"SPARSE=some".hex() in row["payloads"] + ] + sparse_data = next( + at + for at in graph["data_offsets"] + if original[at + payload_header : at + read_u64(original, at + 8)] + == b"SPARSE=some" + ) + sparse_complete = posting_prefix_known( + original, + sparse_data, + graph["entry_offsets"][len(prefix) - 1], + cut, + compact, + ) + for operation in ( + "forward", "reverse", "facade-forward", "facade-reverse" + ): + backwards = operation.endswith("reverse") + probe( + path, + operation, + expected=(list(reversed(sparse_rows)) if backwards else sparse_rows) + if sparse_complete else [], + error=not sparse_complete, + recovery=True, + declared=1005, + recovered_count=len(prefix), + match_hex=b"SPARSE=some".hex(), + ) + probe( + path, + operation, + expected=list(reversed(prefix)) if backwards else prefix, + recovery=True, + declared=1005, + recovered_count=len(prefix), + match_hex=b"COMMON=all".hex(), + ) + probe(path, "forward", expected=[], match_hex=b"ABSENT=value".hex()) + probe( + path, + "seek-forward", + expected=[row for row in prefix if row["realtime"] >= 2000], + recovery=True, + declared=1005, + recovered_count=len(prefix), + realtime=2000, + ) + directory = folder / f"{name}-directory" + directory.mkdir() + (directory / "system.journal").write_bytes(original[:cut]) + probe( + directory, + "forward", + expected=prefix, + recovery=True, + declared=1005, + directory=True, + ) + for operation in ("forward", "reverse", "facade-forward", "facade-reverse"): + backwards = operation.endswith("reverse") + probe( + directory, + operation, + expected=(list(reversed(sparse_rows)) if backwards else sparse_rows) + if sparse_complete else [], + error=not sparse_complete, + recovery=True, + declared=1005, + recovered_count=len(prefix), + match_hex=b"SPARSE=some".hex(), + directory=True, + ) + probe(path, "verify", error=True) + report = { + "oracle": subprocess.run( # nosec B603 - explicit oracle and argument vector. + [str(oracle), "--version"], + capture_output=True, + text=True, + env=env, + check=True, + ).stdout, + "observations": observations, + "status": "PASS", + } + (out / "report.json").write_text(json.dumps(report, indent=2) + "\n") + print(json.dumps({"status": "PASS", "observations": len(observations)}), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tests/interoperability/run_verify_matrix.py b/tests/interoperability/run_verify_matrix.py index da024aac..36d89540 100755 --- a/tests/interoperability/run_verify_matrix.py +++ b/tests/interoperability/run_verify_matrix.py @@ -145,6 +145,7 @@ def build_env() -> dict[str, str]: env.setdefault("GOPATH", str(local / "go")) env.setdefault("CARGO_HOME", str(local / "cargo-home")) env.setdefault("CARGO_TARGET_DIR", str(local / "cargo-target")) + env["CARGO_TARGET_DIR"] = str((REPO_ROOT / env["CARGO_TARGET_DIR"]).resolve()) return env @@ -154,7 +155,7 @@ def build_tools() -> dict[str, str]: go_journalctl = BIN_DIR / "go-journalctl" require_ok( - run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env=env), + run(["go", "build", "-o", str(go_journalctl), "./cmd/journalctl"], cwd=REPO_ROOT / "go", env={**env, "CGO_ENABLED": "1"}), "build go journalctl", ) require_ok( @@ -167,8 +168,9 @@ def build_tools() -> dict[str, str]: ) rust_src = Path(env["CARGO_TARGET_DIR"]) / "debug" / "journalctl" rust_journalctl = BIN_DIR / "rust-journalctl" - if rust_src.exists(): - shutil.copy2(rust_src, rust_journalctl) + if not rust_src.is_file(): + raise RuntimeError(f"expected Rust build output not found: {rust_src}") + shutil.copy2(rust_src, rust_journalctl) for path in (go_journalctl, rust_journalctl): if not path.exists(): diff --git a/tests/interoperability/test_journalctl_oracle.py b/tests/interoperability/test_journalctl_oracle.py new file mode 100644 index 00000000..632353cf --- /dev/null +++ b/tests/interoperability/test_journalctl_oracle.py @@ -0,0 +1,45 @@ +"""Oracle selection checks without building tools or accessing host journals.""" +import subprocess # nosec B404 - CompletedProcess test values only; execution is mocked. +import unittest +from pathlib import Path +from unittest.mock import patch + +import run_journalctl_query_matrix as matrix + + +class OracleSelectionTests(unittest.TestCase): + def test_version_uses_selected_binary_and_required_features(self): + output = "systemd 262 (262)\n+PCRE2 +ZSTD +XZ +LZ4\n" + with patch.object(matrix, "run", return_value=subprocess.CompletedProcess([], 0, output, "")) as run: + self.assertEqual(matrix.systemd_version("/selected/journalctl"), "systemd 262 (262)") + run.assert_called_once_with(["/selected/journalctl", "--version"], timeout=10) + + def test_rejects_other_version_or_missing_feature(self): + for output in ["systemd 260\n+PCRE2 +ZSTD +XZ +LZ4", "systemd 262\n-PCRE2 +ZSTD +XZ +LZ4"]: + with self.subTest(output=output), patch.object(matrix, "run", return_value=subprocess.CompletedProcess([], 0, output, "")): + with self.assertRaises(RuntimeError): + matrix.systemd_version("/selected/journalctl") + + def test_all_command_builders_require_explicit_oracle(self): + tools = {"stock_journalctl": "/selected/journalctl"} + commands = [matrix.reader_command("stock", tools, "file", Path("fixture"), []), + matrix.raw_reader_command("stock", tools, "file", Path("fixture"), []), + matrix.action_command("stock", tools, ["--version"])] + self.assertTrue(all(command[0] == tools["stock_journalctl"] for command in commands)) + with self.assertRaises(KeyError): + matrix.reader_command("stock", {}, "file", Path("fixture"), []) + + def test_empty_cases_use_explicit_oracle(self): + calls = [] + def run(command, **kwargs): + calls.append(command) + return subprocess.CompletedProcess(command, 0, "", "") + tools = {"stock_journalctl": "/selected/journalctl", "go_journalctl": "/selected/go", "rust_journalctl": "/selected/rust"} + with patch.object(matrix, "run", side_effect=run): + matrix.run_empty_result_cases(tools, {"file": Path("fixture")}) + self.assertTrue(calls) + self.assertTrue(all(command[0] in tools.values() for command in calls)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/parser-parity/check_v260_manifest.py b/tests/parser-parity/check_v260_manifest.py index 067f2fc1..2106ea0a 100644 --- a/tests/parser-parity/check_v260_manifest.py +++ b/tests/parser-parity/check_v260_manifest.py @@ -1,9 +1,9 @@ #!/usr/bin/env python3 -"""Cross-check the committed manifest against the systemd v260.1 source. +"""Cross-check the committed manifest against the systemd v262 source. Reads `.agents/sow/specs/journalctl-v260-parity-matrix.md` and `tests/parser-parity/v260-manifest.json` and verifies that every official -v260.1 long option, short option, output mode, and JournalctlAction enum +v262 long option, short option, output mode, and JournalctlAction enum value is represented in both. The systemd source reference is read-only. The script is part of the @@ -22,12 +22,12 @@ REPO_ROOT = Path(__file__).resolve().parents[2] MANIFEST = REPO_ROOT / "tests" / "parser-parity" / "v260-manifest.json" SPEC = REPO_ROOT / ".agents" / "sow" / "specs" / "journalctl-v260-parity-matrix.md" -SOURCE_COMMIT = "c0a5a2516d28" +SOURCE_COMMIT = "8cc40e0c5e9234bf45084751ac53b1fbfe70b492" SYSTEMD_ROOT = Path(os.environ.get("SYSTEMD_SOURCE", "~/src/systemd.git")).expanduser() def git_show(path: str) -> str: - """Read a file from the systemd v260.1 tag in the read-only mirror.""" + """Read a file from the systemd v262 tag in the read-only mirror.""" result = subprocess.run( # nosec B603 B607 ["git", "show", f"{SOURCE_COMMIT}:{path}"], cwd=str(SYSTEMD_ROOT), @@ -38,38 +38,35 @@ def git_show(path: str) -> str: return result.stdout -def official_long_options() -> set[str]: - """Extract `{ "name", ... }` entries from journalctl.c:420.""" +def official_options_source() -> str: + """Read the regular namespace and expand only its referenced common macros.""" source = git_show("src/journal/journalctl.c") - in_table = False - names: set[str] = set() - for line in source.splitlines(): - stripped = line.strip() - if not in_table: - if stripped.startswith("static const struct option options[]"): - in_table = True - continue - if stripped.startswith("{}"): - break - match = re.match(r'\{\s*"([^"]+)"', stripped) - if match: - names.add(match.group(1)) - return names + start = source.index('OptionParser opts = { argc, argv, .namespace = "journalctl" };') + source = source[start:source.index("char **args =", start)] + common = git_show("src/shared/options.h") + for macro in set(re.findall(r"\bOPTION_COMMON_[A-Z_]+\b", source)): + match = re.search(r"^#define " + macro + r"\s+(.*?)(?=^#define|\Z)", common, + re.MULTILINE | re.DOTALL) + if not match: + raise RuntimeError("missing common option macro: " + macro) + source = source.replace(macro, match.group(1).replace("\\\n", " ")) + return re.sub(r"/\*.*?\*/", "", source, flags=re.DOTALL) + + +def official_long_options() -> set[str]: + source = official_options_source() + patterns = [r'\bOPTION\(\s*\'[^\']\'\s*,\s*"([^"]+)"', + r'\bOPTION_FULL\([^,]+,\s*\'[^\']\'\s*,\s*"([^"]+)"', + r'\bOPTION_LONG\(\s*"([^"]+)"', + r'\bOPTION_LONG_FLAGS\([^,]+,\s*"([^"]+)"'] + return {name for pattern in patterns for name in re.findall(pattern, source)} def official_short_options() -> set[str]: - """Extract the short-option string from journalctl.c:520.""" - source = git_show("src/journal/journalctl.c") - match = re.search(r'getopt_long\(argc,\s*argv,\s*"([^"]+)"', source) - if not match: - raise RuntimeError("failed to locate short-option string in journalctl.c") - chars = match.group(1) - out: set[str] = set() - for ch in chars: - if ch == ":": - continue - out.add(ch) - return out + source = official_options_source() + patterns = [r"\bOPTION(?:_SHORT)?\(\s*'([^'])'", + r"\bOPTION_FULL\([^,]+,\s*'([^'])'"] + return {letter for pattern in patterns for letter in re.findall(pattern, source)} def official_output_modes() -> set[str]: @@ -160,7 +157,7 @@ def main(): for line in failures: print(f" - {line}") sys.exit(1) - print("OK: manifest matches systemd v260.1 official surface") + print("OK: manifest matches systemd v262 official surface") if __name__ == "__main__": diff --git a/tests/parser-parity/run_parser_parity.py b/tests/parser-parity/run_parser_parity.py index 9ae40d2b..a9dbe37d 100644 --- a/tests/parser-parity/run_parser_parity.py +++ b/tests/parser-parity/run_parser_parity.py @@ -2,7 +2,7 @@ """Shared parser parity runner. Validates that both Rust and Go journalctl commands recognize every -official systemd v260.1 option, output mode, and parser interaction. +official systemd v262 option, output mode, and parser interaction. For each long option we build a syntactically valid invocation (using a synthetic empty path or `--file=...` placeholder as required) and verify: @@ -22,7 +22,7 @@ verify the parser accepts the mode (no "unknown output format" error). For each parser interaction rule we run a synthetic argument combination -and verify the command reports the official v260.1 conflict error class. +and verify the command reports the official v262 conflict error class. Runtime artifacts stay under `.local/parser-parity/`. The harness only inspects generated output and the in-repo Rust/Go binaries; it never @@ -394,7 +394,7 @@ def exercise_help_stdout(cmd: Command, manifest: dict) -> CaseResult: missing = [ f"--{opt['name']}" for opt in manifest["long_options"] - if f"--{opt['name']}" not in result.stdout + if not opt.get("hidden", False) and f"--{opt['name']}" not in result.stdout ] if not missing: return CaseResult( @@ -425,6 +425,10 @@ def exercise_interaction(cmd: Command, interaction: dict) -> CaseResult: require full journal semantics beyond what the placeholder can verify. """ name = interaction["name"] + if name == "unscoped-field-enumeration": + result = run_command(cmd, ["--file=placeholder.journal", "--fields", "--since=@0"]) + ok = result.returncode != 0 and "cannot be combined with options that limit the journal" in result.stderr + return CaseResult(name, "parser-required", "ok" if ok else "fail", result.stderr.strip()[:300]) if name == "time-bounds-order": # Use the `--since` later than `--until` conflict. try: @@ -850,6 +854,7 @@ def build_go_binary(go_root: Path) -> Path: LOCAL_DIR.parent.mkdir(parents=True, exist_ok=True) cmd = ["go", "build", "-o", str(bin_path), "./cmd/journalctl"] env = os.environ.copy() + env["CGO_ENABLED"] = "1" env.setdefault("GOCACHE", str(REPO_ROOT / ".local" / "go-build")) env.setdefault("GOMODCACHE", str(REPO_ROOT / ".local" / "go-mod-cache")) subprocess.run( # nosec B603 B607 @@ -909,7 +914,7 @@ def main(): rust_bin = args.rust_bin go_bin = args.go_bin rust_cmd = Command(name="rust", path=rust_bin, flag_style="gnu-long") - go_cmd = Command(name="go", path=go_bin, flag_style="go-flag") + go_cmd = Command(name="go", path=go_bin, flag_style="gnu-long") print(f"rust binary: {rust_cmd.path}") print(f"go binary: {go_cmd.path}") diff --git a/tests/parser-parity/v260-manifest.json b/tests/parser-parity/v260-manifest.json index 4d2cd8bd..5e2381da 100644 --- a/tests/parser-parity/v260-manifest.json +++ b/tests/parser-parity/v260-manifest.json @@ -1,6 +1,6 @@ { - "baseline": "systemd v260.1", - "source_commit": "c0a5a2516d28", + "baseline": "systemd v262", + "source_commit": "8cc40e0c5e9234bf45084751ac53b1fbfe70b492", "long_options": [ { "name": "system", @@ -498,6 +498,14 @@ "classification": "recognized-unsupported", "expectation": "recognized-unsupported", "short_alias": null + }, + { + "name": "introspect-cli", + "args": "none", + "classification": "portable-utility-required", + "expectation": "recognized-supported", + "short_alias": null, + "hidden": true } ], "short_options": [ @@ -649,6 +657,7 @@ "ACTION_LIST_NAMESPACES", "ACTION_FLUSH", "ACTION_RELINQUISH_VAR", + "ACTION_SMART_RELINQUISH_VAR", "ACTION_SYNC", "ACTION_ROTATE", "ACTION_VACUUM", @@ -714,6 +723,10 @@ { "name": "case-sensitive-bool-spellings", "description": "Accept official parse_boolean spellings t and f for --case-sensitive=." + }, + { + "name": "unscoped-field-enumeration", + "description": "Reject final -F/-N actions with explicit filters/time/cursor/grep; accept cleared boot/invocation=all and implicit boot." } ] } diff --git a/tests/parser-parity/v260-manifest.py b/tests/parser-parity/v260-manifest.py index 3bca5d31..8f46d16d 100644 --- a/tests/parser-parity/v260-manifest.py +++ b/tests/parser-parity/v260-manifest.py @@ -1,9 +1,10 @@ #!/usr/bin/env python3 -"""Shared v260.1 journalctl parser parity manifest. +"""Shared v262 journalctl parser parity manifest (historical filename retained). Derived from the authoritative option/action list in -`systemd/systemd @ c0a5a2516d28` (`v260.1`) -`src/journal/journalctl.c:420` (long-option table), +`systemd/systemd @ 8cc40e0c5e9234bf45084751ac53b1fbfe70b492` (`v262`) +`src/journal/journalctl.c:383` (ordered option declarations), +`src/shared/options.h` (referenced common option macros), `src/shared/output-mode.c:26` (output mode names), and `src/journal/journalctl.h:7` (JournalctlAction enum). @@ -16,8 +17,8 @@ ```json { - "baseline": "systemd v260.1", - "source_commit": "c0a5a2516d28", + "baseline": "systemd v262", + "source_commit": "8cc40e0c5e9234bf45084751ac53b1fbfe70b492", "long_options": [ { "name": "verify", @@ -56,7 +57,7 @@ def build_long_options(): - """Every official v260.1 long option, with portable expectation.""" + """Every official v262 long option, with portable expectation.""" # Tuple format: (long_name, args, classification, expectation, optional_alias_short) rows = [ # Source options @@ -135,6 +136,7 @@ def build_long_options(): ("list-catalog", "none", "recognized-unsupported", "recognized-unsupported", None), ("dump-catalog", "none", "recognized-unsupported", "recognized-unsupported", None), ("update-catalog", "none", "recognized-unsupported", "recognized-unsupported", None), + ("introspect-cli", "none", "portable-utility-required", "recognized-supported", None), ] out = [] @@ -148,11 +150,13 @@ def build_long_options(): "short_alias": short_alias, } ) + if name == "introspect-cli": + out[-1]["hidden"] = True return out def build_short_options(): - """Official short option letters from journalctl.c:520.""" + """Official short option letters from the ordered v262 declarations.""" return [ {"letter": "h", "long": "help"}, {"letter": "e", "long": "pager-end"}, @@ -186,7 +190,7 @@ def build_short_options(): def build_output_modes(): - """Every official v260.1 output mode string.""" + """Every official v262 output mode string.""" return [ "short", "short-full", @@ -208,7 +212,7 @@ def build_output_modes(): def build_actions(): - """Every official v260.1 JournalctlAction enum value.""" + """Every official v262 JournalctlAction enum value.""" return [ "ACTION_SHOW", "ACTION_NEW_ID128", @@ -226,6 +230,7 @@ def build_actions(): "ACTION_LIST_NAMESPACES", "ACTION_FLUSH", "ACTION_RELINQUISH_VAR", + "ACTION_SMART_RELINQUISH_VAR", "ACTION_SYNC", "ACTION_ROTATE", "ACTION_VACUUM", @@ -234,7 +239,7 @@ def build_actions(): def build_parser_interactions(): - """Parser interaction rules from journalctl.c:1085.""" + """Parser interaction rules from the v262 post-option validation.""" return [ { "name": "source-exclusivity", @@ -302,13 +307,17 @@ def build_parser_interactions(): "name": "case-sensitive-bool-spellings", "description": "Accept official parse_boolean spellings t and f for --case-sensitive=.", }, + { + "name": "unscoped-field-enumeration", + "description": "Reject final -F/-N actions with explicit filters/time/cursor/grep; accept cleared boot/invocation=all and implicit boot.", + }, ] def main(): out = { - "baseline": "systemd v260.1", - "source_commit": "c0a5a2516d28", + "baseline": "systemd v262", + "source_commit": "8cc40e0c5e9234bf45084751ac53b1fbfe70b492", "long_options": build_long_options(), "short_options": build_short_options(), "output_modes": build_output_modes(),