Repository navigation
Conversation
…corpus The server's wrapper (docs/implementation/) starts from the engine itself: a Linux image built from this repo's own scripts (vendor, llama.cpp, ffmpeg), the CLI verified headless in the final image, and Stage 2's proof indexing the pinned Commons corpus and answering natural-language searches with the engine's numbers. Additive: only .gitignore gains lines, and .dockerignore keeps host artifacts out of the build context.
The server talks to DigUp the way the Tauri shell does: spawn digup search-service / index-service / worker as supervised children (the ParentWatch rule), speak their JSON-lines protocols, read status/estimate from the CLI. The engine is never linked, patched, or replaced; the contract tests pin every reply field, event, and exit path the adapter uses, against the shipped Linux payload.
POST /index starts one serialized job (409 busy with the live id); the runner forwards the engine's stop at the safe point, marks interrupted records with partial counts, replays its append-only log across rotations at startup, and reloads+prewarms the searcher after a done job. Root-scope semantics pinned by tests: plain sync retires walked-root ghosts; prune retires out-of-scope ghosts.
FastAPI with typed wire models, the engine's answers verbatim (hits, strong, meaning, ms, warning, the honest empty), engine failures as 5xx quoting the engine, config from the environment only — paths never arrive from a client. localhost by default; in the container the publish line enforces that. The lifespan owns the search-service as a direct child, prewarms at startup, and ends every helper on shutdown.
… by id Vanilla HTML/CSS/JS served same-origin (no build step, no external URLs, pinned by test). Media endpoints take only server-issued media_id handles: /preview (the engine's render JPEGs, symlink-safe to <index>/previews), /text (the passage it indexed), and /file — the file's real bytes, typed by magic bytes, Accept-Ranges and 206 for video seeking (#t=), inline for media kinds, attachment text/plain for code/doc. The UI mirrors shell/ui: keyword-first typing (prefix_last words, then the meaning upgrade after 120 ms with embed_ahead), strong/weaker fold, kind filters, code mode, limit, selection with arrows, Space big look, Open at the video's moment, Estimate panel, activity dot, keys footer — plus the System/Light/Dark appearance choice kept in localStorage, which the OS-following desktop needs no equivalent of. Settings that are OS integrations are documented as deployment env config; reveal and copy are absent, not faked.
server-runtime: engine payload + API + UI on python:3.12-slim, tini as PID 1 (the ParentWatch chain tini->uvicorn->helpers), non-root uid 990, HEALTHCHECK asking the API's own /health, startup prewarm. Compose publishes 127.0.0.1:8787, names its project digup-server, keeps index/models/config/job-log in bind-mounted volumes, and sets stop_grace_period: 90s so a mid-job docker stop finishes the safe-point path (143, interrupted with partial counts) — the 15s default's SIGKILL path recovers by log replay and resumes, both measured (docs stage 8).
The suite's corpus stays exactly 15 files (its rankings normalize over that set), so new test material lives beside it: 20 subjects x 3 Commons photographs fetched as 1400 px renders, pinned in manifest.tsv, out of git, built by build.sh; the H.264 scenes clip travels with it. Provenance and licenses documented; it is testing material for the running server (Compose mount note in its README), not suite input.
…unbook docs .github/workflows/server.yml: build the builder and runtime images, run swift-side payload checks, materialize the corpus, download the pinned models, run the contract suite against the shipped payload, validate the Compose schema, and assert the SIGTERM contract (exit != 137, shutdown complete). The drift-guard step compares ModelFiles.swift's sha256 constants with the workflow's download pins, so an upstream model bump fails CI loudly before the server can silently run a different model. server/README.md and IMPLEMENTATION_STATUS.md carry the runbooks, the merge policy, and what is proven.
…ettings Palettes explicit per theme (measured WCAG ratios), color-scheme per theme, the big-look overlay guarded (#big[hidden]) — it sat at 78% black over the whole page. Search box centered with the hit-limit select at its right end, kind chips as a themed, evenly spaced pill row, gear opens the settings panel: appearance, Index now + prune, Estimate (one honest note line), and the engine's counts. el() takes real child nodes (the [object HTMLElement] text is gone); statusline shows the searchable count by the title.
…e; MP4 by ftyp An ok:false reply (SearchService.swift:106,115) is EngineRefused: raised in the adapter, answered 503 with the engine's sentence — a refusal reply has no hits, and api.py used to crash KeyError on it (measured 500s in the Stage-10 battery). raw() keeps passing the engine's replies verbatim; the runner survives a refused post-job refresh (the job stays done, with a note). The UI ships cache-control: no-cache — measured: without it Chrome served a stylesheet the server no longer had. The magic sniff reads ftyp at offset 4 and the brand there, so scenes.mp4 is video/mp4, not octet-stream.
…d fixes Corpus2 gallery served live: job 0.4 s idempotent (61 files, 65 vectors), six meaning searches all strong 3 (top z 4.46), /file 200·206·416·404 on real bytes, video range + preview frame, estimate ≈ 263.9 s calibrated. Suite: 57 passed, 1 deselected, 241.18 s. The proof battery gates on readiness and retries 503s. .gitignore: the Playwright MCP working folder.
Single-column system: header grid (title left, gear right, the searchable count centered), search row 1fr auto 1fr so the input's center IS the page's center with limit + code at its right end, footer text aligned to the content column, settings anchored under the gear (right edges equal, controls left edges at one x) as a label/control grid with section rules and a close button. The gear is an inline SVG centered 0/0 in its circle, colored by currentColor. Measured at 1280 px and 390 px: center offsets 0, overflowX 0. The gear's svg broke the outside-click close (target !== gear) — closest(). Suite (20): 57 passed, 1 deselected, 236.12 s.
Nine real findings from the adversarial backend review, each fixed with the scenario the reviewer described pinned as a test: - jobs.shutdown() self-deadlocked (a plain Lock re-taken by _mark from inside its own with-block): a docker stop with a job past the join grace hung the lifespan forever. The active job is now read under the lock and marked outside it. - the runner thread died on a TimeoutError from the post-job prewarm (not a RuntimeError): _active stayed set, every later POST /index answered 409 naming a job that could never run. Any escape from _run_one now marks the job interrupted, clears the writer, and keeps the thread alive. - a failed search-service handshake leaked the child (pipes + reader threads unreferenced until its 600 s idle-exit); _spawn_locked reaps on every failure and maps a missing binary to CLIError, so the API quotes the spawn as a 502 instead of dying as a 500. - ok:false WITHOUT an error field passed the refusal guard and hit the API's stripped["hits"] as a KeyError — the guard fires on the shape now. - render held the serialize lock across the engine's ffmpeg work, undoing the engine's own design (render runs beside the read loop on two slots, replies routed by id, "typing never waits on one"). Two slots, lock-free render: a 1.2 s frame no longer freezes keystroke-searches. - Range: bytes=0- (what every media element sends first) buffered the whole file; open/suffix/huge ranges are capped at 8 MiB and streamed from the span's own handle. Type, size and span come from one open (fstat + S_ISREG), root confinement is is_relative_to, and every /file answer carries nosniff. - subprocess.TimeoutExpired is a SubprocessError, not a TimeoutError: mapped in cli.run so a stalled status/estimate quotes the engine (502), not a traceback. - job-log rotation shifted a .1…N chain (O(n) renames, order scramblable by a kill mid-shift: a running line read after its done reported finished jobs as interrupted). One rename to a time_ns suffix; replay is a sort. Replay skips valid JSON that is not a job record (5, null, a record with no id used to brick app start). - HitCache.get() didn't refresh, so eviction was insertion-order: a browsing session evicted its own on-screen handles after ~26 searches (404s, no recovery). move_to_end makes it the LRU the docstring promised. - /status and /health each spawned a digup status (real SQLite counting) per poll; the identity fields ride a StatusCache TTL (DIGUP_SERVER_STATUS_TTL_S, 0 = read every time). SearchReply is extra="allow": engine fields this model has never seen are published, not dropped. Suite: 75 passed, 1 deselected, zero skipped in 228.12 s in digup-server-test:local.
…yncs its aria
The UI review's blockers, verified in a live browser against the running stack:
- Space hijacked every button. Native activation fires on keyup; the document
handler (Quick Look) preventDefault()ed on keydown for everything except a
few tags, so the gear, the close, Index now, Estimate, "all", and the
Open/Download links stopped activating for keyboard users — while the big
look flashed open over the page. Activation targets (closest() over
button/a/[role=button]/input/select/textarea) keep Space; bare page areas
preview. Live: Space on the focused gear opens Settings; Space on a focused
row opens the big look; Space on a focused link is the browser's own.
- a failed job poll killed the loop (unhandled rejection): the job line froze
at its last state and "indexing…" stayed in the footer forever. Polling now
answers "job poll failed: …" honestly, clears the indexing state, and stops;
repeated busy-clicks poll one loop (state.jobPoll), not N.
- FastAPI's list-shaped validation detail printed "[object Object]" on the
footer; api() renders each {loc, msg} and falls back to statusText for a
non-JSON body.
- every settings-close path shares closeSettings(): aria-expanded always
matches the panel's hidden state, opening focuses the first control, closing
returns focus to the gear. Estimate's panel has a generation token, so a
slow crawl's late reply can't write into a panel closed and reopened between.
- roving tabindex (selected 0, rest -1: 40-100 rows no longer turn every
search into a tab storm), aria-selected on rows, the weaker fold says
role=option + aria-expanded/aria-selected, group titles are role=presentation.
- Space in the field on the weaker row toggles the fold (as the document
handler does), the code hint shows for the code-mode empty, and the query
field carries maxlength=512 matching the API's cap.
- CSS: the ticked kind chips mix at 10% (the dark audio pair measured 4.3:1 at
14%, 4.54:1 now — AA), .badge.warn wins the cascade it was losing to .badge,
the audio control is sized where app.js puts it (.audio-picture), and the
keys line unwraps under 720px. A data: favicon silences the browser's
standing 404 in the console (no external URL: the pin tests stay green).
- the test image carries tini as its ENTRYPOINT. The suite's app spawns search-service as a child of the test process, and ParentWatch ends a helper whose parent is PID 1: run the image bare and pytest IS PID 1, so every helper died at once (`exit 0`, no stderr — 59 setup errors, measured). With tini as PID 1 the test process is a live parent, the production shape, and CI needs no --init. - HEALTHCHECK is server/docker/healthcheck.py: 200 healthy, 503 ALIVE (the API answers 503 while the searcher respawns — Docker must not restart a live server over the engine's own refusal; urlopen raises HTTPError on 503 so it is caught), everything else exit 1. The build asserts the script fails with nothing listening; CI polls the live server with the same file. - measured honesty in the Dockerfile: base is Debian 13 (trixie), glibc 2.41-12+deb13u4 (newer than the payload's 2.35 build glibc — forward, never back); no /data/locks (the writer lock is <index>/.digup-server.writer.lock); the inert DIGUP_SERVER_HOST/PORT ENV lines are gone (uvicorn's CMD args win in a container; those envs feed the bare-metal `python -m` entry); `digup --help` claims only that the CLI runs headless — §3's full acceptance is CI's boot step and the proof battery, against a live index. - CI: 180-minute ceiling; the engine chain (ffmpeg, llama.cpp) is built host-side with the repo-pinned actions/cache and the same hash keys ci.yml uses, then seeded into the builder's cache mount through .ci-vendor/ (the vendor scripts land on their own stamps and skip the rebuild; hosted runners start their BuildKit mounts cold, and a fresh 15-minute ffmpeg compile per PR was overrunning the old ceiling). The drift guard runs FIRST and pins ModelFiles.revision + both file names + both sha256s. Zero-skip gate (any SKIPPED line is red; the clip must exist). Compose schema check. The boot step polls /health with the image's own healthcheck.py, asserts exit != 137 + 'Application shutdown complete', and uploads proof/suite logs. - proof.sh counts failures: it piped every gate through tee and exited 0 unconditionally, so a run where the index died and every search answered nothing was 'green'. Gates now record their labels, the pipeline's status is real (pipefail), and the eval ALL row must clear R@1 >= 0.60 / R@5 >= 0.85 (the recorded 0.71/0.88 clear it with margin) before it prints PASSED. - compose: restart: unless-stopped, and DIGUP_PUBLISH_PORT for the host side when 8787 is taken. localhost-only publish unchanged. - .dockerignore excludes the corpus videos/clips, corpus2 media (45 MB rode into the build context) and .playwright-mcp/; .gitignore ignores .ci-vendor/* with a !-exception for the tracked .gitkeep the COPY needs.
- server/README.md: the suite's real number (75 passed / 1 deselected / zero skipped, buckets 17+19+6+13+17+4 collected 76) where two stale counts said 48 and 57; the tini rationale for the test image (the ParentWatch PID-1 trap, measured 59 errors); the -m slow clarification; the quick start's mkdir + chown -R 990 for real Linux hosts (Docker creates bind dirs root-owned, the server crashes EACCES on its lock); restart/publish-port notes; the corpus2 scale note (zGate 2.0 is the engine's gate on every platform - 1-3 hits on 61 files is the corpus, not a filter). - IMPLEMENTATION_STATUS: Stage-11 section, newest-first - suite23/24, the parity identity table (adapter on a sqlite3 .backup snapshot == HTTP, identical hits/strong/z on five queries + keyword-first), the proof battery's 11 gates, the browser verification of the keyboard fixes, the CI/image facts, the commit shape, and the honest open items (CI first green pending, Stages 9-10). - docs/implementation/review-round.md: the full finding -> fix -> test table for backend, UI, and build/CI, plus the deliberate non-fixes with reasons (no auth on a localhost-only server; no pixels cap; no fsync per job-log line; test image runs root; the engine's own limits left alone by the PR's rule). - docs/implementation/stage-11-review-round.md: the stage record - the PID-1 discovery, the parity proof with the probe lesson (a read-only mount of a hot WAL index is not a snapshot: 0 hits, probe artifact), the corpus-scale quantification, the battery gates, the browser checks. - Docs honesty pass, measured not asserted: base image is Debian 13 trixie glibc 2.41-12+deb13u4 (newer than the payload's 2.35 build glibc, forward never back), no /data/locks (the writer lock is <index>/.digup-server.writer.lock), the shipped endpoint list (/preview|/text|/file, no settings endpoints), the timestamp-suffixed rotation shape + the 5 new job tests (13 total), the CORS/localhost wording, the media-surface notes. - Host identity out of the stage records (this is a publication): the three host home paths become ~/Projects paths (a clone works from anywhere), and the exact OS/Xcode build codes (fingerprint details) come out. All build/proof commands in the docs are relative or env-var based: $PWD, $HOME, $RUNNER_TEMP, container paths.
Ocean572
changed the title
… process is the design
…he UI asking once
… resume sums them
…easured on the NVIDIA box
…wn the Gate A commands
…1070 cuda-libs assembles the shipped chain by glob (cp -a keeps symlink+target; 5 lib families + libnccl.so.2, 955 M) — the dotted -dev names are not in a runtime base; the Dockerfile rm -fs Vendor/llama-build/CMakeCache.txt before build-llama: a flavor switch on a shared cache mount must re-configure (stale GGML_CUDA=ON killed it, measured 2026-11-04); gpu-handoff.sh takes CUDA_ARCHS as a build-arg (plain numbers only — apt cmake 3.22.1 rejects 6.1 and native alike), checks the payload in /out, waits /health then retries POST, and samples nvidia-smi at 0.15 s across a real index job (PASS peak > 20% or busy-duty >= 10% with the model in card VRAM; a 2 s sampler saw 0-12% on a provably-GPU job); the nvidia compose override carries target server-runtime-cuda + the matching devel/runtime CUDA bases + CUDA_ARCHS so --profile nvidia up -d --build builds the CUDA image on the box that has the card.
…ssed, 0 skipped DIGUP_SERVER_ESTIMATE_TIMEOUT_S (default 120) and DIGUP_SERVER_JOB_TIMEOUT_S (default 3600): the estimate walk and the index job each get their own ceiling — measured 2026-11-04 on a 187k-file HDD tree, the walk crossed the 120 s default (502, honest) and the first job died exactly at the 3600 s runner ceiling with its partial counters (interrupted, 19,007 files, 0 failed — resume proven). Both knobs pinned by suite tests; .env.example + compose document them. Suite repinned to the x86_64 measurements (engine untouched): - wakeboard top-3 is a SET pin, not an order: x86_64 ranks logan 1.571 > vienna 1.531, arm64 ranks vienna first — 0.040 z apart, inside the embedding matmul's cross-platform noise. The portable contract (which three, strong >= 3, meaning) stays pinned; per-platform order is the engine's, relayed verbatim (pinned by the hit-field tests). - the resume test's shutdown window 6.0 s -> 1.0 s: on a warm box the 8-file scratch embed finished inside 6 s (files_done 8/8 at the shutdown instant), so partial < expected was a timing assumption; 1.0 s lands inside the model load on every platform, partial = 0 stays tolerated, the resume arithmetic untouched.
…pose overrides Server.yml gains a publish job: after a green server job, pushes to main and feat/** (plus dispatch) build ghcr.io/<owner-lowercase>/digup-server — :latest + :latest-cuda from main, :<branch> + :<branch>-cuda from a feature branch, so PR ARahim3#184's build is pullable as :feat-server. CUDA shape: nvidia/cuda:12.4.1 devel+runtime build-args, fat CUDA_ARCHS=61,70,75,80,86,89,90 (61 compiles deprecated on 12.4.1; CUDA 13 drops it); pinned actions by commit SHA; separate gha cache scopes; a hygiene step deletes untagged versions (free tier is metered by GB). Never on pull_request — that token is scoped to the base repo's registry. x86_64 only, README §11's rule. Push paths now include Sources/**, Package.swift, scripts/**: the published images republish on engine changes, ghcr never serves a stale engine. deploy/docker-compose.ghcr.yml (build: !reset null + image, DIGUP_IMAGE_TAG knob) and docker-compose.ghcr.nvidia.yml (profile nvidia, :<tag>-cuda, the same device reservation + DIGUP_NVIDIA_GPU pin as the build override): all DIGUP_* env, knobs and volumes unchanged from the base file. All four merge shapes validated with docker compose config --quiet on this box, and CI's schema step now checks the same four shapes. README documents pull-and-run; status doc records the suite counts, pins and registry facts measured 2026-11-05.
…st match its ABI Measured on the fork's first CI run (Server ARahim3#1, 2026-11-04): the host-side Vendor build on the ubuntu-24.04 runner (glibc 2.39) was seeded into the ubuntu:22.04 engine-builder (glibc 2.35); the stamped scripts landed on the seed's stamps and skipped the rebuild, and swift test died in 2 minutes — a host-built libllama/libpdfium cannot load against the container's older glibc. This was the seed's first-ever CI run (upstream main has 0 Server runs), so the design was untested until now. Both jobs now run the same stamped, idempotent scripts inside ubuntu:22.04 (the builder's base), cache the result host-side, and seed .ci-vendor/ exactly as before; the CUDA flavor still re-pays only the GPU compile (the llama stamp embeds $FLAVOR). Verified on this box: clean-clone simulation, 22.04-built seed, engine-builder swift build (48 s) + swift test (116 tests, 16 suites) + CLI check — green.
…easured) Wikimedia blocks Microsoft Azure IP ranges on purpose (Phabricator T420833: all of AS8075 blocked since 2026-04-03, T399485: GitHub Actions runners live in exactly those ranges). Measured on the Server workflow's run ARahim3#2: photos verify from the checkout pins (no network), the manifest's first video row hit 403 seconds in — the on-demand fetch is structurally unrunnable on hosted runners. The videos join the photos' committed-and-pinned precedent (same manifest.tsv sha256 verification, both digests re-checked 2026-11-05): the suite's fixtures are now fully checkout- deterministic, the zero-skip gate has nothing left to fetch, and the pins still catch any upstream re-render. 93 MB is one-time PR weight; files stay under GitHub's limits, .gitignore untouched (git add -f: tracked beats ignored).
…fmpeg never has them Measured on the Server workflow's run ARahim3#3 (2026-11-05): the corpus step died in under a second with ffmpeg's exit 8. The step-4 Vendor cache restores Vendor/ into the runner workspace, build.sh picked its candidate-1 (Vendor/ffmpeg/bin/ffmpeg, executable, real), and the clip command asked it for -c:v libx264 — which scripts/build-ffmpeg.sh never enables: the engine's build is decode-only (--enable-encoder=pcm_s16le,mjpeg,ppm,rawvideo, no mp4 muxer; docs/implementation/stage-1-linux-build.md says so in plain words). The engine never encodes x264; the synthetic clip can only come from a full-featured ffmpeg. Fix: build.sh now probes each candidate's -encoders/-muxers lists and accepts only one that has libx264 and the mp4 muxer — Vendor is probed out, the system ffmpeg (preinstalled on hosted runners, apt/Homebrew on dev boxes) is accepted; an explicit FFMPEG=… is honoured as given. Verified on this box in three shapes: clone with a CI-restored Vendor tree (probe rejects, clip made, exit 0); clone without Vendor (system ffmpeg, clip made); explicit FFMPEG=Vendor (honoured, fails loudly as documented). The corpus README's Mac guidance named the impossible candidate; corrected. Gate step unchanged — it still fails the build plainly if the clip is missing.
…hips none Measured on the Server workflow's run ARahim3#4 (2026-11-05): the corpus step passed and the clip gate failed — build.sh, with its new capability probe (b6235a7), correctly probed out the engine's decode-only Vendor ffmpeg restored into the workspace by the Vendor cache, and found no encoder-capable ffmpeg on the runner: the ubuntu-24.04 hosted image ships NO ffmpeg (its Tools inventory lists MediaInfo; the apt list jumps ftp→g++). The encoders were never on the runner at all — runs ARahim3#2/ARahim3#3 died before any probe existed. Fix: the corpus step installs apt's ffmpeg before build.sh — verified on this box in a clean ubuntu:24.04 container: apt ffmpeg passes the probe (2 'libx264' encoder hits, 1 'mp4' muxer hit, shape-identical list lines). Dev boxes keep using their system ffmpeg; Macs use Homebrew's. The runner-step comment that claimed ffmpeg ships on the host is corrected. No build.sh change needed: the probe did its job exactly as designed, naming the impossible candidate out and leaving a plain gate failure — not a silent skip.
…n-root, the runner dirs deny them Measured on the Server workflow's run ARahim3#5 (2026-11-05): the contract suite passed (4 min 49 s, zero skips — the fixture chain is fully green on hosted runners now), and the headless proof died in 58 s with all 27 gates red. Cause: ownership, not the engine. The engine-runtime and server-runtime images ship USER digup (uid 990); the proof's /index and /out/proof live under $RUNNER_TEMP, dirs the runner owns (mode 775) and dirs docker auto-creates as root — uid 990 can write NEITHER: tee died on 'Permission denied', the index on 'unable to open database file'. The suite passed because its test image has no USER directive (runs as root) — the asymmetry hid the trap until the first step that runs as non-root. Reproduced on this box with the exact CI mounts: runner-shape dirs → PROOF FAILED 27 gates (identical signature); 777 dirs → PROOF PASSED, all files 990-owned, eval ALL row R@1 0.76 / R@5 0.88 over the gates. Fix: proof and boot steps chmod 777 their $RUNNER_TEMP write dirs (CI scratch, by design: containers create files there under host-foreign uids). The proof also gets its own index dir — hermetic: the suite's /index holds root-owned rows from test runs; the engine's proof must start from an empty index. The boot step's data/index + data/config get the same treatment (the writer lock + job log live there, uid 990).
…-builder — CUDA publish now builds Measured on the Server workflow's run ARahim3#6 (2026-11-05): the server job went fully green (12 min 52 s, all 17 steps) and publish pushed the CPU image — ghcr.io/ocean572/digup-server :feat-server is live (anonymous pull verified: token flow, tags list, manifest 200; on this box: docker pull → compose merge with docker-compose.ghcr.yml on port 8788 → Up (healthy), /api/v1/health ok:true engine b11461, UI served at /, docker stop --time 15 → exit 143 + Application shutdown complete). The CUDA image died in 2 min 44 s: build-llama.sh passes CUDA_ARCHS verbatim to -DCMAKE_CUDA_ARCHITECTURES, CMake lists are SEMICOLON-separated, and publish's 61,70,75,80,86,89,90 is ONE invalid arch token — configure died in ~35 s per attempt, three attempts + 60 s of sleeps = 164 s, matching the step to the second. scripts/** is read-only, so the fix lives at the server's own boundary: the engine-builder RUN exports CUDA_ARCHS through tr ',' ';' — plain numbers, native and semicolon lists pass through unchanged. Verified on this box with a configure-only probe (real script's cmake invocation, nvidia/cuda:12.4.1-devel-ubuntu22.04, seeded Vendor): comma shape fails exactly where CI died (ggml-cuda CMakeLists enable_language), semicolon shape prints CONFIGURE-OK with compiler-ID cubins for 61, 70, 75, 80, 86, 89 and 90 — cmake 3.22.1 + nvcc 12.4.1 accept the full fat list. Dockerfile/compose comments updated (the old '6.1 for Pascal' example predates the plain-numbers finding); status doc carries the run ARahim3#6 numbers.
…te, best-effort warning Measured on the Server workflow's run ARahim3#7 (2026-11-05): the server job was green again (12 min 26 s) and publish BUILT AND PUSHED THE CUDA IMAGE — step 10 green in 1 h 28 m 57 s: ghcr.io/ocean572/digup-server now carries feat-server AND feat-server-cuda (anonymous pull verified from this box: token flow, tags list, manifest 200 on both). The CUDA arch fix (7400c82) worked exactly as probed. The only red step: registry hygiene, exit 1 in 1 s. Two bugs, both fixed: (1) the python heredoc body sat deeper than its delimiters, so after the run block's indentation was applied python got leading whitespace on every line — IndentationError, instant exit 1; the body now sits at the same column as the delimiters, verified locally by extracting the run block from the YAML and executing it (reaches the API: 401 with a dummy token, warning printed, exit 0). (2) GitHub's 2022 verb migration: deleting package versions is POST /user/packages/{type}/{name}/versions/delete — the old DELETE answers 404. Hygiene is best-effort from now on: publish has already succeeded at that point, so cleanup problems print a loud ::warning and exit 0 — never a red run, never silence.
…070 (healthy 11 s, 1780 MiB resident, exit 143) + hygiene-step fixes verified
… tags embed the pushed engine
… 131,972 files / 131,758 searchable / 403,831 vectors / 1.39 GB Job 3 record (engine's own): started 01:32:33 → ended 07:13:03 box-local, 20,225.0 s, files_done 7,097, files_failed 1, errors 0. Whole library across the three kept records: 131,543 files completed, 421 recorded failures (video decode refusals), 0 server-side errors; engine working time 59,940 s = 16 h 39 min, 2.195 files/s average — 33.75% of the estimate forecast's 177,575.3 s (GPU-assisted). Three verified ceiling-stop/resume cycles. Sampler peaks: 99% GPU (job 2) / 90% (job 3), model resident 1,780–1,802 MiB; container 207.75% CPU, 3.436 GiB RAM. Final timed searches 0.6875–0.8004 s, five queries, all 10 hits/10 strong/meaning true. Index 1,387,327,274 bytes; / free 7.0 G at end. Full sampler table + arithmetic in server/data/bench-18m-stats.md (git-ignored).
…rge shapes pinned in CI, ghost-row recreation lesson measured and documented docker-compose.lan.yml appends a LAN host binding for the API (env-forced DIGUP_LAN_BIND — refuses to apply without it; the base file's 127.0.0.1 mapping stays: compose merges ports lists). Header states the trade plainly: the server has no auth, so a LAN bind exposes the whole indexed library to the network; home-LAN use only, the §7 shape for anything else is an auth'd proxy. Field-proven on this box merged with the GPU override: healthy, UI + /api/v1 + searches on the host's LAN address, 10 hits/10 strong in 0.71-0.81 s at the 132k-file index. CI now schema-checks seven merge shapes (adds base+lan, lan+nvidia, ghcr+lan; uses RFC 5737 TEST-NET addresses — documentation addresses, nobody's). The server/README.md LAN section documents the shape and pins the recreation rule the field run proved the hard way: recreations must pass the SAME file chain AND env every time — dropped env moved the media bind one folder off the roots the index rows carry, every row fell out of scope, and the API answered 0 hits (keyword queries included) while health/files/vectors/job-log stayed normal; same command with the original env restored 10/10 strong, no prune, no reindex, the three job records untouched.
…58 s (7 merge shapes), publish 1 h 21 m 44 s, both tags republished
…ng compose state, git-ignored by design)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add self-hosted Docker server for DigUp
This PR introduces a self-hosted Docker server for DigUp, enabling users to search and browse their media library via a lightweight web interface or REST API.
The server operates as an adapter around the existing DigUp engine rather than implementing a separate search system, ensuring consistent search behavior across desktop, CLI, and server environments.
Closes #183
Key Features
/api/v1.Dockerfileanddocker-composesetups. Data paths (index,models,config, job logs) use bind mounts to ensure container recreation never destroys index state.:latest) and CUDA (:latest-cuda) images to GHCR with pre-configured Compose override files. Opt-in NVIDIA GPU acceleration is measured and validated on physical hardware.Once running, the interface is accessible at:
http://127.0.0.1:8787/How It Works
The server delegates all indexing and search logic to the underlying DigUp engine via its published interfaces (
search-service/index-serviceJSON-lines protocols, standard CLI, and--jsonoutput).SearchServiceand the HTTP API on identical database snapshots.Testing & Performance Benchmarks
All performance metrics were gathered on physical Linux x86_64 hardware (CI hosted runners and a host running a Ryzen 9 7950X, 64 GB RAM, GTX 1070).
1. Automated Test Suite
2. CI Pipeline (
server.yml)swift test), CLI headless checks, contract tests, search evaluations (R@1 0.76 / R@5 0.88), Compose schema checks, and gracefulSIGTERMcontainer termination (exit 143).Sources/DigUpKit/ModelFiles.swift.3. Docker & GPU Validation
4. Large-Scale Library Benchmark (131,972 Indexable Files)
Validated against a 187,055-file real-world library on USB storage over three sequential jobs:
POST /api/v1/index). Single-writer locks correctly returned409on concurrent index requests.Continuous Integration & Delivery
The
server.ymlGitHub Actions workflow triggers on modifications to server code, deployment files, engine dependencies, or scripts::latest(CPU) and:latest-cudatags toghcr.ioformainbranch updates, as well as feature-branch tags (:feat-server/:feat-server-cuda). Unused interim tags are routinely cleaned up via registry hygiene steps.Known Limitations
docker-compose.lan.ymloverride. Public internet usage requires an authenticating reverse proxy.x86_64only. Other architectures (e.g., Apple Silicon) build natively viadocker compose up -d --build.cuda). AMD/Vulkan setups are left for future PRs.Quickstart Guide
Option 1: Local Build (Any Architecture)
cd deploy docker compose up -d --buildOption 2: Pre-built CPU Image (x86_64)
cd deploy DIGUP_INDEX_DIR=/path/to/index \ DIGUP_MODELS_DIR=/path/to/models \ DIGUP_MEDIA_DIR=/path/to/media \ DIGUP_IMAGE_TAG=latest \ docker compose -f docker-compose.yml -f docker-compose.ghcr.yml up -dOption 3: Pre-built NVIDIA GPU Image
Option 4: Home LAN Exposure (Unauthenticated)
Access the UI at
http://127.0.0.1:8787/. Trigger indexing manually via the UI orPOST /api/v1/index.PR Footprint
server/IMPLEMENTATION_STATUS.mdanddocs/implementation/.