Fix/video pipeline hardening

.gitignore

@@ -9,3 +9,8 @@ videos/
 node_modules/
 test-results/
 playwright-report/
+# Python test bytecode
+__pycache__/
+*.pyc
+# local generated/read-only review pages
+docs/

README.md

@@ -62,7 +62,7 @@ site/work/<slug>/
 
 A slug is never silently replaced; choose another slug if that directory already exists. The publish operation also rebuilds `site/catalog.json`, which the gallery at `site/index.html` reads. Published viewer settings come from embedded configuration rather than browser storage, so one work cannot inherit another work's camera or shader.
 
-All resources used by an individual work are relative to its own directory. You can copy `site/work/<slug>/` wholesale to an S3 bucket or any other static web server without changing a base URL. It contains no Studio/API dependency and makes no network requests for runtime libraries. Serve the directory over HTTP rather than opening `index.html` as a `file://` URL; ES modules are commonly blocked or restricted from local files. This layout is intended to support a future BrightSign packaging workflow with little or no transformation, though device-specific browser/WebGL validation is still required.
+All resources used by an individual work are relative to its own directory. You can copy `site/work/<slug>/` wholesale to an S3 bucket or any other static web server without changing a base URL. It contains no Studio/API dependency and makes no network requests for runtime libraries. Serve the directory over HTTP rather than opening `index.html` as a `file://` URL; ES modules are commonly blocked or restricted from local files. Published environments also load on plain HTTP origins without Web Crypto; SHA-256 hashing is optional only for this static playback path. Export-capable skybox loading still requires Web Crypto (HTTPS or localhost) to verify content and protect resume identity. This layout is intended to support a future BrightSign packaging workflow with little or no transformation, though device-specific browser/WebGL validation is still required.
 
 Preview the gallery locally with:
 
@@ -106,7 +106,30 @@ The collapsed **Video timing** section provides export FPS, loop frame count, du
 
 **Export video…** opens a confirmation dialog for MP4 or MKV container, resolution, quality and range. The default is a 30-second MP4 clip starting at the current frame, shortened to fit when the loop is shorter than 30 seconds. Whole-loop export remains available. A clip window uses a start and duration in seconds, `MM:SS`, or `HH:MM:SS`; the window must remain inside one loop. The confirmation lists exact frames, duration and a bitrate-based size estimate before any work begins. If that file name already labels a clip in `videos/`, the viewer asks before adding another.
 
-Rendering and encoding are intentionally split. The active browser renders the exact live WebGL scene one deterministic PNG frame at a time. The server encodes independent H.264/Matroska checkpoint segments, then losslessly concatenates them into the selected MP4 or MKV under `videos/`. Frames are not retained as an image sequence or accumulated in browser memory. Completed videos include an adjacent JSON provenance file. Static published works can display the export UI but cannot encode video without the local API server.
+Rendering and encoding are intentionally split. The active browser renders the exact live WebGL scene one deterministic PNG frame at a time. The server encodes independent H.264/Matroska checkpoint segments, then losslessly concatenates them into the selected MP4 or MKV under `videos/`. Frames are not retained as an image sequence or accumulated in browser memory. Capture uses a reusable OffscreenCanvas with promise-based PNG encoding where supported, falling back to synchronous capture for compatibility. The selected skybox must finish loading before export; its actual face hashes are part of the resume signature, and environment changes are locked during rendering. Completed videos include an adjacent JSON provenance file. Static published works can display the export UI but cannot encode video without the local API server.
+
+New exports explicitly convert full-range RGB to limited-range BT.709 YUV and tag BT.709 matrix, transfer and primaries in H.264 and the container. Checkpoint concatenation preserves these tags, verified with real `ffprobe` tests for MP4 and MKV. Existing checkpoint jobs without a color profile retain their original conversion rather than mixing old and new colors within one movie. Checkpoint encoder failures preserve the last 4 KB of diagnostics in the interrupted job's reason.
+
+### Video validation commands
+
+```sh
+make test-fast           # routine editing: no Chromium, networking or real encoder
+make test-python         # durable store, API and real ffmpeg recovery tests
+make test-browser        # isolated capture + complete browser integration
+make test-video-capture  # pixel integrity and capture-fallback test; no HTTP server
+make test-video-encoder  # store + real encoder tests without localhost networking
+make test-video-soak VIDEO_SOAK_SOURCE="videos/example.mp4"
+# Longer encoder validation is opt-in; duration is output video seconds:
+make test-video-soak VIDEO_SOAK_SOURCE="videos/example.mp4" VIDEO_SOAK_SECONDS=600
+```
+
+Use `make test-fast` during development and select `test-video-capture` or `test-video-encoder` only for relevant changes. Run `make test-browser` once for final integration validation; do not automatically retry failed full browser suites.
+
+Resource-heavy targets share a per-user nonblocking lock across checkout directories, a wall-clock timeout, and an owned-process cleanup guard. Concurrent runs are refused rather than queued. Network-heavy targets refuse to start at 8,000 or more host `TIME_WAIT` sockets; nested stages recheck pressure before proceeding. Browser tests also use this guard when invoked directly. The guard attempts graceful cleanup, then terminates only the invocation's process group and recorded descendants whose PID/start time/command still match—not the user's server or browser. Test encoders are limited to two threads; production exports keep their normal thread policy.
+
+Browser cleanup is awaited on success, failure and SIGINT/SIGTERM. The temporary server pauses its jobs before exit, stderr is continuously drained with an 8 KB diagnostic tail, and temporary files are removed only after server exit. The API test helpers reuse an HTTP connection per test and close it during teardown. The export API already supports HTTP/1.1 keep-alive for sequential browser uploads.
+
+The soak uses three samples from an existing artwork video at 1080p/30 FPS, two store-restart/resume cycles, exact frame count and duration checks, BT.709 verification and encoder-leak checks. It uses temporary storage and does not alter the source. The default six-second run is deliberately bounded; it is not a claim of multi-hour browser endurance. Long hidden-tab/4K exports still need workload-specific endurance validation.
 
 ### Long-running video recovery
 

makefile

@@ -1,4 +1,6 @@
-.PHONY: serve static install sync update test test-python test-browser unstick
+.PHONY: serve static install sync update test test-fast test-guard test-python test-browser test-video-capture test-video-encoder test-video-soak _test-all _test-browser unstick
+
+TEST_GUARD = python3 scripts/test_guard.py
 
 serve:
 	python3 studio/server.py --port 1313
@@ -29,11 +31,45 @@ update:
 	$(MAKE) sync
 	$(MAKE) test
 
+# Full validation is explicit and serialized. Prefer test-fast while editing.
 test:
-	bun run test
+	$(TEST_GUARD) --network --timeout 600 -- $(MAKE) _test-all
+
+_test-all:
+	$(MAKE) test-fast
+	$(MAKE) test-python
+	$(MAKE) test-browser
+
+# No browser, network server, real encoder, or production assets.
+test-fast:
+	bun run test:vendor
+	bun run test:syntax
+	node studio/test_resolution.cjs
+	node studio/test_lifecycle_test.cjs
+	node studio/test_skybox_hashing.cjs
+	python3 -m unittest discover -s studio -p 'test_video_resume.py'
+	$(MAKE) test-guard
+
+test-guard:
+	python3 -m unittest discover -s scripts -p 'test_test_guard.py'
 
 test-python:
-	python3 -m unittest discover -s studio -p 'test_*.py'
+	$(TEST_GUARD) --network --timeout 180 -- python3 -m unittest discover -s studio -p 'test_*.py'
+
+# Opt-in production-resolution encoder test. Uses temporary storage, never overwrites source.
+VIDEO_SOAK_SECONDS ?= 6
+test-video-soak:
+	$(TEST_GUARD) --timeout 7200 -- python3 studio/video_soak.py --source "$(VIDEO_SOAK_SOURCE)" --seconds "$(VIDEO_SOAK_SECONDS)"
+
+test-video-encoder:
+	$(TEST_GUARD) --timeout 90 -- python3 -m unittest studio.test_video_resume studio.test_video_resume_integration
+
+test-video-capture:
+	$(TEST_GUARD) --timeout 60 -- node studio/test_video_capture.cjs
 
 test-browser:
-	bun run test:browser
+	$(TEST_GUARD) --network --timeout 360 -- $(MAKE) _test-browser
+
+_test-browser:
+	$(MAKE) test-video-capture
+	node studio/test_browser.cjs

package.json

@@ -3,11 +3,12 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "test": "bun run test:vendor && bun run test:syntax && bun run test:unit && bun run test:browser",
+    "test": "make test",
+    "test:fast": "make test-fast",
     "test:vendor": "node scripts/sync-vendor.mjs --check",
-    "test:syntax": "node --check tesseract.js && node --check viewer-skyboxes.js && node --check studio/studio.js && node --check studio/pipeline.js",
+    "test:syntax": "node --check tesseract.js && node --check viewer-skyboxes.js && node --check studio/studio.js && node --check studio/pipeline.js && node --check studio/test_browser.cjs && node --check studio/test_video_capture.cjs && node --check studio/test_lifecycle.cjs",
     "test:unit": "node studio/test_resolution.cjs && make test-python",
-    "test:browser": "node studio/test_browser.cjs",
+    "test:browser": "make test-browser",
     "sync:vendor": "node scripts/sync-vendor.mjs"
   },
   "dependencies": {

scripts/test_guard.py

@@ -0,0 +1,176 @@
+#!/usr/bin/env python3
+"""Serialize resource-heavy tests, bound runtime, and clean only owned processes (POSIX)."""
+import argparse
+import fcntl
+import json
+import os
+from pathlib import Path
+import signal
+import subprocess
+import sys
+import tempfile
+import time
+import uuid
+
+LOCK_PATH = Path(tempfile.gettempdir()) / f'tesseract-heavy-tests-{os.getuid()}.lock'
+TOKEN_KEY = 'TESSERACT_TEST_GUARD_TOKEN'
+
+
+def socket_pressure():
+    result = subprocess.run(['netstat', '-an'], capture_output=True, text=True, timeout=5)
+    if result.returncode:
+        raise RuntimeError('Cannot inspect socket pressure; refusing network-heavy tests')
+    return sum('TIME_WAIT' in line for line in result.stdout.splitlines())
+
+
+def process_snapshot():
+    result = subprocess.run(['ps', '-axo', 'pid=,ppid=,lstart=,command='],
+                            capture_output=True, text=True, timeout=5)
+    if result.returncode:
+        raise RuntimeError('Cannot inspect test-owned processes')
+    entries = {}
+    for line in result.stdout.splitlines():
+        fields = line.split(None, 7)
+        if len(fields) == 8 and fields[0].isdigit() and fields[1].isdigit():
+            entries[int(fields[0])] = (int(fields[1]), ' '.join(fields[2:7]), fields[7])
+    return entries
+
+
+def remember_descendants(root_pid, owned, snapshot):
+    parents = {root_pid} | {pid for pid, identity in owned.items()
+                               if pid in snapshot and snapshot[pid][1:] == identity}
+    changed = True
+    while changed:
+        changed = False
+        for pid, (parent, started, command) in snapshot.items():
+            if parent in parents and pid not in parents:
+                parents.add(pid)
+                owned[pid] = (started, command)
+                changed = True
+
+
+def signal_owned(owned, sig):
+    snapshot = process_snapshot()
+    for pid, identity in owned.items():
+        if pid in snapshot and snapshot[pid][1:] == identity:
+            try:
+                os.kill(pid, sig)
+            except ProcessLookupError:
+                pass
+
+
+def cleanup(child, owned):
+    try:
+        remember_descendants(child.pid, owned, process_snapshot())
+    except (OSError, RuntimeError, subprocess.SubprocessError) as error:
+        print(f'Cleanup process inspection failed: {error}', file=sys.stderr)
+    # Allow JS finally/signal handlers to close Chromium and the server first.
+    if child.poll() is None:
+        child.send_signal(signal.SIGTERM)
+        try:
+            child.wait(timeout=10)
+        except subprocess.TimeoutExpired:
+            pass
+    # Detached browser processes can have their own groups. Their PID + start
+    # time + command must still match our recorded descendants before signalling.
+    try:
+        signal_owned(owned, signal.SIGTERM)
+        deadline = time.monotonic() + 3
+        while time.monotonic() < deadline:
+            snapshot = process_snapshot()
+            if not any(pid in snapshot and snapshot[pid][1:] == identity
+                       for pid, identity in owned.items()):
+                break
+            time.sleep(.2)
+        signal_owned(owned, signal.SIGKILL)
+    except (OSError, RuntimeError, subprocess.SubprocessError) as error:
+        print(f'Detached-process inspection failed; terminating only the owned test group: {error}', file=sys.stderr)
+    # Also cover ordinary children reparented during forced shutdown. This group
+    # was exclusively created for this test invocation, not the user's server.
+    try:
+        os.killpg(child.pid, signal.SIGKILL)
+    except ProcessLookupError:
+        pass
+    if child.poll() is None:
+        child.wait(timeout=5)
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--timeout', type=float, default=300)
+    parser.add_argument('--network', action='store_true')
+    parser.add_argument('command', nargs=argparse.REMAINDER)
+    args = parser.parse_args(argv)
+    command = args.command[1:] if args.command[:1] == ['--'] else args.command
+    if not command or args.timeout <= 0:
+        parser.error('A command and positive timeout are required')
+
+    with LOCK_PATH.open('a+') as lock:
+        # Nested Make targets inherit the outer guard and its process group.
+        lock.seek(0)
+        try:
+            owner = json.loads(lock.read() or '{}')
+        except ValueError:
+            owner = {}
+        if os.environ.get(TOKEN_KEY) and os.environ[TOKEN_KEY] == owner.get('token'):
+            try:
+                fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+            except BlockingIOError:
+                # The token is valid only while the outer guard still owns its
+                # lock. A stale inherited environment must not bypass guarding.
+                if args.network and socket_pressure() >= 8000:
+                    print('Host socket pressure rose during this run; stopping before launching more network-heavy tests.', file=sys.stderr)
+                    return 2
+                os.execvp(command[0], command)
+        try:
+            fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            print('Heavy tests are already running. Wait for that run to finish; do not retry concurrently.', file=sys.stderr)
+            return 2
+        if args.network:
+            pressure = socket_pressure()
+            if pressure >= 8000:
+                print(f'Refusing network-heavy tests: {pressure:,} TIME_WAIT sockets. '
+                      'Use make test-fast or make test-video-encoder; retry only after host pressure subsides.', file=sys.stderr)
+                return 2
+        token = uuid.uuid4().hex
+        lock.seek(0); lock.truncate()
+        lock.write(json.dumps({'pid': os.getpid(), 'token': token, 'command': command}))
+        lock.flush()
+        environment = dict(os.environ, **{TOKEN_KEY: token, 'TESSERACT_TEST_ENCODER_THREADS': '2'})
+        child = subprocess.Popen(command, env=environment, start_new_session=True)
+        owned = {}
+        interrupted = []
+        def stop(sig, _frame):
+            interrupted.append(sig)
+        previous = {sig: signal.signal(sig, stop) for sig in (signal.SIGINT, signal.SIGTERM)}
+        started = time.monotonic()
+        timed_out = False
+        try:
+            while child.poll() is None:
+                remember_descendants(child.pid, owned, process_snapshot())
+                if interrupted:
+                    break
+                if time.monotonic() - started > args.timeout:
+                    timed_out = True
+                    print(f'Test exceeded {args.timeout:g}s; cleaning up this invocation only.', file=sys.stderr)
+                    break
+                time.sleep(.5)
+        finally:
+            try:
+                cleanup(child, owned)
+            finally:
+                for sig, handler in previous.items():
+                    signal.signal(sig, handler)
+                lock.seek(0); lock.truncate(); lock.flush()
+        if interrupted:
+            return 128 + interrupted[0]
+        return 124 if timed_out else child.returncode
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main())
+    except (OSError, RuntimeError, subprocess.SubprocessError) as error:
+        print(f'Test guard: {error}', file=sys.stderr)
+        sys.exit(2)