diff --git a/ARGO/oceancurrent/README.md b/ARGO/oceancurrent/README.md index 1078184b..a5c0794f 100644 --- a/ARGO/oceancurrent/README.md +++ b/ARGO/oceancurrent/README.md @@ -278,6 +278,7 @@ python3 ARGO/oceancurrent/gliders_catalogue.py \ --input /tmp/gliders-index.html \ --output /tmp/gliders-catalogue.json python3 -m pytest -q ARGO/oceancurrent/test_gliders_catalogue.py +python3 -m pytest -q ARGO/oceancurrent/test_publish_gliders_catalogue.py ``` The standard library runner is also supported: @@ -294,31 +295,128 @@ validates before writing, uses a temporary file alongside the output, then atomically replaces the destination. Parsing, validation or replacement failures leave the previous catalogue intact, remove the temporary file and exit nonzero. -### Refresh and limitations - -No cron, deployment, scanner or scientific processing changes are included. -A future separately reviewed scheduling change should run this generator after -the legacy catalogue has finished publishing, with explicit input and output -paths, in the existing OceanCurrent job environment. It should log the record -count and failures and alert on unexpected count reductions. Avoid reading a -partially written HTML file: use the publisher's completion signal or a completed -snapshot. Start with a manual comparison before enabling recurring refreshes; -refresh at the legacy catalogue's publication cadence rather than running a new -service. - -The four table anchors, column headers and coordinate units are deliberately -validated. Missing tables, empty main views and conflicting metadata for the -same name/version fail the run. The event table may be empty. Year/summary rows -and incomplete or unrecognised mission rows are skipped. This is a structural -sanity check, not proof that a partially published catalogue is complete: there -is no fixed mission-count threshold. Historical letter suffixes and `_1`/`_2` -legs are supported; new naming/header conventions need parser review. Scientific -and visualisation links are preserved but their availability is not checked. -The source's precision and errors remain authoritative; no scientific metadata, -annual-map hotspots or missing links are inferred. - -Future `ocean-current-api` work should define a dedicated catalogue response and -loader for this schema, retain the four order lists and mission/version identity, -and associate visualisation references with the existing Gliders asset indexes. -The catalogue must not be interpreted as an image index. Endpoints, frontend -browsing and caching/refresh behaviour remain separate integration work. +### Production workflow and scheduling + +Confirmed investigation: CSIRO publishes `gliders/index.html` to `webapp-prod-aws-syd` +through the `oceancurrent_website` rsync module rooted at `/mnt/oceancurrent/website`, +historically connecting from `140.79.40.143`. Publication intervals vary (roughly +two-hourly or daily); no fixed completion schedule or signal is confirmed. The image +scanner runs every two hours independently; legacy `index.php` reads the HTML. + +**Proposed initial schedule: check every two hours, independently of the scanner.** +Manage a separate `po_gliders_catalogue` job through +`chef-private/data_bags/cronjobs`, registered in the webapp node's `cronjobs` list. +Use `minute: "0"`, `hour: "*/2"`, `shell: "/bin/bash"`, the verified execution user +(proposed `projectofficer`), existing operations `mailto` and `monitored: true`. +Its command should source the verified job environment and run the command below. +This is a polling interval, not an assumed upstream publication time. No production +cron configuration or automatic publication is installed by this change. + +### Publication + +Install both Python modules together. Set `PYTHON` to the verified Python 3 interpreter +with `beautifulsoup4`, and `DATA_SERVICES_DIR` to the installed checkout. The dependency +is declared in `requirements.txt`, but production availability is unverified. +`projectofficer` is the proposed user; confirm source readability and permissions to +write temporary files and replace JSON in the destination directory. Set +`GLIDERS_LOCK_FILE` to an operations-provisioned writable lock outside the rsync tree; +use the same lock for scheduled and manual runs. + +```bash +"${PYTHON:?}" "${DATA_SERVICES_DIR:?}/ARGO/oceancurrent/publish_gliders_catalogue.py" \ + --input /mnt/oceancurrent/website/gliders/index.html \ + --output /mnt/oceancurrent/website/gliders/gliders-catalogue.json \ + --lock-file "${GLIDERS_LOCK_FILE:?}" \ + --stability-seconds 30 +``` + +Each check compares HTML bytes across a configurable observation window (default +30 seconds), aborts before parsing if they or the file identity/stat change, and +checks again before publication. Changes are detected even with preserved mtimes; +resulting JSON content determines whether a write is needed. No timestamp-only +shortcut or persistent change-tracking state is used. + +The 30-second window is an initial heuristic, **not proof of completion**: an +incomplete transfer can pause longer, or restart after the final check. Coordinate +with the upstream publisher for stronger serialization/completion guarantees. +`--source-complete` skips the wait only for an explicitly confirmed manual run. + +The publisher validates schema version 1, mission records and all four order lists. +Every mission-count or per-order count reduction is blocked by default; +`--max-reduction-percent P` permits an explicitly reviewed percentage (`0 <= P < 100`). +Unchanged JSON is not rewritten. The existing generator's atomic writer preserves +the previous output on failure. The lock serializes publishers, not upstream writes; +never unlink it while in use. Logs on stderr include start/completion, paths, duration, +counts, changes and failures. Preserve stderr/status in existing mail/Nagios reporting: +success returns 0, failure 1, and invalid CLI syntax 2. + +### Rsync deletion protection (required before deployment) + +The writable module's local configuration has no explicit deletion/filter protection; +the upstream command and effective daemon settings remain unverified. Rsync deletion +could remove locally generated JSON absent from the sender. A local dry run confirmed +this for the JSON, default lock and atomic temporary files. Missing JSON fails closed +on the next check without a reviewed initial baseline; polling is not a mitigation. + +Coordinate receiver-side deletion protection **and** sender-side exclusion to prevent +overwrite. For a transfer rooted at the website directory, proposed rules are: + +```text +--filter='P /gliders/gliders-catalogue.json*' +--filter='H /gliders/gliders-catalogue.json*' +--filter='P /gliders/.gliders-catalogue.json*' +--filter='H /gliders/.gliders-catalogue.json*' +``` + +Confirm paths/rule precedence against the actual transfer; test with `--dry-run` +and `--itemize-changes`, including its deletion options. A plain exclude can lose deletion +protection under `--delete-excluded`; explicit receiver protection is required +([rsync documentation](https://download.samba.org/pub/rsync/rsync.1#FILTER_RULES)). +Operations must implement/verify equivalent protections before enabling publication; +no upstream rsync configuration is changed here. + +### First publication and recovery + +1. Coordinate with operations and the upstream owner; confirm HTML generation and + transfer completed and prevent another update during review/publication. +2. Verify `"$PYTHON" -c 'import bs4'` and directory permissions as the execution user. + Copy the completed HTML to a temporary directory and run the local generator above. +3. Inspect the candidate with `"$PYTHON" -m json.tool`. Compare mission/version counts, + metadata and all four orders with the legacy HTML; back up any existing JSON. +4. Confirm the live HTML still matches the reviewed copy. Run the publication command, + adding `--initial-mission-count N` with the reviewed count when no output exists. + Compare the published JSON with the candidate, then verify remote access below. +5. Confirm rsync protection and failure reporting, then approve the separate Chef job. + Scheduled runs omit both `--source-complete` and `--initial-mission-count`. + +On failure, inspect stderr and fix the source/environment/permissions. A changing +source is retried on the next scheduled check; manual retries use the same command. +Review reductions before allowing a one-time tolerance; do not delete the baseline +to bypass the guard. If existing JSON is invalid, preserve a copy and restore the +last known valid catalogue before retrying. Count guards cannot detect every +incomplete source, so upstream coordination remains necessary. + +### Remote and API verification + +The inspected edge deployment configures `REMOTE_BASE_URL=https://oceancurrent.edge.aodn.org.au`; +`REMOTE_RESOURCE_PATH` defaults to `/resource/`. Its catalogue URL is +. +Reconfirm actual deployed values before testing: + +```bash +: "${REMOTE_BASE_URL:?}" "${REMOTE_RESOURCE_PATH:?}" +resource_url="${REMOTE_BASE_URL%/}/${REMOTE_RESOURCE_PATH#/}" +curl --fail --show-error --location "${resource_url%/}/gliders/gliders-catalogue.json" +curl --fail --show-error 'https://oceancurrent.edge.aodn.org.au/api/v1/gliders/missions?size=100' +``` + +Require HTTP 200, schema version 1 and metadata/orders matching the published JSON; +check API pagination, each order and an exact mission ID. Both edge URLs returned 404 +on 2026-10-19, so availability has not been established. An endpoint 404 also requires +a deployment/routing check. For API 503, check the configured resource URL, upstream +status/JSON, publication logs, permissions and API fetch logs. + +The API refresh interval is `GLIDERS_CATALOGUE_REFRESH_INTERVAL` (default `1h`): after +it elapses, a request triggers asynchronous refresh while serving cached data. Check +subsequent reads; failed initial loads retry on later requests. Always verify the +resource separately because a cached snapshot can conceal an upstream failure. diff --git a/ARGO/oceancurrent/gliders_catalogue.py b/ARGO/oceancurrent/gliders_catalogue.py index 85c8b3cb..07f88e28 100644 --- a/ARGO/oceancurrent/gliders_catalogue.py +++ b/ARGO/oceancurrent/gliders_catalogue.py @@ -158,6 +158,13 @@ def generate_catalogue(input_path, output_path, source_url=SOURCE_URL): if input_path.resolve() == output_path.resolve(): raise ValueError("Input and output paths must differ") catalogue = parse_catalogue(input_path.read_text(encoding="utf-8"), source_url) + write_catalogue(catalogue, output_path) + return catalogue + + +def write_catalogue(catalogue, output_path): + """Atomically publish an already parsed and validated catalogue.""" + output_path = Path(output_path) payload = json.dumps(catalogue, indent=2, ensure_ascii=False, allow_nan=False) + "\n" temporary = None try: diff --git a/ARGO/oceancurrent/publish_gliders_catalogue.py b/ARGO/oceancurrent/publish_gliders_catalogue.py new file mode 100644 index 00000000..f2b68ed2 --- /dev/null +++ b/ARGO/oceancurrent/publish_gliders_catalogue.py @@ -0,0 +1,177 @@ +"""Check stable Gliders HTML and atomically publish validated mission metadata.""" +import argparse +import fcntl +import json +import logging +import math +from pathlib import Path +import sys +import time +from urllib.parse import quote, urlparse + +LOG = logging.getLogger("gliders_catalogue_publication") + + +def validate_catalogue(catalogue): + """Validate generated JSON and the previous baseline before comparing counts.""" + from gliders_catalogue import MISSION, VIEWS + + def require(condition, message): + if not condition: + raise ValueError(message) + + def public_url(value): + return isinstance(value, str) and urlparse(value).scheme in ("http", "https") and bool(urlparse(value).netloc) + + require(isinstance(catalogue, dict) and type(catalogue.get("schema_version")) is int + and catalogue["schema_version"] == 1, "Expected schema_version == 1") + require(public_url(catalogue.get("source_url")), "Invalid source_url") + missions = catalogue.get("missions") + require(isinstance(missions, list) and bool(missions), "Missing or empty missions") + ids = [] + for record in missions: + require(isinstance(record, dict), "Invalid mission record") + name, version = record.get("mission_name"), record.get("processing_version") + require(isinstance(name, str) and MISSION.fullmatch(name), "Invalid mission_name") + require(version is None or isinstance(version, str) and bool(version.strip()), "Invalid processing_version") + require(record.get("id") == quote(name, safe="") + "@" + quote(version or "unknown", safe=""), "Invalid mission id") + require(record.get("mission_id") == (name[:-4] if name.endswith("_nrt") else name), "Invalid mission_id") + require(type(record.get("is_nrt")) is bool and record["is_nrt"] == name.endswith("_nrt"), "Invalid is_nrt") + numbers = record.get("catalogue_numbers") + require(isinstance(numbers, list) and bool(numbers) + and all(isinstance(n, str) and n.isdigit() for n in numbers), "Invalid catalogue_numbers") + for field in ("dates_raw", "start_date", "end_date", "glider_id", "processing_version"): + require(field in record and (record[field] is None or isinstance(record[field], str)), "Invalid " + field) + for field, low, high in (("latitude", -90, 90), ("longitude", -180, 180), + ("duration_days", 0, math.inf), ("distance_travelled_km", 0, math.inf)): + value = record.get(field) + require(field in record and (value is None or type(value) in (int, float) + and math.isfinite(value) and low <= value <= high), "Invalid " + field) + for field in ("temperature_salinity_url", "scientific_data_url"): + require(field in record and (record[field] is None or public_url(record[field])), "Invalid " + field) + visuals = record.get("visualisations") + require(isinstance(visuals, dict), "Invalid visualisations") + for kind in ("details", "animations"): + links = visuals.get(kind) + require(isinstance(links, list) and all(isinstance(link, dict) + and isinstance(link.get("label"), str) and public_url(link.get("url")) + for link in links), "Invalid visualisations: " + kind) + ids.append(record["id"]) + require(len(ids) == len(set(ids)), "Duplicate mission ids") + orders = catalogue.get("orders") + require(isinstance(orders, dict), "Missing orders") + for view in VIEWS.values(): + order = orders.get(view) + require(isinstance(order, list) and all(isinstance(item, str) for item in order), "Missing or invalid order: " + view) + require(len(order) == len(set(order)) and set(order).issubset(ids), "Invalid order references: " + view) + require(bool(order) or view == "event_based_sampling", "Empty order: " + view) + return len(missions) + + +def _read_source(path): + """Read one snapshot, rejecting replacement or in-place changes during the read.""" + before = path.stat() + content = path.read_bytes() + after = path.stat() + signature = lambda stat: (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns, stat.st_ctime_ns) + if signature(before) != signature(after): + raise ValueError("Source changed while reading; wait for upstream publication to complete") + return content, signature(after) + + +def publish_catalogue(source, destination, *, source_complete=False, + stability_seconds=30, lock_file=None, + max_reduction_percent=0, initial_mission_count=None): + """Observe source stability, or accept explicitly confirmed manual completion. + + Quiet snapshots cannot prove upstream completion. The lock serializes catalogue + publishers only; stronger guarantees require cooperation from the HTML producer. + """ + # Lazy import lets the CLI report missing BeautifulSoup through normal logging. + from gliders_catalogue import parse_catalogue, write_catalogue + + source, destination = Path(source), Path(destination) + if not math.isfinite(stability_seconds) or stability_seconds <= 0: + raise ValueError("stability-seconds must be finite and positive") + if not math.isfinite(max_reduction_percent) or not 0 <= max_reduction_percent < 100: + raise ValueError("max-reduction-percent must be finite and in [0, 100)") + if initial_mission_count is not None and initial_mission_count <= 0: + raise ValueError("initial-mission-count must be positive") + lock_path = Path(lock_file) if lock_file is not None else Path(str(destination) + ".lock") + if len({source.resolve(), destination.resolve(), lock_path.resolve()}) != 3: + raise ValueError("Source must differ from destination and lock; destination must differ from lock") + # Never unlink the lock: removing it would allow two processes to lock different inodes. + with lock_path.open("a") as lock: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError as error: + raise ValueError("Another catalogue publication holds the output lock") from error + html, signature = _read_source(source) + if not source_complete: + LOG.info("Observing HTML stability for %s seconds (not an upstream completion signal)", stability_seconds) + time.sleep(stability_seconds) + observed_html, observed_signature = _read_source(source) + if observed_html != html or observed_signature != signature: + raise ValueError("Source changed during stability observation; retry on the next check") + catalogue = parse_catalogue(html.decode("utf-8")) + new_count = validate_catalogue(catalogue) + previous_bytes = destination.read_bytes() if destination.exists() else None + previous = json.loads(previous_bytes) if previous_bytes is not None else None + previous_count = validate_catalogue(previous) if previous_bytes is not None else None + LOG.info("Mission counts previous=%s new=%d", previous_count, new_count) + if previous is None: + if initial_mission_count != new_count: + raise ValueError("First publication requires --initial-mission-count matching the reviewed candidate (%d)" % new_count) + else: + counts = [("missions", previous_count, new_count)] + counts.extend((view, len(old), len(catalogue["orders"][view])) + for view, old in previous["orders"].items() if view in catalogue["orders"]) + for label, old, new in counts: + if old and (old - new) * 100 > old * max_reduction_percent: + raise ValueError("Reduction requires manual review: %s previous=%d new=%d threshold=%s%%" + % (label, old, new, max_reduction_percent)) + current_html, current_signature = _read_source(source) + if current_html != html or current_signature != signature: + raise ValueError("Source changed during generation; publication aborted") + # Semantic equality avoids rewriting even a differently formatted valid baseline. + changed = previous != catalogue + if changed: + write_catalogue(catalogue, destination) + LOG.info("Catalogue publication changed=%s missions=%d destination=%s", changed, new_count, destination) + return catalogue + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--input", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--source-complete", action="store_true", + help="Manual confirmation of upstream completion; skips the stability wait") + parser.add_argument("--stability-seconds", type=float, default=30, + help="Quiet observation window for periodic checks; default 30, not proof of completion") + parser.add_argument("--lock-file", type=Path, + help="Shared lock outside incoming rsync tree (default: .lock)") + parser.add_argument("--max-reduction-percent", type=float, default=0, + help="Maximum reviewed reduction in missions or any order; default 0 blocks every decrease") + parser.add_argument("--initial-mission-count", type=int, + help="Reviewed exact mission count, required only when output does not exist") + args = parser.parse_args(argv) + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s") + started = time.monotonic() + LOG.info("Publication started source=%s destination=%s", args.input, args.output) + status = 1 + try: + publish_catalogue(args.input, args.output, source_complete=args.source_complete, + stability_seconds=args.stability_seconds, lock_file=args.lock_file, + max_reduction_percent=args.max_reduction_percent, + initial_mission_count=args.initial_mission_count) + status = 0 + except (OSError, ValueError, ImportError) as error: + LOG.error("Validation/publication failed; previous catalogue preserved: %s", error) + finally: + LOG.info("Publication completed status=%d duration_seconds=%.3f", status, time.monotonic() - started) + return status + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ARGO/oceancurrent/test_publish_gliders_catalogue.py b/ARGO/oceancurrent/test_publish_gliders_catalogue.py new file mode 100644 index 00000000..de3ea276 --- /dev/null +++ b/ARGO/oceancurrent/test_publish_gliders_catalogue.py @@ -0,0 +1,318 @@ +import copy +import fcntl +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +from bs4 import BeautifulSoup + +import publish_gliders_catalogue as publisher +from gliders_catalogue import parse_catalogue + +FIXTURE = Path(__file__).parent / "fixtures" / "gliders_catalogue.html" +SCRIPT = Path(__file__).with_name("publish_gliders_catalogue.py") + + +class TestPublication(unittest.TestCase): + def setUp(self): + self.folder = tempfile.TemporaryDirectory() + self.addCleanup(self.folder.cleanup) + self.source = Path(self.folder.name) / "index.html" + self.destination = Path(self.folder.name) / "gliders-catalogue.json" + self.html = FIXTURE.read_text(encoding="utf-8") + self.source.write_text(self.html, encoding="utf-8") + self.count = len(parse_catalogue(self.html)["missions"]) + + def publish(self, **kwargs): + options = dict(source_complete=True, initial_mission_count=self.count) + options.update(kwargs) + return publisher.publish_catalogue(self.source, self.destination, **options) + + def remove_mission(self, name, only_view=None): + soup = BeautifulSoup(self.html, "html.parser") + container = soup.find("a", id=only_view).find_next("table") if only_view else soup + for row in container.find_all("tr"): + cells = row.find_all("td", recursive=False) + if len(cells) == 12 and cells[-1].get_text(strip=True) == name: + row.decompose() + self.source.write_text(str(soup), encoding="utf-8") + + def test_initial_publication_and_unchanged_output(self): + with self.assertLogs(publisher.LOG, level="INFO") as logs: + result = self.publish() + self.assertEqual(json.loads(self.destination.read_text()), parse_catalogue(self.html)) + self.assertEqual(len(result["missions"]), self.count) + self.assertEqual(self.destination.stat().st_mode & 0o777, 0o644) + self.assertIn("previous=None new=8", "\n".join(logs.output)) + original = self.destination.stat() + self.source.write_text(self.html + "\n", encoding="utf-8") + with patch.object(publisher, "LOG") as log, patch("gliders_catalogue.write_catalogue") as write: + self.publish() + write.assert_not_called() + self.assertEqual(self.destination.stat().st_mtime_ns, original.st_mtime_ns) + self.assertIn("False", str(log.info.call_args_list)) + + def test_updated_mission_information(self): + self.publish() + self.source.write_text(self.html.replace("SG540", "SG541"), encoding="utf-8") + result = self.publish() + self.assertEqual(json.loads(self.destination.read_text()), result) + self.assertTrue(any(r["glider_id"] == "SG541" for r in result["missions"])) + + def test_invalid_or_incomplete_source_preserves_output(self): + self.publish() + original = self.destination.read_bytes() + for invalid in ("", "Maintenance", self.html[:self.html.index("")]): + with self.subTest(invalid=invalid[:30]): + self.source.write_text(invalid) + with self.assertRaises(ValueError): + self.publish() + self.assertEqual(self.destination.read_bytes(), original) + + def test_reduction_and_explicit_threshold(self): + self.publish() + original = self.destination.read_bytes() + self.remove_mission("Brisbane20180306A_nrt") + for threshold in (0, 10): + with self.assertRaisesRegex(ValueError, "Reduction requires manual review"): + self.publish(max_reduction_percent=threshold) + self.assertEqual(self.destination.read_bytes(), original) + result = self.publish(max_reduction_percent=12.5) + self.assertEqual(len(result["missions"]), self.count - 1) + + def test_partial_view_reduction_even_when_total_unchanged(self): + self.publish() + original = self.destination.read_bytes() + self.remove_mission("TwoRocks20260327", only_view="byebs") + with self.assertRaisesRegex(ValueError, "event_based_sampling"): + self.publish() + self.assertEqual(self.destination.read_bytes(), original) + + def test_missing_unreadable_and_changing_source(self): + self.publish() + original = self.destination.read_bytes() + self.source.unlink() + with self.assertRaises(FileNotFoundError): + self.publish() + self.source.write_text(self.html) + with patch.object(Path, "read_bytes", side_effect=PermissionError("unreadable")): + with self.assertRaises(PermissionError): + self.publish() + read_source = publisher._read_source + calls = [] + + def changed_source(path): + if calls: + path.write_text(self.html + "") + calls.append(True) + return read_source(path) + + with patch.object(publisher, "_read_source", side_effect=changed_source): + with self.assertRaisesRegex(ValueError, "Source changed"): + self.publish() + self.assertEqual(self.destination.read_bytes(), original) + + def test_changes_during_snapshot_read_rejected(self): + real_read = Path.read_bytes + + def changing_read(path): + result = real_read(path) + path.write_bytes(result + b" ") + return result + + with patch.object(Path, "read_bytes", changing_read): + with self.assertRaisesRegex(ValueError, "Source changed while reading"): + self.publish() + self.assertFalse(self.destination.exists()) + + def test_configuration_and_first_publication_review_required(self): + for options in (dict(initial_mission_count=None), + dict(initial_mission_count=1), dict(initial_mission_count=0), + dict(stability_seconds=0), dict(stability_seconds=-1), + dict(stability_seconds=float("nan")), dict(stability_seconds=float("inf")), + dict(max_reduction_percent=-1), dict(max_reduction_percent=100), + dict(max_reduction_percent=float("nan")), dict(max_reduction_percent=float("inf"))): + with self.subTest(options=options), self.assertRaises(ValueError): + self.publish(**options) + self.assertFalse(self.destination.exists()) + with self.assertRaisesRegex(ValueError, "Source must differ"): + publisher.publish_catalogue(self.source, self.source, source_complete=True) + + def test_invalid_baseline_requires_recovery_preserves_bytes(self): + for value in (None, {}, {"schema_version": 2}, "invalid JSON"): + self.destination.write_text(json.dumps(value)) + original = self.destination.read_bytes() + with self.assertRaises(ValueError): + self.publish() + self.assertEqual(self.destination.read_bytes(), original) + self.destination.write_text("{broken JSON") + with self.assertRaises(ValueError): + self.publish() + self.assertEqual(self.destination.read_text(), "{broken JSON") + + def test_schema_records_and_orders_validation(self): + valid = parse_catalogue(self.html) + changes = [lambda c: c.update(schema_version=True), + lambda c: c.update(missions=[]), + lambda c: c["missions"][0].update(mission_name="invalid"), + lambda c: c["missions"][0].update(latitude=999), + lambda c: c["missions"].append(c["missions"][0]), + lambda c: c["orders"].pop("mission_name"), + lambda c: c["orders"].update(deployment_date=["unknown"]), + lambda c: c["orders"].update(event_based_sampling=[[], []])] + for change in changes: + candidate = copy.deepcopy(valid) + change(candidate) + with self.assertRaises(ValueError): + publisher.validate_catalogue(candidate) + + def test_concurrent_publication_fails(self): + self.publish() + original = self.destination.read_bytes() + lock_path = Path(str(self.destination) + ".lock") + with lock_path.open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + result = subprocess.run([sys.executable, str(SCRIPT), "--input", str(self.source), + "--output", str(self.destination), "--source-complete"], + capture_output=True, text=True) + self.assertEqual(result.returncode, 1) + self.assertIn("holds the output lock", result.stderr) + self.assertEqual(self.destination.read_bytes(), original) + self.assertTrue(lock_path.exists()) + + def test_periodic_check_waits_and_publishes_only_stable_source(self): + with patch.object(publisher.time, "sleep") as sleep: + result = self.publish(source_complete=False, stability_seconds=15) + sleep.assert_called_once_with(15) + self.assertEqual(json.loads(self.destination.read_text()), result) + original = self.destination.stat().st_mtime_ns + with patch.object(publisher.time, "sleep"), patch("gliders_catalogue.write_catalogue") as write: + self.publish(source_complete=False) + write.assert_not_called() + self.assertEqual(self.destination.stat().st_mtime_ns, original) + + def test_periodic_source_change_aborts_before_generation(self): + self.publish() + original = self.destination.read_bytes() + + def update_source(seconds): + self.source.write_text(self.html.replace("SG540", "SG541")) + + with patch.object(publisher.time, "sleep", side_effect=update_source): + with patch("gliders_catalogue.parse_catalogue") as parse: + with self.assertRaisesRegex(ValueError, "stability observation"): + self.publish(source_complete=False) + parse.assert_not_called() + self.assertEqual(self.destination.read_bytes(), original) + + def test_content_update_with_preserved_mtime_is_published(self): + self.publish() + stat = self.source.stat() + self.source.write_text(self.html.replace("SG540", "SG541")) + os.utime(self.source, ns=(stat.st_atime_ns, stat.st_mtime_ns)) + with patch.object(publisher.time, "sleep"): + result = self.publish(source_complete=False) + self.assertTrue(any(r["glider_id"] == "SG541" for r in result["missions"])) + self.assertEqual(json.loads(self.destination.read_text()), result) + + def test_observation_compares_content_even_if_stat_signature_matches(self): + self.publish() + original = self.destination.read_bytes() + html, signature = publisher._read_source(self.source) + with patch.object(publisher.time, "sleep"), patch.object( + publisher, "_read_source", side_effect=[(html, signature), (html + b" ", signature)]): + with self.assertRaisesRegex(ValueError, "stability observation"): + self.publish(source_complete=False) + self.assertEqual(self.destination.read_bytes(), original) + + def test_external_lock_and_manual_confirmation(self): + lock_path = Path(self.folder.name) / "outside-transfer.lock" + with patch.object(publisher.time, "sleep") as sleep: + self.publish(lock_file=lock_path) + sleep.assert_not_called() + self.assertTrue(lock_path.exists()) + self.assertFalse(Path(str(self.destination) + ".lock").exists()) + with lock_path.open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + with self.assertRaisesRegex(ValueError, "holds the output lock"): + self.publish(lock_file=lock_path) + with self.assertRaisesRegex(ValueError, "destination must differ"): + self.publish(lock_file=self.destination) + + def test_periodic_cli_failure_exit_and_logging(self): + self.publish() + original = self.destination.read_bytes() + lock_path = Path(self.folder.name) / "outside-transfer.lock" + + def update_source(seconds): + self.source.write_text(self.html.replace("SG540", "SG541")) + + with patch.object(publisher.time, "sleep", side_effect=update_source) as sleep: + with self.assertLogs(publisher.LOG, level="INFO") as logs: + status = publisher.main(["--input", str(self.source), "--output", str(self.destination), + "--stability-seconds", "5", "--lock-file", str(lock_path)]) + self.assertEqual(status, 1) + sleep.assert_called_once_with(5) + self.assertTrue(lock_path.exists()) + self.assertEqual(self.destination.read_bytes(), original) + messages = "\n".join(logs.output) + for expected in ("Observing HTML stability", "stability observation", "status=1", "duration_seconds="): + self.assertIn(expected, messages) + + def test_publication_failure_preserves_previous_and_cleans_temp(self): + self.publish() + original = self.destination.read_bytes() + self.source.write_text(self.html.replace("SG540", "SG541")) + with patch("gliders_catalogue.os.replace", side_effect=OSError("disk failure")): + with self.assertRaisesRegex(OSError, "disk failure"): + self.publish() + self.assertEqual(self.destination.read_bytes(), original) + self.assertEqual({p.name for p in Path(self.folder.name).iterdir()}, + {self.source.name, self.destination.name, self.destination.name + ".lock"}) + + def test_cli_exit_codes_and_logging(self): + args = ["--input", str(self.source), "--output", str(self.destination), + "--source-complete", "--initial-mission-count", str(self.count)] + with self.assertLogs(publisher.LOG, level="INFO") as logs: + self.assertEqual(publisher.main(args), 0) + messages = "\n".join(logs.output) + for expected in ("Publication started", str(self.source), str(self.destination), + "previous=None new=8", "changed=True", "status=0", "duration_seconds="): + self.assertIn(expected, messages) + with patch.dict(sys.modules, {"gliders_catalogue": None}): + with self.assertLogs(publisher.LOG, level="INFO") as logs: + self.assertEqual(publisher.main(args), 1) + self.assertIn("Validation/publication failed", "\n".join(logs.output)) + self.assertIn("status=1", "\n".join(logs.output)) + result = subprocess.run([sys.executable, str(SCRIPT), "--input", str(self.source), + "--output", str(self.destination), "--max-reduction-percent", "invalid"], + capture_output=True, text=True) + self.assertEqual(result.returncode, 2) + self.assertIn("invalid float value", result.stderr) + + def test_cli_missing_beautifulsoup_and_source_log_failures(self): + self.publish() + original = self.destination.read_bytes() + args = [str(SCRIPT), "--input", str(self.source), + "--output", str(self.destination), "--source-complete"] + # -S removes site-packages, simulating the production dependency being absent. + result = subprocess.run([sys.executable, "-S"] + args, capture_output=True, text=True) + self.assertEqual(result.returncode, 1) + self.assertIn("No module named 'bs4'", result.stderr) + self.assertIn("Validation/publication failed", result.stderr) + self.assertIn("status=1", result.stderr) + self.source.unlink() + result = subprocess.run([sys.executable] + args, capture_output=True, text=True) + self.assertEqual(result.returncode, 1) + self.assertIn("No such file", result.stderr) + self.assertIn("duration_seconds=", result.stderr) + self.assertEqual(self.destination.read_bytes(), original) + + +if __name__ == "__main__": + unittest.main()