Add ndi.update: check for and install a newer NDI-python (mirrors NDI-matlab#1038)

.github/workflows/installer.yml

@@ -209,6 +209,31 @@ jobs:
           done
           "$bin/uv$x" --version
 
+      - name: The receipt says what was installed, and ndi.update reads it
+        working-directory: ${{ runner.temp }}
+        run: |
+          if [ "$RUNNER_OS" = Windows ]; then
+            py="$HOME/.ndi/venvs/ndi/Scripts/python.exe"
+          else
+            py="$HOME/.ndi/venvs/ndi/bin/python"
+          fi
+          cat "$HOME/.ndi/install.json"
+          # Written by the installer, read by ndi.update and by NDI-matlab's
+          # ndi.update.checkPython. git was hidden during the install, so a
+          # local-checkout install records no commit -- only the time.
+          "$py" -c "
+          import re, sys
+          from ndi.update import installType
+          kind, info = installType()
+          r = info['Receipt']
+          assert kind == 'installer', kind
+          assert r is not None, 'no readable ~/.ndi/install.json'
+          assert r['installer'] in ('install-ndi.sh', 'install-ndi.ps1'), r
+          assert re.fullmatch(r'\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ', r['installedAt']), r
+          assert r['commit'] is None or re.fullmatch('[0-9a-f]{40}', r['commit']), r
+          print('receipt ok:', r)
+          "
+
       - name: The tool environment is this platform's, and pyraview a release wheel
         working-directory: ${{ runner.temp }}
         run: |

CHANGELOG.md

@@ -8,6 +8,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- **Checking for, and installing, a newer NDI-python** (mirroring
+  VH-Lab/NDI-matlab#1038). New `ndi.update`: `check()` compares this copy with
+  the latest commit on main -- a git clone through `git fetch` (plus the
+  `~/.ndi/tools` clones `ndi_install.py` makes, and the user's own changed
+  files, new files and local commits), an installer install through the commit
+  in `~/.ndi/install.json` -- and `install()` updates it: `git pull --ff-only`
+  then `ndi_install.py --update`, or the NDI installer again. It refuses a
+  clone not on main, and lists the user's own changes and asks (default No)
+  before touching them. `ndi update` / `ndi update --check [--json]` on the
+  command line, an **Update** button in the navigator's NDI pane, and a quiet
+  once-a-day check when the navigator opens (`Update.CheckOnStartup`,
+  `Update.CheckIntervalDays`, shared with NDI-matlab; `Update.Python.LastCheckTime`).
+  The installers now install the exact commit main is at and record it, with
+  the install time, in `~/.ndi/install.json` (removed by `--uninstall`), which
+  NDI-matlab's `ndi.update.checkPython` reads too. `ndi.version()` reports that
+  commit for an installer install.
+
 - **Users in several NDI Cloud organizations** (mirroring VH-Lab/NDI-matlab#1037).
   Login no longer keeps only the first organization: it caches the user's
   whole list -- id, name and upload permission -- as JSON in

install/README.md

@@ -38,13 +38,14 @@ terminal window** and try `ndi check`.
 
 Everything goes in a hidden `.ndi` folder in your home folder
 (`~/.ndi` on macOS and Linux, `%USERPROFILE%\.ndi` on Windows), which
-NDI-matlab uses too. The installer owns only three folders there:
+NDI-matlab uses too. The installer owns only three folders and one file there:
 
-| folder | what is in it |
+| folder or file | what is in it |
 |---|---|
 | `.ndi/bin` | the `ndi`, `napariViewGEF` and `napariViewLightsheet` commands, and [uv](https://docs.astral.sh/uv/), the tool that installs them |
 | `.ndi/python` | a private copy of Python 3.12 |
 | `.ndi/venvs` | NDI and the packages it needs |
+| `.ndi/install.json` | which commit of NDI was installed, and when, so `ndi update` (and NDI-matlab) can tell whether there is a newer one |
 
 It never touches anything else in `.ndi`, such as your
 `NDI_Preferences.json`. Outside `.ndi` it only adds `.ndi/bin` to the end of
@@ -55,7 +56,11 @@ on Windows), which makes updates faster; `~/.ndi/bin/uv cache clean` empties it.
 
 ## Update
 
-Run the installer again, the same way. It replaces NDI with the latest version.
+`ndi update --check` says whether a newer NDI is available, and `ndi update`
+installs it (after asking). The navigator's **Update** button does the same,
+and NDI-matlab's `ndi.update.install()` / Update button update this
+installation too. Or run the installer again, the same way: it replaces NDI
+with the latest version.
 
 ## Uninstall
 
@@ -68,8 +73,9 @@ powershell -ExecutionPolicy Bypass -NoProfile -c "& ([scriptblock]::Create((irm
 ```
 
 (or run `install-ndi.sh --uninstall` / `install-ndi.ps1 -Uninstall` from this
-folder). This removes `.ndi/bin`, `.ndi/python`, `.ndi/venvs` and the `PATH`
-lines, and leaves the rest of `.ndi` (your settings) alone.
+folder). This removes `.ndi/bin`, `.ndi/python`, `.ndi/venvs`,
+`.ndi/install.json` and the `PATH` lines, and leaves the rest of `.ndi` (your
+settings) alone.
 
 ## Options
 

install/install-ndi.ps1

@@ -10,10 +10,13 @@
   the ndi, napariViewGEF and napariViewLightsheet commands.
 
   WHAT IT TOUCHES. Everything lives under %USERPROFILE%\.ndi, which NDI-matlab
-  shares, and this script owns ONLY three folders there:
-    .ndi\bin      uv itself, plus ndi.exe / napariViewGEF.exe / napariViewLightsheet.exe
-    .ndi\python   the Python interpreter uv downloads (UV_PYTHON_INSTALL_DIR)
-    .ndi\venvs    the environment NDI is installed into (UV_TOOL_DIR)
+  shares, and this script owns ONLY three folders and one file there:
+    .ndi\bin           uv itself, plus ndi.exe / napariViewGEF.exe / napariViewLightsheet.exe
+    .ndi\python        the Python interpreter uv downloads (UV_PYTHON_INSTALL_DIR)
+    .ndi\venvs         the environment NDI is installed into (UV_TOOL_DIR)
+    .ndi\install.json  which commit was installed, and when, so that
+                       ndi.update (and NDI-matlab's ndi.update.checkPython)
+                       can tell whether main has moved on since
   Everything else in .ndi (NDI_Preferences.json, tools, caches, ...) is never
   touched, by install or by -Uninstall. Outside .ndi it changes only your
   user-level PATH (unless -NoPath); and uv keeps its download cache in its
@@ -65,6 +68,9 @@ $NdiHome = Join-Path $env:USERPROFILE '.ndi'
 $Bin = Join-Path $NdiHome 'bin'
 $PyDir = Join-Path $NdiHome 'python'
 $Venvs = Join-Path $NdiHome 'venvs'
+$Receipt = Join-Path $NdiHome 'install.json'
+$CommitsApi = 'https://api.github.com/repos/Waltham-Data-Science/NDI-python/commits/main'
+$ArchiveBase = 'https://github.com/Waltham-Data-Science/NDI-python/archive'
 
 function Fail([string]$Reason) {
     # One line saying why, and a nonzero exit. [Console]::Error so it is
@@ -138,13 +144,17 @@ function Remove-NdiFromPath {
 # ---- uninstall ---------------------------------------------------------------
 
 if ($Uninstall) {
-    Write-Host "Removing NDI from $NdiHome (bin, python, venvs) ..."
+    Write-Host "Removing NDI from $NdiHome (bin, python, venvs, install.json) ..."
     foreach ($d in @($Bin, $PyDir, $Venvs)) {
         if (Test-Path -LiteralPath $d) {
             try { Remove-Item -LiteralPath $d -Recurse -Force }
             catch { Fail "could not remove $d (is an NDI program still running?)" }
         }
     }
+    if (Test-Path -LiteralPath $Receipt) {
+        try { Remove-Item -LiteralPath $Receipt -Force }
+        catch { Fail "could not remove $Receipt" }
+    }
     Remove-NdiFromPath
     Write-Host "Your NDI settings and anything else in $NdiHome were left in place."
     Write-Output "NDI uninstalled: $Bin"
@@ -201,8 +211,31 @@ $env:UV_PYTHON_PREFERENCE = 'only-managed' # never a system/Store/conda Python t
 $env:UV_NO_CONFIG = '1'                    # ignore any uv.toml of yours
 $env:UV_NO_MODIFY_PATH = '1'
 
+# Which commit is being installed, for .ndi\install.json (step 4). For the
+# default source, ask GitHub for main's current commit and install exactly
+# that commit's archive, so the receipt cannot name a different commit from
+# the one installed. If GitHub's API cannot be reached (offline, rate limit),
+# install main as before and record no commit; ndi.update then falls back to
+# the install time. A local checkout records its HEAD when git is there.
+$Commit = $null
 $Source = [Environment]::GetEnvironmentVariable('NDI_INSTALL_SOURCE')
-if (-not $Source) { $Source = $DefaultSource }
+if (-not $Source) {
+    $Source = $DefaultSource
+    try {
+        $sha = (Invoke-RestMethod -Uri $CommitsApi -TimeoutSec 30 -UseBasicParsing).sha
+        if ($sha -match '^[0-9a-f]{40}$') {
+            $Commit = $sha
+            $Source = "$ArchiveBase/$Commit.zip"
+        }
+    } catch {
+        Write-Host '  (could not ask GitHub which commit main is at; installing main)'
+    }
+} elseif ((Test-Path -LiteralPath $Source -PathType Container) -and (Get-Command git -ErrorAction SilentlyContinue)) {
+    $ErrorActionPreference = 'Continue'
+    $head = & git -C $Source rev-parse HEAD 2>$null
+    if ($LASTEXITCODE -eq 0 -and $head -match '^[0-9a-f]{40}$') { $Commit = $head }
+    $ErrorActionPreference = 'Stop'
+}
 if (Test-Path -LiteralPath $Source) {
     # A local checkout (CI): a file:/// URL to its absolute path.
     $Source = ([System.Uri](Resolve-Path -LiteralPath $Source).ProviderPath).AbsoluteUri
@@ -253,7 +286,21 @@ if ($checkExit -ne 0) {
 }
 Write-Host '  ndi check: all checks passed'
 
-# 4. PATH.
+# 4. The receipt: what was installed, from where, and when. Written only
+# after the check passed, so it never describes an install that failed.
+# UTF-8 without a byte-order mark: Windows PowerShell 5.1's -Encoding UTF8
+# writes one, and JSON readers (Python's json, MATLAB's jsondecode) reject it.
+$receiptJson = [ordered]@{
+    commit      = $Commit
+    source      = $Source
+    installedAt = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd'T'HH:mm:ss'Z'")
+    installer   = 'install-ndi.ps1'
+} | ConvertTo-Json
+try { [System.IO.File]::WriteAllText($Receipt, $receiptJson, (New-Object System.Text.UTF8Encoding $false)) }
+catch { Fail "could not write $Receipt" }
+if ($Commit) { Write-Host "  installed commit $Commit" }
+
+# 5. PATH.
 if (-not $NoPath) {
     Add-NdiToPath
     Write-Host 'Open a new terminal window to use the ndi, napariViewGEF and napariViewLightsheet commands.'

install/install-ndi.sh

@@ -13,10 +13,13 @@
 #   --uninstall  remove what this script installed, and nothing else
 #
 # WHAT IT TOUCHES. Everything lives under ~/.ndi, which NDI-matlab shares,
-# and this script owns ONLY three folders there:
-#   ~/.ndi/bin      uv itself, plus the ndi / napariViewGEF / napariViewLightsheet commands
-#   ~/.ndi/python   the Python interpreter uv downloads (UV_PYTHON_INSTALL_DIR)
-#   ~/.ndi/venvs    the environment NDI is installed into (UV_TOOL_DIR)
+# and this script owns ONLY three folders and one file there:
+#   ~/.ndi/bin           uv itself, plus the ndi / napariViewGEF / napariViewLightsheet commands
+#   ~/.ndi/python        the Python interpreter uv downloads (UV_PYTHON_INSTALL_DIR)
+#   ~/.ndi/venvs         the environment NDI is installed into (UV_TOOL_DIR)
+#   ~/.ndi/install.json  which commit was installed, and when, so that
+#                        ndi.update (and NDI-matlab's ndi.update.checkPython)
+#                        can tell whether main has moved on since
 # Everything else in ~/.ndi (NDI_Preferences.json, tools/, caches, ...) is
 # never touched, by install or by --uninstall. Outside ~/.ndi it edits only
 # your shell start-up files, inside a marked block (see add_to_path), unless
@@ -53,6 +56,9 @@ NDI_HOME="$HOME/.ndi"
 BIN="$NDI_HOME/bin"
 PYDIR="$NDI_HOME/python"
 VENVS="$NDI_HOME/venvs"
+RECEIPT="$NDI_HOME/install.json"
+COMMITS_API="https://api.github.com/repos/Waltham-Data-Science/NDI-python/commits/main"
+ARCHIVE_BASE="https://github.com/Waltham-Data-Science/NDI-python/archive"
 
 PATH_BEGIN="# >>> NDI: added by install-ndi.sh; remove with install-ndi.sh --uninstall >>>"
 PATH_END="# <<< NDI <<<"
@@ -184,12 +190,13 @@ remove_from_path() {
 # ---- uninstall ---------------------------------------------------------------
 
 if [ "$UNINSTALL" = 1 ]; then
-    say "Removing NDI from $NDI_HOME (bin, python, venvs) ..."
+    say "Removing NDI from $NDI_HOME (bin, python, venvs, install.json) ..."
     for d in "$BIN" "$PYDIR" "$VENVS"; do
         # uv marks some of its files read-only; make them removable first.
         if [ -d "$d" ]; then chmod -R u+w "$d" 2>/dev/null || true; fi
         rm -rf "$d" || fail "could not remove $d"
     done
+    rm -f "$RECEIPT" || fail "could not remove $RECEIPT"
     remove_from_path
     say "Your NDI settings and anything else in $NDI_HOME were left in place."
     say "NDI uninstalled: $BIN"
@@ -258,7 +265,27 @@ export UV_PYTHON_PREFERENCE=only-managed  # never a system/Homebrew/conda Python
 export UV_NO_CONFIG=1                   # ignore any uv.toml of yours
 export UV_NO_MODIFY_PATH=1
 
+# Which commit is being installed, for ~/.ndi/install.json (step 4). For the
+# default source, ask GitHub for main's current commit and install exactly
+# that commit's archive, so the receipt cannot name a different commit from
+# the one installed. If GitHub's API cannot be reached (offline, rate limit),
+# install main as before and record no commit; ndi.update then falls back to
+# the install time. A local checkout records its HEAD when git is there.
+COMMIT=""
 SOURCE="${NDI_INSTALL_SOURCE:-$DEFAULT_SOURCE}"
+if [ -z "${NDI_INSTALL_SOURCE:-}" ]; then
+    tmp=$(mktemp)
+    if download "$COMMITS_API" "$tmp" 2>/dev/null; then
+        # The commit's own sha is the first "sha" at the top level (2 spaces in).
+        COMMIT=$(sed -n 's/^  "sha": "\([0-9a-f]\{40\}\)".*$/\1/p' "$tmp" | head -n 1)
+    fi
+    rm -f "$tmp"
+    if [ -n "$COMMIT" ]; then
+        SOURCE="$ARCHIVE_BASE/$COMMIT.zip"
+    fi
+elif [ -d "$SOURCE" ] && command -v git >/dev/null 2>&1; then
+    COMMIT=$(git -C "$SOURCE" rev-parse HEAD 2>/dev/null || true)
+fi
 if [ -d "$SOURCE" ]; then
     # A local checkout (CI): a file URL to its absolute path.
     SOURCE="file://$(cd "$SOURCE" && pwd -P | sed 's/ /%20/g')"
@@ -303,7 +330,17 @@ fi
 rm -f "$log"
 say "  ndi check: all checks passed"
 
-# 4. PATH.
+# 4. The receipt: what was installed, from where, and when. Written only
+# after the check passed, so it never describes an install that failed.
+if [ -n "$COMMIT" ]; then commit_json="\"$COMMIT\""; else commit_json=null; fi
+printf '{\n  "commit": %s,\n  "source": "%s",\n  "installedAt": "%s",\n  "installer": "install-ndi.sh"\n}\n' \
+    "$commit_json" "$SOURCE" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$RECEIPT" \
+    || fail "could not write $RECEIPT"
+if [ -n "$COMMIT" ]; then
+    say "  installed commit $COMMIT"
+fi
+
+# 5. PATH.
 if [ "$NO_PATH" = 0 ]; then
     add_to_path
     say "Open a new terminal window to use the ndi, napariViewGEF and napariViewLightsheet commands."

src/ndi/__init__.py

@@ -88,8 +88,11 @@ def version() -> tuple:
 
     Returns:
         Tuple of ``(version_string, url)``.  The version string is the
-        Git commit hash when running inside a checkout, otherwise the
-        package version.
+        Git commit hash when running inside a checkout; for a copy the NDI
+        installer put in ``~/.ndi``, the commit its ``install.json`` records
+        (MATLAB likewise reports an installed toolbox's release version);
+        otherwise the package version. To check for a newer version, see
+        :func:`ndi.update.check`.
     """
     import subprocess
     from pathlib import Path as _Path
@@ -109,6 +112,15 @@ def version() -> tuple:
             return result.stdout.strip(), url
     except Exception:
         pass
+    try:
+        from .update.install_type import installType
+
+        kind, info = installType()
+        commit = (info.get("Receipt") or {}).get("commit") if kind == "installer" else None
+        if commit:
+            return str(commit)[:7], url
+    except Exception:
+        pass
     return __version__, url
 
 

src/ndi/__main__.py

@@ -18,6 +18,7 @@ def main() -> int:
         print("Commands:")
         print("  check     Validate NDI installation")
         print("  version   Show version information")
+        print("  update    Update NDI-python (--check: only report; --help: options)")
         print("  napari    View a spatial gene expression pyramid")
         return 0
 
@@ -33,6 +34,8 @@ def main() -> int:
         from ndi.gui.app.genepyramid.cli import main as napari_main
 
         return napari_main()
+    elif command == "update":
+        return _update_main(sys.argv[1:])
     elif command == "version":
         from ndi import version
 
@@ -46,5 +49,42 @@ def main() -> int:
         return 1
 
 
+def _update_main(argv: list[str]) -> int:
+    """``ndi update``: check for, and install, a newer NDI-python."""
+    import argparse
+    import json
+
+    parser = argparse.ArgumentParser(
+        prog="ndi update",
+        description="Update NDI-python to the latest version on its main branch.",
+    )
+    parser.add_argument("--check", action="store_true", help="only report; change nothing")
+    parser.add_argument(
+        "--json", action="store_true", help="with --check, print the full result as JSON"
+    )
+    parser.add_argument("--yes", action="store_true", help="do not ask before updating")
+    parser.add_argument(
+        "--allow-modified",
+        action="store_true",
+        help="with --yes, update even if you have changed NDI-python's code",
+    )
+    parser.add_argument("--force", action="store_true", help="reinstall even if up to date")
+    args = parser.parse_args(argv)
+
+    from ndi import update
+
+    if args.check:
+        result = update.check(Verbose=not args.json)
+        if args.json:
+            print(json.dumps(result, indent=2))
+        return 0
+    try:
+        update.install(Confirm=not args.yes, AllowModified=args.allow_modified, Force=args.force)
+    except update.UpdateError as exc:
+        print(exc, file=sys.stderr)
+        return 1
+    return 0
+
+
 if __name__ == "__main__":
     sys.exit(main())