Unify docker compose so local dev needs only an .env, no override yml

.gitignore

@@ -47,3 +47,7 @@ python_data_import/debug.log.txt
 python_data_import/logs.txt
 python_data_import/date.txt
 */__pycache__/
+
+# Local-dev compose env created by copying docker/.env.local.example. Per-developer,
+# never committed. The .example template IS committed.
+/docker/.env.local

.nvmrc

@@ -0,0 +1 @@
+18.20.5

AGENTS.md

@@ -0,0 +1,206 @@
+# AGENTS.md — testing this branch (dataquest dspace-angular)
+
+Goal: spin up the **backend in Docker** and the **frontend natively** (live-editable
+`ng serve`) and confirm they talk to each other, with the fewest moving parts.
+
+Validated end-to-end on `internal/unify-docker-compose` (Windows + Docker Desktop, Compose
+v2.40): BE up from one `.env.local`, FE on `ng serve` rendering the LINDAT/CLARIAH-CZ home in
+a real browser with demo content (6 communities, 471 indexed items), 200s, no CORS errors.
+Builds on PR #1289, which makes `docker/docker-compose-rest.yml` drivable from one `--env-file`.
+
+> Several non-obvious traps below cost real time to find — read **Gotchas** before starting.
+> **Match the BE to the FE version:** this FE reports **7.6.5** (see the browser console
+> startup banner), so use a **7.6.5** backend image. A 7.5 BE (`dtq-dev-7.5`) parses most
+> things but errors on browse definitions.
+
+---
+
+## Prerequisites
+
+- **Docker** running.
+- **Node 18** for the frontend. NOT 20+/24 — the Angular 15 toolchain breaks on newer Node in
+  ways CI doesn't catch (CI runs `build:prod`, never `ng serve`). Portable install, no global
+  change:
+  ```bash
+  D=/c/Users/$USER/AppData/Local/Temp/node18setup; mkdir -p "$D" && cd "$D"
+  curl -fsSL -o n18.zip https://nodejs.org/dist/v18.20.5/node-v18.20.5-win-x64.zip
+  /c/Windows/System32/tar.exe -xf n18.zip                 # MSYS tar can't unzip; use Windows bsdtar
+  ./node-v18.20.5-win-x64/npm.cmd install -g yarn@1.22.19
+  export PATH="$D/node-v18.20.5-win-x64:$PATH"            # prepend for FE commands
+  ```
+- **The `ng serve` fix** (Gotcha #2): the branch pins `copy-webpack-plugin@^6.4.1`, which
+  breaks `ng serve`. Bump it to `^11.0.0` in `package.json` and `yarn install`. (This is now
+  committed on PR #1289.)
+
+---
+
+## TL;DR — two commands
+
+```bash
+# 0) one-time: use Node 18 (see .nvmrc) and install deps
+nvm use            # or: see Prerequisites for a portable Node 18
+yarn install
+
+# 1) BACKEND: DSpace 7.6.5 + demo content at http://127.0.0.1:8087/server  (~2-4 min first run)
+build-scripts/run/dev.backend.sh           # add 'fresh' to wipe volumes + reload from scratch
+
+# 2) FRONTEND: live-reload dev server on http://localhost:4000
+yarn start:dev:local
+```
+
+`dev.backend.sh` brings the backend up (correct 7.6.5 images, IPv4 host, CORS, demo dataset)
+and reindexes Solr; `start:dev:local` is `ng serve` with the right `DSPACE_REST_*` env baked in.
+Stop/wipe the backend with `docker compose -f docker/docker-compose-rest.yml -f docker/db.entities.yml down -v`.
+
+Everything below is the manual/explained version of those two commands (for customizing the
+instance, image set, or running the FE in Docker). The FE maps env vars onto `config/config.yml`
+(`rest.host`→`DSPACE_REST_HOST`, … ; env wins last — see `src/config/config.server.ts`).
+
+---
+
+## Backend (Docker)
+
+`docker/docker-compose-rest.yml` fully defines the network, so the BE comes up from that file
+(plus `db.entities.yml` for demo data) — no `docker-compose.yml` (that's the FE container).
+
+`docker/.env.local` for this recipe:
+
+```ini
+INSTANCE=7
+# 127.0.0.1, NOT localhost (Gotcha #5). BE self-links use REST_URL, so keep it on 127.0.0.1.
+DSPACE_HOST=127.0.0.1
+DSPACE_REST_PORT=808${INSTANCE}            # -> 8087
+DSPACE_REST_NAMESPACE=/server
+REST_URL=http://127.0.0.1:808${INSTANCE}/server
+UI_URL=http://localhost:4000               # native FE dev-server port, NOT 400${INSTANCE}
+HOST_IP=127.0.0.1
+DSPACE_SUBNET_PREFIX=10.10${INSTANCE}
+REST_CORS_ALLOWED_ORIGINS=http://localhost:4000,http://127.0.0.1:4000
+
+# --- Option A: demo content (VERIFIED) — upstream 7.6.5 + db.entities.yml -------------------
+# Matches the FE's 7.6.5 and loads the official DSpace demo entities dataset. NOTE: this is
+# UPSTREAM DSpace data on an UPSTREAM BE, so CLARIN-specific FE calls 404 (harmless to render).
+DSPACE_REST_IMAGE=dspace/dspace:dspace-7_x
+DSPACE_DB_IMAGE=dspace/dspace-postgres-pgcrypto:dspace-7_x   # db.entities.yml overrides to -loadsql
+DSPACE_SOLR_IMAGE=dspace/dspace-solr:dspace-7_x
+DOCKER_REGISTRY=docker.io                  # consumed by db.entities.yml to resolve the -loadsql image
+DOCKER_OWNER=dspace
+DSPACE_VER=dspace-7_x
+
+# --- Option B: CLARIN fidelity (version-matched, but no content unless you have a dump) ------
+# DSPACE_REST_IMAGE=dataquest/dspace:dspace-7_x            # DSpace 7.6.5, CLARIN tables/endpoints
+# DSPACE_DB_IMAGE=dataquest/dspace-postgres-pgcrypto:dspace-7_x
+# DSPACE_SOLR_IMAGE=dataquest/dspace-solr:dspace-7_x
+# Fresh DB = empty homepage. For real CLARIN content, restore a dataquest/LINDAT DB dump into
+# dspacedb7 (pg_restore/psql) instead of layering db.entities.yml, then reindex.
+```
+
+- Bring up Option A with **both** files: `-f docker/docker-compose-rest.yml -f docker/db.entities.yml`.
+  The `-loadsql` Postgres downloads + imports `dspace7-entities-data.sql` on first boot; the BE
+  then runs `database migrate ignored`. **Then reindex Solr** (command above) or the homepage's
+  browse/search/"What's New" stay empty even though the DB has data.
+- For Option B (no `db.entities.yml`), use just `-f docker/docker-compose-rest.yml`.
+- **Ports** (INSTANCE=7): REST `8087`, JVM debug `8007`, Postgres `5437`, Solr `8987`. Change
+  `INSTANCE` to re-target all at once. 5 and 8 are reserved by `.github/workflows/deploy.yml`.
+- Admin user (optional): `… -f docker/cli.yml run --rm dspace-cli create-administrator -e admin@test.dev -f admin -l user -p admin -c en -o dataquest`
+
+### Verify the BE
+```bash
+curl -s http://127.0.0.1:8087/server/api                       # dspaceVersion => DSpace 7.6.5
+curl -s http://127.0.0.1:8087/server/api/core/communities/search/top  # totalElements: 6 (demo)
+curl -s -i -X OPTIONS http://127.0.0.1:8087/server/api/core/items \
+  -H 'Origin: http://localhost:4000' -H 'Access-Control-Request-Method: GET' \
+  | grep -i access-control-allow-origin                         # => http://localhost:4000
+```
+
+---
+
+## Frontend (native, live-editable)
+
+```bash
+export PATH="/c/Users/$USER/AppData/Local/Temp/node18setup/node-v18.20.5-win-x64:$PATH"
+yarn install        # after the copy-webpack-plugin bump, or if node_modules is partial
+DSPACE_REST_SSL=false DSPACE_REST_HOST=127.0.0.1 DSPACE_REST_PORT=8087 \
+  DSPACE_UI_HOST=localhost DSPACE_UI_PORT=4000 \
+  yarn start:dev      # ng serve, live-reload, http://localhost:4000
+```
+
+- `yarn start:dev` = `ng serve` (CSR dev server, fast rebuilds — what you want for editing).
+  `yarn start` does a full SSR production build instead.
+- The browser calls the BE directly at `http://127.0.0.1:8087/server`; the CORS list must include
+  `http://localhost:4000` (it does).
+- Don't set `DSPACE_REST_NAMESPACE` from a Git-Bash shell (Gotcha #1); config default `/server`
+  is correct.
+
+### What success looks like
+`http://localhost:4000` redirects to `/home` and renders the **LINDAT/CLARIAH-CZ Repository
+Home**: search + facets (Author/Subject/Language), a **"What's New"** list of items, and the
+footer. DevTools → Network shows `…8087/server/api/*` 200s with no CORS errors. Benign console
+errors are normal: `favicon.ico` 404, Matomo refused, `google.analytics.key` 404, `/security/csrf`
+404, and (with Option A's upstream BE) some CLARIN-specific 404s.
+
+---
+
+## Gotchas (each one cost time to find)
+
+1. **Git Bash mangles leading-slash paths.** `DSPACE_REST_NAMESPACE=/server` becomes
+   `C:/Program Files/Git/server`, and `docker exec dspace7 /dspace/bin/dspace …` becomes
+   `…/Git/dspace/bin/dspace` (no such file). Fix: prefix the command with
+   `MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*'`. (Values inside the `.env.local` file are NOT
+   mangled — only shell args.)
+
+2. **`ng serve` is broken on this branch out of the box.** `package.json` pins
+   `copy-webpack-plugin@^6.4.1`, but build-angular@15 (via `@angular-builders/custom-webpack`)
+   emits asset-copy patterns using the `priority` option (v9+) → `Copy Plugin … unknown property
+   'priority'`. CI misses it (it runs `build:prod`, not `ng serve`). Fix: `^11.0.0` (matches
+   build-angular) + `yarn install`. (Committed on PR #1289.)
+
+3. **Stale / partial `node_modules`.** `Can't resolve 'd3'` / `'ngx-skeleton-loader'` + a cascade
+   of `NG6002`/`NG8004` → run a full `yarn install`.
+
+4. **Stale Postgres volume version mismatch.** Re-using an old `dspace-7_pgdata` initialised by a
+   different Postgres major gives `FATAL: database files are incompatible with server`; the BE
+   then hangs forever in its DB-wait loop. Fix: `down -v` for a clean start (`--remove-orphans`
+   clears a leftover `dspace-angular<INSTANCE>` container).
+
+5. **`localhost` ≠ `127.0.0.1` for the FE→BE hop.** Node 18 resolves `localhost` to IPv6 `::1`
+   with no IPv4 fallback, but the BE publishes on `127.0.0.1` only, so Node fetches `ECONNREFUSED`
+   → `undefined doesn't contain the link sites`. Worse, a host MISMATCH (FE on `127.0.0.1` but BE
+   self-links on `localhost`) made the HAL parser resolve objects under two hostnames and recurse
+   → `RangeError: Maximum call stack size exceeded` in `DspaceRestResponseParsingService`. Keep
+   **both** `DSPACE_REST_HOST` and `REST_URL` on `127.0.0.1`. (`curl` hides this — it falls back
+   to IPv4.)
+
+6. **Match the BE to the FE version.** The FE reports **7.6.5**, so use a **7.6.5** BE
+   (`dspace/dspace:dspace-7_x` or `dataquest/dspace:dspace-7_x`). The `dataquest/dspace:dtq-dev-7.5`
+   image is **DSpace 7.5** and causes `An error occurred while retrieving the browse definitions`
+   in the console.
+
+7. **Demo data ≠ empty repo, and Solr needs reindexing.** A fresh DB shows an empty homepage —
+   that's expected, not a bug. Layer `db.entities.yml` (Option A) for sample content, then run
+   `index-discovery -b` so browse/search/"What's New" populate. (The public
+   `dspace7-entities-data.sql` now ships future 8.0/9.0/10.0 flyway entries; a 7.6.5 BE still
+   reads the data fine via `migrate ignored`.)
+
+8. **Orphaned `ng serve` keeps port 4000.** Stopping `nodemon` doesn't kill its `ng serve` child,
+   so the next start crashes on `Port 4000 is already in use`. Kill the listener first:
+   ```bash
+   pid=$(netstat -ano | grep LISTENING | grep ':4000' | awk '{print $NF}' | head -1)
+   taskkill //F //T //PID $pid
+   ```
+   And do NOT switch git branches while `ng serve` runs — it watches the working tree and will
+   recompile against the wrong files. Stop it first (or use a separate `git worktree`).
+
+---
+
+## Alternative: fully containerized FE (PR #1289's documented path)
+
+To run the FE in Docker too, use `docker/.env.local.example` as-is (`host.docker.internal`,
+`HOST_IP=0.0.0.0`, CORS on `400${INSTANCE}`) and bring up **both** compose files with `--build`:
+```bash
+docker compose --env-file docker/.env.local \
+  -f docker/docker-compose.yml -f docker/docker-compose-rest.yml up -d --build
+# FE on http://localhost:4007 ; this build path exercises the Dockerfile cp fix from PR #1289.
+```
+`HOST_IP=0.0.0.0` publishes every BE port (incl. Postgres pwd `dspace`, JVM debug) on all
+interfaces — dev-only, not for an untrusted network.

Dockerfile

@@ -28,5 +28,13 @@ ENV NODE_ENV=development
 RUN apk add tzdata
 RUN yarn build:prod
 RUN npm install pm2 -g
-CMD /bin/sh -c "pm2-runtime start docker/dspace-ui.json > /dev/null 2> /dev/null"
+
+# Mirror Dockerfile.dist's layout so docker-compose.yml's entrypoint
+# (`pm2-runtime start dspace-ui.json`, no `docker/` prefix) works for both
+# the locally-built dev image and the published dist image. Before this,
+# locally-built containers ENOENT-looped because compose's entrypoint
+# pointed at the dist path while the file sat at /app/docker/dspace-ui.json.
+RUN cp docker/dspace-ui.json /app/dspace-ui.json
+
+CMD /bin/sh -c "pm2-runtime start dspace-ui.json > /dev/null 2> /dev/null"
 

build-scripts/run/dev.backend.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+#
+# One-command local-dev BACKEND for working on the native frontend.
+#
+# Brings up a DSpace 7.6.5 backend (matching this FE's version) in Docker, loaded with the
+# official demo entities dataset, reachable at http://127.0.0.1:8087/server, then indexes Solr
+# so browse/search/"What's New" are populated. After it prints "Backend ready", start the FE:
+#
+#     yarn start:dev:local      # ng serve, live-reload, http://localhost:4000  (needs Node 18 — see .nvmrc)
+#
+# Usage:
+#     build-scripts/run/dev.backend.sh           # up (reuses existing containers/data)
+#     build-scripts/run/dev.backend.sh fresh     # wipe DB/Solr volumes first, then up (clean slate)
+#
+# Notes:
+#   - 127.0.0.1 (not localhost): Node resolves localhost to IPv6 ::1, but the BE binds IPv4 only.
+#   - Demo data is UPSTREAM DSpace (not CLARIN), so some CLARIN-specific FE calls 404 — harmless.
+#     For CLARIN content, point DSPACE_REST_IMAGE at dataquest/dspace:dspace-7_x and restore a
+#     dataquest/LINDAT DB dump instead of using db.entities.yml.
+#
+set -uo pipefail
+cd "$(dirname "$0")/../.." || exit 1
+
+export INSTANCE="${INSTANCE:-7}"
+export COMPOSE_PROJECT_NAME="dspace-${INSTANCE}"
+export DSPACE_HOST=127.0.0.1
+export DSPACE_REST_NAMESPACE=/server
+export REST_URL="http://127.0.0.1:808${INSTANCE}/server"
+export UI_URL="http://localhost:4000"
+export HOST_IP=127.0.0.1
+export DSPACE_SUBNET_PREFIX="10.10${INSTANCE}"
+export REST_CORS_ALLOWED_ORIGINS="http://localhost:4000,http://127.0.0.1:4000"
+# DSpace 7.6.5 to match the FE; upstream images + the demo entities dataset (db.entities.yml).
+export DSPACE_REST_IMAGE=dspace/dspace:dspace-7_x
+export DSPACE_DB_IMAGE=dspace/dspace-postgres-pgcrypto:dspace-7_x
+export DSPACE_SOLR_IMAGE=dspace/dspace-solr:dspace-7_x
+export DOCKER_REGISTRY=docker.io DOCKER_OWNER=dspace DSPACE_VER=dspace-7_x
+
+COMPOSE=(docker compose -f docker/docker-compose-rest.yml -f docker/db.entities.yml)
+REST="http://127.0.0.1:808${INSTANCE}/server/api"
+
+if [ "${1:-}" = "fresh" ]; then
+  echo ">> wiping previous dev backend (down -v)"
+  "${COMPOSE[@]}" down -v --remove-orphans || true
+fi
+
+echo ">> starting backend: DSpace 7.6.5 + demo entities (project ${COMPOSE_PROJECT_NAME})"
+if ! "${COMPOSE[@]}" up -d; then
+  echo "!! 'up' failed. If it's a Postgres version/volume mismatch, run: $0 fresh" >&2
+  exit 1
+fi
+
+echo -n ">> waiting for REST API (${REST}) "
+for _ in $(seq 1 90); do
+  [ "$(curl -s -o /dev/null -w '%{http_code}' "$REST" 2>/dev/null)" = "200" ] && { echo " ready"; break; }
+  printf '.'; sleep 5
+done
+if [ "$(curl -s -o /dev/null -w '%{http_code}' "$REST" 2>/dev/null)" != "200" ]; then
+  echo " timed out. Check: ${COMPOSE[*]} logs dspace${INSTANCE}" >&2
+  exit 1
+fi
+
+echo ">> indexing Solr discovery (so browse/search/What's New populate)"
+# MSYS_NO_PATHCONV stops Git Bash from rewriting /dspace/... into a Windows path.
+MSYS_NO_PATHCONV=1 docker exec "dspace${INSTANCE}" /dspace/bin/dspace index-discovery -b \
+  || echo "   (reindex failed — rerun: MSYS_NO_PATHCONV=1 docker exec dspace${INSTANCE} /dspace/bin/dspace index-discovery -b)"
+
+cat <<MSG
+
+==================================================================
+ Backend ready:  ${REST%/api}     (DSpace 7.6.5, demo content)
+ Start the FE :  yarn start:dev:local      (Node 18 — see .nvmrc)
+ Then open    :  http://localhost:4000/
+ Stop / wipe  :  ${COMPOSE[*]} down -v
+==================================================================
+MSG