diff --git a/docker-compose-examples/README.md b/docker-compose-examples/README.md index 792a1e3aa0..7d8393a2be 100644 --- a/docker-compose-examples/README.md +++ b/docker-compose-examples/README.md @@ -34,11 +34,19 @@ Chroma, OpenSearch, pgvector vector DBs, and a ClickHouse database) you can run alongside SEMOSS, plus notes on how to make the SEMOSS container connect to them. See [engines/README.md](engines/README.md). +[engines/semoss-maintained/](engines/semoss-maintained/) holds the services +SEMOSS itself publishes: [unoserver](engines/semoss-maintained/unoserver/) for +document conversion and [node-builder](engines/semoss-maintained/node-builder/) +for building project client apps. Every variant already sets `UNOSERVER` and +`NODE_SERVER_ENDPOINT` to their container names, so bringing them up is all it +takes to turn those features on. + ## Usage All variants attach to a shared Docker network named `semoss-net` (so the optional -services in [engines/](engines/) can reach SEMOSS by container name). Create it -once before your first `up`: +services in [engines/](engines/) can reach SEMOSS by container name). The +Monolith `local-docker-testing` stacks for locally built images join the same +network, so they see the same services. Create it once before your first `up`: ```bash docker network create semoss-net @@ -99,3 +107,6 @@ ZooKeeper, `echo ruok | nc localhost 2181` should return `imok`. only - change the credentials and integrate an external SSO before exposing any of this. - Python is enabled (`NETTY_PYTHON` / `NATIVE_PY_SERVER`); R is off (`R_ON: 'false'`). +- The node agent execution environment is on (`AGENT_DEFAULT_TOOLS_ENABLE_NODE: 'true'`), + which registers the `ExecuteNodeCode` agent tool. The image already sets + `NODE_HOME=/opt/node`; the compose files set it as well so the path is visible. diff --git a/docker-compose-examples/engines/README.md b/docker-compose-examples/engines/README.md index 7b2d61cce9..257ca621b5 100644 --- a/docker-compose-examples/engines/README.md +++ b/docker-compose-examples/engines/README.md @@ -31,6 +31,24 @@ curl http://localhost:8123/ping # clickhouse (returns " curl -i http://localhost:9100/minio/health/live # minio (204 when ready) docker exec semoss-sftp nc -z localhost 22 && echo ok # sftp curl http://localhost:8085/api/service/readiness # mail (greenmail) +curl http://localhost:8082/health # unoserver +curl http://localhost:3000/health # node-builder +``` + +## SEMOSS-maintained services + +[semoss-maintained/](semoss-maintained/) holds the services SEMOSS publishes +itself. They are not engines: SEMOSS reaches them through a platform setting, +which every SEMOSS compose file already points at the container name. + +| Folder | Image | Host port | SEMOSS setting | +|--------|-------|-----------|----------------| +| [unoserver/](semoss-maintained/unoserver/README.md) | `quay.io/semoss/unoserver:latest` | 8082 (-> 8080) | `UNOSERVER` | +| [node-builder/](semoss-maintained/node-builder/README.md) | `quay.io/semoss/smss-node-builder:latest` | 3000 | `NODE_SERVER_ENDPOINT` | + +```bash +docker compose -f semoss-maintained/unoserver/semoss-unoserver.yml up -d +docker compose -f semoss-maintained/node-builder/semoss-node-builder.yml up -d ``` ## Pointing SEMOSS at one @@ -71,6 +89,8 @@ jdbc:clickhouse://semoss-clickhouse:8123/semoss (clickhouse, HTTP port 8123) http://semoss-minio:9000 (minio, internal S3 API port 9000) semoss-sftp:22 (sftp, internal port 22) semoss-mail:3025 / :3110 / :3143 (mail, smtp / pop3 / imap) +http://semoss-unoserver:8080 (unoserver) +http://semoss-node-builder:3000 (node-builder) ``` **SEMOSS on your host** (not in Docker) - use `localhost` and the published port: @@ -84,6 +104,8 @@ jdbc:clickhouse://localhost:8123/semoss (clickhouse, host port 8123) http://localhost:9100 (minio, host port 9100) localhost:2222 (sftp, host port 2222) localhost:3025 / :3110 / :3143 (mail, same ports on the host) +http://localhost:8082 (unoserver, host port 8082) +http://localhost:3000 (node-builder) ``` > **pgvector port note:** pgvector is just Postgres, the same as the SEMOSS `db` diff --git a/docker-compose-examples/engines/semoss-maintained/node-builder/README.md b/docker-compose-examples/engines/semoss-maintained/node-builder/README.md new file mode 100644 index 0000000000..d951936a4d --- /dev/null +++ b/docker-compose-examples/engines/semoss-maintained/node-builder/README.md @@ -0,0 +1,43 @@ +# node-builder + +The SEMOSS build service, published as `quay.io/semoss/smss-node-builder`. +When a project is published (`BuildAndPublishApp`, and project restore), SEMOSS +zips the project's `assets/client` folder, posts it here, and unpacks the built +app it gets back into the project's `portals` folder. + +```bash +docker compose -f semoss-node-builder.yml up -d +curl http://localhost:3000/health # {"status":"ok","activeBuilds":0,"maxConcurrent":3} +docker compose -f semoss-node-builder.yml down +``` + +## API + +| Method | Path | What it does | +|--------|------|--------------| +| `GET` | `/health` | build counts, always HTTP 200 | +| `GET` | `/ready` | HTTP 503 while every build slot is busy | +| `POST` | `/build?buildCmd=...` | zip upload in the multipart `source` field, runs the build and returns `portals.zip` | + +## Pointing SEMOSS at it + +SEMOSS reads `NODE_SERVER_ENDPOINT` from RDF_Map.prop. In the SEMOSS image, +`runCS.sh` copies the `NODE_SERVER_ENDPOINT` environment variable into +RDF_Map.prop at startup. + +| Where SEMOSS runs | Setting | +|-------------------|---------| +| one of the SEMOSS compose files (already set) | `NODE_SERVER_ENDPOINT: 'http://semoss-node-builder:3000'` | +| on your host | `NODE_SERVER_ENDPOINT http://localhost:3000` in RDF_Map.prop | + +## Settings + +All optional, set as environment variables on the `node-builder` service: + +| Variable | Default | What it does | +|----------|---------|--------------| +| `MAX_CONCURRENT_BUILDS` | `3` | builds that run at once; more get HTTP 503 | +| `BUILD_MEMORY_LIMIT_MB` | `1024` | node heap cap (`--max-old-space-size`) for each build | +| `BUILD_TIMEOUT_MS` | `300000` | how long a build may run | +| `MAX_UPLOAD_MB` | `100` | largest accepted upload | +| `PORT` | `3000` | listen port inside the container | diff --git a/docker-compose-examples/engines/semoss-maintained/node-builder/semoss-node-builder.yml b/docker-compose-examples/engines/semoss-maintained/node-builder/semoss-node-builder.yml new file mode 100644 index 0000000000..b4159d05e3 --- /dev/null +++ b/docker-compose-examples/engines/semoss-maintained/node-builder/semoss-node-builder.yml @@ -0,0 +1,24 @@ +name: semoss-node-builder + +# node-builder - builds project client apps when they are published. +# +# Run: docker compose -f semoss-node-builder.yml up -d +# Stop: docker compose -f semoss-node-builder.yml down +# Check: curl http://localhost:3000/health +# +# Equivalent to: +# docker run -d --name semoss-node-builder -p 3000:3000 quay.io/semoss/smss-node-builder:latest +services: + node-builder: + image: quay.io/semoss/smss-node-builder:latest + container_name: semoss-node-builder + pull_policy: always + ports: + - "3000:3000" # HTTP API + networks: + - semoss-net + +networks: + # shared with the SEMOSS compose files so SEMOSS can reach this by container name + semoss-net: + external: true diff --git a/docker-compose-examples/engines/semoss-maintained/unoserver/README.md b/docker-compose-examples/engines/semoss-maintained/unoserver/README.md new file mode 100644 index 0000000000..bed7988077 --- /dev/null +++ b/docker-compose-examples/engines/semoss-maintained/unoserver/README.md @@ -0,0 +1,38 @@ +# unoserver + +LibreOffice document conversion over HTTP, published as +`quay.io/semoss/unoserver`. SEMOSS sends it a file and gets the converted file +back. It backs the `ConvertFileUnoserver` reactor, the file conversion agent hook, +and the `InspectPptx` agent tool, which renders a deck to PDF before a vision +model reviews it. + +```bash +docker compose -f semoss-unoserver.yml up -d +curl http://localhost:8082/health # {"ok":true} once LibreOffice is up +docker compose -f semoss-unoserver.yml down +``` + +The image is published for `linux/amd64` only, so the compose file sets +`platform: linux/amd64`. On Apple Silicon it runs under emulation, so startup and +conversions are slower. + +## API + +| Method | Path | What it does | +|--------|------|--------------| +| `GET` | `/health` | `{"ok": true}` when LibreOffice answers, otherwise HTTP 503 | +| `POST` | `/convert?to=pdf` | multipart upload in the `file` field, returns the converted file | + +## Pointing SEMOSS at it + +SEMOSS reads `UNOSERVER` from RDF_Map.prop and falls back to an environment +variable of the same name. + +| Where SEMOSS runs | Setting | +|-------------------|---------| +| one of the SEMOSS compose files (already set) | `UNOSERVER: 'http://semoss-unoserver:8080'` | +| on your host | `UNOSERVER http://localhost:8082` in RDF_Map.prop | + +`UNOSERVER_TIMEOUT_SECONDS` (RDF_Map.prop or environment) bounds how long SEMOSS +waits for a health check or conversion response: default `180`, allowed `1` to +`1800`. diff --git a/docker-compose-examples/engines/semoss-maintained/unoserver/semoss-unoserver.yml b/docker-compose-examples/engines/semoss-maintained/unoserver/semoss-unoserver.yml new file mode 100644 index 0000000000..97f38d5c29 --- /dev/null +++ b/docker-compose-examples/engines/semoss-maintained/unoserver/semoss-unoserver.yml @@ -0,0 +1,27 @@ +name: semoss-unoserver + +# unoserver - LibreOffice document conversion over HTTP (file conversion, pptx +# inspection). +# +# Run: docker compose -f semoss-unoserver.yml up -d +# Stop: docker compose -f semoss-unoserver.yml down +# Check: curl http://localhost:8082/health +# +# Equivalent to: +# docker run -d --name semoss-unoserver --platform linux/amd64 -p 8082:8080 quay.io/semoss/unoserver:latest +services: + unoserver: + image: quay.io/semoss/unoserver:latest + container_name: semoss-unoserver + pull_policy: always + # published for amd64 only, so it runs under emulation on Apple Silicon + platform: linux/amd64 + ports: + - "8082:8080" # HTTP API + networks: + - semoss-net + +networks: + # shared with the SEMOSS compose files so SEMOSS can reach this by container name + semoss-net: + external: true diff --git a/docker-compose-examples/semoss-with-postgres-minio-redis.yml b/docker-compose-examples/semoss-with-postgres-minio-redis.yml index b9eff4502b..4735192fdd 100644 --- a/docker-compose-examples/semoss-with-postgres-minio-redis.yml +++ b/docker-compose-examples/semoss-with-postgres-minio-redis.yml @@ -24,6 +24,14 @@ x-semoss-env: &semoss-env CHROOT_SYMLINK_PATHS: /usr/lib/python FAKECHROOT_EXCLUDE_PATH: /dev + # node agent execution environment (ExecuteNodeCode agent tool) + NODE_HOME: /opt/node + AGENT_DEFAULT_TOOLS_ENABLE_NODE: 'true' + + # semoss-maintained services, run from engines/semoss-maintained/ + UNOSERVER: 'http://semoss-unoserver:8080' + NODE_SERVER_ENDPOINT: 'http://semoss-node-builder:3000' + # local master CUSTOM_LM_RDBMS_TYPE: "postgres" CUSTOM_LM_DRIVER: "org.postgresql.Driver" diff --git a/docker-compose-examples/semoss-with-postgres-minio-zk.yml b/docker-compose-examples/semoss-with-postgres-minio-zk.yml index 2e27d2330a..2cfdf12ad9 100644 --- a/docker-compose-examples/semoss-with-postgres-minio-zk.yml +++ b/docker-compose-examples/semoss-with-postgres-minio-zk.yml @@ -24,6 +24,14 @@ x-semoss-env: &semoss-env CHROOT_SYMLINK_PATHS: /usr/lib/python FAKECHROOT_EXCLUDE_PATH: /dev + # node agent execution environment (ExecuteNodeCode agent tool) + NODE_HOME: /opt/node + AGENT_DEFAULT_TOOLS_ENABLE_NODE: 'true' + + # semoss-maintained services, run from engines/semoss-maintained/ + UNOSERVER: 'http://semoss-unoserver:8080' + NODE_SERVER_ENDPOINT: 'http://semoss-node-builder:3000' + # local master CUSTOM_LM_RDBMS_TYPE: "postgres" CUSTOM_LM_DRIVER: "org.postgresql.Driver" diff --git a/docker-compose-examples/semoss-with-postgres-minio.yml b/docker-compose-examples/semoss-with-postgres-minio.yml index 893ae86840..c7dfaa6975 100644 --- a/docker-compose-examples/semoss-with-postgres-minio.yml +++ b/docker-compose-examples/semoss-with-postgres-minio.yml @@ -32,6 +32,14 @@ services: CHROOT_SYMLINK_PATHS: /usr/lib/python FAKECHROOT_EXCLUDE_PATH: /dev + # node agent execution environment (ExecuteNodeCode agent tool) + NODE_HOME: /opt/node + AGENT_DEFAULT_TOOLS_ENABLE_NODE: 'true' + + # semoss-maintained services, run from engines/semoss-maintained/ + UNOSERVER: 'http://semoss-unoserver:8080' + NODE_SERVER_ENDPOINT: 'http://semoss-node-builder:3000' + # local master CUSTOM_LM_RDBMS_TYPE : "postgres" CUSTOM_LM_DRIVER: "org.postgresql.Driver" diff --git a/docker-compose-examples/semoss-with-postgres.yml b/docker-compose-examples/semoss-with-postgres.yml index 987ee8631e..226619c9d1 100644 --- a/docker-compose-examples/semoss-with-postgres.yml +++ b/docker-compose-examples/semoss-with-postgres.yml @@ -32,6 +32,14 @@ services: CHROOT_SYMLINK_PATHS: /usr/lib/python FAKECHROOT_EXCLUDE_PATH: /dev + # node agent execution environment (ExecuteNodeCode agent tool) + NODE_HOME: /opt/node + AGENT_DEFAULT_TOOLS_ENABLE_NODE: "true" + + # semoss-maintained services, run from engines/semoss-maintained/ + UNOSERVER: "http://semoss-unoserver:8080" + NODE_SERVER_ENDPOINT: "http://semoss-node-builder:3000" + # local master CUSTOM_LM_RDBMS_TYPE: "postgres" CUSTOM_LM_DRIVER: "org.postgresql.Driver"