diff --git a/README.md b/README.md index 175aaee..719579d 100644 --- a/README.md +++ b/README.md @@ -1,213 +1,32 @@ # ATT&CK Workbench Deployment -This repository contains deployment files for the ATT&CK Workbench, a web application for editing ATT&CK data represented in STIX. It is composed of a frontend SPA, a backend REST API, and a database. Optionally, you can deploy a "sidecar service" that makes your Workbench data available over a TAXII 2.1 API. +This repository contains deployment files for the ATT&CK Workbench, a web application for editing ATT&CK data represented in STIX. +It is composed of a frontend Single Page App (SPA), a backend REST API, and a database. +Optionally, you can deploy a "sidecar service" that makes your Workbench data available over a TAXII 2.1 API. -## Deployment Options +## Docker Setup -### Docker Compose +To quickly create and deploy a custom Workbench instance using Docker Compose use the interactive setup script in the `docker/` directory. -The ATT&CK Workbench can be deployed using Docker Compose with two different configurations: +See [docker/README](docker/README.md) for detailed instructions. -#### 1. Using Pre-built Images (Recommended) - -Use `compose.yaml` to pull pre-built images directly from GitHub Container Registry (GHCR): - -```bash -# Deploy with pre-built images -docker compose up -d - -# Deploy with TAXII server -docker compose --profile with-taxii up -d - -# Stop the deployment -docker compose down -``` - -#### 2. Building from Source - -Use `compose.dev.yaml` in combination with `compose.yaml` to build images from source code: - -```bash -# Build and deploy from source -docker compose -f compose.yaml -f compose.dev.yaml up -d --build - -# Build and deploy with TAXII server -docker compose -f compose.yaml -f compose.dev.yaml --profile with-taxii up -d --build - -# Stop the deployment -docker compose -f compose.yaml -f compose.dev.yaml down -``` - -**Note**: When building from source, you need the following three source repositories to be available as sibling directories to this deployment repository: - -- [attack-workbench-frontend](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/) -- [attack-workbench-rest-api](https://github.com/center-for-threat-informed-defense/attack-workbench-rest-api/) -- [attack-workbench-taxii-server](https://mitre-attack/attack-workbench-taxii-server/) - -The directory structure should look like this: - -```bash -. -├── attack-workbench-deployment -├── attack-workbench-frontend -├── attack-workbench-rest-api -└── attack-workbench-taxii-server (optional) -``` - -### Kubernetes +## Kubernetes Setup For production deployments, Kubernetes manifests with Kustomize are available in the `k8s/` directory. -See [k8s/README.md](k8s/README.md) for detailed instructions. - -## Configuration - -### Environment Variables - -We make heavy use of string interpolation to minimize having to modify the Docker Compose manifest files (e.g., [compose.yaml](./compose.yaml)). Consequently, that means you must set a bunch of environment variables when using these templates. Fortunately, we've provided a dotenv template that you can source. - -Copy `template.env` to `.env` and customize the values as needed: - -```bash -cp template.env .env -``` - -Available environment variables: - -| Variable | Default Value | Description | -|----------|---------------|-------------| -| **Docker Image Tags** | | | -| `ATTACKWB_FRONTEND_VERSION` | `latest` | Frontend Docker image tag | -| `ATTACKWB_RESTAPI_VERSION` | `latest` | REST API Docker image tag | -| `ATTACKWB_TAXII_VERSION` | `latest` | TAXII server Docker image tag | -| **HTTP Listener Ports** | | | -| `ATTACKWB_FRONTEND_HTTP_PORT` | `80` | Frontend HTTP port | -| `ATTACKWB_FRONTEND_HTTPS_PORT` | `443` | Frontend HTTPS port | -| `ATTACKWB_RESTAPI_HTTP_PORT` | `3000` | REST API port | -| `ATTACKWB_DB_PORT` | `27017` | MongoDB port | -| `ATTACKWB_TAXII_HTTP_PORT` | `5002` | TAXII server port | -| **SSL/TLS Configuration** | | | -| `ATTACKWB_FRONTEND_CERTS_PATH` | `./certs` | Path to SSL certificates | -| **TAXII Configuration** | | | -| `ATTACKWB_TAXII_ENV` | `dev` | Specifies the name of the dotenv file to load (e.g., A value of `dev` tells the TAXII server to load `dev.env`) | - -### Service-Specific Configuration - -Each service has its own configuration directory: - -#### Frontend - -**Default config files**: `configs/frontend/` - -The frontend container is an Nginx instance which serves the frontend SPA and reverse proxies requests to the backend REST API. -We provide a basic `nginx.conf` template in the aforementioned directory that should get you started. -Refer to the [frontend documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend) -for further details on customizing the SPA. - -#### REST API - -> [!IMPORTANT] -> The REST API service requires the `SESSION_SECRET` environment variable to be set in order to deploy. -> Without it set, `docker compose up` will fail to start this required service. - -**Default config files**: `configs/rest-api/` - -The backend REST API loads runtime configurations from environment variables, as well as from a JSON configuration file. -Templates are provided in the aforementioned directory. -Refer to the [REST API usage documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-rest-api/blob/main/USAGE.md#configuration) -for further details on customizing the backend. - -#### TAXII Server - -**Default config files**: `configs/taxii/config/` - -The TAXII server loads all runtime configuration parameters from a dotenv file. -The specific filename of the dotenv file is specified by the `ATTACKWB_TAXII_ENV` environment variable. -For example, a value of `dev` tells the TAXII server to load `dev.env`. - -## Quick Start - -1. Clone this repository: - - ```bash - git clone https://github.com/center-for-threat-informed-defense/attack-workbench-deployment.git - cd attack-workbench-deployment - ``` - -2. Configure environment variables (optional): - - ```bash - cp template.env .env - # Edit .env with your preferred settings - ``` - -3. Configure REST API environment variables (required): - - ```bash - cp configs/rest-api/template.env configs/rest-api/.env - ``` - - Generate a secure random secret - - ```bash - node -e "console.log(require('crypto').randomBytes(48).toString('base64'))" - ``` - - Set the above secret in `configs/rest-api/.env` - - ```bash - SESSION_SECRET= - ``` - -4. Deploy using pre-built images: - - ```bash - docker compose up -d - ``` - -5. Access the application at `http://localhost` (or your configured port) - -6. To include the TAXII server: - - ```bash - docker compose --profile with-taxii up -d - ``` - -## Data Persistence - -MongoDB data is persisted in the `workspace-data` named Docker volume. Thus, the `database` service can be deleted and re-deployed without losing access to the database. The database volume will be remounted to the `database` service upon deployment. - -## Troubleshooting - -### Check Service Status - -```bash -# View running containers -docker compose ps - -# Show logs for all running containers -docker compose logs - -# Follow logs -docker compose logs -f - -# Show logs for a specific container -docker compose logs frontend -docker compose logs rest-api -docker compose logs database -docker compose logs taxii -``` -## Contributing +See [k8s/README](k8s/README.md) for detailed instructions. -Please refer to the [contribution guide](./docs/CONTRIBUTING.md) for contribution guidelines, as well as the [developer guide](./docs/DEVELOPMENT.md) for information on our release process. +## Troubleshooting & Support -## License +- View logs: `docker compose logs -f` +- Check running containers: `docker compose ps` -This project is licensed under the Apache License 2.0. See the [LICENSE](./LICENSE) file for details. +More tips in [docs/troubleshooting](docs/troubleshooting.md). -## Support +For questions or issues, visit the [GitHub issues page](https://github.com/mitre-attack/attack-workbench-deployment/issues). -For issues and questions: +## Contributing & License -- Check the [deployment repository issues](https://github.com/center-for-threat-informed-defense/attack-workbench-deployment/issues) -- Refer to the main [ATT&CK Workbench documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend) +- Contribution guide: [contribution guide](./docs/CONTRIBUTING.md) +- Developer guide: [developer guide](./docs/DEVELOPMENT.md) +- License: [Apache License 2.0](./LICENSE) diff --git a/certs/README.md b/certs/README.md index ab60ab3..a294f6e 100644 --- a/certs/README.md +++ b/certs/README.md @@ -36,7 +36,7 @@ If you're using environment variables in your shell, you can use: ```yaml volumes: - - .${HOST_CERTS_PATH}:/usr/src/app/certs + - ${HOST_CERTS_PATH}:/usr/src/app/certs environment: - NODE_EXTRA_CA_CERTS=./certs/${CERTS_FILENAME} ``` diff --git a/compose.dev.yaml b/compose.dev.yaml deleted file mode 100644 index 9c18d81..0000000 --- a/compose.dev.yaml +++ /dev/null @@ -1,13 +0,0 @@ -services: - - frontend: - image: attack-workbench-frontend - build: ../attack-workbench-frontend - - rest-api: - image: attack-workbench-rest-api - build: ../attack-workbench-rest-api - - taxii: - image: attack-workbench-taxii-server - build: ../attack-workbench-taxii-server diff --git a/compose.yaml b/compose.yaml deleted file mode 100644 index d6ec03b..0000000 --- a/compose.yaml +++ /dev/null @@ -1,86 +0,0 @@ -services: - - frontend: - container_name: attack-workbench-frontend - image: ghcr.io/center-for-threat-informed-defense/attack-workbench-frontend:${ATTACKWB_FRONTEND_VERSION:-latest} - depends_on: - - rest-api - ports: - - "${ATTACKWB_FRONTEND_HTTP_PORT:-80}:${ATTACKWB_FRONTEND_HTTP_PORT:-80}" - - "${ATTACKWB_FRONTEND_HTTPS_PORT:-443}:${ATTACKWB_FRONTEND_HTTPS_PORT:-443}" - volumes: - - ./configs/frontend/nginx.conf:/etc/nginx/nginx.conf:ro - - "${ATTACKWB_FRONTEND_CERTS_PATH:-./certs}:/etc/nginx/certs:ro" - restart: unless-stopped - logging: - driver: "json-file" - options: - max-size: "10m" - max-file: "5" - - rest-api: - container_name: attack-workbench-rest-api - image: ghcr.io/center-for-threat-informed-defense/attack-workbench-rest-api:${ATTACKWB_RESTAPI_VERSION:-latest} - depends_on: - - database - ports: - - "${ATTACKWB_RESTAPI_HTTP_PORT:-3000}:${ATTACKWB_RESTAPI_HTTP_PORT:-3000}" - volumes: - - ./configs/rest-api/rest-api-service-config.json:/usr/src/app/resources/rest-api-service-config.json:ro - env_file: - - ./configs/rest-api/.env - restart: unless-stopped - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:3000/api/health/ping"] - interval: 30s - timeout: 10s - retries: 3 - logging: - driver: "json-file" - options: - max-size: "10m" - max-file: "5" - - database: - container_name: attack-workbench-database - image: mongo:8 - ports: - - "${ATTACKWB_DB_PORT:-27017}:${ATTACKWB_DB_PORT:-27017}" - volumes: - - workspace-data:/data/db - - ./database-backup:/dump - restart: unless-stopped - healthcheck: - test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"] - interval: 30s - timeout: 10s - retries: 5 - logging: - driver: "json-file" - options: - max-size: "10m" - max-file: "5" - - taxii: - container_name: attack-workbench-taxii-server - image: ghcr.io/mitre-attack/attack-workbench-taxii-server:${ATTACKWB_TAXII_VERSION:-latest} - depends_on: - - rest-api - ports: - - "${ATTACKWB_TAXII_HTTP_PORT:-5002}:${ATTACKWB_TAXII_HTTP_PORT:-5002}" - volumes: - - ./configs/taxii/config:/app/config:ro - environment: - - TAXII_ENV=${ATTACKWB_TAXII_ENV:-dev} - - TAXII_HYDRATE_ON_BOOT=true - profiles: - - with-taxii - restart: unless-stopped - logging: - driver: "json-file" - options: - max-size: "10m" - max-file: "5" - -volumes: - workspace-data: diff --git a/configs/frontend/nginx.conf b/configs/frontend/nginx.conf deleted file mode 100644 index 62ddc56..0000000 --- a/configs/frontend/nginx.conf +++ /dev/null @@ -1,48 +0,0 @@ -worker_processes 1; - -events { - worker_connections 1024; -} - -http { - server { - listen 80; - server_name localhost; - - root /usr/share/nginx/html; - index index.html index.htm; - include /etc/nginx/mime.types; - - gzip on; - gzip_min_length 1000; - gzip_proxied expired no-cache no-store private auth; - gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; - - location / { - try_files $uri $uri/ /index.html; - } - - location /api { - client_max_body_size 50M; - - # Disable buffering for SSE streams - proxy_buffering off; - proxy_cache off; - - # Keep connection alive for long-running requests - proxy_read_timeout 600s; - proxy_connect_timeout 300s; - proxy_send_timeout 300s; - - # Required headers for SSE - proxy_set_header Connection ''; - proxy_http_version 1.1; - chunked_transfer_encoding on; - - # Disable compression for SSE - gzip off; - - proxy_pass http://attack-workbench-rest-api:3000; - } - } -} diff --git a/configs/rest-api/template.env b/configs/rest-api/template.env deleted file mode 100644 index 7b8874e..0000000 --- a/configs/rest-api/template.env +++ /dev/null @@ -1,66 +0,0 @@ -# HTTP Listener Port -PORT=3000 - -# CORS (`*`, `disable`, or comma-separated list of FQDNs) -CORS_ALLOWED_ORIGINS=* - -# Environment -NODE_ENV=development - -# Database -DATABASE_URL=mongodb://attack-workbench-database/attack-workspace - -# Database Migration -WB_REST_DATABASE_MIGRATION_ENABLE=true - -# Authentication Mechanism -AUTHN_MECHANISM=anonymous - -# OIDC Authentication -AUTHN_OIDC_CLIENT_ID= -AUTHN_OIDC_CLIENT_SECRET= -AUTHN_OIDC_ISSUER_URL= -AUTHN_OIDC_REDIRECT_ORIGIN=http://localhost:3000 - -# Service Account Authentication - OIDC Client Credentials -SERVICE_ACCOUNT_OIDC_ENABLE=false -JWKS_URI= - -# Service Account Authentication - Challenge API Key -WB_REST_SERVICE_ACCOUNT_CHALLENGE_APIKEY_ENABLE=false -WB_REST_TOKEN_SIGNING_SECRET= -WB_REST_TOKEN_TIMEOUT=300 - -# Service Account Authentication - Basic API Key -WB_REST_SERVICE_ACCOUNT_BASIC_APIKEY_ENABLE=false - -# Collection Index Interval -DEFAULT_INTERVAL=300 - -# Configuration File Path -JSON_CONFIG_PATH= - -# Logging -LOG_LEVEL=info - -# Static Marking Definitions Path -WB_REST_STATIC_MARKING_DEFS_PATH=./app/lib/default-static-marking-definitions/ - -# Allowed Values Configuration File Path -ALLOWED_VALUES_PATH=./app/config/allowed-values.json - -# Scheduler Settings -CHECK_WORKBENCH_INTERVAL=10 -ENABLE_SCHEDULER=true - -########## -# OPTIONAL -########## - -# Session Configuration -# Generate a secure random secret with: node -e "console.log(require('crypto').randomBytes(48).toString('base64'))" -# If not provided, the REST API will generate one for you at startup (not recommended for production) -#SESSION_SECRET= - -# Path to additional CA certificates file in PEM format -#NODE_EXTRA_CA_CERTS= diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000..718dcc0 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,30 @@ +# Docker Compose Setup + +Use the interactive setup script `setup-workbench.sh` to quickly create and deploy a custom Workbench instance: + +```bash +# Clone and run setup script +git clone https://github.com/mitre-attack/attack-workbench-deployment.git +cd attack-workbench-deployment/docker/ +./setup-workbench.sh +``` + +After running the script, deploy with: + +```bash +cd ../instances/your-instance-name + +# deploy with docker compose +docker compose up -d + +# or deploy in development mode +docker compose up -d --build +``` + +Access Workbench at + +Full variable descriptions and examples are available in [docs/configuration](docs/configuration.md). + +For source builds or TAXII setup, see [docs/deployment](docs/deployment.md). + +For information on how to backup or restore the mongo database, see [docs/database-backups](docs/database-backups.md). diff --git a/compose.certs.yaml b/docker/example-setup/compose.certs.yaml similarity index 87% rename from compose.certs.yaml rename to docker/example-setup/compose.certs.yaml index 19667b8..ad5102a 100644 --- a/compose.certs.yaml +++ b/docker/example-setup/compose.certs.yaml @@ -18,6 +18,6 @@ services: rest-api: volumes: - - .${HOST_CERTS_PATH}:/usr/src/app/certs + - ${HOST_CERTS_PATH:-./certs}:/usr/src/app/certs environment: - - NODE_EXTRA_CA_CERTS=./certs/${CERTS_FILENAME} + - NODE_EXTRA_CA_CERTS=./certs/${CERTS_FILENAME:-custom-certs.pem} diff --git a/docker/example-setup/compose.dev.yaml b/docker/example-setup/compose.dev.yaml new file mode 100644 index 0000000..75366ac --- /dev/null +++ b/docker/example-setup/compose.dev.yaml @@ -0,0 +1,54 @@ +services: + + frontend: + image: attack-workbench-frontend + build: ../../../attack-workbench-frontend + develop: + watch: + # Sync source files for hot-reload + - action: sync + path: ../../../attack-workbench-frontend/src + target: /app/src + ignore: + - node_modules/ + # Rebuild on package.json changes + - action: rebuild + path: ../../../attack-workbench-frontend/package.json + + rest-api: + image: attack-workbench-rest-api + build: ../../../attack-workbench-rest-api + develop: + watch: + # Sync app source files + - action: sync + path: ../../../attack-workbench-rest-api/app + target: /usr/src/app/app + ignore: + - node_modules/ + # Restart on config changes + - action: sync+restart + path: ../../../attack-workbench-rest-api/resources + target: /usr/src/app/resources + # Rebuild on package.json changes + - action: rebuild + path: ../../../attack-workbench-rest-api/package.json + + taxii: + image: attack-workbench-taxii-server + build: ../../../attack-workbench-taxii-server + develop: + watch: + # Sync source files + - action: sync + path: ../../../attack-workbench-taxii-server/taxii + target: /app/taxii + ignore: + - node_modules/ + # Sync config files and restart + - action: sync+restart + path: ../../../attack-workbench-taxii-server/config + target: /app/config + # Rebuild on package.json changes + - action: rebuild + path: ../../../attack-workbench-taxii-server/package.json diff --git a/docker/example-setup/compose.insights.yaml b/docker/example-setup/compose.insights.yaml new file mode 100644 index 0000000..03b5d2a --- /dev/null +++ b/docker/example-setup/compose.insights.yaml @@ -0,0 +1,47 @@ +services: + + grafana: + image: grafana/grafana-oss:${ATTACKWB_GRAFANA_VERSION:-latest} + depends_on: + mongodb: + condition: service_healthy + ports: + - "${ATTACKWB_GRAFANA_PORT:-3001}:3000" + volumes: + - grafana-data:/var/lib/grafana + - "${ATTACKWB_GRAFANA_PROVISIONING_PATH:-./configs/grafana/provisioning}:/etc/grafana/provisioning:ro" + env_file: + - "${ATTACKWB_GRAFANA_ENV_FILE:-./configs/grafana/.env}" + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"] + interval: 15s + timeout: 5s + retries: 5 + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + + insights-agent: + image: attack-workbench-insights-agent + build: ../../insights-agent + depends_on: + mongodb: + condition: service_healthy + grafana: + condition: service_healthy + volumes: + - "${ATTACKWB_INSIGHTS_AGENT_SYSTEM_PROMPT_FILE:-../../insights-agent/system-prompt.md}:/app/config/system-prompt.md:ro" + env_file: + - "${ATTACKWB_INSIGHTS_AGENT_ENV_FILE:-./configs/insights-agent/.env}" + environment: + - ATTACKWB_CUSTOM_CA_CERT=/run/attackwb/custom-ca.pem + - SYSTEM_PROMPT_PATH=/app/config/system-prompt.md + restart: "no" + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" diff --git a/docker/example-setup/compose.taxii.yaml b/docker/example-setup/compose.taxii.yaml new file mode 100644 index 0000000..5aa1dac --- /dev/null +++ b/docker/example-setup/compose.taxii.yaml @@ -0,0 +1,18 @@ +services: + + taxii: + image: ghcr.io/mitre-attack/attack-workbench-taxii-server:${ATTACKWB_TAXII_VERSION:-latest} + depends_on: + - rest-api + ports: + - "${ATTACKWB_TAXII_HTTP_PORT:-8000}:8000" + volumes: + - "${ATTACKWB_TAXII_CONFIG_DIR:-./configs/taxii/config}:/app/config:ro" + environment: + - TAXII_ENV=${ATTACKWB_TAXII_ENV:-dev} + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" diff --git a/docker/example-setup/compose.yaml b/docker/example-setup/compose.yaml new file mode 100644 index 0000000..c264fd7 --- /dev/null +++ b/docker/example-setup/compose.yaml @@ -0,0 +1,63 @@ +services: + + frontend: + image: ghcr.io/center-for-threat-informed-defense/attack-workbench-frontend:${ATTACKWB_FRONTEND_VERSION:-latest} + depends_on: + - rest-api + ports: + - "${ATTACKWB_FRONTEND_HTTP_PORT:-80}:80" + - "${ATTACKWB_FRONTEND_HTTPS_PORT:-443}:443" + volumes: + - "${ATTACKWB_FRONTEND_NGINX_CONFIG_FILE:-./configs/frontend/nginx.api.conf}:/etc/nginx/conf.d/default.conf:ro" + - "${ATTACKWB_FRONTEND_CERTS_PATH:-./certs}:/etc/nginx/certs:ro" + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + + rest-api: + image: ghcr.io/center-for-threat-informed-defense/attack-workbench-rest-api:${ATTACKWB_RESTAPI_VERSION:-latest} + depends_on: + - mongodb + ports: + - "${ATTACKWB_RESTAPI_HTTP_PORT:-3000}:3000" + volumes: + - "${ATTACKWB_RESTAPI_CONFIG_FILE:-./configs/rest-api/rest-api-service-config.json}:/usr/src/app/resources/rest-api-service-config.json:ro" + env_file: + - "${ATTACKWB_RESTAPI_ENV_FILE:-./configs/rest-api/.env}" + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:3000/api/health/ping"] + interval: 30s + timeout: 10s + retries: 3 + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + + mongodb: + image: mongo:8 + ports: + - "127.0.0.1:${ATTACKWB_DB_PORT:-27017}:27017" + volumes: + - workspace-data:/data/db + - "${ATTACKWB_DB_BACKUP_PATH:-./database-backup}:/dump" + restart: unless-stopped + healthcheck: + test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"] + interval: 30s + timeout: 10s + retries: 5 + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + +volumes: + workspace-data: + grafana-data: diff --git a/docker/example-setup/configs/frontend/nginx.api.conf b/docker/example-setup/configs/frontend/nginx.api.conf new file mode 100644 index 0000000..6789577 --- /dev/null +++ b/docker/example-setup/configs/frontend/nginx.api.conf @@ -0,0 +1,42 @@ +server { + listen 80; + http2 on; + server_name _; + + root /usr/share/nginx/html; + index index.html index.htm; + include /etc/nginx/mime.types; + + gzip on; + gzip_min_length 1000; + gzip_proxied expired no-cache no-store private auth; + gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; + + location / { + try_files $uri $uri/ /index.html; + } + + location /api { + client_max_body_size 50M; + + # Disable buffering for SSE streams + proxy_buffering off; + proxy_cache off; + + # Keep connection alive for long-running requests + proxy_read_timeout 600s; + proxy_connect_timeout 300s; + proxy_send_timeout 300s; + + # Required headers for SSE + proxy_set_header Connection ''; + proxy_http_version 1.1; + chunked_transfer_encoding on; + + # Disable compression for SSE + gzip off; + + proxy_pass http://rest-api:3000; + } + +} diff --git a/docker/example-setup/configs/frontend/nginx.api.ssl.conf b/docker/example-setup/configs/frontend/nginx.api.ssl.conf new file mode 100644 index 0000000..2c2bb87 --- /dev/null +++ b/docker/example-setup/configs/frontend/nginx.api.ssl.conf @@ -0,0 +1,51 @@ +server { + listen 80; + server_name _; + return 301 https://$host$request_uri; +} + +server { + listen 443 ssl default_server; + http2 on; + server_name _; + + ssl_certificate /etc/nginx/certs/server.pem; + ssl_certificate_key /etc/nginx/certs/server.key; + + root /usr/share/nginx/html; + index index.html index.htm; + include /etc/nginx/mime.types; + + gzip on; + gzip_min_length 1000; + gzip_proxied expired no-cache no-store private auth; + gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; + + location / { + try_files $uri $uri/ /index.html; + } + + location /api { + client_max_body_size 50M; + + # Disable buffering for SSE streams + proxy_buffering off; + proxy_cache off; + + # Keep connection alive for long-running requests + proxy_read_timeout 600s; + proxy_connect_timeout 300s; + proxy_send_timeout 300s; + + # Required headers for SSE + proxy_set_header Connection ''; + proxy_http_version 1.1; + chunked_transfer_encoding on; + + # Disable compression for SSE + gzip off; + + proxy_pass http://rest-api:3000; + } + +} diff --git a/docker/example-setup/configs/frontend/nginx.conf b/docker/example-setup/configs/frontend/nginx.conf new file mode 100644 index 0000000..6123cec --- /dev/null +++ b/docker/example-setup/configs/frontend/nginx.conf @@ -0,0 +1,19 @@ +server { + listen 80; + http2 on; + server_name _; + + root /usr/share/nginx/html; + index index.html index.htm; + include /etc/nginx/mime.types; + + gzip on; + gzip_min_length 1000; + gzip_proxied expired no-cache no-store private auth; + gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; + + location / { + try_files $uri $uri/ /index.html; + } + +} diff --git a/docker/example-setup/configs/frontend/nginx.ssl.conf b/docker/example-setup/configs/frontend/nginx.ssl.conf new file mode 100644 index 0000000..d423ee4 --- /dev/null +++ b/docker/example-setup/configs/frontend/nginx.ssl.conf @@ -0,0 +1,28 @@ +server { + listen 80; + server_name _; + return 301 https://$host$request_uri; +} + +server { + listen 443 ssl default_server; + http2 on; + server_name _; + + ssl_certificate /etc/nginx/certs/server.pem; + ssl_certificate_key /etc/nginx/certs/server.key; + + root /usr/share/nginx/html; + index index.html index.htm; + include /etc/nginx/mime.types; + + gzip on; + gzip_min_length 1000; + gzip_proxied expired no-cache no-store private auth; + gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; + + location / { + try_files $uri $uri/ /index.html; + } + +} diff --git a/docker/example-setup/configs/grafana/provisioning/dashboards/.gitkeep b/docker/example-setup/configs/grafana/provisioning/dashboards/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/docker/example-setup/configs/grafana/provisioning/dashboards/.gitkeep @@ -0,0 +1 @@ + diff --git a/docker/example-setup/configs/grafana/provisioning/datasources/.gitkeep b/docker/example-setup/configs/grafana/provisioning/datasources/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/docker/example-setup/configs/grafana/provisioning/datasources/.gitkeep @@ -0,0 +1 @@ + diff --git a/docker/example-setup/configs/grafana/template.env b/docker/example-setup/configs/grafana/template.env new file mode 100644 index 0000000..faced41 --- /dev/null +++ b/docker/example-setup/configs/grafana/template.env @@ -0,0 +1,13 @@ +# Grafana OSS - Environment Configuration Template + +# Administrator account +GF_SECURITY_ADMIN_USER=admin +GF_SECURITY_ADMIN_PASSWORD=admin + +# Access controls +GF_AUTH_ANONYMOUS_ENABLED=true +GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer + +# Plugins +# Comma-separated plugin IDs to install at startup. +GF_INSTALL_PLUGINS=grafana-mongodb-datasource diff --git a/docker/example-setup/configs/insights-agent/template.env b/docker/example-setup/configs/insights-agent/template.env new file mode 100644 index 0000000..106a1ac --- /dev/null +++ b/docker/example-setup/configs/insights-agent/template.env @@ -0,0 +1,29 @@ +# ATT&CK Workbench Insights Agent - Environment Configuration Template + +# Database +# Prefer a single MongoDB URL with the database name included. +# Example (Docker): mongodb://mongodb/attack-workspace +# Example (local): mongodb://localhost:27017/attack-workspace +MONGODB_DATABASE_URL= + +# Legacy fallback settings. These are only used if MONGODB_DATABASE_URL is empty. +#MONGODB_URI=mongodb://mongodb:27017 +#MONGODB_DATABASE=attack-workspace + +# Grafana +GRAFANA_URL=http://grafana:3000 +#GRAFANA_API_KEY= +GRAFANA_ADMIN_USER=admin +GRAFANA_ADMIN_PASSWORD=admin + +# LLM provider +LLM_BASE_URL= +LLM_API_KEY= +LLM_API_TYPE=auto +#LLM_API_VERSION= +#LLM_AZURE_DEPLOYMENT= +LLM_MODEL=gpt-4o + +# Scheduling +# 0 means run once and exit. +RUN_INTERVAL_SECONDS=0 diff --git a/configs/rest-api/rest-api-service-config.json b/docker/example-setup/configs/rest-api/rest-api-service-config.json similarity index 100% rename from configs/rest-api/rest-api-service-config.json rename to docker/example-setup/configs/rest-api/rest-api-service-config.json diff --git a/docker/example-setup/configs/rest-api/template.env b/docker/example-setup/configs/rest-api/template.env new file mode 100644 index 0000000..42dfb3f --- /dev/null +++ b/docker/example-setup/configs/rest-api/template.env @@ -0,0 +1,147 @@ +# Attack Workbench REST API - Environment Configuration Template +# Guidance: +# - Booleans: use true or false +# - Lists: use comma-separated values + +# Server +# PORT (int) - HTTP server port +# Default: 3000 +#PORT=3000 + +# Database (REQUIRED) +# DATABASE_URL (string) - MongoDB connection string +# Example (Docker): +#DATABASE_URL=mongodb://mongodb/attack-workspace +# Example (local): +#DATABASE_URL=mongodb://localhost:27017/attack-workspace +DATABASE_URL= + +# CORS_ALLOWED_ORIGINS (domains) - Allowed origins for REST API +# Accepts: +# * : allow any origin +# disable : disable CORS +# Comma-separated list of origins (http/https), e.g.: +# http://localhost:3000,https://example.com,https://sub.domain.org:8443 +# Supports localhost, private IPv4 (10.x, 172.16-31.x, 192.168.x), and FQDNs. +# Default: * +#CORS_ALLOWED_ORIGINS=* + +# Application +# NODE_ENV (string) - Environment name +# Options: development, production, test +# Default: development +#NODE_ENV=development + +# WB_REST_DATABASE_MIGRATION_ENABLE (bool) - Auto-run DB migrations on startup +# Default: true +#WB_REST_DATABASE_MIGRATION_ENABLE=true + +# Logging +# LOG_LEVEL (string) - Console log level +# Options: error, warn, http, info, verbose, debug +# Default: info +#LOG_LEVEL=info + +# Workbench Collection Indexes +# DEFAULT_INTERVAL (int, seconds) - Default polling interval for new indexes +# Note: does not affect existing indexes +# Default: 300 +#DEFAULT_INTERVAL=300 + +# Configuration Files +# JSON_CONFIG_PATH (string) - Path to a JSON file with additional configuration. +# Use this to provide arrays for service accounts and OIDC clients +# +# Some example values which align to some sample configurations which can be found here: +# https://github.com/center-for-threat-informed-defense/attack-workbench-rest-api/tree/main/resources/sample-configurations +# +# ./resources/collection-manager-apikey.json +# ./resources/collection-manager-oidc-keycloak.json +# ./resources/collection-manager-oidc-okta.json +# +# Default: empty (disabled) +#JSON_CONFIG_PATH= + +# ALLOWED_VALUES_PATH (string) - Path to allowed values configuration file +# Default: ./app/config/allowed-values.json +#ALLOWED_VALUES_PATH=./app/config/allowed-values.json + +# WB_REST_STATIC_MARKING_DEFS_PATH (string) - Directory of static marking definition JSON files +# Default: ./app/lib/default-static-marking-definitions/ +#WB_REST_STATIC_MARKING_DEFS_PATH=./app/lib/default-static-marking-definitions/ + +# Scheduler +# ENABLE_SCHEDULER (bool) - Enable background scheduler +# Default: true +#ENABLE_SCHEDULER=true + +# CHECK_WORKBENCH_INTERVAL (int, seconds) - Scheduler start interval +# Default: 10 +#CHECK_WORKBENCH_INTERVAL=10 + +# Session +# SESSION_SECRET (string) - Secret to sign session cookies. +# Default: securely generated at startup (changes on restart; not recommended for production). +# Generate with: node -e "console.log(require('crypto').randomBytes(48).toString('base64'))" +#SESSION_SECRET= + +# MONGOSTORE_CRYPTO_SECRET (string) - Secret to encrypt session data in MongoDB. +# Default: securely generated at startup (changes on restart; not recommended for production). +# Generate with: node -e "console.log(require('crypto').randomBytes(48).toString('base64'))" +#MONGOSTORE_CRYPTO_SECRET= + +# User Authentication +# AUTHN_MECHANISM (enum) - User login mechanism +# Options: anonymous, oidc +# Default: anonymous +#AUTHN_MECHANISM=anonymous + +# OIDC settings (required if AUTHN_MECHANISM=oidc) +# AUTHN_OIDC_ISSUER_URL (string) - OIDC issuer URL (e.g., https://idp.example.com) +# Default: empty +#AUTHN_OIDC_ISSUER_URL= +# AUTHN_OIDC_CLIENT_ID (string) - OIDC client ID +# Default: empty +#AUTHN_OIDC_CLIENT_ID= +# AUTHN_OIDC_CLIENT_SECRET (string) - OIDC client secret +# Default: empty +#AUTHN_OIDC_CLIENT_SECRET= +# AUTHN_OIDC_REDIRECT_ORIGIN (string) - Origin used to build redirect URI +# Example: http://localhost:3000 -> http://localhost:3000/authn/oidc/callback +# Default: http://localhost:3000 +#AUTHN_OIDC_REDIRECT_ORIGIN=http://localhost:3000 + +# Service Authentication +# OIDC Client Credentials (service-to-service) +# SERVICE_ACCOUNT_OIDC_ENABLE (bool) - Enable client credentials flow +# Default: false +#SERVICE_ACCOUNT_OIDC_ENABLE=false +# JWKS_URI (string) - JWKS endpoint for IdP public keys (required if enabled) +# Default: empty +#JWKS_URI= + +# Challenge API Key (token exchange) +# WB_REST_SERVICE_ACCOUNT_CHALLENGE_APIKEY_ENABLE (bool) - Enable challenge flow +# Default: false +#WB_REST_SERVICE_ACCOUNT_CHALLENGE_APIKEY_ENABLE=false +# WB_REST_TOKEN_SIGNING_SECRET (string) - Token signing secret +# Default: securely generated at startup (changes on restart; set for production) +#WB_REST_TOKEN_SIGNING_SECRET= +# WB_REST_TOKEN_TIMEOUT (int, seconds) - Access token lifetime +# Default: 300 +#WB_REST_TOKEN_TIMEOUT=300 + +# Basic API Key (no challenge) +# WB_REST_SERVICE_ACCOUNT_BASIC_APIKEY_ENABLE (bool) - Enable basic apikey auth +# Default: false +#WB_REST_SERVICE_ACCOUNT_BASIC_APIKEY_ENABLE=false + +# TLS/Certificates +# NODE_EXTRA_CA_CERTS (string) - Path to additional CA certs in PEM format +# Useful when MongoDB or IdP uses a private CA +# Default: empty +#NODE_EXTRA_CA_CERTS= + +# Validation +# VALIDATE_WITH_ADM_SCHEMAS - Toggle request body validation with ATT&CK Data Model schemas +#VALIDATE_WITH_ADM_SCHEMAS=false diff --git a/configs/taxii/README.md b/docker/example-setup/configs/taxii/README.md similarity index 100% rename from configs/taxii/README.md rename to docker/example-setup/configs/taxii/README.md diff --git a/configs/taxii/config/template.env b/docker/example-setup/configs/taxii/config/template.env similarity index 94% rename from configs/taxii/config/template.env rename to docker/example-setup/configs/taxii/config/template.env index 13d07a4..286e9da 100644 --- a/configs/taxii/config/template.env +++ b/docker/example-setup/configs/taxii/config/template.env @@ -9,7 +9,7 @@ TAXII_ENV=dev # ***** SERVER SETTINGS ********************************************************************************************** # ******************************************************************************************************************** TAXII_APP_ADDRESS=0.0.0.0 -TAXII_APP_PORT=5002 +TAXII_APP_PORT=8000 TAXII_HTTPS_ENABLED=false TAXII_SSL_PRIVATE_KEY= TAXII_SSL_PUBLIC_KEY= @@ -21,8 +21,8 @@ TAXII_MAX_CONTENT_LENGTH=0 # ******************************************************************************************************************** # ***** NGINX SSL/TLS CERTIFICATE AUTO REG/RENEW ********************************************************************* # ******************************************************************************************************************** -CERTBOT_LE_FQDN=attack-taxii.mitre.org -CERBOT_LE_EMAIL=attack@mitre.org +CERTBOT_LE_FQDN=taxii.example.com +CERTBOT_LE_EMAIL=noreply@example.com CERTBOT_LE_ACME_SERVER=https://acme-v02.api.letsencrypt.org/directory CERTBOT_LE_RSA_KEY_SIZE=4096 @@ -54,7 +54,7 @@ TAXII_CACHE_RECONNECT=true # ******************************************************************************************************************** # ***** STIX/WORKBENCH SETTINGS ************************************************************************************** # ******************************************************************************************************************** -TAXII_STIX_SRC_URL=http://attack-workbench-rest-api:3000 +TAXII_STIX_SRC_URL=http://rest-api:3000 TAXII_STIX_DATA_SRC=workbench TAXII_WORKBENCH_AUTH_HEADER=dGF4aWktc2VydmVyOnNlY3JldC1zcXVpcnJlbA== @@ -63,7 +63,7 @@ TAXII_WORKBENCH_AUTH_HEADER=dGF4aWktc2VydmVyOnNlY3JldC1zcXVpcnJlbA== # ******************************************************************************************************************** # ***** DATABASE SETTINGS ******************************************************************************************** # ******************************************************************************************************************** -TAXII_MONGO_URI=mongodb://attack-workbench-database/taxii +TAXII_MONGO_URI=mongodb://mongodb:27017/taxii TAXII_HYDRATE_ON_BOOT=true diff --git a/configs/taxii/nginx.conf b/docker/example-setup/configs/taxii/nginx.conf similarity index 77% rename from configs/taxii/nginx.conf rename to docker/example-setup/configs/taxii/nginx.conf index 43c3ca5..9ff97e0 100644 --- a/configs/taxii/nginx.conf +++ b/docker/example-setup/configs/taxii/nginx.conf @@ -1,14 +1,8 @@ -worker_processes 1; - -events { - worker_connections 1024; -} - http { - # Server block for private TAXII administrative traffic. Routes traffic for downstream Workbench services. server { listen 80; - server_name localhost; + http2 on; + server_name _; root /usr/share/nginx/html; index index.html index.htm; @@ -25,16 +19,34 @@ http { location /api { client_max_body_size 50M; - proxy_pass http://attack-workbench-rest-api:3000; + + # Disable buffering for SSE streams + proxy_buffering off; + proxy_cache off; + + # Keep connection alive for long-running requests + proxy_read_timeout 600s; + proxy_connect_timeout 300s; + proxy_send_timeout 300s; + + # Required headers for SSE + proxy_set_header Connection ''; + proxy_http_version 1.1; + chunked_transfer_encoding on; + + # Disable compression for SSE + gzip off; + + proxy_pass http://rest-api:3000; } + } # Server block for TAXII server's LetsEncrypt handshake process server { listen 80 default_server; - listen [::]:80 default_server; - server_name attack-taxii.mitre.org; + server_name taxii.example.com; location /.well-known/acme-challenge { resolver 127.0.0.11 valid=30s; # If you're wondering if 127.0.0.11 is a typo – it's not – it is actually the @@ -72,7 +84,7 @@ http { proxy_set_header X-Forwarded-Proto $scheme; location /taxii { - proxy_pass http://attack-workbench-taxii-server:5000; + proxy_pass http://taxii:8000; # limit_req zone=one burst=5; } diff --git a/docker/example-setup/template.env b/docker/example-setup/template.env new file mode 100644 index 0000000..e69a0ac --- /dev/null +++ b/docker/example-setup/template.env @@ -0,0 +1,41 @@ +# Docker Image Tags +ATTACKWB_FRONTEND_VERSION=latest +ATTACKWB_RESTAPI_VERSION=latest +ATTACKWB_GRAFANA_VERSION=latest +ATTACKWB_TAXII_VERSION=latest + +# Frontend +#ATTACKWB_FRONTEND_HTTP_PORT=80 +#ATTACKWB_FRONTEND_HTTPS_PORT=443 +#ATTACKWB_FRONTEND_NGINX_CONFIG_FILE=./configs/frontend/nginx.api.conf +# Used for setting SSL certs in nginx +#ATTACKWB_FRONTEND_CERTS_PATH=./certs + +# REST API +#ATTACKWB_RESTAPI_HTTP_PORT=3000 +#ATTACKWB_RESTAPI_CONFIG_FILE=./configs/rest-api/rest-api-service-config.json +#ATTACKWB_RESTAPI_ENV_FILE=./configs/rest-api/.env + +# Grafana +#ATTACKWB_GRAFANA_PORT=3001 +#ATTACKWB_GRAFANA_ENV_FILE=./configs/grafana/.env +#ATTACKWB_GRAFANA_PROVISIONING_PATH=./configs/grafana/provisioning + +# Insights Agent +#ATTACKWB_INSIGHTS_AGENT_ENV_FILE=./configs/insights-agent/.env +#ATTACKWB_INSIGHTS_AGENT_SYSTEM_PROMPT_FILE=../../insights-agent/system-prompt.md + +# REST API Custom SSL certs (optional) +# These will be used for the REST API and, when enabled, the Grafana/Insights stack. +# See compose.certs.yaml for REST API details. +#HOST_CERTS_PATH=./certs +#CERTS_FILENAME=custom-certs.pem + +# Database +#ATTACKWB_DB_PORT=27017 +#ATTACKWB_DB_BACKUP_PATH=./database-backup + +# TAXII Server +#ATTACKWB_TAXII_HTTP_PORT=5002 +#ATTACKWB_TAXII_CONFIG_DIR=./configs/taxii/config +#ATTACKWB_TAXII_ENV=dev diff --git a/docker/setup-workbench.sh b/docker/setup-workbench.sh new file mode 100755 index 0000000..6396b90 --- /dev/null +++ b/docker/setup-workbench.sh @@ -0,0 +1,1356 @@ +#!/usr/bin/env bash + +# ATT&CK Workbench Deployment Setup Script +# This script helps you quickly set up a custom ATT&CK Workbench instance +# +# SCRIPT ORGANIZATION: +# 1. Constants and Configuration +# 2. Color and Output Functions +# 3. Validation Functions +# 4. Helper Functions (prompts, file operations) +# 5. Instance Management Functions +# 6. Configuration Functions (database, environment, certificates) +# 7. Deployment Option Functions (TAXII, insights, dev mode) +# 8. Compose Override Generation Functions +# 9. Output Functions (summary, instructions) +# 10. Main Execution Flow + +#=============================================================================== +# CONSTANTS +#=============================================================================== + +readonly DEPLOYMENT_REPO_URL="https://github.com/mitre-attack/attack-workbench-deployment.git" +readonly CTID_GITHUB_ORG="https://github.com/center-for-threat-informed-defense" +readonly MITRE_GITHUB_ORG="https://github.com/mitre-attack" + +readonly REPO_FRONTEND="attack-workbench-frontend" +readonly REPO_REST_API="attack-workbench-rest-api" +readonly REPO_TAXII="attack-workbench-taxii-server" + +readonly DB_URL_DOCKER="mongodb://mongodb/attack-workspace" +readonly DB_URL_LOCAL="mongodb://localhost:27017/attack-workspace" + +#=============================================================================== +# COLORS & OUTPUT FUNCTIONS +#=============================================================================== + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +info() { echo -e "${BLUE}$1${NC}"; } +success() { echo -e "${GREEN}$1${NC}"; } +warning() { echo -e "${YELLOW}$1${NC}"; } +error() { echo -e "${RED}$1${NC}"; } + +#=============================================================================== +# VALIDATION FUNCTIONS +#=============================================================================== + +# Validate that a required command is available +# Usage: require_command "git" "Please install git" +require_command() { + local command="$1" + local message="$2" + + if ! command -v "$command" &> /dev/null; then + error "$command is not installed or not in PATH" + if [[ -n "$message" ]]; then + echo " $message" + fi + return 1 + fi + return 0 +} + +# Validate that a file exists +# Usage: require_file "/path/to/file" "File description" +require_file() { + local file_path="$1" + local description="$2" + + if [[ ! -f "$file_path" ]]; then + error "${description:-File} not found: $file_path" + return 1 + fi + return 0 +} + +# Validate that a directory exists +# Usage: require_directory "/path/to/dir" "Directory description" +require_directory() { + local dir_path="$1" + local description="$2" + + if [[ ! -d "$dir_path" ]]; then + error "${description:-Directory} not found: $dir_path" + return 1 + fi + return 0 +} + +#=============================================================================== +# HELPER FUNCTIONS +#=============================================================================== + +# Prompt for yes/no answer with validation +# Usage: prompt_yes_no "Question?" "Y" +# Args: $1=question, $2=default (Y/N) +prompt_yes_no() { + # If defaults are automatically accepted, use the provided default and skip prompting + if $ACCEPT_DEFAULTS; then + PROMPT_YES_NO_RESULT="${2:-N}" + return + fi + + local question="$1" + local default="$2" + PROMPT_YES_NO_RESULT="" + + while true; do + read -p "$question [y/N] " -r answer + answer=${answer:-$default} + + if [[ $answer =~ ^[YyNn]$ ]]; then + PROMPT_YES_NO_RESULT="$answer" + break + else + error "Invalid option. Please enter 'y' for yes or 'n' for no." + fi + done +} + +# Prompt for menu selection with validation +# Usage: prompt_menu "default_index" "option1" "option2" "option3" +# Args: $1=default index (1-based), remaining args are menu options +prompt_menu() { + # If defaults are automatically accepted, use the default index and skip prompting + if $ACCEPT_DEFAULTS; then + PROMPT_MENU_RESULT="${1}" + return + fi + + local default_index="$1" + shift + local -a options=("$@") + local num_options=${#options[@]} + + PROMPT_MENU_RESULT="" + while true; do + for i in "${!options[@]}"; do + echo "$((i + 1))) ${options[$i]}" + done + echo "" + read -p "Select option 1-$num_options: [1] " -r choice + choice=${choice:-$default_index} + + if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "$num_options" ]; then + PROMPT_MENU_RESULT="$choice" + break + else + error "Invalid option. Please select 1-$num_options." + echo "" + fi + done + + echo "" +} + +# Prompt for non-empty string with validation +# Usage: prompt_non_empty "Question" +prompt_non_empty() { + local question="$1" + PROMPT_NON_EMPTY_RESULT="" + + while true; do + read -p "$question " PROMPT_NON_EMPTY_RESULT + if [[ -n "$PROMPT_NON_EMPTY_RESULT" ]]; then + break + else + error "Input cannot be empty" + fi + done +} + +# Expand shell-style home directory paths after they have been read into variables. +# Usage: expand_user_path "~/path" +expand_user_path() { + local input_path="$1" + + if [[ "$input_path" == "~" ]]; then + echo "$HOME" + elif [[ "$input_path" == "~/"* ]]; then + echo "$HOME/${input_path#"~/"}" + else + echo "$input_path" + fi +} + +# Resolve a path to an absolute path, using a base directory for relative inputs +# Usage: resolve_absolute_path "./relative/path" "/base/dir" +resolve_absolute_path() { + local input_path="$1" + local base_dir="$2" + local resolved_path="" + + input_path="$(expand_user_path "$input_path")" + + if [[ "$input_path" == /* ]]; then + resolved_path="$input_path" + else + resolved_path="$base_dir/$input_path" + fi + + local parent_dir + parent_dir="$(dirname "$resolved_path")" + mkdir -p "$parent_dir" + parent_dir="$(cd "$parent_dir" && pwd)" + echo "$parent_dir/$(basename "$resolved_path")" +} + +# Update or add a key=value in an env file +# Usage: update_env_file "/path/to/.env" "KEY" "value" +update_env_file() { + local env_file="$1" + local key="$2" + local value="$3" + + if grep -q "^${key}=" "$env_file"; then + local tmp + tmp="$(mktemp /tmp/env.XXXXXX)" + sed "s|^${key}=.*|${key}=${value}|" "$env_file" > "$tmp" && mv "$tmp" "$env_file" + elif grep -q "^#${key}=" "$env_file"; then + local tmp + tmp="$(mktemp /tmp/env.XXXXXX)" + sed "s|^#${key}=.*|${key}=${value}|" "$env_file" > "$tmp" && mv "$tmp" "$env_file" + else + echo "${key}=${value}" >> "$env_file" + fi +} + +# Copy a file into a bind-mounted file target, removing stale Docker-created +# directories that can appear when a missing file bind source is started. +copy_mount_file() { + local source_file="$1" + local target_file="$2" + local target_dir + local target_name + + target_dir="$(dirname "$target_file")" + target_name="$(basename "$target_file")" + mkdir -p "$target_dir" + + if [[ -d "$target_file" ]]; then + warning "Found a directory where a file mount target should be: $target_file" + warning "Removing stale Docker-created directory: $target_file" + rm -rf -- "$target_file" + fi + + find "$target_dir" -maxdepth 1 -type d -name "${target_name}.stale-dir-*" -exec rm -rf -- {} + + cp "$source_file" "$target_file" +} + +# Check if a repository exists in parent directory +# Usage: check_repo_exists "/parent/dir" "repo-name" +check_repo_exists() { + local parent_dir="$1" + local repo_name="$2" + + [[ -d "$parent_dir/$repo_name" ]] +} + +# Get GitHub URL for a repository +# Usage: get_repo_url "repo-name" +get_repo_url() { + local repo_name="$1" + + if [[ "$repo_name" == "$REPO_TAXII" ]]; then + echo "$MITRE_GITHUB_ORG/$repo_name.git" + else + echo "$CTID_GITHUB_ORG/$repo_name.git" + fi +} + +#=============================================================================== +# INSTANCE MANAGEMENT FUNCTIONS +#=============================================================================== + +# Prompt for and validate instance name +get_instance_name() { + local default_instance_name="my-workbench" + + # If instance name was provided via cli, use the cli value and skip prompting + if [[ -n "${AUTO_INSTANCE_NAME-}" ]]; then + GET_INSTANCE_NAME_NAME_REF="${AUTO_INSTANCE_NAME}" + return + fi + + # If defaults are automatically accepted, use the default name and skip prompting + if $ACCEPT_DEFAULTS; then + GET_INSTANCE_NAME_NAME_REF="${default_instance_name}" + return + fi + + read -p "Enter instance name [my-workbench]: " GET_INSTANCE_NAME_NAME_REF + GET_INSTANCE_NAME_NAME_REF=${GET_INSTANCE_NAME_NAME_REF:-$default_instance_name} + + # Validate instance name + if [[ ! "$GET_INSTANCE_NAME_NAME_REF" =~ ^[a-zA-Z0-9_-]+$ ]]; then + error "Instance name can only contain letters, numbers, hyphens, and underscores" + exit 1 + fi +} + +# Check if instance exists and handle overwrite +handle_existing_instance() { + local instance_dir="$1" + local instance_name="$2" + + if [[ ! -d "$instance_dir" ]]; then + return 0 + fi + + warning "Instance '$instance_name' already exists at $instance_dir" + echo "" + + prompt_yes_no "Would you like to overwrite it?" "N" + local overwrite="$PROMPT_YES_NO_RESULT" + + if [[ ! $overwrite =~ ^[Yy]$ ]]; then + error "Aborted" + exit 1 + fi + + warning "Removing existing instance directory..." + rm -rf "$instance_dir" + echo "" +} + +# Create instance directory and copy template files +create_instance() { + local instance_dir="$1" + local deployment_dir="$2" + local source_dir="$deployment_dir/docker/example-setup" + + info "Creating instance directory: $instance_dir" + echo "" + mkdir -p "$instance_dir" + + info "Copying template files..." + # Copy all files except compose templates (they're handled by this script) + find "$source_dir" -maxdepth 1 \ + ! -name "compose.dev.yaml" \ + ! -name "compose.certs.yaml" \ + ! -name "compose.insights.yaml" \ + ! -name "compose.taxii.yaml" \ + ! -path "$source_dir" \ + -exec cp -r {} "$instance_dir/" \; + success "Template files copied" + echo "" +} + +#=============================================================================== +# CONFIGURATION FUNCTIONS +#=============================================================================== + +# Configure database connection and return the selected DATABASE_URL +configure_database() { + CONFIGURE_DATABASE_DB_URL_REF="" + + # echo "" + info "Configure MongoDB connection:" + echo "" + + # If instance name was provided via cli, use the cli value and skip prompting + if [[ -n "${AUTO_DATABASE_URL-}" ]]; then + CONFIGURE_DATABASE_DB_URL_REF="${AUTO_DATABASE_URL}" + info "Using custom connection: $CONFIGURE_DATABASE_DB_URL_REF" + return + fi + + prompt_menu 1 \ + "Docker setup ($DB_URL_DOCKER)" \ + "Local MongoDB ($DB_URL_LOCAL)" \ + "Custom connection string" + local db_choice="$PROMPT_MENU_RESULT" + + case $db_choice in + 1) + CONFIGURE_DATABASE_DB_URL_REF="$DB_URL_DOCKER" + info "Using Docker setup: $CONFIGURE_DATABASE_DB_URL_REF" + ;; + 2) + CONFIGURE_DATABASE_DB_URL_REF="$DB_URL_LOCAL" + info "Using local MongoDB: $CONFIGURE_DATABASE_DB_URL_REF" + ;; + 3) + echo "" + prompt_non_empty "Enter MongoDB connection string:" + CONFIGURE_DATABASE_DB_URL_REF="$PROMPT_NON_EMPTY_RESULT" + info "Using custom connection: $CONFIGURE_DATABASE_DB_URL_REF" + ;; + esac + echo "" +} + +# Configure the host directory that will store all files mounted into containers +configure_host_mount_dir() { + CONFIGURE_HOST_MOUNT_DIR_REF="" + + local default_mount_dir="$INSTANCE_DIR" + + info "Configure host directory for container-mounted files:" + echo "" + + if [[ -n "${AUTO_HOST_MOUNT_DIR-}" ]]; then + CONFIGURE_HOST_MOUNT_DIR_REF="${AUTO_HOST_MOUNT_DIR}" + elif $ACCEPT_DEFAULTS; then + CONFIGURE_HOST_MOUNT_DIR_REF="$default_mount_dir" + else + read -p "Enter host directory for files mounted into containers [$default_mount_dir]: " user_mount_dir + CONFIGURE_HOST_MOUNT_DIR_REF=${user_mount_dir:-$default_mount_dir} + fi + + CONFIGURE_HOST_MOUNT_DIR_REF="$(resolve_absolute_path "$CONFIGURE_HOST_MOUNT_DIR_REF" "$INSTANCE_DIR")" + info "Using host mount directory: $CONFIGURE_HOST_MOUNT_DIR_REF" + echo "" +} + +# Set up all environment files for the instance +setup_environment_files() { + local database_url="$1" + + info "Setting up environment files..." + + # Main .env file + if [[ -f "$INSTANCE_DIR/template.env" ]]; then + mv "$INSTANCE_DIR/template.env" "$INSTANCE_DIR/.env" + success "Created $INSTANCE_DIR/.env" + fi + + # REST API .env file + if [[ -f "$INSTANCE_DIR/configs/rest-api/template.env" ]]; then + local rest_api_env="$INSTANCE_DIR/configs/rest-api/.env" + mv "$INSTANCE_DIR/configs/rest-api/template.env" "$rest_api_env" + update_env_file "$rest_api_env" "DATABASE_URL" "$database_url" + success "Created $rest_api_env with DATABASE_URL configured" + fi + + # Grafana .env file (optional) + if [[ -f "$INSTANCE_DIR/configs/grafana/template.env" ]]; then + mv "$INSTANCE_DIR/configs/grafana/template.env" "$INSTANCE_DIR/configs/grafana/.env" + success "Created $INSTANCE_DIR/configs/grafana/.env" + fi + + # Insights agent .env file (optional) + if [[ -f "$INSTANCE_DIR/configs/insights-agent/template.env" ]]; then + local insights_env="$INSTANCE_DIR/configs/insights-agent/.env" + mv "$INSTANCE_DIR/configs/insights-agent/template.env" "$insights_env" + update_env_file "$insights_env" "MONGODB_DATABASE_URL" "$database_url" + success "Created $insights_env with MONGODB_DATABASE_URL configured" + + local system_prompt_file="$INSTANCE_DIR/configs/insights-agent/system-prompt.md" + cp "$DEPLOYMENT_DIR/insights-agent/system-prompt.md" "$system_prompt_file" + update_env_file "$INSTANCE_DIR/.env" "ATTACKWB_INSIGHTS_AGENT_SYSTEM_PROMPT_FILE" "./configs/insights-agent/system-prompt.md" + success "Created $system_prompt_file" + fi + + # TAXII .env file (optional) + if [[ -f "$INSTANCE_DIR/configs/taxii/config/template.env" ]]; then + mv "$INSTANCE_DIR/configs/taxii/config/template.env" "$INSTANCE_DIR/configs/taxii/config/dev.env" + success "Created $INSTANCE_DIR/configs/taxii/config/dev.env" + fi + + echo "" +} + +# Copy files that must be mounted into containers into the selected host directory +sync_container_mount_files() { + local source_instance_dir="$1" + local host_mount_dir="$2" + local env_file="$source_instance_dir/.env" + + if [[ "$host_mount_dir" == "$source_instance_dir" ]]; then + return + fi + + info "Syncing container-mounted files to: $host_mount_dir" + + mkdir -p "$host_mount_dir/configs" "$host_mount_dir/certs" "$host_mount_dir/database-backup" + + if [[ -d "$source_instance_dir/configs" ]]; then + cp -R "$source_instance_dir/configs/." "$host_mount_dir/configs/" + fi + + if [[ -d "$source_instance_dir/certs" ]]; then + cp -R "$source_instance_dir/certs/." "$host_mount_dir/certs/" + fi + + if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + local custom_certs_path + custom_certs_path="$(resolve_absolute_path "$HOST_CERTS_PATH" "$source_instance_dir")" + local custom_cert_file="$custom_certs_path/$CERTS_FILENAME" + if [[ -f "$custom_cert_file" ]]; then + copy_mount_file "$custom_cert_file" "$host_mount_dir/certs/$CERTS_FILENAME" + else + warning "Custom certificate file not found: $custom_cert_file" + warning "Place it at $host_mount_dir/certs/$CERTS_FILENAME before starting Docker Compose." + fi + fi + + if [[ -d "$source_instance_dir/database-backup" ]]; then + cp -R "$source_instance_dir/database-backup/." "$host_mount_dir/database-backup/" + fi + + update_env_file "$env_file" "ATTACKWB_FRONTEND_NGINX_CONFIG_FILE" "$host_mount_dir/configs/frontend/nginx.api.conf" + update_env_file "$env_file" "ATTACKWB_FRONTEND_CERTS_PATH" "$host_mount_dir/certs" + update_env_file "$env_file" "ATTACKWB_RESTAPI_CONFIG_FILE" "$host_mount_dir/configs/rest-api/rest-api-service-config.json" + update_env_file "$env_file" "ATTACKWB_RESTAPI_ENV_FILE" "$host_mount_dir/configs/rest-api/.env" + update_env_file "$env_file" "ATTACKWB_DB_BACKUP_PATH" "$host_mount_dir/database-backup" + + if [[ $ENABLE_TAXII =~ ^[Yy]$ ]]; then + update_env_file "$env_file" "ATTACKWB_TAXII_CONFIG_DIR" "$host_mount_dir/configs/taxii/config" + fi + + if [[ $ENABLE_INSIGHTS =~ ^[Yy]$ ]]; then + update_env_file "$env_file" "ATTACKWB_GRAFANA_ENV_FILE" "$host_mount_dir/configs/grafana/.env" + update_env_file "$env_file" "ATTACKWB_GRAFANA_PROVISIONING_PATH" "$host_mount_dir/configs/grafana/provisioning" + update_env_file "$env_file" "ATTACKWB_INSIGHTS_AGENT_ENV_FILE" "$host_mount_dir/configs/insights-agent/.env" + update_env_file "$env_file" "ATTACKWB_INSIGHTS_AGENT_SYSTEM_PROMPT_FILE" "$host_mount_dir/configs/insights-agent/system-prompt.md" + fi + + if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + update_env_file "$env_file" "HOST_CERTS_PATH" "$host_mount_dir/certs" + HOST_CERTS_PATH="$host_mount_dir/certs" + fi + + success "Container-mounted files synced to $host_mount_dir" + echo "" +} + +# Configure custom SSL certificates for REST API +configure_custom_certificates() { + CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF="" + CONFIGURE_CUSTOM_CERTIFICATES_CERTS_FILENAME_REF="" + + # echo "" + info "Custom SSL certificates allow containerized services to trust additional CA certificates." + info "This is useful when behind a firewall that performs SSL inspection." + echo "" + + if [[ -z "${AUTO_HOST_CERTS_PATH}" ]]; then + read -p "Enter host certificates path [./certs]: " user_certs_path + CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF=${user_certs_path:-./certs} + else + CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF="${AUTO_HOST_CERTS_PATH}" + fi + CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF="$(expand_user_path "$CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF")" + + if [[ -z "${AUTO_CERTS_FILENAME}" ]]; then + read -p "Enter certificate filename [custom-certs.pem]: " user_certs_filename + CONFIGURE_CUSTOM_CERTIFICATES_CERTS_FILENAME_REF=${user_certs_filename:-custom-certs.pem} + else + CONFIGURE_CUSTOM_CERTIFICATES_CERTS_FILENAME_REF="${AUTO_CERTS_FILENAME}" + fi + + echo "" + info "Using certificates from: $CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF/$CONFIGURE_CUSTOM_CERTIFICATES_CERTS_FILENAME_REF" + # echo "" + + # Add custom cert configuration to .env + local env_file="$INSTANCE_DIR/.env" + update_env_file "$env_file" "HOST_CERTS_PATH" "$CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF" + update_env_file "$env_file" "CERTS_FILENAME" "$CONFIGURE_CUSTOM_CERTIFICATES_CERTS_FILENAME_REF" + success "Added certificate configuration to $env_file" + echo "" +} + +#=============================================================================== +# DEPLOYMENT OPTION FUNCTIONS +#=============================================================================== + +# Add TAXII service to compose.yaml by inserting before the volumes section +add_taxii_to_compose() { + local compose_file="$INSTANCE_DIR/compose.yaml" + local taxii_template="$DEPLOYMENT_DIR/docker/example-setup/compose.taxii.yaml" + local temp_file="$INSTANCE_DIR/compose.yaml.tmp" + + info "Adding TAXII server to compose.yaml..." + + # Insert TAXII service before the "volumes:" section + sed '/^volumes:/,$d' "$compose_file" > "$temp_file" + # Extract only the service definition, skipping the "services:" header + sed -n '/^services:/,${/^services:/!p;}' "$taxii_template" >> "$temp_file" + echo "" >> "$temp_file" + sed -n '/^volumes:/,$p' "$compose_file" >> "$temp_file" + mv "$temp_file" "$compose_file" + + success "TAXII server added to compose.yaml" + echo "" +} + +# Add Grafana and insights-agent services to compose.yaml by inserting before the volumes section +add_insights_to_compose() { + local compose_file="$INSTANCE_DIR/compose.yaml" + local insights_template="$DEPLOYMENT_DIR/docker/example-setup/compose.insights.yaml" + local temp_file="$INSTANCE_DIR/compose.yaml.tmp" + + info "Adding Grafana and insights-agent services to compose.yaml..." + + sed '/^volumes:/,$d' "$compose_file" > "$temp_file" + sed -n '/^services:/,${/^services:/!p;}' "$insights_template" >> "$temp_file" + echo "" >> "$temp_file" + sed -n '/^volumes:/,$p' "$compose_file" >> "$temp_file" + mv "$temp_file" "$compose_file" + + success "Grafana and insights-agent services added to compose.yaml" + echo "" +} + +# Verify all required source repositories exist for developer mode +verify_dev_mode_repos() { + local parent_dir="$1" + local enable_taxii="$2" + local -a missing_repos=() + + if ! check_repo_exists "$parent_dir" "$REPO_FRONTEND"; then + missing_repos+=("$REPO_FRONTEND") + fi + + if ! check_repo_exists "$parent_dir" "$REPO_REST_API"; then + missing_repos+=("$REPO_REST_API") + fi + + if [[ $enable_taxii =~ ^[Yy]$ ]] && ! check_repo_exists "$parent_dir" "$REPO_TAXII"; then + missing_repos+=("$REPO_TAXII") + fi + + if [[ ${#missing_repos[@]} -gt 0 ]]; then + warning "Missing required repositories:" + for repo in "${missing_repos[@]}"; do + echo " - $repo" + done + echo "" + warning "Please clone the missing repositories to:" + echo " $parent_dir/" + echo "" + echo "Clone commands:" + for repo in "${missing_repos[@]}"; do + echo " git clone $(get_repo_url "$repo") $parent_dir/$repo" + done + else + success "All required repositories found!" + fi + echo "" +} + +# Display expected directory structure for developer mode +show_dev_mode_structure() { + local deployment_dir="$1" + local enable_taxii="$2" + local enable_insights="$3" + + # echo "" + info "Developer mode requires source repositories to be cloned as siblings to the deployment repository." + echo "" + echo "Expected directory structure:" + echo " $(dirname "$deployment_dir")/" + echo " ├── attack-workbench-deployment/" + echo " ├── $REPO_FRONTEND/" + echo " ├── $REPO_REST_API/" + if [[ $enable_taxii =~ ^[Yy]$ ]]; then + echo " └── $REPO_TAXII/" + fi + echo "" + if [[ $enable_insights =~ ^[Yy]$ ]]; then + echo "The insights agent source is maintained in this repository:" + echo " $deployment_dir/insights-agent/" + echo "" + fi +} + +#=============================================================================== +# COMPOSE OVERRIDE GENERATION FUNCTIONS +#=============================================================================== + +# Generate the frontend service override configuration for dev mode +generate_frontend_override() { + cat << EOF + + frontend: + image: $REPO_FRONTEND + build: ../../../$REPO_FRONTEND + develop: + watch: + # Sync source files for hot-reload + - action: sync + path: ../../../$REPO_FRONTEND/src + target: /app/src + ignore: + - node_modules/ + # Rebuild on package.json changes + - action: rebuild + path: ../../../$REPO_FRONTEND/package.json +EOF +} + +# Generate the rest-api service override configuration for dev mode +generate_rest_api_dev_override() { + cat << EOF + + rest-api: + image: $REPO_REST_API + build: ../../../$REPO_REST_API +EOF + + # Add custom cert volumes if enabled + if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + cat << 'EOF' + volumes: + - ${HOST_CERTS_PATH:-./certs}:/usr/src/app/certs + environment: + - NODE_EXTRA_CA_CERTS=./certs/${CERTS_FILENAME:-custom-certs.pem} +EOF + fi + + # Add develop watch configuration + cat << EOF + develop: + watch: + # Sync app source files + - action: sync + path: ../../../$REPO_REST_API/app + target: /usr/src/app/app + ignore: + - node_modules/ + # Restart on config changes + - action: sync+restart + path: ../../../$REPO_REST_API/resources + target: /usr/src/app/resources + # Rebuild on package.json changes + - action: rebuild + path: ../../../$REPO_REST_API/package.json +EOF +} + +# Generate the rest-api service override for production mode with custom certs +generate_rest_api_certs_override() { + cat << 'EOF' + + rest-api: + volumes: + - ${HOST_CERTS_PATH:-./certs}:/usr/src/app/certs + environment: + - NODE_EXTRA_CA_CERTS=./certs/${CERTS_FILENAME:-custom-certs.pem} +EOF +} + +# Generate the insights-agent service override configuration for dev/custom-cert mode +generate_insights_override() { + if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + cat << 'EOF' + + grafana: + volumes: + - type: bind + source: ${HOST_CERTS_PATH:-./certs}/${CERTS_FILENAME:-custom-certs.pem} + target: /etc/ssl/certs/ca-certificates.crt + read_only: true + bind: + create_host_path: false + environment: + - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt +EOF + fi + + cat << 'EOF' + + insights-agent: +EOF + + if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + cat << 'EOF' + volumes: + - type: bind + source: ${HOST_CERTS_PATH:-./certs}/${CERTS_FILENAME:-custom-certs.pem} + target: /run/attackwb/custom-ca.pem + read_only: true + bind: + create_host_path: false +EOF + fi + + if [[ $DEV_MODE =~ ^[Yy]$ ]]; then + cat << 'EOF' + develop: + watch: + - action: sync+restart + path: ../../insights-agent + target: /app + ignore: + - __pycache__/ + - .venv/ + - Dockerfile + - requirements.txt + - action: rebuild + path: ../../insights-agent/requirements.txt + - action: rebuild + path: ../../insights-agent/Dockerfile +EOF + fi +} + +# Generate the TAXII service override configuration for dev mode +generate_taxii_override() { + cat << EOF + + taxii: + image: $REPO_TAXII + build: ../../../$REPO_TAXII + develop: + watch: + # Sync source files + - action: sync + path: ../../../$REPO_TAXII/taxii + target: /app/taxii + ignore: + - node_modules/ + # Sync config files and restart + - action: sync+restart + path: ../../../$REPO_TAXII/config + target: /app/config + # Rebuild on package.json changes + - action: rebuild + path: ../../../$REPO_TAXII/package.json +EOF +} + +# Generate the complete compose.override.yaml file +generate_compose_override() { + local override_file="$1" + + # Write header + cat > "$override_file" << 'EOF' +# This file was generated by setup-workbench.sh +# It will be automatically merged with compose.yaml when running docker compose commands + +services: +EOF + + # Add service configurations based on mode + if [[ $DEV_MODE =~ ^[Yy]$ ]]; then + generate_frontend_override >> "$override_file" + generate_rest_api_dev_override >> "$override_file" + + if [[ $ENABLE_TAXII =~ ^[Yy]$ ]]; then + generate_taxii_override >> "$override_file" + fi + if [[ $ENABLE_INSIGHTS =~ ^[Yy]$ ]]; then + generate_insights_override >> "$override_file" + fi + else + # Production mode - only add rest-api if custom certs are enabled + if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + generate_rest_api_certs_override >> "$override_file" + if [[ $ENABLE_INSIGHTS =~ ^[Yy]$ ]]; then + generate_insights_override >> "$override_file" + fi + fi + fi + + # Add newline at end + echo "" >> "$override_file" +} + +#=============================================================================== +# OUTPUT FUNCTIONS +#=============================================================================== + +# Display configuration summary +show_configuration_summary() { + local instance_dir="$1" + local override_file="$2" + local dev_mode="$3" + local enable_taxii="$4" + local enable_insights="$5" + local enable_custom_certs="$6" + local host_mount_dir="$7" + + local config_root="$instance_dir" + if [[ -n "$host_mount_dir" ]]; then + config_root="$host_mount_dir" + fi + + info "Configuration files:" + echo " Main: $instance_dir/.env" + echo " Compose: $instance_dir/compose.yaml" + if [[ $dev_mode =~ ^[Yy]$ ]] || [[ $enable_custom_certs =~ ^[Yy]$ ]]; then + echo " + Override: $override_file" + fi + if [[ "$config_root" != "$instance_dir" ]]; then + echo " Mount Dir: $config_root" + fi + echo " REST API: $config_root/configs/rest-api/.env" + echo " REST API: $config_root/configs/rest-api/rest-api-service-config.json" + if [[ $enable_taxii =~ ^[Yy]$ ]]; then + echo " TAXII: $config_root/configs/taxii/config/.env" + fi + if [[ $enable_insights =~ ^[Yy]$ ]]; then + echo " Grafana: $config_root/configs/grafana/.env" + echo " Insights: $config_root/configs/insights-agent/.env" + echo " Prompt: $config_root/configs/insights-agent/system-prompt.md" + fi + echo "" +} + +# Display custom SSL certificate information +show_certificate_info() { + local instance_dir="$1" + local host_certs_path="$2" + local certs_filename="$3" + + info "Custom SSL certificates:" + echo " Path: $host_certs_path" + echo " Filename: $certs_filename" + echo "" + warning "Make sure to place your certificate file at:" + if [[ "$host_certs_path" = ./* ]] || [[ "$host_certs_path" = ../* ]]; then + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + echo " $instance_dir/$host_certs_path/$certs_filename" + else + echo " $host_certs_path/$certs_filename" + fi + echo "" +} + +# Display deployment instructions +show_deployment_instructions() { + local instance_dir="$1" + local deployment_dir="$2" + local dev_mode="$3" + local enable_insights="$4" + local host_mount_dir="$5" + + info "To deploy your instance:" + echo " cd $instance_dir" + if [[ $dev_mode =~ ^[Yy]$ ]]; then + echo " docker compose up -d --build" + echo "" + info "For hot-reloading in developer mode, use watch:" + echo " docker compose watch" + else + echo " docker compose up -d" + fi + echo "" + + info "After deployment, access your Workbench at:" + echo " http://localhost" + if [[ $enable_insights =~ ^[Yy]$ ]]; then + echo " Grafana: http://localhost:3001" + fi + echo "" + + if [[ -n "$host_mount_dir" && "$host_mount_dir" != "$instance_dir" ]]; then + info "Mounted configs and certificates live under:" + echo " $host_mount_dir" + echo "" + fi + + info "For more information, see:" + echo " Configuration: $deployment_dir/docs/configuration.md" + echo " Deployment: $deployment_dir/docs/deployment.md" + echo "" +} + +# Display script usage information +usage() { + echo "Usage: $(basename "$0") [--accept-defaults] [--dev-mode] [--instance-name ] [-h | --help] [--mongodb-connection ] [--taxii-server] [--insights] [--host-mount-dir ]" + echo + echo "Generate Docker Compose configurations to deploy local workbench instances." + echo + echo "Options:" + echo " --accept-defaults Run non-interactively using default selections unless overriden by other options" + echo " --dev-mode Setup in developer mode (build from source)" + echo " --instance-name Name of the generated configuration" + echo " -h, --help Show this help and exit" + echo " --mongodb-connection MongoDB connection string" + echo " --taxii-server Deploy with the TAXII server" + echo " --insights Deploy with Grafana and the insights agent" + echo " --host-mount-dir Host directory for files mounted into containers" + echo " --ssl-host-certs-path Host certificates directory path (default \"./certs\")" + echo " --ssl-certs-file Certificates filename (default \"custom-certs.pem\")" +} + +#=============================================================================== +# ARGUMENT PARSING +#=============================================================================== + +# Parse optional CLI arguments +ACCEPT_DEFAULTS=false +AUTO_ENABLE_TAXII=false +AUTO_ENABLE_INSIGHTS=false +AUTO_DEV_MODE=false +AUTO_INSTANCE_NAME="" +AUTO_DATABASE_URL="" +AUTO_HOST_MOUNT_DIR="" +AUTO_HOST_CERTS_PATH="" +AUTO_CERTS_FILENAME="" +while [[ $# -gt 0 ]]; do + case "$1" in + --accept-defaults) + ACCEPT_DEFAULTS=true + shift + ;; + --dev-mode) + AUTO_DEV_MODE=true + shift + ;; + -h|--help) + usage + exit 0 + ;; + --instance-name) + if [[ $# -lt 2 || "${2:-}" == -* ]]; then + echo "Error: --instance-name requires a value." >&2 + echo "" + usage + exit 1 + fi + AUTO_INSTANCE_NAME="$2" + shift 2 + ;; + --instance-name=*) + AUTO_INSTANCE_NAME="${1#*=}" + if [[ -z "$AUTO_INSTANCE_NAME" ]]; then + echo "Error: --instance-name requires a value." >&2 + echo "" + usage + exit 1 + fi + shift + ;; + --mongodb-connection) + if [[ $# -lt 2 || "${2:-}" == -* ]]; then + echo "Error: --mongodb-connection requires a value." >&2 + echo "" + usage + exit 1 + fi + AUTO_DATABASE_URL="$2" + shift 2 + ;; + --mongodb-connection=*) + AUTO_DATABASE_URL="${1#*=}" + if [[ -z "$AUTO_DATABASE_URL" ]]; then + echo "Error: --mongodb-connection requires a value." >&2 + echo "" + usage + exit 1 + fi + shift + ;; + --host-mount-dir) + if [[ $# -lt 2 || "${2:-}" == -* ]]; then + echo "Error: --host-mount-dir requires a value." >&2 + echo "" + usage + exit 1 + fi + AUTO_HOST_MOUNT_DIR="$2" + shift 2 + ;; + --host-mount-dir=*) + AUTO_HOST_MOUNT_DIR="${1#*=}" + if [[ -z "$AUTO_HOST_MOUNT_DIR" ]]; then + echo "Error: --host-mount-dir requires a value." >&2 + echo "" + usage + exit 1 + fi + shift + ;; + --ssl-host-certs-path) + if [[ $# -lt 2 || "${2:-}" == -* ]]; then + echo "Error: --ssl-host-certs-path requires a value." >&2 + echo "" + usage + exit 1 + fi + AUTO_HOST_CERTS_PATH="$2" + shift 2 + ;; + --ssl-host-certs-path=*) + AUTO_HOST_CERTS_PATH="${1#*=}" + if [[ -z "$AUTO_HOST_CERTS_PATH" ]]; then + echo "Error: --ssl-host-certs-path requires a value." >&2 + echo "" + usage + exit 1 + fi + shift + ;; + --ssl-certs-file) + if [[ $# -lt 2 || "${2:-}" == -* ]]; then + echo "Error: --ssl-certs-file requires a value." >&2 + echo "" + usage + exit 1 + fi + AUTO_CERTS_FILENAME="$2" + shift 2 + ;; + --ssl-certs-file=*) + AUTO_CERTS_FILENAME="${1#*=}" + if [[ -z "$AUTO_CERTS_FILENAME" ]]; then + echo "Error: --ssl-certs-file requires a value." >&2 + echo "" + usage + exit 1 + fi + shift + ;; + --taxii-server) + AUTO_ENABLE_TAXII=true + shift + ;; + --insights) + AUTO_ENABLE_INSIGHTS=true + shift + ;; + --) + shift + break + ;; + -*) + echo "Error: Unknown option: $1" >&2 + echo "" + usage + exit 1 + ;; + esac +done + +#=============================================================================== +# BANNER +#=============================================================================== + +echo "" +echo "╔════════════════════════════════════════════════════════════╗" +echo "║ ATT&CK Workbench Deployment Setup ║" +echo "╚════════════════════════════════════════════════════════════╝" +echo "" + +#=============================================================================== +# LOCATE DEPLOYMENT REPOSITORY +#=============================================================================== + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEPLOYMENT_DIR="" + +# Check if we're already in the deployment repo +if [[ -f "$SCRIPT_DIR/example-setup/compose.yaml" ]]; then + DEPLOYMENT_DIR="$(dirname $SCRIPT_DIR)" + info "Running from deployment repository: $DEPLOYMENT_DIR" +elif [[ -d "$SCRIPT_DIR/attack-workbench-deployment" ]]; then + DEPLOYMENT_DIR="$SCRIPT_DIR/attack-workbench-deployment" + info "Found deployment repository: $DEPLOYMENT_DIR" +elif [[ -f "./docker/example-setup/compose.yaml" ]]; then + DEPLOYMENT_DIR="$(pwd)" + info "Running from current directory: $DEPLOYMENT_DIR" +else + # Not in the repo - need to clone or find it + warning "Not in the attack-workbench-deployment repository" + + # Check if git is available + if ! command -v git &> /dev/null; then + error "Git is not installed. Please install git or manually clone the repository." + exit 1 + fi + + echo "" + prompt_yes_no "Would you like to clone the repository?" "Y" + CLONE_REPO="$PROMPT_YES_NO_RESULT" + + if [[ $CLONE_REPO =~ ^[Yy]$ ]]; then + info "Cloning repository from $DEPLOYMENT_REPO_URL..." + + CLONE_DIR="./attack-workbench-deployment" + if [[ -d "$CLONE_DIR" ]]; then + error "Directory $CLONE_DIR already exists" + exit 1 + fi + + git clone "$DEPLOYMENT_REPO_URL" "$CLONE_DIR" + DEPLOYMENT_DIR="$(cd "$CLONE_DIR" && pwd)" + success "Repository cloned to $DEPLOYMENT_DIR" + else + error "Cannot proceed without the deployment repository" + exit 1 + fi +fi +echo "" + +cd "$DEPLOYMENT_DIR" + +#=============================================================================== +# PREREQUISITE CHECKS +#=============================================================================== + +# Check for Docker (warn but don't fail - user might not deploy immediately) +if ! require_command "docker" "Please install Docker to deploy the Workbench. Visit: https://docs.docker.com/get-docker/"; then + warning "Docker is not installed - you will need it to deploy the Workbench" +fi + +# Check for Docker Compose (warn but don't fail) +if ! docker compose version &> /dev/null 2>&1; then + warning "Docker Compose is not available" + echo " Please install Docker Compose" +fi + + +#=============================================================================== +# MAIN EXECUTION FLOW +#=============================================================================== + +#--------------------------------------- +# Instance Setup +#--------------------------------------- + +# echo "" +info "Setting up your Workbench instance..." +echo "" + +get_instance_name +INSTANCE_NAME="$GET_INSTANCE_NAME_NAME_REF" +INSTANCE_DIR="$DEPLOYMENT_DIR/instances/$INSTANCE_NAME" + +handle_existing_instance "$INSTANCE_DIR" "$INSTANCE_NAME" +create_instance "$INSTANCE_DIR" "$DEPLOYMENT_DIR" + +#--------------------------------------- +# Deployment Options +#--------------------------------------- + +# echo "" +info "Configuring deployment options..." +echo "" + +if $AUTO_ENABLE_TAXII; then + ENABLE_TAXII="y" +else + prompt_yes_no "Do you want to deploy with the TAXII server?" "N" + ENABLE_TAXII="$PROMPT_YES_NO_RESULT" + echo "" +fi + +if $AUTO_ENABLE_INSIGHTS; then + ENABLE_INSIGHTS="y" +else + prompt_yes_no "Do you want to deploy with Grafana and the insights agent?" "N" + ENABLE_INSIGHTS="$PROMPT_YES_NO_RESULT" + echo "" +fi + +if [[ ! $ENABLE_TAXII =~ ^[Yy]$ ]]; then + # Remove TAXII configs if not needed + if [[ -d "$INSTANCE_DIR/configs/taxii" ]]; then + rm -rf "$INSTANCE_DIR/configs/taxii" + fi +fi + +if [[ ! $ENABLE_INSIGHTS =~ ^[Yy]$ ]]; then + if [[ -d "$INSTANCE_DIR/configs/grafana" ]]; then + rm -rf "$INSTANCE_DIR/configs/grafana" + fi + if [[ -d "$INSTANCE_DIR/configs/insights-agent" ]]; then + rm -rf "$INSTANCE_DIR/configs/insights-agent" + fi +fi + +#--------------------------------------- +# Environment Configuration +#--------------------------------------- + +configure_host_mount_dir +HOST_MOUNT_DIR="$CONFIGURE_HOST_MOUNT_DIR_REF" + +configure_database +DATABASE_URL="$CONFIGURE_DATABASE_DB_URL_REF" +setup_environment_files "$DATABASE_URL" + +if [[ $ENABLE_TAXII =~ ^[Yy]$ ]]; then + add_taxii_to_compose +fi + +if [[ $ENABLE_INSIGHTS =~ ^[Yy]$ ]]; then + add_insights_to_compose +fi + +# echo "" +success "Instance '$INSTANCE_NAME' created successfully!" +echo "" + +#--------------------------------------- +# Additional Options +#--------------------------------------- + +if $AUTO_DEV_MODE; then + DEV_MODE="y" +else + prompt_yes_no "Do you want to set up in developer mode (build from source)?" "N" + DEV_MODE="$PROMPT_YES_NO_RESULT" +fi + +if [[ -z "${AUTO_HOST_CERTS_PATH}" && -z "${AUTO_CERTS_FILENAME}" ]]; then + prompt_yes_no "Do you want to configure custom SSL certificates for containerized services?" "N" + ENABLE_CUSTOM_CERTS="$PROMPT_YES_NO_RESULT" +else + ENABLE_CUSTOM_CERTS="y" +fi + +HOST_CERTS_PATH="./certs" +CERTS_FILENAME="custom-certs.pem" + +echo "" +if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + configure_custom_certificates + HOST_CERTS_PATH="$CONFIGURE_CUSTOM_CERTIFICATES_HOST_CERTS_REF" + CERTS_FILENAME="$CONFIGURE_CUSTOM_CERTIFICATES_CERTS_FILENAME_REF" +fi + +sync_container_mount_files "$INSTANCE_DIR" "$HOST_MOUNT_DIR" + +#--------------------------------------- +# Developer Mode Setup +#--------------------------------------- + +if [[ $DEV_MODE =~ ^[Yy]$ ]]; then + show_dev_mode_structure "$DEPLOYMENT_DIR" "$ENABLE_TAXII" "$ENABLE_INSIGHTS" + PARENT_DIR="$(dirname "$DEPLOYMENT_DIR")" + verify_dev_mode_repos "$PARENT_DIR" "$ENABLE_TAXII" +fi + +#--------------------------------------- +# Generate Compose Override +#--------------------------------------- + +OVERRIDE_FILE="$INSTANCE_DIR/compose.override.yaml" + +if [[ $DEV_MODE =~ ^[Yy]$ ]] || [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + info "Generating compose.override.yaml..." + generate_compose_override "$OVERRIDE_FILE" + success "Created $OVERRIDE_FILE" + echo "" +fi + +#--------------------------------------- +# Summary +#--------------------------------------- + +show_configuration_summary "$INSTANCE_DIR" "$OVERRIDE_FILE" "$DEV_MODE" "$ENABLE_TAXII" "$ENABLE_INSIGHTS" "$ENABLE_CUSTOM_CERTS" "$HOST_MOUNT_DIR" +if [[ $ENABLE_CUSTOM_CERTS =~ ^[Yy]$ ]]; then + show_certificate_info "$INSTANCE_DIR" "$HOST_CERTS_PATH" "$CERTS_FILENAME" +fi +show_deployment_instructions "$INSTANCE_DIR" "$DEPLOYMENT_DIR" "$DEV_MODE" "$ENABLE_INSIGHTS" "$HOST_MOUNT_DIR" diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..db8d057 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,151 @@ +# Configuration + +## Docker Compose Environment Variables + +We make heavy use of string interpolation to minimize having to modify the Docker Compose files. +Consequently, that means you must set a bunch of environment variables when using these templates. +Fortunately, we've provided a dotenv template that you can source. + +Copy `template.env` to `.env` and customize the values as needed: + +```bash +cp template.env .env +``` + +If you use `docker/setup-workbench.sh`, the script can also rewrite these path +variables to point at a dedicated host directory via `--host-mount-dir`. This +is useful in environments that require all Docker mounts to come from a +specific location such as `/Users/Shared/Docker/...`. + +Available environment variables: + +### Docker Image Tags + +| Variable | Default Value | Description | +|-----------------------------|---------------|-------------------------------| +| `ATTACKWB_FRONTEND_VERSION` | `latest` | Frontend Docker image tag | +| `ATTACKWB_RESTAPI_VERSION` | `latest` | REST API Docker image tag | +| `ATTACKWB_GRAFANA_VERSION` | `latest` | Grafana Docker image tag | +| `ATTACKWB_TAXII_VERSION` | `latest` | TAXII server Docker image tag | + +### Frontend + +| Variable | Default Value | Description | +|---------------------------------------|-------------------------------------|------------------------------------| +| `ATTACKWB_FRONTEND_HTTP_PORT` | `80` | Frontend HTTP port | +| `ATTACKWB_FRONTEND_HTTPS_PORT` | `443` | Frontend HTTPS port | +| `ATTACKWB_FRONTEND_NGINX_CONFIG_FILE` | `./configs/frontend/nginx.api.conf` | Path to nginx config file | +| `ATTACKWB_FRONTEND_CERTS_PATH` | `./certs` | Path to SSL certificates for nginx | + +There are four sample nginx config files that can be used as reference: + +- `nginx.conf`: Minimal nginx configuration that only routes the Workbench frontend. +- `nginx.ssl.conf`: Same as `nginx.conf` but with an SSL redirect. You need to provide your own SSL certs in the `ATTACKWB_FRONTEND_CERTS_PATH` directory. +- `nginx.api.conf` (default): Nginx configuration with an additional `/api` location block for connecting to the REST API container. +- `nginx.api.ssl.conf`: Same as `nginx.api.conf` but with an SSL redirect. You need to provide your own SSL certs in the `ATTACKWB_FRONTEND_CERTS_PATH` directory. + +### REST API + +| Variable | Default Value | Description | +|--------------------------------|---------------------------------------------------|---------------------------------------------------| +| `ATTACKWB_RESTAPI_HTTP_PORT` | `3000` | REST API port | +| `ATTACKWB_RESTAPI_CONFIG_FILE` | `./configs/rest-api/rest-api-service-config.json` | Path to REST API JSON config file | +| `ATTACKWB_RESTAPI_ENV_FILE` | `./configs/rest-api/.env` | Path to REST API environment variable config file | + +### Grafana + +| Variable | Default Value | Description | +|----------------------------------|---------------------------------------|-----------------------------------------------| +| `ATTACKWB_GRAFANA_PORT` | `3001` | Grafana HTTP port | +| `ATTACKWB_GRAFANA_ENV_FILE` | `./configs/grafana/.env` | Path to Grafana environment variable config | +| `ATTACKWB_GRAFANA_PROVISIONING_PATH` | `./configs/grafana/provisioning` | Path to Grafana provisioning directory | + +### Insights Agent + +| Variable | Default Value | Description | +|------------------------------------|--------------------------------------|-----------------------------------------------------| +| `ATTACKWB_INSIGHTS_AGENT_ENV_FILE` | `./configs/insights-agent/.env` | Path to the insights agent environment config file | +| `ATTACKWB_INSIGHTS_AGENT_SYSTEM_PROMPT_FILE` | `../../insights-agent/system-prompt.md` | Path to the mounted system prompt file | + +### REST API Custom SSL certs (Optional) + +These will be used to set `NODE_EXTRA_CA_CERTS` in the REST API container and +to mount a custom CA file for the optional Grafana and insights-agent services. +See `compose.certs.yaml` for the REST API reference overlay. + +| Variable | Default Value | Description | +|-------------------|--------------------|-------------------------------| +| `HOST_CERTS_PATH` | `./certs` | Path to custom cert directory | +| `CERTS_FILENAME` | `custom-certs.pem` | Filename of custom cert | + +### Database + +| Variable | Default Value | Description | +|---------------------------|---------------------|--------------------------| +| `ATTACKWB_DB_PORT` | `27017` | MongoDB port | +| `ATTACKWB_DB_BACKUP_PATH` | `./database-backup` | MongoDB backup directory | + +### TAXII Server + +| Variable | Default Value | Description | +|-----------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------| +| `ATTACKWB_TAXII_HTTP_PORT` | `5002` | TAXII server port | +| `ATTACKWB_TAXII_CONFIG_DIR` | `./configs/taxii/config` | DIrectory to find TAXII config file in | +| `ATTACKWB_TAXII_ENV` | `dev` | Specifies the name of the dotenv file to load (e.g., A value of `dev` tells the TAXII server to load `dev.env`) | + +## Service-Specific Configuration + +Each service has its own configuration directory: + +### Frontend + +**Config files**: [configs/frontend/](../docker/example-setup/configs/frontend/) + +The frontend container is an Nginx instance which serves the frontend SPA and reverse proxies requests to the backend REST API. +We provide a basic `nginx.conf` template in the aforementioned directory that should get you started. +Refer to the [frontend documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend) +for further details on customizing the Workbench frontend. + +### REST API + +**Config files**: [configs/rest-api/](../docker/example-setup/configs/rest-api/) + +The backend REST API loads runtime configurations from environment variables, as well as from a JSON configuration file. +Templates are provided in the aforementioned directory. +Refer to the [REST API usage documentation](https://github.com/center-for-threat-informed-defense/attack-workbench-rest-api/blob/main/USAGE.md#configuration) +for further details on customizing the backend. + +**Important**: For production deployments, set the following environment variables in your `.env` file to ensure persistent secrets across server restarts: + +- `SESSION_SECRET` - Secret used to sign session cookies +- `MONGOSTORE_CRYPTO_SECRET` - Secret used to encrypt session data in MongoDB + +Generate secure secrets using: `node -e "console.log(require('crypto').randomBytes(48).toString('base64'))"` + +### Grafana + +**Config files**: [configs/grafana/](../docker/example-setup/configs/grafana/) + +The Grafana container loads runtime configuration from its own dotenv file. +The provisioning directory is mounted read-only so you can add datasources and +dashboard providers without modifying the compose template. + +### Insights Agent + +**Config files**: [configs/insights-agent/](../docker/example-setup/configs/insights-agent/) + +The insights agent loads all runtime configuration from a dedicated dotenv file. +Set `MONGODB_DATABASE_URL` to the same connection string used by the REST API +and configure the LLM settings there. + +The system prompt is loaded from a separate mounted file referenced by +`ATTACKWB_INSIGHTS_AGENT_SYSTEM_PROMPT_FILE`, so prompt changes do not require +rebuilding the insights-agent image. + +### TAXII Server + +**Config files**: [configs/taxii/config/](../docker/example-setup/configs/taxii/config/) + +The TAXII server loads all runtime configuration parameters from a dotenv file. +The specific filename of the dotenv file is specified by the `ATTACKWB_TAXII_ENV` environment variable. +For example, a value of `dev` tells the TAXII server to load `dev.env`. diff --git a/docs/database-backups.md b/docs/database-backups.md new file mode 100644 index 0000000..3900dbf --- /dev/null +++ b/docs/database-backups.md @@ -0,0 +1,39 @@ +# Database Backups + +TODO: Clean up this documentation! Make it way easier to manage! + +The MongoDB commands `mongodump` and `mongorestore` can be used to create the database backup files and to restore the database using those files. + +The `compose.yaml` file maps the `ATTACKWB_DB_BACKUP_PATH` directory (defaults to `./database-backup`) on the host to the `/dump` directory +in the container in order to ease access to the backup files and to make sure those files exist even if the container is deleted. +This directory is listed in the `.gitignore` file so the backup files will not be added to the git repo. + +To access the command line inside the container, run this command from the host: + +```shell +docker exec -it attack-workbench-database bash +``` + +## Creating a Database Backup + +Create the backup as a compressed archive file: + +```shell +# From inside the attack-workbench-database container +mongodump --db attack-workspace --gzip --archive=dump/workspace.archive.gz +``` + +This creates a file in `/dump` in the container (`$ATTACKWB_DB_BACKUP_PATH` on the host). + +## Restoring the Database from the Backup + +The backup file must be in `$ATTACKWB_DB_BACKUP_PATH` on the host. + +Restoring from the compressed archive file: + +```shell +# From inside the attack-workbench-database container +mongorestore --drop --gzip --archive=dump/workspace.archive.gz +``` + +This drops the collections from the database, recreates the collections, loads the backed up documents into those collections, and rebuilds the indexes. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..a212b80 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,73 @@ +# Deployment Options + +## Docker Compose + +The ATT&CK Workbench can be deployed using Docker Compose with two different configurations: + +The easiest way to generate a deployment-ready instance is with +`docker/setup-workbench.sh`, which can also enable optional services such as +TAXII, Grafana, and the insights agent: + +```bash +./docker/setup-workbench.sh --instance-name my-workbench --insights +``` + +If your environment requires mounted files to live under a dedicated host +directory, generate the instance with: + +```bash +./docker/setup-workbench.sh --instance-name my-workbench --insights \ + --host-mount-dir /Users/Shared/Docker/my-workbench +``` + +### 1. Using Pre-built Images (Recommended) + +Use `compose.yaml` to pull pre-built images directly from GitHub Container Registry (GHCR): + +```bash +# Deploy with pre-built images +docker compose up -d + +# Deploy with TAXII server +docker compose --profile with-taxii up -d + +# Stop the deployment +docker compose down +``` + +### 2. Building from Source + +Use `compose.dev.yaml` in combination with `compose.yaml` to build images from source code: + +```bash +# Build and deploy from source +docker compose -f compose.yaml -f compose.dev.yaml up -d --build + +# Build and deploy with TAXII server +docker compose -f compose.yaml -f compose.dev.yaml --profile with-taxii up -d --build + +# Stop the deployment +docker compose -f compose.yaml -f compose.dev.yaml down +``` + +**Note**: When building from source, you need the following three source repositories to be available as sibling directories to this deployment repository: + +- [attack-workbench-frontend](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/) +- [attack-workbench-rest-api](https://github.com/center-for-threat-informed-defense/attack-workbench-rest-api/) +- [attack-workbench-taxii-server](https://github.com/mitre-attack/attack-workbench-taxii-server) + +The directory structure should look like this: + +```bash +. +├── attack-workbench-deployment +├── attack-workbench-frontend +├── attack-workbench-rest-api +└── attack-workbench-taxii-server (optional) +``` + +### Data Persistence + +MongoDB data is persisted in the `workspace-data` named Docker volume. +Thus, the `database` service can be deleted and re-deployed without losing access to the database. +The database volume will be remounted to the `database` service upon deployment. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..c21d998 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,20 @@ +# Troubleshooting + +Here are a few commands you can use to troubleshoot the docker compose setup. + +```bash +# View running containers +docker compose ps + +# Show logs for all running containers +docker compose logs + +# Follow logs +docker compose logs -f + +# Show logs for a specific container +docker compose logs frontend +docker compose logs rest-api +docker compose logs database +docker compose logs taxii +``` diff --git a/insights-agent/Dockerfile b/insights-agent/Dockerfile new file mode 100644 index 0000000..6029245 --- /dev/null +++ b/insights-agent/Dockerfile @@ -0,0 +1,14 @@ +FROM python:3.12-slim + +WORKDIR /app + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY . . + +CMD ["python", "-u", "main.py"] diff --git a/insights-agent/agent.py b/insights-agent/agent.py new file mode 100644 index 0000000..4338865 --- /dev/null +++ b/insights-agent/agent.py @@ -0,0 +1,240 @@ +"""LLM agent that generates Grafana dashboards from database insights. + +Uses OpenAI-compatible chat completions with tool calling. The agent receives +a structured database summary, reasons about what dashboards would be useful, +and creates them via Grafana API tool calls. +""" + +import json +import logging +from pathlib import Path +from urllib.parse import parse_qs, urlsplit, urlunsplit + +import httpx +from openai import AzureOpenAI, OpenAI + +from grafana import GrafanaClient + +log = logging.getLogger(__name__) + +# ── Tool definitions (OpenAI function-calling format) ──────────────────────── + +TOOLS = [ + { + "type": "function", + "function": { + "name": "upsert_dashboard", + "description": ( + "Create or update a Grafana dashboard. Provide the full Grafana " + "dashboard JSON model (panels, title, uid, etc.). The dashboard " + "will be placed in the 'ATT&CK Insights' folder." + ), + "parameters": { + "type": "object", + "properties": { + "dashboard": { + "type": "object", + "description": "Complete Grafana dashboard JSON model.", + } + }, + "required": ["dashboard"], + }, + }, + }, + { + "type": "function", + "function": { + "name": "list_datasources", + "description": "List all Grafana datasources. Returns an array of datasource objects.", + "parameters": {"type": "object", "properties": {}}, + }, + }, +] + + +def _execute_tool_call(grafana: GrafanaClient, folder_uid: str, name: str, arguments: dict) -> str: + """Execute a tool call and return the JSON result string.""" + if name == "upsert_dashboard": + result = grafana.upsert_dashboard(arguments["dashboard"], folder_uid=folder_uid) + return json.dumps(result) + if name == "list_datasources": + result = grafana.list_datasources() + return json.dumps(result) + return json.dumps({"error": f"Unknown tool: {name}"}) + + +def _load_system_prompt(system_prompt_path: str | None) -> str: + prompt_path = Path(system_prompt_path) if system_prompt_path else Path(__file__).with_name("system-prompt.md") + prompt = prompt_path.read_text(encoding="utf-8").strip() + if not prompt: + raise ValueError(f"System prompt file is empty: {prompt_path}") + log.info("Loaded system prompt from %s", prompt_path) + return prompt + + +def _looks_like_azure_url(base_url: str) -> bool: + split_url = urlsplit(base_url) + query = parse_qs(split_url.query) + path = split_url.path.rstrip("/") + return "api-version" in query or "/openai/deployments/" in path or path.endswith("/openai") + + +def _build_azure_client_config( + llm_base_url: str, + llm_model: str, + llm_api_version: str | None, + llm_azure_deployment: str | None, +) -> tuple[str, str, str]: + split_url = urlsplit(llm_base_url) + path = split_url.path.rstrip("/") + query = parse_qs(split_url.query) + + deployment = llm_azure_deployment + endpoint_path = path + if "/openai/deployments/" in path: + endpoint_path, deployment_path = path.split("/openai/deployments/", 1) + deployment_from_url = deployment_path.split("/", 1)[0] + deployment = deployment or deployment_from_url + elif path.endswith("/openai"): + endpoint_path = path[: -len("/openai")] + + api_version = llm_api_version or next(iter(query.get("api-version", [])), None) + if not api_version: + raise ValueError( + "Azure/OpenAI-compatible endpoints require LLM_API_VERSION or api-version in LLM_BASE_URL" + ) + + deployment = deployment or llm_model + endpoint = urlunsplit((split_url.scheme, split_url.netloc, endpoint_path, "", "")) + return endpoint, api_version, deployment + + +def _build_llm_client( + llm_base_url: str, + llm_api_key: str, + llm_model: str, + llm_api_type: str, + llm_api_version: str | None, + llm_azure_deployment: str | None, + ca_bundle_path: str, +) -> tuple[OpenAI | AzureOpenAI, httpx.Client, str]: + http_client = httpx.Client(verify=ca_bundle_path) + api_type = llm_api_type.lower() + use_azure = api_type == "azure" or (api_type == "auto" and _looks_like_azure_url(llm_base_url)) + + if use_azure: + endpoint, api_version, deployment = _build_azure_client_config( + llm_base_url=llm_base_url, + llm_model=llm_model, + llm_api_version=llm_api_version, + llm_azure_deployment=llm_azure_deployment, + ) + log.info( + "Using Azure-compatible chat completions endpoint: endpoint=%s deployment=%s api_version=%s", + endpoint, + deployment, + api_version, + ) + client = AzureOpenAI( + api_key=llm_api_key, + api_version=api_version, + azure_endpoint=endpoint, + http_client=http_client, + ) + return client, http_client, deployment + + log.info("Using OpenAI-compatible chat completions endpoint: base_url=%s model=%s", llm_base_url, llm_model) + client = OpenAI(base_url=llm_base_url, api_key=llm_api_key, http_client=http_client) + return client, http_client, llm_model + + +def run( + llm_base_url: str, + llm_api_key: str, + llm_model: str, + llm_api_type: str, + llm_api_version: str | None, + llm_azure_deployment: str | None, + system_prompt_path: str | None, + grafana: GrafanaClient, + db_summary: dict, + ca_bundle_path: str, +) -> str: + """Run the agent loop: prompt LLM → handle tool calls → return summary.""" + + client, http_client, request_model = _build_llm_client( + llm_base_url=llm_base_url, + llm_api_key=llm_api_key, + llm_model=llm_model, + llm_api_type=llm_api_type, + llm_api_version=llm_api_version, + llm_azure_deployment=llm_azure_deployment, + ca_bundle_path=ca_bundle_path, + ) + + try: + system_prompt = _load_system_prompt(system_prompt_path) + + # Ensure the target folder exists + folder = grafana.ensure_folder("ATT&CK Insights", uid="attack-insights") + folder_uid = folder["uid"] + + # Build the user message with the crawl data + user_message = ( + "Here is the structured summary of the ATT&CK Workbench database. " + "Analyse it and create Grafana dashboards that would be most useful " + "to ATT&CK content authors.\n\n" + f"```json\n{json.dumps(db_summary, indent=2, default=str)}\n```" + ) + + messages = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_message}, + ] + + # Agent loop — keep going until the LLM stops making tool calls + max_iterations = 20 + for i in range(max_iterations): + log.info("Agent iteration %d/%d", i + 1, max_iterations) + + response = client.chat.completions.create( + model=request_model, + messages=messages, + tools=TOOLS, + tool_choice="auto", + ) + + choice = response.choices[0] + assistant_msg = choice.message + + # Append assistant message to history + messages.append(assistant_msg.model_dump(exclude_none=True)) + + # If no tool calls, the agent is done + if not assistant_msg.tool_calls: + log.info("Agent finished with text response") + return assistant_msg.content or "(no summary)" + + # Process each tool call + for tc in assistant_msg.tool_calls: + fn_name = tc.function.name + fn_args = json.loads(tc.function.arguments) + log.info("Tool call: %s", fn_name) + + try: + result = _execute_tool_call(grafana, folder_uid, fn_name, fn_args) + except Exception as e: + log.error("Tool call %s failed: %s", fn_name, e) + result = json.dumps({"error": str(e)}) + + messages.append( + { + "role": "tool", + "tool_call_id": tc.id, + "content": result, + } + ) + + return "Agent reached maximum iterations without finishing." + finally: + http_client.close() diff --git a/insights-agent/crawler.py b/insights-agent/crawler.py new file mode 100644 index 0000000..5a283d9 --- /dev/null +++ b/insights-agent/crawler.py @@ -0,0 +1,275 @@ +"""MongoDB schema crawler for ATT&CK Workbench databases. + +Introspects the database, gathers collection stats, samples documents, and +computes domain-specific summaries that give an LLM enough context to design +useful Grafana dashboards. +""" + +import logging +from collections import Counter + +from pymongo import MongoClient +from pymongo.database import Database +from pymongo.uri_parser import parse_uri + +log = logging.getLogger(__name__) + +# STIX type → human-friendly label (for prompt readability) +STIX_TYPE_LABELS = { + "attack-pattern": "Technique", + "intrusion-set": "Group", + "malware": "Software (Malware)", + "tool": "Software (Tool)", + "course-of-action": "Mitigation", + "campaign": "Campaign", + "x-mitre-tactic": "Tactic", + "x-mitre-data-source": "Data Source", + "x-mitre-data-component": "Data Component", + "x-mitre-matrix": "Matrix", + "x-mitre-collection": "Collection Bundle", + "x-mitre-asset": "Asset", + "x-mitre-analytic": "Analytic", + "x-mitre-detection-strategy": "Detection Strategy", + "note": "Note", + "marking-definition": "Marking Definition", + "identity": "Identity", + "relationship": "Relationship", +} + +ATTACK_DOMAINS = ["enterprise-attack", "mobile-attack", "ics-attack"] +WORKFLOW_STATES = ["work-in-progress", "awaiting-review", "reviewed", "static", "draft"] + + +def connect(uri: str, db_name: str = "attack-workspace") -> Database: + client = MongoClient(uri) + db = client[db_name] + # Quick connectivity check + db.command("ping") + log.info("Connected to MongoDB: %s / %s", uri, db_name) + return db + + +def get_database_name(database_url: str, default_name: str = "attack-workspace") -> str: + """Return the database name embedded in a MongoDB URL, if any.""" + parsed = parse_uri(database_url) + return parsed.get("database") or default_name + + +def _latest_versions_pipeline(match: dict | None = None) -> list[dict]: + """Standard pipeline: group by stix.id, take latest stix.modified.""" + pipeline: list[dict] = [] + if match: + pipeline.append({"$match": match}) + pipeline += [ + {"$sort": {"stix.modified": -1}}, + {"$group": {"_id": "$stix.id", "doc": {"$first": "$$ROOT"}}}, + {"$replaceRoot": {"newRoot": "$doc"}}, + ] + return pipeline + + +def crawl(db: Database) -> dict: + """Return a structured summary of the ATT&CK Workbench database. + + The output is designed to be serialised into an LLM prompt. + """ + summary: dict = {} + + # ------------------------------------------------------------------ + # 1. Collection-level stats + # ------------------------------------------------------------------ + collections = db.list_collection_names() + col_stats = {} + for name in sorted(collections): + count = db[name].estimated_document_count() + col_stats[name] = count + summary["collections"] = col_stats + + # ------------------------------------------------------------------ + # 2. STIX object type distribution (latest versions only) + # ------------------------------------------------------------------ + attack_objects = db["attackObjects"] + type_counts: Counter = Counter() + for doc in attack_objects.aggregate( + _latest_versions_pipeline() + [{"$group": {"_id": "$stix.type", "count": {"$sum": 1}}}] + ): + stix_type = doc["_id"] + label = STIX_TYPE_LABELS.get(stix_type, stix_type) + type_counts[label] = doc["count"] + + # Include relationships from their dedicated collection + if "relationships" in collections: + rel_pipeline = _latest_versions_pipeline() + [{"$count": "total"}] + rel_result = list(db["relationships"].aggregate(rel_pipeline)) + if rel_result: + type_counts["Relationship"] = rel_result[0]["total"] + + summary["stix_type_distribution"] = dict(type_counts.most_common()) + + # ------------------------------------------------------------------ + # 3. Domain distribution + # ------------------------------------------------------------------ + domain_counts: Counter = Counter() + for doc in attack_objects.aggregate( + _latest_versions_pipeline() + + [ + {"$unwind": {"path": "$stix.x_mitre_domains", "preserveNullAndEmptyArrays": False}}, + {"$group": {"_id": "$stix.x_mitre_domains", "count": {"$sum": 1}}}, + ] + ): + domain_counts[doc["_id"]] = doc["count"] + summary["domain_distribution"] = dict(domain_counts.most_common()) + + # ------------------------------------------------------------------ + # 4. Workflow state distribution + # ------------------------------------------------------------------ + workflow_counts: Counter = Counter() + for doc in attack_objects.aggregate( + _latest_versions_pipeline() + + [{"$group": {"_id": "$workspace.workflow.state", "count": {"$sum": 1}}}] + ): + state = doc["_id"] or "unset" + workflow_counts[state] = doc["count"] + summary["workflow_state_distribution"] = dict(workflow_counts.most_common()) + + # ------------------------------------------------------------------ + # 5. Relationship type distribution + # ------------------------------------------------------------------ + if "relationships" in collections: + rel_type_counts: Counter = Counter() + for doc in db["relationships"].aggregate( + _latest_versions_pipeline() + + [{"$group": {"_id": "$stix.relationship_type", "count": {"$sum": 1}}}] + ): + rel_type_counts[doc["_id"]] = doc["count"] + summary["relationship_type_distribution"] = dict(rel_type_counts.most_common()) + + # ------------------------------------------------------------------ + # 6. Platform coverage (techniques only) + # ------------------------------------------------------------------ + platform_counts: Counter = Counter() + for doc in attack_objects.aggregate( + _latest_versions_pipeline({"stix.type": "attack-pattern"}) + + [ + {"$unwind": {"path": "$stix.x_mitre_platforms", "preserveNullAndEmptyArrays": False}}, + {"$group": {"_id": "$stix.x_mitre_platforms", "count": {"$sum": 1}}}, + ] + ): + platform_counts[doc["_id"]] = doc["count"] + summary["technique_platform_coverage"] = dict(platform_counts.most_common()) + + # ------------------------------------------------------------------ + # 7. Deprecated / revoked counts + # ------------------------------------------------------------------ + deprecated = list( + attack_objects.aggregate( + _latest_versions_pipeline() + + [ + { + "$group": { + "_id": None, + "deprecated": { + "$sum": {"$cond": [{"$eq": ["$stix.x_mitre_deprecated", True]}, 1, 0]} + }, + "revoked": { + "$sum": {"$cond": [{"$eq": ["$stix.revoked", True]}, 1, 0]} + }, + "total": {"$sum": 1}, + } + } + ] + ) + ) + if deprecated: + d = deprecated[0] + summary["object_health"] = { + "total_latest_objects": d["total"], + "deprecated": d["deprecated"], + "revoked": d["revoked"], + "active": d["total"] - d["deprecated"] - d["revoked"], + } + + # ------------------------------------------------------------------ + # 8. Tactic → technique counts (kill-chain mapping) + # ------------------------------------------------------------------ + tactic_technique: Counter = Counter() + for doc in attack_objects.aggregate( + _latest_versions_pipeline({"stix.type": "attack-pattern"}) + + [ + {"$unwind": "$stix.kill_chain_phases"}, + {"$group": {"_id": "$stix.kill_chain_phases.phase_name", "count": {"$sum": 1}}}, + ] + ): + tactic_technique[doc["_id"]] = doc["count"] + summary["techniques_per_tactic"] = dict(tactic_technique.most_common()) + + # ------------------------------------------------------------------ + # 9. Validation error counts + # ------------------------------------------------------------------ + validation_errors: Counter = Counter() + for doc in attack_objects.aggregate( + _latest_versions_pipeline() + + [ + {"$match": {"workspace.validation.errors": {"$exists": True, "$ne": []}}}, + {"$unwind": "$workspace.validation.errors"}, + {"$group": {"_id": "$workspace.validation.errors.field", "count": {"$sum": 1}}}, + ] + ): + validation_errors[doc["_id"]] = doc["count"] + if validation_errors: + summary["validation_error_hotspots"] = dict(validation_errors.most_common(20)) + + # ------------------------------------------------------------------ + # 10. Version depth (how many historical versions per object) + # ------------------------------------------------------------------ + version_depth = list( + attack_objects.aggregate( + [ + {"$group": {"_id": "$stix.id", "versions": {"$sum": 1}}}, + { + "$group": { + "_id": None, + "avg_versions": {"$avg": "$versions"}, + "max_versions": {"$max": "$versions"}, + } + }, + ] + ) + ) + if version_depth: + v = version_depth[0] + summary["version_depth"] = { + "avg_versions_per_object": round(v["avg_versions"], 2), + "max_versions": v["max_versions"], + } + + # ------------------------------------------------------------------ + # 11. Sample documents (one per major STIX type, latest version) + # ------------------------------------------------------------------ + samples = {} + sample_types = ["attack-pattern", "intrusion-set", "malware", "relationship"] + for stype in sample_types: + col = db["relationships"] if stype == "relationship" else attack_objects + doc = col.find_one( + {"stix.type": stype}, + sort=[("stix.modified", -1)], + projection={ + "_id": 0, + "stix.type": 1, + "stix.id": 1, + "stix.name": 1, + "stix.description": 1, + "stix.x_mitre_domains": 1, + "stix.kill_chain_phases": 1, + "stix.relationship_type": 1, + "stix.source_ref": 1, + "stix.target_ref": 1, + "workspace.attack_id": 1, + "workspace.workflow.state": 1, + }, + ) + if doc: + samples[stype] = doc + summary["sample_documents"] = samples + + return summary diff --git a/insights-agent/grafana.py b/insights-agent/grafana.py new file mode 100644 index 0000000..dc37135 --- /dev/null +++ b/insights-agent/grafana.py @@ -0,0 +1,120 @@ +"""Grafana HTTP API client for dashboard CRUD.""" + +import json +import logging +import time + +import requests + +log = logging.getLogger(__name__) + + +class GrafanaClient: + """Thin wrapper around the Grafana HTTP API for dashboard management.""" + + def __init__( + self, + base_url: str, + api_key: str | None = None, + admin_user: str | None = None, + admin_password: str | None = None, + ): + self.base_url = base_url.rstrip("/") + self.session = requests.Session() + if api_key: + self.session.headers["Authorization"] = f"Bearer {api_key}" + elif admin_user and admin_password: + self.session.auth = (admin_user, admin_password) + + # ------------------------------------------------------------------ + # Health + # ------------------------------------------------------------------ + + def wait_until_ready(self, timeout: int = 120, interval: int = 5) -> None: + """Block until Grafana responds to health checks.""" + deadline = time.time() + timeout + while time.time() < deadline: + try: + r = self.session.get(f"{self.base_url}/api/health", timeout=5) + if r.ok: + log.info("Grafana is ready") + return + except requests.ConnectionError: + pass + log.info("Waiting for Grafana …") + time.sleep(interval) + raise TimeoutError(f"Grafana not ready after {timeout}s") + + # ------------------------------------------------------------------ + # Datasources + # ------------------------------------------------------------------ + + def list_datasources(self) -> list[dict]: + r = self.session.get(f"{self.base_url}/api/datasources") + r.raise_for_status() + return r.json() + + def get_datasource_by_name(self, name: str) -> dict | None: + for ds in self.list_datasources(): + if ds["name"] == name: + return ds + return None + + # ------------------------------------------------------------------ + # Folders + # ------------------------------------------------------------------ + + def ensure_folder(self, title: str, uid: str | None = None) -> dict: + """Return existing folder or create a new one.""" + r = self.session.get(f"{self.base_url}/api/folders") + r.raise_for_status() + for f in r.json(): + if f["title"] == title: + return f + payload = {"title": title} + if uid: + payload["uid"] = uid + r = self.session.post(f"{self.base_url}/api/folders", json=payload) + r.raise_for_status() + return r.json() + + # ------------------------------------------------------------------ + # Dashboards + # ------------------------------------------------------------------ + + def search_dashboards(self, query: str = "", folder_id: int | None = None) -> list[dict]: + params: dict = {"query": query} + if folder_id is not None: + params["folderIds"] = folder_id + r = self.session.get(f"{self.base_url}/api/search", params=params) + r.raise_for_status() + return r.json() + + def get_dashboard(self, uid: str) -> dict | None: + r = self.session.get(f"{self.base_url}/api/dashboards/uid/{uid}") + if r.status_code == 404: + return None + r.raise_for_status() + return r.json() + + def upsert_dashboard(self, dashboard_json: dict, folder_uid: str | None = None) -> dict: + """Create or update a dashboard. + + ``dashboard_json`` is the *inner* dashboard model (with panels, title, etc.). + The wrapper envelope (folderUid, overwrite) is added here. + """ + payload: dict = { + "dashboard": dashboard_json, + "overwrite": True, + } + if folder_uid: + payload["folderUid"] = folder_uid + r = self.session.post(f"{self.base_url}/api/dashboards/db", json=payload) + r.raise_for_status() + result = r.json() + log.info("Upserted dashboard %s → %s", dashboard_json.get("title"), result.get("url")) + return result + + def delete_dashboard(self, uid: str) -> bool: + r = self.session.delete(f"{self.base_url}/api/dashboards/uid/{uid}") + return r.ok diff --git a/insights-agent/main.py b/insights-agent/main.py new file mode 100644 index 0000000..8c53407 --- /dev/null +++ b/insights-agent/main.py @@ -0,0 +1,127 @@ +"""ATT&CK Workbench Insights Agent — entrypoint. + +Crawls the MongoDB database, feeds a structured summary to an LLM, and creates +Grafana dashboards via tool-calling. + +Configuration is via environment variables: + MONGODB_DATABASE_URL – MongoDB URL with database name (preferred) + MONGODB_URI – MongoDB connection string (default: mongodb://mongodb:27017) + MONGODB_DATABASE – Database name (default: attack-workspace) + GRAFANA_URL – Grafana base URL (default: http://grafana:3000) + GRAFANA_API_KEY – Grafana service account token (optional for anonymous/admin) + LLM_BASE_URL – OpenAI base URL or Azure endpoint / request URL (required) + LLM_API_KEY – API key for the LLM (required) + LLM_API_TYPE – openai | azure | auto (default: auto) + LLM_API_VERSION – Azure API version override (optional) + LLM_AZURE_DEPLOYMENT – Azure deployment override (optional) + LLM_MODEL – Model identifier / Azure deployment fallback (default: gpt-4o) + SYSTEM_PROMPT_PATH – Optional path to the system prompt file + RUN_INTERVAL_SECONDS – Re-run interval; 0 = run once and exit (default: 0) +""" + +import logging +import os +import sys +import time + +import agent +import crawler +from grafana import GrafanaClient +from tls import configure_runtime_ca_bundle + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(name)s] %(levelname)s %(message)s", + stream=sys.stdout, +) +log = logging.getLogger("insights-agent") + + +def main() -> None: + # ── Required config ────────────────────────────────────────────────── + llm_base_url = os.environ.get("LLM_BASE_URL") + llm_api_key = os.environ.get("LLM_API_KEY") + if not llm_base_url or not llm_api_key: + log.error("LLM_BASE_URL and LLM_API_KEY must be set") + sys.exit(1) + + # ── Optional config with defaults ──────────────────────────────────── + mongodb_database_url = os.environ.get("MONGODB_DATABASE_URL") + mongodb_uri = os.environ.get("MONGODB_URI", "mongodb://mongodb:27017") + mongodb_database = os.environ.get("MONGODB_DATABASE", "attack-workspace") + grafana_url = os.environ.get("GRAFANA_URL", "http://grafana:3000") + grafana_api_key = os.environ.get("GRAFANA_API_KEY") + grafana_admin_user = os.environ.get("GRAFANA_ADMIN_USER", "admin") + grafana_admin_password = os.environ.get("GRAFANA_ADMIN_PASSWORD", "admin") + llm_model = os.environ.get("LLM_MODEL", "gpt-4o") + llm_api_type = os.environ.get("LLM_API_TYPE", "auto") + llm_api_version = os.environ.get("LLM_API_VERSION") + llm_azure_deployment = os.environ.get("LLM_AZURE_DEPLOYMENT") + system_prompt_path = os.environ.get("SYSTEM_PROMPT_PATH") + run_interval = int(os.environ.get("RUN_INTERVAL_SECONDS", "0")) + custom_ca_path = os.environ.get("ATTACKWB_CUSTOM_CA_CERT") + runtime_ca_bundle = configure_runtime_ca_bundle(custom_ca_path) + if mongodb_database_url: + mongodb_uri = mongodb_database_url + mongodb_database = crawler.get_database_name(mongodb_database_url, default_name=mongodb_database) + + log.info("Configuration:") + log.info(" MongoDB: %s / %s", mongodb_uri, mongodb_database) + log.info(" Grafana: %s", grafana_url) + log.info(" LLM: %s @ %s", llm_model, llm_base_url) + log.info(" LLM API: %s", llm_api_type) + if llm_api_version: + log.info(" LLM Ver: %s", llm_api_version) + if llm_azure_deployment: + log.info(" LLM Dep: %s", llm_azure_deployment) + if system_prompt_path: + log.info(" Prompt: %s", system_prompt_path) + log.info(" Interval: %s", f"{run_interval}s" if run_interval else "once") + log.info(" CA certs: %s", runtime_ca_bundle) + + # ── Connect to dependencies ────────────────────────────────────────── + grafana = GrafanaClient( + grafana_url, + api_key=grafana_api_key, + admin_user=grafana_admin_user, + admin_password=grafana_admin_password, + ) + grafana.wait_until_ready() + + db = crawler.connect(mongodb_uri, mongodb_database) + + # ── Run loop ───────────────────────────────────────────────────────── + while True: + log.info("Starting database crawl …") + summary = crawler.crawl(db) + log.info( + "Crawl complete: %d object types, %d total latest objects", + len(summary.get("stix_type_distribution", {})), + summary.get("object_health", {}).get("total_latest_objects", "?"), + ) + + log.info("Running LLM agent …") + result = agent.run( + llm_base_url=llm_base_url, + llm_api_key=llm_api_key, + llm_model=llm_model, + llm_api_type=llm_api_type, + llm_api_version=llm_api_version, + llm_azure_deployment=llm_azure_deployment, + system_prompt_path=system_prompt_path, + grafana=grafana, + db_summary=summary, + ca_bundle_path=runtime_ca_bundle, + ) + log.info("Agent result:\n%s", result) + + if run_interval <= 0: + log.info("Single run complete. Exiting.") + break + + log.info("Sleeping %ds before next run …", run_interval) + time.sleep(run_interval) + + +if __name__ == "__main__": + main() diff --git a/insights-agent/requirements.txt b/insights-agent/requirements.txt new file mode 100644 index 0000000..0b74105 --- /dev/null +++ b/insights-agent/requirements.txt @@ -0,0 +1,6 @@ +pymongo>=4.7,<5 +openai>=1.30,<2 +pydantic>=2.7,<3 +requests>=2.32,<3 +httpx>=0.27,<1 +certifi>=2024.2.2,<2027 diff --git a/insights-agent/system-prompt.md b/insights-agent/system-prompt.md new file mode 100644 index 0000000..0da0cea --- /dev/null +++ b/insights-agent/system-prompt.md @@ -0,0 +1,43 @@ +You are an expert data analyst for the MITRE ATT&CK knowledge base. You are +given a structured summary of an ATT&CK Workbench MongoDB database and your job +is to create a set of Grafana dashboards that surface actionable insights for +the ATT&CK content authors who maintain this workspace. + +## ATT&CK domain knowledge + +- **Techniques** (attack-pattern) are the core of ATT&CK. They belong to one + or more **Tactics** via kill_chain_phases, and may target specific platforms. +- **Groups** (intrusion-set), **Software** (malware/tool), and **Campaigns** + map to real-world threat activity. +- **Relationships** link objects together (e.g. a Group *uses* a Technique). + Relationship density indicates how well-connected the knowledge graph is. +- **Data Sources / Data Components** describe what defenders can collect to + detect techniques. +- Objects move through **workflow states**: work-in-progress -> awaiting-review + -> reviewed. Objects may be **deprecated** or **revoked**. +- ATT&CK spans three **domains**: enterprise-attack, mobile-attack, ics-attack. + +## Dashboard design guidelines + +- Create **multiple focused dashboards** rather than one monolithic one. + Good groupings: Overview / Object Health, Technique Coverage, Relationship + Graph Density, Workflow & Editorial, Validation Errors (if data exists). +- Each dashboard should have a clear **uid** (kebab-case, descriptive). +- Use panel types appropriate to the data: stat for single numbers, barchart or + piechart for distributions, table for lists, text for annotations. +- For panels that display distributions from the crawl summary, embed the data + directly in the panel using the "text" type with HTML/markdown tables, OR + use the built-in "barchart" panel with a static datasource frame. +- Prefer the **"-- Grafana --"** built-in datasource for static/precomputed + data. Do NOT reference datasource UIDs you haven't confirmed exist. +- Set sensible panel sizes (gridPos: each row is h=8, full width is w=24). +- Always set `"schemaVersion": 39` and `"id": null` (Grafana will assign IDs). + +## Tool usage + +You have a tool `upsert_dashboard` to create or update dashboards. Call it +once per dashboard with the complete dashboard JSON model. You also have +`list_datasources` to check available Grafana datasources before using them. + +When you are done creating all dashboards, return a final text message +summarising what you created. diff --git a/insights-agent/tls.py b/insights-agent/tls.py new file mode 100644 index 0000000..fe14211 --- /dev/null +++ b/insights-agent/tls.py @@ -0,0 +1,59 @@ +"""Runtime TLS helpers for outbound HTTPS clients.""" + +import logging +import os +import re +from pathlib import Path + +import certifi + +log = logging.getLogger(__name__) + +_PEM_CERT_PATTERN = re.compile( + r"-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----", + re.DOTALL, +) + + +def configure_runtime_ca_bundle( + custom_ca_path: str | None, + output_path: str | None = None, +) -> str: + """Create a CA bundle that combines public roots with optional custom CAs.""" + + base_bundle_path = Path(certifi.where()) + runtime_bundle_path = Path( + output_path or os.environ.get("ATTACKWB_RUNTIME_CA_BUNDLE", "/tmp/attackwb-ca-bundle.crt") + ) + runtime_bundle_path.parent.mkdir(parents=True, exist_ok=True) + runtime_bundle_path.write_bytes(base_bundle_path.read_bytes()) + + if custom_ca_path: + _append_custom_certificates(Path(custom_ca_path), runtime_bundle_path) + + bundle_path = str(runtime_bundle_path) + for env_var in ("SSL_CERT_FILE", "REQUESTS_CA_BUNDLE", "CURL_CA_BUNDLE", "PIP_CERT"): + os.environ[env_var] = bundle_path + + return bundle_path + + +def _append_custom_certificates(source_path: Path, bundle_path: Path) -> None: + if not source_path.exists(): + log.warning("Custom CA file %s does not exist; using default trust store only", source_path) + return + + pem_blocks = _PEM_CERT_PATTERN.findall(source_path.read_text(encoding="utf-8", errors="ignore")) + if not pem_blocks: + log.warning( + "Custom CA file %s did not contain PEM certificates; using default trust store only", + source_path, + ) + return + + with bundle_path.open("a", encoding="utf-8") as bundle_file: + bundle_file.write("\n") + bundle_file.write("\n".join(pem_blocks)) + bundle_file.write("\n") + + log.info("Appended %d custom certificate(s) from %s", len(pem_blocks), source_path) diff --git a/k8s/base/configmap-taxii.yaml b/k8s/base/configmap-taxii.yaml index dbf7d23..b2c7da6 100644 --- a/k8s/base/configmap-taxii.yaml +++ b/k8s/base/configmap-taxii.yaml @@ -6,7 +6,7 @@ metadata: data: TAXII_ENV: "prod" TAXII_APP_ADDRESS: "0.0.0.0" - TAXII_APP_PORT: "5002" + TAXII_APP_PORT: "8000" TAXII_HTTPS_ENABLED: "false" TAXII_SSL_PRIVATE_KEY: "" TAXII_SSL_PUBLIC_KEY: "" diff --git a/k8s/overlays/dev/configmap-taxii-dev.yaml b/k8s/overlays/dev/configmap-taxii-dev.yaml index d0d282b..4fab48c 100644 --- a/k8s/overlays/dev/configmap-taxii-dev.yaml +++ b/k8s/overlays/dev/configmap-taxii-dev.yaml @@ -6,7 +6,7 @@ metadata: data: TAXII_ENV: "dev" TAXII_APP_ADDRESS: "0.0.0.0" - TAXII_APP_PORT: "5002" + TAXII_APP_PORT: "8000" TAXII_HTTPS_ENABLED: "false" TAXII_SSL_PRIVATE_KEY: "" TAXII_SSL_PUBLIC_KEY: "" diff --git a/k8s/overlays/prod/configmap-taxii-prod.yaml b/k8s/overlays/prod/configmap-taxii-prod.yaml index 2c3d59b..02d45e9 100644 --- a/k8s/overlays/prod/configmap-taxii-prod.yaml +++ b/k8s/overlays/prod/configmap-taxii-prod.yaml @@ -6,7 +6,7 @@ metadata: data: TAXII_ENV: "prod" TAXII_APP_ADDRESS: "0.0.0.0" - TAXII_APP_PORT: "5002" + TAXII_APP_PORT: "8000" TAXII_HTTPS_ENABLED: "true" TAXII_SSL_PRIVATE_KEY: "/etc/ssl/private/tls.key" TAXII_SSL_PUBLIC_KEY: "/etc/ssl/certs/tls.crt" diff --git a/template.env b/template.env deleted file mode 100644 index 0f33b85..0000000 --- a/template.env +++ /dev/null @@ -1,22 +0,0 @@ -# Docker Image Tags -ATTACKWB_FRONTEND_VERSION=latest -ATTACKWB_RESTAPI_VERSION=latest -ATTACKWB_TAXII_VERSION=latest - -# HTTP Listener Ports -ATTACKWB_FRONTEND_HTTP_PORT=80 -ATTACKWB_FRONTEND_HTTPS_PORT=443 -ATTACKWB_RESTAPI_HTTP_PORT=3000 -ATTACKWB_DB_PORT=27017 -ATTACKWB_TAXII_HTTP_PORT=5002 - -# Nginx SSL/TLS certs path -ATTACKWB_FRONTEND_CERTS_PATH=./certs - -# TAXII dotenv filename -ATTACKWB_TAXII_ENV=dev - -# For setting custom SSL/TLS certs in nginx -# See compose.certs.yaml for details -HOST_CERTS_PATH= -CERTS_FILENAME= \ No newline at end of file